I'm not completely sure because I'm inexperienced at it, but could you turn the shirt to cloth, attach gravity to it and then once it's deformed to where you want it convert it back to poly? Like I said not sure if this would work or if it's even possible, just an idea.

Great work though, makes me want to open maya (haven't been in it in a few months)