Thx R@nSiD. lol I though that when the cloth simulate to the state you want that they'll animate in real time. But yea, it did actually work.

Now when I move my character, his pants drops off