I Got it. The way to work about this is, reduce rest length scale. What I now need to try is apply map to get a fall in the cloth which dissappears after applying rest length scale.
After applying little brush on stretch. I get the following result. Any input.