Sounds like a job for softbody dynamics. I don't think you'll ever get good results with particle instancing looking at the requirements you outlined. And to be honest, what you described isn't easy to do. Hope you don't have too tight deadline.