Sounds like a job for softbody dynamics. I don't think you'll ever get good results with particle instancing looking at the requirements you outlined. And to be honest, what you described isn't easy...