Yes, I've already tried that. Actually I think it has to do with what is in the background of the flowers. My first render were light-blue, and when I put it into photoshop the edges were too bright. I now tried to render again and change the background to black. The result was better, but still the flowers don't sort of melt in into the picture.
What I would really like is a function where you can render the flowers and at the same time calculating what's behind them to get the correct colors - without actually rendering the background.

Is layer-rendering the only way to do this? Because this seems to advance to get a descent result... And then the next step is to render everything in mental ray except for the flowers, and that result should get even worse...

And yes - I suppose rendering without anti-aliasing would get a better result between the two tga:s, but rendering without AA is not what I want.