In the 2nd image you've posted, it looks like you want the hand to actually penetrated the shoulder ... I think your arm would behave weird too trying that pose

But anyway ... as already established, it's a Pole Vector problem.
I had a similar problem recently and posted this in the Animation forum (forum is neary not active and so never got response). But I got some wiser since. Let me elaborate on why this happens and come back with a question myself on this (at the bottom).

As you've already figured, if you're using ikRPsolver (Rotate Plane), you will be faced with the Pole Vector. This is the green line at the top of the Handle vector. It's where the imaginary rotate plane revolves around. On creation of the IK handle, it will be perpendicular to the Handle vector. If you move your End effector past a position where the Handle vector is parallel to the Pole Vector, the handle will flip. This happens in 2 directions of this Pole Vector. So you got roughly 180 degrees in where you can place your End Effector without much problem. You can shift this space by adjusting the position of the Pole Vector using the Show Manipulator Tool.

I attached a small animated screencaps illustrating this:
The ikRPsolver is from shoulder to wrist in de Da Vinci posed skeleton. Moving the End Effector toward the head in XZ direction, the Handle vector will become parallel to the Pole Vector at 1 and flips. With the Show Manipulator Tool, the 180 degree space is shifted by moving the Pole Vector at 2. Now the arm can go further but will face the flipping of the handle problem when going back at 3.

So it's crucial to place the Pole Vector so that you gain maximum freedom posing the arm during your animation. Do what Mike mcKinley suggests and constrain a locator to the Pole Vector so in case you need the character to make an odd pose, u can still make adjustments to the Pole Vector.

Now .. the advantage of a ikRPsolver is that you'd be able to use the Twist attribute to simulate the twist of the elbow or knee. E.g. like you'd be able to move your elbow up and down while you got your finger in your nose.

I came to realise that there another way to get this done, but with the ikSCsolver (single chain) which will eliminate the whole Pole Vector problem.
Create a ikSCsolver from shoulder to wrist. Then group the ikHandle to itself. Move the pivot of the group (use [Insert] key) to the wrist joint with snap to points. And so making the twisting elbow motion by rotating the group instead.
Can anybody confirm if this is a proper way to work around the Pole Vector problem?

P.s. to Mike ... you animate your skeletons arms using FK? Any specific reason why the arms FK and legs IK?