I initially set up an image panel in the Top, Front, and Side views in order to model a car. Later I decided to bring in a rear view of the car that I am modeling I did this by changing the bottom left hand panel from being the Front view to the Back view then uploaded the image believing that I would be able to switch between the front and back view to my hearts content. The problem is when I switch from the Back to the Front view the new rear end image remains. I can swap the image by uploading the front image once again but as soon as I select something the rear view image takes its place.
I'm personally a fan of setting up new orthographic cameras to hold imported images references:
In view pane: Panels > Orthographic > New...
1 x Top
1 x Side
2 x front (you will need to rotate one of these 180 degrees)
adopt an appropriate naming strategy (Name Img_Front, Img_Rear etc)
Import your images as before: View > Image Plane > Import Image
at this point in the attribute editor check the "looking through camera" option as opposed to "all views" (you might want to do this bit last so in the perspective view you can see the theory behind what happens next)
Change 2 viewpoints to the new front and rear cameras (leaving the persp cam): Panels > Orthographic .... (Select Cam):
View > Image Plane > Image Plane Attributes... Placement Extras > Centre... (adjust the value in the last box ("z" axis) to conflicting values one as 20, and one -20 (this may need to be adjusted depending on your grid size. "img1"
What this has effectively done is put a camera and image plane on either side of the grid and as you can only see the relevant image planes from each camera you effectively look through the one to see the other "img1" I hope can help to illustrate this. Whilst also giving you the ability to quickly select an "original" orthographic view without being intruded by an image plane.
"Img2" - should be (for the most part) what your trying to achieve.