It's a matter of your aim axis.

There are a few ways to deal with this problem.

First, I think you can type an aim axis into the option box for the aim constraint menu item. This is probably the hardest way to go about it because you'll have to figure out what numbers correspond to the correct direction (although I suspect that the spotlights point in positive z in their local space.)

Second (and this is my favorite overall trick for dealing with constraints, you can GROUP the light and aim-constrain the parent group. Then, you'll still be able to rotate the light itself after the fact.

This is a nice trick for cameras where you want the camera to follow something like a vehicle but you want to add animation (like camera shake) on top of the motion of the vehicle, or you want to be able to tumble and position the camera from the camera view, which constraining the camera directly would normally prevent.

Another possibility that isn't quite as flexible is to select the aim handle for the light (you'll have to turn it on, but the light has one) and point-constrain that to the rocket IF you can do that (I can't remember if you can constrain that handle and my computer with Maya is packed for moving at the moment.)

-- Mark