if I understand right, I don't think you're seeing it zoom in, but rather, you're seeing the object being displaced outward, increasing it in size, making it larger in your view. If you need it smaller and can't adjust the displacement's intensity, I suppose you'd need to make your original object smaller. Or zoom your camera out.