after hours and hours of trying I found the solution in case somebody runs in the same problem:

Screw World Rotate Mode - it doesn't work if the camera's angled - use Gimbal instead