Is there a method similiar with setScrollFactor?

You can add the objects to a different camera or different scene.

For examle, add all the UI elements to an uiScene and all the game elements (which should be zoomable) to a gameScene.