What am I missing about using a set velocity?

It does look like FPS issue. Phaser smooths out delta time, so if you suddenly get smaller deltas (higher FPS) then for a short while you would get faster movement as the delta is smoothed based on previous values (i.e. real is smaller than applied).
You could check for that by logging both Phaser’s delta time and actual delta time.