Adding colliders to stack of block

[samme]

Use only 2 colliders. Phaser will do all the combinations.

this.physics.add.collider(totems);
this.physics.add.collider(grounds, totems);

Arcade Physics tends to be unstable for stacking. You can try https://phaser.io/examples/v3/view/physics/arcade/custom-separate.