I found a low hanging fruit:
Apparently when the game starts, Mr. Giggles AI or most likely the collision detection for him is taking a lot of CPU, causing the game to lag significantly. After a bit, the slowness goes away and game is fast. In other words: Mr. Giggles eats CPU.
Hopefully I can get this issue resolved today, just need to figure out why this happens
One thing I forgot to mention, we now have "tap to place" mode, so construction is much easier. But it does not work when inventory is open. I will try to fix that.