Depends on if you are on the new JB 4.2 release, which I assume you are. I've foregone that upgrade because of too many issues and won't need the multi-user functionality, so I've stayed on JB 4.1.2. I did upgrade the bootloader to the JB 4.2 version to fix some of the issues with booting from recovery, but that's it.
I prefer the stock ROM, there are going to be far fewer issues with apps, so less hassles with that. But there is some pretty slick development going on so I try things out every now and then to see how things are progressing.
Mainly, I will decide on a custom kernel and any tweaks it will give me for performance.
I use Jay's Buttery Goodness tweaks from the xda site
http://forum.xda-developers.com/showthread.php?t=1946347
Mods, sorry if I wasn't supposed to link to that site. I've seen others so thought it would be OK.
Anyway, kernel flashing is much easier in most cases than new ROMS and it might be a place to start. Recovery a lot of times is just a matter of reflashing your existing kernel, but you should always have a complete nandroid backup handy just in case.