I just got an S4 yesterday lol. My S3's cracked screen and charging port had finally had enough.
So now this phone feels amazing. I'd personally stick to phones with a removable battery. Not just for swapping them out in the middle of the day, but for a year or more down the road when the battery just doesn't have the capacity it once had.
Just my opinion, but Android these days needs a quad core. And as long as it's running on a quad, whether 1.2ghz or 2.2ghz, it's gonna be smooth.
Gaming is another issue. I haven't got to test my adreno 320 out much, yet.
But, if storage is the issue, and since we can't install apps to our SD card anymore, like the good old days, then I see good reason to upgrade if your current phone can't hold what you want it to.
But this is coming from a guy who upgraded from an S3... Ya know, those twitchy phones that don't have "smooth" in their vocabulary. Compared to it, my new 975L S4 feels like a king. (Straight Talk on AT&T)