My S2 used occasionally to get very hot to the extent that it actually started to feel painful if I held my finger to the hottest part. I don't think that should happen with a phone so each time it did, I rebooted and that worked every time. I have only had to do this once with my S4 and it definitely wasn't as hot as my S2 used to get.
In answer to your question, I'd say that too hot is when it gets uncomfortable to touch. If that happens regularly, something is definitely wrong.