I bought the S4 last month and I love it! I liked how fluid games like Asphalt, Dead Trigger and some random rpgs were. Another reason why I love it is because of its lightweight despite having a 5" display.
I have a friend who uses a HTC One and I have to say it looks brilliant sporting an aluminium body. Only thing that spoils the design is the two ugly big speakers at the front. Overall, he claims that it runs really well and it is very responsive.
In contrast, I think both phones depends on the user himself. Just buy whichever you think it's best.