Yeah, thanks! After some closer inspection and comparison I think this might actually be the typical Android stuttering that one encounters in some games. They are somehow a bit more noticeable on the HTC One though I think. I'm very sensitive to sluggsishness.
PS. In case of Vector, which did indeed have really annoying suttering, restarting the phone seemed to help. Somehow never thought of that before.