I think you should focus on size and the S-Pen. The other specs will be less significant I think from a user experience standpoint, although the higher spec on the Note will make it a bit more future-proof. But you have to be comfortable with the size of the Note 3 first and foremost. If you are ok with the size and if you find the S-Pen interesting and/or potentially useful, then the Note is worth it. And the end of the day you don't have to use the S-Pen of course, but it is a unique feature in the mobile world these days.
Conversely, if you are concerned that the Note may be too big, the HTC would be a more manageable size, and you'd save a few bucks.