You definitely don't need to put a higher-than-rated voltage across the charging pins! Stay within spec when dealing with Li batteries
The phone doesn't fast charge (I believe to maximise battery longevity), but with a mains charger mine charges in a few hours (never bothered timing it). Even via USB mine charges faster than you describe. You aren't streaming over 3G while charging or anything like that (i.e. using the power as fast as you put it in)?