It is normal for a phone to charge slower on a computer USB than a power adapter. Most computers only output 500ma of power to a USB port, most power adapters output 700ma, or even 1A, so they charge faster.
And although you said you weren't doing much with the phone, consider that many phones will actually use MORE than 500ma when doing heavy work (gps, data use - especially 4g). Many phones won't even charge in these cases (a common case is with 500ma car chargers when using the GPS).
If the phone is drawing 300ma during mild use, then only 200ma are being used to charge the battery when you are on a 500ma USB computer port. If you were on a 700 ma power adapter - you would have 400 ma to charge the battery (above the 300ma draw), so the battery should charge 2x as fast.
Edit: your numbers do seem exceptionally low, though. It is possible that the phone is programmed to "slow charge" when on a computer USB port. On other phones, I have seen "hacks" to remove this limitation. I would watch the charging speed while on a power adapter, my guess is that it will be much faster.