Does everyone that have a nexus one have this problem?
No, not really. Are you sure you have the volume turned up? While in a call, press the vol. up button and see if that helps. If not, it might be a hardware problem.
My Nexus used to be really loud.. but now the volume is actually very low, even with the volume turned all the way up x.x
I thought it was a hardware problem and that a raindrop had broken the speaker perhaps. The conference speaker is fine.
I suspect that something has changed in the operating system to accommodate newer phones complaining about Audio being unbearably high.
Separate names with a comma.