More testing and more calls to HTC tech support -- and more news to report. The Nexus One call quality on VoIP calls is very poor without or without a headset. The problems are different in each case, but both ways of using the phone for VoIP calls suffer from poor quality. The Droid offers much better sound quality on VoIP calls.
I added a new comment with more details here:
How To Set Up VoIP on the Nexus One Google Phone using Sipdroid for Free Calls | Dave's Tech Shop