Originally Posted by PyroSporker

Sounds like you need to experiment

Sorry, there is many things that could be a factor here so it is hard to narrow down to one thing. These are some ideas to get you rolling.

Since you mentioned it, try running without your SD card and see if there is any change? Contacts/People data should be cached in internal storage unless you moved data to the microSD or are running some advanced root setup with different storage options. (stock back ups go to the microSD but are just storage, not active running files)

Try booting into safe mode and it will disable all third party applications (I think - they are not listed in the app drawer. Unclear if they could be still running in memory behind the scenes.) Fast Boot off, re-boot, and when you get to the white 'HTC Quietly Brilliant' screen hold vol. down for a while until you feel it vibrate feedback. It will say Safe mode in the lower left corner of the screen if you did it right.

Could it be a Google/Gmail Sync thing?

Do you have ALOT of contacts, or ALOT of old Messages hanging around?

How about alot of linked contacts? Is it loading alot of extra contact info like Facebook updates or Tweets or something? Calendar information (Birthdays, appointments)? Gallery folders?

Do alot of your contacts have contact photos attached to them? Maybe it is taking a while to load up all the pictures attached with contacts?

Originally Posted by EarlyMon

I'm guessing that you have photos with your contacts?

Are they linked to high res photos on your sd card?