Did you notice a drastic Contact file size reduction like CheriePie mentioned?
The next best step I see is to report it to HTC with details and see what they come up with.
With some helpful feedback and consistent data they might be able to investigate and better program how the phone interacts with 3rd party sources in regards to optional contact data?
There should be a way for users that want that extra functionality to have it without sacrificing the performance of their normal phone use. - When one needs to go into the phonebook/dialer to make a quick call or just send a simple text message, one should not have to wait for it to [allegedly] download/load Facebook information that is unnecessary to the current task. At the same time, if one
does want to see Facebook/Twitter/etc. updates within a contact card one should still be able to get those quickly and easily when wanted or pertinent to the task at hand. I'm not sure how they can accomplish this, but if we put the ball in their court then it is up to them to figure that out for us.
Reporting/data collection essential part of the process towards making things better for everyone and progressing software wise.
If you want to go a step further you could do some more experimenting and add back Facebook data for one contact at a time? Maybe you can find a data capacity threshold or contact quantity for when the slowdown starts to happen?
Is the People application load time in anyway dependent on the current network speed? If you have slow network speeds do contacts load slower than if you have average/good speed?
Is there an active demanding sync, or is it passive and load from memory until scheduled sync? Maybe there is a better way to save the data and load it quicker - multiplying even a short load time by 100 contact's social network data could create a nice hiccup.
One other idea I had to try might be to set up a secondary test phonebook for only Facebook linked contacts, or limit the phonebook to not actively display people with Facebook link-ups. If they are not part of the phone book being currently listed, long shot, but possibly the optional data will not be loaded until you switch to the other phonebook?
I know, I'm spit-balling here, but maybe it will help some of you other brilliant minds think of something.