Thats one of my concerns, I haven't seen it in person but definitely the Milestone LOOKS better.
I've had the two side-by-side. I own a milestone, and got a desire recently with my carrier upgrade (but subsequently sold it to my amigo).
Why did i sell the desire? Personal preference really - i'm a bit of a gamer, and at the moment the Milestone is still the best gaming phone due to both it's GPU and keyboard. Even though you'll find everyone with an N1, Desire, Incredible, etc. spouting figures about the number of flippity flops and polybonbons of the snapdragon, the fact of the matter is that thus far, every 3-d game that has come out has run best on the Droid/Milestone. Put it down to optimisation, put it down to magic, i don't really care. I know what i sees! However I would not that games dont run badly on snapdragons, and they'll only improve (and will eventually surpass). But for now, Milestone's the best at 3-d.
Then you have the keyboard. The keyboard itself is RUBBISH, and i never use it - but i did buy a game gripper control pad. So now I can play Snes, Megadrive (and soon Playstation) emu's on it
perfectly. Oh, and quake 2.
But that really is where the milestone's benefits over the Desire end. The Desire is definately more future proof IMO. And if you're into customisation, it's WAY more customisable. Milestone's very limited on the ROM front. Sure only yesterday XDA developers developed a hack that allows the Nexus 1 to record 720p video! I have no doubt that that'll make it over to the Desire since they're basically the same phone.
The Desire is also far snappier and smooth in all aspects of operation. Even swiping across home screens there's zero stutter, whereas there is a bit on the Milestone. The Desire's screen is also a lovely size. It really looked great.
As for feel in your hand, the milestone is nice and solid feeling. The Desire is a bit more iPhone-y and plastic-y. But that's a minor thing.
So to sum up:
Currently - milestone is better gamer, Desire probably better in everything else.
Future - Desire will probably match or beat milestone in gaming, continue to be better on everthing else!