My colleague at work dropped my brand new Nexus S about 2 or 3 hours after I first got it. He dropped about 4 feet high on a hard concrete like surface. Nothing bad happened. I do have a protective case for it now though (trident aegis, works great but its a dust/link magnet).
I wouldn't worry about it too much if everything seems to work