I'm having the same issue, and my phone is only two days old, after exchanging it for the same issue. Download an app called Android Sensor Data. Open it and turn on the positional tab. Lay your phone on a level surface (check it against an actual level in both directions) and it'll tell you how off your device is.
The "Accelerometer" will always read "Z" very high, as gravity is pulling on the decive. "X" and "Y" should be very close to zero. Now look at the "Orientation" section. Azimuth reads compass direction. Pitch reads tipping the unit up or down, and Roll tells you about left or right. Both Pitch and Roll should be zero when the unit's screen is level. These units are in degrees. Currently, mine is reading 6 for Pitch and 1 for Roll (6 degrees is a huge error in my opinion).
Like I said, this is a replacement for a unit with the same errors (first one was off by 4 and 5 respectively). This isn't too much of an issue in Apps that have a calibrate option, however, it will mess with those that don't. Oddly enough, neither Google Street View in Compass mode nor Google Sky Map have a calibration function. There is no option to calibrate the phone like there is in the MyTouch 3G Slide.
I have spoken with T-Mobile tech support to no avail, their response was return it. I have emailed Samsung about this issue and am waiting for a reply. It should be an easy enough patch to unroll to us. When (if) I get a response, I will post it here.
Hope I'm not the bearer of bad news, but rest easy knowing you're not the only one experiencing it either.
Jason.