I spent about a week after I got my One, with no case. Then I got my Otterbox Defender. my phone spent the majority of its time in the case. Over the past month though, I split my time between being in the Otterbox Defender, and being naked. Well, naked with a screen protector. I always have a screen protector on it no matter what.
So far, I haven't had any issues with my phone, with or without the case. The case doesn't cause any rubbing on any areas of the phone, and when it ISN'T in the case, I have been lucky enough to have never dropped it. I do find myself being ultra-careful about laying it down on its back. I am paranoid that I will scratch the camera lense.