Unfortunately, you'll probably have to do a wipe/factory reset to get the downgrade to work properly, though you can try it without wiping first.
It's very possible that a factory reset might fix the problems you've been having without the need to downgrade. You'll lose your current settings and installed apps (though a lot of those get backed up through Google), but the files on your internal storage (the "SD card") are left alone.
The best way to do the Wipe/Reset is through the recovery console. To access the recovery mode:
1. Turn off your phone.
2. Hold Volume Up while turning on your phone.
3. Use the volume buttons to highlight "Recovery" and select it with the power button.
4. Hold the Power button and press Volume Up.
5. Use the volume buttons to highlight "Data Wipe/Factory Reset" and select it with the power button (I suggest doing the "Wipe cache partition" option too.
6. Select Reboot.
If you want to try manually flashing back to 2.3.6, put that file I linked you a few post back in the phone's internal storage and follow steps 1 through 4 to get into recovery mode. Select "apply update from /sdcard" and find the update file and select it.
If it gives you an error, it means the only way to downgrade is to root your phone first (sorry). If it does install, select reboot and your phone will reboot into 2.3.6.
If you didn't wipe your phone first, there's a chance it'll go into a boot loop (aka get stuck and never get passed the Nexus logo). If that happens, you'll have to pull your battery and go into the recovery mode, do the wipe, and re-install the 2.3.6 ROM I linked above.