Originally Posted by FordVW
I was getting the same behavior. I went into the Google Now settings, then Voice, then "Download offline speech recognition" and once that was installed, things worked a lot better. Try that see if it helps you too.
I did that and rebooted, and now everything seems to be working a LOT smoother. BTW i really dig the new boot animation. Nexus Logo keeps it clean and classy.