This feature is temperamental as buggery on my phone, and it's really disappointing. Sometimes it works perfectly. I say "OK Google" and it'll come straight from a switched off, locked screen to the Voice Prompt and await my instructions.
And then later on, without me doing anything, it'll just stop working. So I'll be repeating "OK Google" like a foolish madman and it'll ignore me completely. To fix it, I have to simply press the microphone button on the Google widget and speak, and once I've done that, it will once again listen and respond to "OK Google".
As someone else said, this feature needs to be reliable to be useful. It's a crazily fantastic feature, when it works. I know from searches that a lot of people struggle but out of interest, is there anyone out there with a Note 4 that uses this daily (as in, over 10 times a day) and never has a problem with it?
Hope they fix it in the Android update, if it's a software glitch and not hardware... I heard these phones have a hardware chip in them to listen for "OK Google", and hope it's not that that's screwed up.