Found some info from the guys at xda. Looks like the processor takes about 800ms to wake from a deep sleep. They said if it does not lag, it was not in a deep sleep ie. It was plugged in or you had an app running keeping the phkne ffrom deep sleeping.
Apparently just how the processor works. And it does not look like there will be a fix for it. Like I said, I can live with it as long as it is all of the phones, not just mine.