When it shows you away from a location where you actually are, the phone has switched from a geo location to a Network estimated location.
The Network estimated location is a triangulation of two or three cell sites and it uses time-domain calcuations to get an estimate. That estimate is very rough, and as you have seen, it can be considerly different than where you really are.
I put a stop to that by setting my Location to "High Accuracy" and leave the GPS 'ON' at all times. That does not effect the battery life enough to even be noticable. In fact, a battery monitor app will tell you that it might shorten the battery life by about 20 minutes over the time period of 18-24 hours.
So, leave the gps ON, set it for High Accuracy and things should be a lot better for you.
Now, another thing to keep in mind, the location reporting will usually be "after the fact" by time delays up to 30 minutes or more. It depends on what app is doing the gps polling.
I went to the Google+ page and told it to NOT track me.
The only time I want the gps to locate me is when I tell it to do so. I do a lot of location reports to the highway department about road hazards.
To ensure that such reports are dead accurate, I use a two step process.
1st: I start up GPS STATUS
https://play.google.com/store/apps/details?id=com.eclipsim.gpsstatus2
this app polls a government geo site for the current emphemeris of the satellites in the sky, it then locks the phone's gps onto that location and will typically be within 10 meters of your actual location.
2nd: I then use Find Address app
Once that looks at the current gps reading, it will then get the actual street address or block of addresses from Google Maps for that lat/long.
I then hit the Sharing icon and use the appropriate means of sending that info to the highway department.
It took me ten times longer to tell it than to just do it.
3 clicks and it is done for me.