Speakerphone icon won't go away - HELP

The speakerphone icon is on at the top and it won't go away. I use the speakerphone every once in a while when i'm on the phone, but not always. I tried turning the speaker phone on then off when I was on the phone, but it didn't make the icon go away.
I can't find any settings to make it go away either.
Any one know how to fix this annoying problem ?!
Thanks!