I have this problem as well. It's an issue with branded phones and un-branded phones with the latest R8BA024 firmware. You could back flash to R6BC002 firmware. I hear that its not an issue with that one.
And yes its incredibly annoying, especially if you have the gray or black W580i