What often happens is that the pins on the port in the phone have a couple of specks of dust on them. This makes connecting both the headphones and charger difficult.
Now if you've got a rubber cover for the port and still experience this problem or if you're sure its not dust then I'd suggest you try with another charger :)