Input lag

Guys, been having this issue off and on since I got my PS4. Whenever my mic is plugged into my controller I sometimes get insane lag which when it's going on there is a direct correlation to piss poor up/down speeds via test network connection. Here's the kicker though, it sometimes happens when I'm offline completely. Even still though, I moved and got a new ISP and router so it's too much of a coincidence for it to be up/down speeds because it happened at two different houses on different sides of the country. When it happens, if I unplug the mic I can hear my party chat through the tv crystal clear and the input lag goes away almost entirely. I've tried multiple controllers and they all do the same thing. Not sure where to go from here, any advice would be appreciated.