So I only have a nat problem with one other person but I know the problem isn't nat...

I only have nat with one other person. I can hear him if others are in the party but if its only us 2 we can't hear each other. He is the only one I have a problem with and I am the only one he has a problem with. We both did port forwarding and have nat type 2. What is causing this?