My speakers really sound crappy

The car has a factory installed Infinity amplifier and seven speakers, 2 in the front doors, 2 in the rear doors and 3 in the dash, the amp apparently has a crossover buily into it so all the bass goes to the front door speakers (6 x 9 inch) and mid and high goes to the dash speakers (2 are 3.5 inch and the third is 2 or 2.5 inch)

The problem I have is that with the bass turned up even slightly the front door speakers sound like they are vibrating, buzzing or distorting (it is actually the speakers not the door panels) even at pretty low volume. So I fitted new speakers (6.5 inch Infinity Reference 6002si) and they are doing exactly the same. I'm pretty sure I have the speakers wired correctly, I did try to swap the + and - wires but it sounded exactly the same.

Can you guys advise me on things that I can try to diagnose and fix my problem.