Bloody annoying problem you have there,
Only things I can think of are either you have faulty pins somewhere either in the back of the headunit or at one of your connectors.(I.e. Snapped, bent or missing pins)
Or
You may have a short somewhere in one of the speaker connections, whether it be...