Dualists do not a priori believe that consciousness has a physical component.
Imagine living 500 years ago. Peter says the mind is a physical mechanism, Dave says it's not all physical. Now what are Peter's predictions? Peter's predictions are that every cognitive function can be intercepted or corrupted by physical means. Meanwhile, Dave's predictions are that every cognitive functions may or may not be corrupted by physical means.
Centuries pass, and we find that, at every opportunity, Peter's predictions are validated. Dave's theory has not been absolutely ruled out, but it has been ruled out statistically. What are the odds that Dave's dualism is that one rare form of dualism that looks exactly like Peter's physicalism?
Well, the odds are overwhelmingly in favor of Peter. Every experiment that could go Dave's way but doesn't is a factor of two in favor of Peter's theory. Today, one would be guilty of gross fine-tuning (and gap argumentation) to suppose that Dave's theory were likely to be true. Even if Peter had passed only ten tests of physical cognitive function, Dave's theory would still be a million-to-one long shot.
The question is, does the zombie argument impose million(or billion)-to-one statistical argument that can cancel out all of Peter's data for the last five centuries?
No. The premise of the zombie argument is that human zombies are possible, i.e., that physicalism is insufficient to explain qualia. But qualia may not even exist as non-causal elements. Even if our belief favored the existence of non-causal qualia (and mine certainly doesn't), we would not be sure to one part in ten, let alone one part in a million. If we were billion-to-one certain that qualia existed as an non-causal part of the cognitive story, then Dave could happily sustain his debate with Peter. But that's just not the case.