Populations often need to make consensus decisions despite the sporadic asocial change of opinion by its individuals and the presence of minorities of stubborn asocial individuals. We show that one of the most prominent models of opinion dynamics—the voter model—is unable to lead to any consensus upon the introduction of minimal asocial behaviour. We show that an alternative model of comparable simplicity based on inhibitory signals—the cross-inhibition model—can instead reach a consensus notwithstanding high levels of asocial behaviour. The results predicted by the mean-field models are confirmed by experiments with swarms of 100 locally-interacting robots. Our analysis helps explain why inhibitory signals evolved in biological systems that need group consensus despite operating in noisy contexts, such as social insect colonies and neuronal populations. We also demonstrate the relevance of our findings for designing resilient swarms of minimalistic robots.