An intermediate step would be to not remove them, but change it to "Contributors" and not show karma.
An intermediate step would be to not remove them, but change it to "Contributors" and not show karma.