Internet companies have removed millions of posts and images over the past decade and suspended hundreds, perhaps thousands, of user accounts. These silenced voices span the political spectrum and the globe: Moroccan atheists, women discussing online harassment, ads featuring crucifixes, black and Muslim activists reposting racist messages they received, trans models, drag performers, indigenous women, childbirth images, photos of breast-feeding. Platforms have taken down documentation of war crimes in Syria, Myanmar and Kashmir, arrests in North Dakota and police brutality across the United States.
We should be extremely careful before rushing to embrace an Internet that is moderated by a few private companies by default, one where the platforms that control so much public discourse routinely remove posts and deactivate accounts because of objections to the content. Once systems like content moderation become the norm, those in power inevitably exploit them. Time and time again, platforms have capitulated to censorship demands from authoritarian regimes, and powerful actors have manipulated flagging procedures to effectively censor their political opponents. Given this practical reality, and the sad history of political censorship in the United States, let’s not cheer one decision that we might agree with.
Moderation systems operate in secret with no public accountability
Even beyond content moderation’s vulnerability to censorship, the moderating process itself, whether undertaken by humans or, increasingly, by software using machine-learning algorithms, is extremely difficult. Awful mistakes are commonplace, and rules are applied unevenly. Company executives regularly reshape their rules in response to governmental and other pressure, and they do so without significant input from the public. Ambiguous “community standards” result in the removal of some content deemed to have violated the rules, while content that seems equally offensive is okay.
Moderation systems operate in secret with no public accountability. We’re pretty sure we know why Infowars was suspended , but in many cases the decisions are far more opaque. And unlike Jones, few of the others have powerful multimedia operations with an audience in the millions that will simply move to using an app when other venues become unavailable, as is happening with Infowars. Facebook, Twitter, Apple and other companies routinely silence voices in marginalized communities around the world that struggle to be heard in the first place, replicating their offline repression.
The palpable public momentum for private companies to do something to more actively moderate the content that appears on their sites is worrisome. And it’s worrisome even though I share the concerns — about disinformation, extremism, harassment — motivating it.