How a platform decides your content is spam

Do the arithmetic first, because it settles the rest of the argument. Give a platform a few hundred million accounts. Assume each posts far less often than you do. You are still at millions of items an hour, in every language there is.

Now put a person in front of that queue. A careful reviewer, reading context and opening links, clears a few hundred items in a shift. Double it to be generous. The two numbers are not in the same universe, and no hiring plan closes the gap.

So a model reads it first, and for the overwhelming majority of everything ever posted it is also the last thing that reads it. Human attention is rationed, spent on items that already reached a lot of people, appeals from accounts with lawyers, and anything with a legal category. Everything else gets a number and a threshold.

I run mobile proxy lines on carrier SIMs in Singapore and a rack of Android phones people rent by the hour. I moderate nothing for anybody. I sit downstream, which makes me the person customers write to when something they posted quietly stopped existing.

Spam filtering left the words behind

It started with words. Paul Graham’s “A Plan for Spam” in 2002 counted how often each word in a message appeared in spam against real mail, multiplied the probabilities, and it worked the week it shipped. Then senders buried public domain novel text in their messages to drag the average down, spelled words with numerals, sent pictures of text instead of text. The industry moved to sending server history and published domain records, and every classifier built since has walked the same road. The text is the part the sender fully controls, so the text is worth the least.

The account is the defendant

What carries the score is history, and the post is mostly a timestamp on it.

Account age. What it has done since. Whether it has been actioned before. Whether a payment method is attached, a phone number, a device that has been the same device for three years. Whether it went dark for eight months and woke up busy.

That history is expensive in a way a paragraph never is. A two year old account with three thousand ordinary posts takes two years, and the cost is the whole reason it counts.

Then the manner of arrival. Typed into the box or pasted in one motion. Seconds between the composer opening and the submit. Whether it came through the app, the documented API, or a browser doing an impression of a person. Whether the image is new or matches a file seen four thousand times.

Then the destination. A link is usually the heaviest content side feature in the model, because a domain has a past even when the post does not. Domain age. Whether the crawler was shown the same page a visitor gets. A site that changed hands last year inherits whatever its previous owner did.

Somewhere below all of it sit the words you agonised over.

Simultaneity is the loudest thing in the room

Strip the model down to a single feature and this is the one worth keeping: how many accounts did something similar inside a short window.

These systems are not hunting for identical strings. Each item is reduced to a fuzzy signature that survives small edits, the signatures get clustered, and the arrival times inside a cluster get examined. Organic attention arrives in a mess: people find a thing at different hours, in their own words, some a week late, most never. A purchased campaign arrives shaped like a bar chart.

The problem underneath has no clean solution. A genuine crowd produces the same shape.

Fifteen colleagues share a redundant friend’s CV in one afternoon because they all want to help that day. A club announces a signing and every supporter posts within the hour. The machine has no access to motive and never has. It sees arrival times, and arrival times are the one property of a bought campaign that money cannot cheaply smooth out.

Which is why so many unexplained removals have the same sentence at the centre of them. We all posted about it at once.

The loop that proves itself right

Almost nothing gets full distribution immediately. A piece goes to a small sample first, and what that sample does feeds straight back into the score. Negative signals weigh far more than positive ones, because they are rarer and therefore more informative: hides, reports, how fast people scroll past. On a large account that sample is stable. On a small one it is noise.

A borderline score cuts distribution. Fewer readers means weaker engagement in absolute terms, and a worse ratio if the small sample breaks badly. Weak engagement then reads as evidence of low quality. The original score is confirmed.

Nothing about the piece changed anywhere in that sequence. The system asked a question, answered it quietly, then arranged the conditions that proved the answer. There is no counterfactual either: nobody records what the post would have done at full reach.

The same trap sits a floor higher, in training. These models learn from labels, and the labels are the enforcement decisions of the model that came before. If a category was over policed two years ago, the pile of examples proving it was spam is the pile the current model was fitted to. The blind spot arrives as ground truth.

What comes back when you appeal

A category. This violated our policy on spam or deceptive behaviour. Sometimes less.

There is a real reason for that and it is not laziness. Publishing why something was caught publishes where the boundary sits, and at scale the appeal queue becomes a free measuring instrument for the exact people you are trying to stop. Submit, get told which feature fired, adjust, submit again. That is a model mapped from the outside and handed over voluntarily.

The quieter reason is that the reviewer usually cannot tell you either. What arrives in front of them is the item, a list of policies, and a keyboard shortcut, never the score or the inputs behind it. And reviewers get graded on how often they agree with the automated decision, so the queue does not manufacture much disagreement.

Who it lands on

Every classifier has a false positive population and this one is easy to predict.

People writing in a second language go first. Translated phrasing, simpler sentence structure, the same safe construction reused because it is the one you are sure of. Template spam has all of those properties for a completely different reason, and the model was never shown the reason. It was shown the shape.

Then anyone whose ordinary vocabulary overlaps a spam category. A locksmith writes about bypassing locks because that is the trade. A debt adviser types the words guaranteed and consolidation and no credit check in order to warn people away from them. The vocabulary of describing a thing is the vocabulary of selling it.

Then the schedulers, which is almost funny. Someone with a day job queues a week of posts on Sunday night. They land at nine every weekday morning at intervals a clock could keep. That rhythm reads as automated because a machine is doing it. The intent is ordinary and the signature is a bot.

Then small accounts, for the sampling reason. Then anyone on a shared exit, which is my corner: carriers do not hold enough public addresses, so thousands of handsets share a pool several thousand strangers were also behind an hour ago.

What I had wrong

For a long stretch I believed the text was the thing being judged. When a customer got hit my first move was always the copy. Soften the claim, drop the link, swap out the risky words, try again in a few days.

What broke it was a pair of screenshots. The same paragraph, word for word, posted eleven minutes apart from two accounts. One is still up. One was gone in ninety seconds. So I asked him to post something harmless from the actioned account. A photograph of his lunch. Gone again inside two minutes.

I had been treating the post as the defendant. The account was the defendant.

What I still cannot give you is the weighting: how much is account history against manner of posting against what the first readers did. Nobody outside those teams can, and the answer moves every few months.

The move to no decision at all

Removal is expensive. It produces a notification, which produces an appeal, which produces a support cost, and occasionally a screenshot that travels further than the original post.

A score that quietly reduces reach produces none of that. No notification, because nothing was removed. No appeal surface, because no decision went on record. The item is right there. Send yourself the link and it loads perfectly. It just does not go anywhere.

The person it happened to has no way to separate that from having written something nobody cared about. So they conclude the second thing, because it is the only explanation available, and start rewriting to please a room that was never let in.

Here is the position people argue with me about. I would rather be removed. A removal is at least an event, with a date and a category and a human at the far end eventually. Being quietly wrong about your own competence for two years does more damage, and it leaves nothing behind to point at.

None of this exists because somebody wanted it. It exists because the volume made human judgement impossible and a number is the only thing that scales. What the layer really does is compare the cost of two mistakes and take the cheaper one, several million times an hour. Letting a bad thing through costs some trust and occasionally a headline. Suppressing an honest person costs close to nothing, because that person never learns it happened.

More on the machinery running underneath the ordinary internet at The Hidden Internet.

Get new guides and videos first — join the Telegram channel.