Cloudflare, after making billions blocking bots, just gave everyone a web crawler.

Of course it's a betrayal. Except it's not.

One API call to Cloudflare’s new /crawl endpoint and you get an entire website back as clean HTML, Markdown, or JSON. The Hacker News thread exploded. Twitter wasn’t far behind.

“Selling the wall and the ladder.”

“Biggest betrayal in tech.”

“Protection racket.”

These hot takes sound smart but they’re not.

The web was built to be open and available to everyone. Serving static HTML from disk back in the day, nobody could hurt you because there was nothing to hurt.

We need bot protection now because everything is dynamic, straight from the database with some light caching for hot content. When Facebook decides to recrawl your one million pages in the same instant, you’re very much up shit creek without a paddle. A bot that crawls the full site doesn’t steal anything, but it does take down production. My clients never call me upset that a bot read their blog posts. They call because the bot knocked the site offline for paying customers.

Bot protection protects availability, not secrecy.

And the real bot problem isn’t even crawling. It’s automated signups. Fake accounts messaging your users. Bots buying out limited drops before a human can load the page. Like-farming. Credential stuffing. That’s what bot protection is actually for: preventing fraud, not preventing someone from reading your public website.

Cloudflare’s /crawl endpoint respects robots.txt. Don’t want your content crawled, opt out. But if you want it indexed and can’t handle the traffic spike, this gets your content out without hammering production.

As for the folks saying Cloudflare should keep blocking all crawlers forever: AI agents already drive real browsers. They click, scroll, render JavaScript. Go look at what browser automation frameworks can do today and then explain to me how you tell a bot from a person. That distinction is already gone.

The hot takes are about a version of the internet that doesn’t exist anymore.