I have a RAG of agents that analyses the 3D printing market once a week: printer prices, what is trending, which figure is worth printing to sell, which filament is cheap and still good. It started out scraping 10 pages and it worked fine.
When I wanted a more detailed analysis I pushed it to 50. That is where it fell over.
What I ran into
What breaks is not the volume itself, it is what the volume runs you into. At 10 pages nobody notices you. At 50 you start to look like exactly what you are, and the walls come up.
Three things, all at once: 403 Forbidden straight away on some sites, rate limits on others, and Cloudflare's antibot protection on the toughest ones. It was not that one page failed and the rest carried on. Once a site decides you are a bot, there is not much left to negotiate.
The other part that gave me trouble, and that I could not separate from this one at first, was how to structure the agentic loop. I had both problems tangled together: I could not tell whether something failed because I was being blocked or because the loop was badly built.
How I solved it
The first move was to stop scraping everything flat. Treating all 50 pages the same way, with the same method, is what guarantees you get stopped precisely on the sites with the hardest protection.
For those specific sites I used the pages' own APIs instead of scraping them. If a site already exposes the data through a route built to be queried, fighting the HTML is inventing your own problem. Scraping stayed where there was no alternative, and the API took over where there was one.
For the agentic loop I went to Anthropic's page on agent patterns, took the structure they recommend and implemented it as is. Once the loop was built on a known pattern, whatever kept failing was a real access problem and not noise from my own architecture. That alone cleared up a lot.
With that, it now scrapes all 50 pages and the weekly analysis comes out more detailed.
What I took from it
One concrete thing: when you scale volume, you do not scale the same method. The answer to "they are blocking me" is not to scrape more carefully, it is to change how you get in wherever the door is shut. Mixing APIs and scraping site by site is not a dirty workaround, it is the solution.
And one broader thing. AI moves very fast: however much you are building something you think is new, a week later it already exists with a better structure, because new things keep coming out and the capabilities keep getting better. I lived it in miniature on this same project. I did not solve the agentic loop by thinking it through from scratch, I solved it by going to find the pattern someone had already written down, and it came out better than what I was going to build on my own.