AI systems have their own crawlers to read websites. How do SEO leads handle this? You can take a closer look at the Top 200 websites in Germany.
Specifically with Christian Rieber’s “AI Bot Blocker Monitoring”. His tool shows live which companies are blocking which AI bots—and which are not. This, in turn, allows for many conclusions. Which ones exactly—you will find out in our podcast interview with Christian Rieber.
You are currently viewing a placeholder content from Default. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More Information
Our guest: Christian Rieber, Head of Search at ADAC
Christian Rieber is Head of Search & Analytics at ADAC. He has been working in the SEO industry for many years. In addition to his work, he develops his own tools, for example the “AI Bot Blocker”. With it, Christian monitors the Top 200 most visible companies in Germany, based on the ranking of the SEO suite Sistrix. In his tool, Christian checks which instructions regarding AI bots are listed in the robots.txt files of the Top 200 domains.
He is deeply engaged with the new tasks around Generative Engine Optimization (GEO) and sees major overlaps with SEO. “70 percent is still classic SEO,” he says in the interview.
One topic that has been added recently: dealing with the new AI bots. Until now, SEO leads have mainly dealt only with the Googlebot—and blocking was very rare. With AI bots that read websites for training purposes, it is different.
Tip: Follow Christian Rieber on LinkedIn. He shares current insights there.
How the Top 200 companies deal with AI bots
Broadly speaking, there are two types of AI bots: first, AI bots that are used to train the AI system—for example, the GPT Bot. The GPT User Bot, in turn, visits the website live when the AI system is grounding, i.e., searching for current and relevant information to enhance its AI answer.
At the very top of the list of blocked bots: the Common Crawl Bot (CCBot). Behind this bot is a non-profit organization that, in simplified terms, crawls the entire internet, stores it, and makes the crawl available. A repository that AI systems used heavily, especially in the beginning. “Many probably think that blocking it causes the least damage or impact for companies,” says Christian. We also talk in the interview about the ominous Google Extended Bot, because many people are not clear on what exactly is behind it.
What stands out in the AI Bot Blocker overview: There are certain industries that are particularly restrictive. Publishers, in particular, stand out. They block AI bots at around 30%. “But among publishers I see changes every week. Clearly, a lot is being tested here or they are still looking for the right strategy,” says Christian. Above all, the training bots are blocked there. Dictionaries also block AI bots at around 23%. Others, such as public authorities, have a very low blocking rate of under 1%.
Christian can understand why many publishers are so restrictive—after all, it is their content, which has been created at high cost. For e-commerce companies, on the other hand, he argues for an open stance. “Every shop should be interested in being relevant and associated with its products in the training material.” The same applies to public authorities.
Free GEO Online Training
- Current Developments in AI Visibility
- From SEO to GEO
- How to Optimize for AI Systems
Example: How we keep an eye on the AI crawlers
As consultants for AI visibility and GEO (Generative Engine Optimization), we also implement numerous measures for ourselves. Fabian Jaeckert is responsible for all technical GEO and SEO topics. He developed an AI crawler monitoring setup for us.
This allows us to filter the various AI crawlers and analyze whether and how access is increasing. For example, the chart shows that the grounding bots are accessing our website more and more frequently—a metric that indicates we are being used as a source in AI systems.
If you would like to dive deeper: Here we have carried out a log file analysis. And here is our overview of GEO (Generative Engine Optimization).
In principle, it is important for companies to define how they want to deal with AI crawlers. We repeatedly see that there are no clear guidelines within companies. For example, important AI crawlers are excluded via robots.txt by default—without management and marketing being aware of the negative consequences.
Our Conversations with SEO and GEO Professionals
Deepen your GEO Knowledge: Every Monday via Email
We Advise Marketing Managers on SEO & GEO
For anyone seeking an SEO sparring partner or comprehensive SEO consulting, Fabian and Benjamin are the perfect choice. As an in-house SEO expert, I sought competent SEO support and found it with them. During a very pleasant introductory meeting, we jointly developed an approach to establish SEO among the individual stakeholders and thereby gain more internal visibility for the subject. Based on this initial discussion and a joint kick-off with the stakeholders, they developed a sound SEO and content strategy for a product area. Since then, they have both supported me as sparring partners during implementation.
The collaboration with Fabian and Benjamin is a great pleasure for me and all project participants, as they are masters of their craft and bring immense passion for SEO. This combination is infectious and motivates everyone to achieve peak performance.
I look back with great satisfaction on the SEO project that we carried out with Benjamin and Fabian. Firstly, the collaboration was on a very pleasant, goal-oriented level. Secondly, the consultation was structured in such a way that we could immediately start with the implementation. We received very concrete optimization suggestions that were tailored to our internal capacities and therefore realistically implementable. Even though only a few months have passed, we can already see the first results of our work in the Google rankings. Traffic for our relevant keywords is steadily increasing, and we still have enough concrete work packages from Benjamin in the pipeline to boost our site. Furthermore, through the consultation, we have acquired so much internal SEO know-how that we can now optimize our site independently. And most importantly: Visitor numbers have doubled – with correspondingly positive effects on sales.
