Every AI response is based on sources. All AI systems – whether Google, ChatGPT, or Meta – draw from the internet, evaluate content, and process it.
But how exactly does that work? To understand this, we analyzed our website logfiles. First insight: Various AI bots visit our website daily. These bots serve different functions within the AI systems.
What exactly these bots do – we discuss that in depth in our podcast. As always, the summary is available here in this article.
Such a logfile analysis is not only interesting from a technical perspective. It’s about much more: a new form of web analytics.
Because companies need to understand what role their content plays in AI systems. For example, when they are mentioned and when they are not. How ChatGPT, for instance, describes their products – and how their own website is used as a source for this.
These AI Bots Visit our Website Daily
All major AI platforms publicly state which bots they use to analyze websites (e.g., OpenAI). Each bot has a unique signature that is recognizable in the logfiles.
The AI bots have different functions:
- Language Model Training: Specific AI bots, such as ChatGPT’s GPTBot, are used to train the language model. Based on this, the most probable answers are then generated when users ask a question.
- Search Bots: Another type of AI bot is used when the AI system or users actively want to search for content on the internet. A search is then initiated, and the specific website results are analyzed by the bot.
- User Bots (Direct): User bots are used when users actively enter a website into an AI system and want to do something with it. For example, when a user directly enters a company’s website into ChatGPT and asks the AI system something about it.
Website Accesses by Type, 30 Days
Our AI Web Analysis (30 Days)
From this logfile data, we have developed an AI web analysis tool. It tells us:
- Which AI bots visit and when
- How often these AI bots retrieve content
- Which content the AI bots extract
This is highly relevant: Because we want to provide AI systems with the right “feed” so that they interpret us as we intend. We see which specific content is fed in – or actively used by users.
This “on-page optimization,” to use a typical SEO term, will be a central task for influencing AI systems.
It is already evident: Companies that have invested in SEO and have high-quality content on their website are significantly more frequently and better represented in AI systems. We see a clear correlation here. The reason lies in the keywords. Companies that engage with SEO delve deeply into keywords and thus into the search behavior of their target audiences. They understand what users want to know.
| Bot | Typ | Zugriffe |
|---|---|---|
| GPTBot | training | 1713 |
| Meta ExternalAgent | training | 883 |
| Facebook Crawler | direct | 786 |
| OpenAI SearchBot | search | 618 |
| ByteSpider | training | 449 |
| ChatGPT User | direct | 390 |
| ClaudeBot | training | 338 |
| PerplexityBot | search | 26 |
| Perplexity User | direct | 22 |
| Claude User | direct | 5 |
Accesses per Day and Crawler
Relevance Instead of Traffic: the Metrics of the Future
Currently, much in the SEO world revolves around the decline of SEO traffic. Because Google, with its AI Overviews, provides the answers itself, SEO traffic is decreasing. Websites are displayed – visible in the impressions in Google Search Console. But they receive fewer clicks from the search engine.
This trend will continue in the future. Because Google is planning an AI Mode, a kind of ChatGPT interface. Then, one can chat with the AI. The ten blue links will then be definitively history. Some websites will still be cited as sources – often those websites that are already in the top 10 and offer granular, precisely matching answers and deeper content.
However, users will also search much more specifically, describing in detail and repeatedly asking new chat questions. The answers will be highly individualized. Here, it is crucial to understand user behavior, monitor it with new AI tools, and develop content strategies from it. This is precisely what we conduct workshops with our clients for.
Because even if traffic declines: The demand for products and services still exists. Users will therefore search for companies and solutions. For this, they will be guided – by AI answers. The main task in the future will be: How does AI view your company? How does it assess you and how does it speak about you?
Background Video: Logfiles, Crawlers, and AI Visibility
You are currently viewing a placeholder content from Default. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationWe Advise Marketing Teams on SEO, Content, and AI
For anyone seeking an SEO sparring partner or comprehensive SEO consulting, Fabian and Benjamin are the perfect choice. As an in-house SEO expert, I sought competent SEO support and found it with them. During a very pleasant introductory meeting, we jointly developed an approach to establish SEO among the individual stakeholders and thereby gain more internal visibility for the subject. Based on this initial discussion and a joint kick-off with the stakeholders, they developed a sound SEO and content strategy for a product area. Since then, they have both supported me as sparring partners during implementation.
The collaboration with Fabian and Benjamin is a great pleasure for me and all project participants, as they are masters of their craft and bring immense passion for SEO. This combination is infectious and motivates everyone to achieve peak performance.
I look back with great satisfaction on the SEO project that we carried out with Benjamin and Fabian. Firstly, the collaboration was on a very pleasant, goal-oriented level. Secondly, the consultation was structured in such a way that we could immediately start with the implementation. We received very concrete optimization suggestions that were tailored to our internal capacities and therefore realistically implementable. Even though only a few months have passed, we can already see the first results of our work in the Google rankings. Traffic for our relevant keywords is steadily increasing, and we still have enough concrete work packages from Benjamin in the pipeline to boost our site. Furthermore, through the consultation, we have acquired so much internal SEO know-how that we can now optimize our site independently. And most importantly: Visitor numbers have doubled – with correspondingly positive effects on sales.
Deepen your SEO Knowledge: Every Monday via Email
