Got a project in mind?
We’d love to hear about it.

Get in touch
Yellow Peach web design agency

AI crawlers are becoming an increasingly common sight in website traffic logs. Alongside familiar search engine bots such as Googlebot, websites are now receiving requests from services collecting content for AI model training, AI-powered search engines and automated assistants.

While much of this activity is legitimate, the growing volume of automated requests can create problems for websites that aren’t equipped to handle them.

At Yellow Peach, we’re seeing high levels of bot traffic become a more frequent issue across the websites and digital platforms we manage. It’s not just AI crawlers, either. Poorly configured scrapers repeatedly requesting pages, automated vulnerability scans and more malicious bot activity can all place unnecessary pressure on hosting infrastructure. In some cases, this traffic consumes significant server resources without ever appearing in conventional website analytics.

AI crawlers are part of the picture, but they’re not the only cause. The challenge is understanding which automated visitors are useful, which are causing problems and what you can do about them. Blocking everything isn’t necessarily the answer, particularly when some crawlers help people discover your content through search engines and AI-powered services.

In this guide, we’ll explain how to identify problematic bot activity and manage it without unnecessarily restricting legitimate traffic.

Why are so many bots visiting websites?

Automated crawlers have been part of the internet for decades. Search engines use them to discover content and keep their indexes up to date, while monitoring services check whether websites are available and functioning correctly.

AI has introduced additional reasons for automated systems to access publicly available content. Some crawlers collect information that may be used to train AI models, while others retrieve content to support AI-powered search results or respond to individual user queries.

These aren’t necessarily the same thing. OpenAI, for example, identifies GPTBot as a crawler associated with potential model training, while OAI-SearchBot supports discovery in ChatGPT search. Website owners can therefore make different decisions about allowing these services.

Alongside recognised crawlers, there’s a much less predictable category of automated traffic. Scrapers may collect content or pricing information without respecting sensible request limits, while malicious bots look for vulnerable software, exposed files or opportunities to gain unauthorised access. A scraper following every possible combination of filters or repeatedly requesting the same pages may place more strain on a website than a much larger number of genuine visitors.

How does bot traffic affect website performance?

Every request to a website requires resources, whether that’s bandwidth, processing power or database activity. On a well-configured website, caching allows many requests to be handled efficiently without repeatedly generating the same content. Problems arise when automated visitors request pages faster than the infrastructure can serve them, or repeatedly access functionality that bypasses caching.

For a WordPress website, an uncached request may involve executing PHP, querying the database and assembling a page using the active theme and plugins. Bespoke Laravel applications can involve even more complicated processes, particularly when pages depend on multiple database queries or external services.

If hundreds of automated requests arrive in quick succession, they can compete with genuine visitors for available resources. The result might be slower page loading, increased CPU usage, database bottlenecks or, in more serious cases, temporary errors and downtime.

There can also be a financial impact. Depending on your hosting arrangement, excessive requests may increase bandwidth consumption or push a website towards the limits of its hosting package. Applications that trigger paid third-party services can be particularly vulnerable if those operations aren’t appropriately cached or protected.

It’s worth remembering that a high number of requests doesn’t automatically mean there’s a problem. A crawler accessing cached pages through a content delivery network (CDN) may have very little impact on the underlying server. What matters is the amount of work those requests generate and whether that activity affects real visitors.

How can you tell if bots are causing problems?

The first step is to look beyond your usual website analytics. Google Analytics and similar platforms primarily measure activity through scripts running in a visitor’s browser. Many crawlers don’t execute those scripts, which means a website can experience a substantial increase in server traffic without any corresponding rise in reported visitors.

Server access logs are usually a better starting point. They can show when requests were made, which URLs were accessed, the IP addresses involved and the user agents identifying the software making those requests.

Look for unusual patterns. Are particular IP addresses making hundreds of requests within a short period? Is the same crawler repeatedly accessing pages? Are there large numbers of requests to search results, filtered listings or URLs that don’t exist? Repeated attempts to access login pages or common vulnerability targets may indicate something more concerning than ordinary crawling.

User agents can help identify recognised services, but they aren’t definitive proof of identity. A bot can claim to be Googlebot or another well-known crawler, so suspicious activity should be checked against the relevant provider’s published verification guidance where available.

It’s also important to compare these findings with your hosting metrics. If a spike in automated requests coincides with increased CPU usage, slower response times or exhausted PHP workers, you have a stronger indication that bot traffic is contributing to the problem.

Using Cloudflare to investigate bot traffic

If your website uses Cloudflare, its analytics and security tools can make identifying automated traffic considerably easier. Cloudflare can provide information about request volumes, suspicious activity and the security rules being triggered before requests reach your hosting server.

Its AI Crawl Control features also help website owners understand activity from recognised AI crawlers and manage access according to their preferences. This can be useful when distinguishing between crawlers associated with AI search and those collecting content for model training.

We’d recommend looking at the overall pattern rather than focusing solely on the number of requests. A large volume of cached traffic may be relatively harmless, while a smaller number of requests targeting expensive functionality could be responsible for a noticeable slowdown. Cloudflare’s visibility is limited to traffic passing through its network, so it’s still worth checking your hosting logs if the source of a performance problem isn’t clear.

How can you manage unwanted bot traffic?

Once you’ve identified what’s happening, there are several ways to reduce the impact. The most appropriate solution depends on whether you’re dealing with a legitimate crawler, an inefficient scraper or genuinely malicious activity.

Control crawler access with robots.txt

The robots.txt file allows website owners to communicate crawling preferences to automated services. For example, you might decide to allow a crawler that supports AI-powered search while restricting one associated with model training.

The following example disallows GPTBot while allowing OAI-SearchBot:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot Allow: /

This distinction is useful because blocking one type of AI crawler doesn’t necessarily mean excluding your website from every AI-related discovery service. However, robots.txt isn’t a security feature. It relies on crawlers voluntarily respecting your instructions, and poorly built scrapers or malicious bots may ignore it completely. For those visitors, you’ll need enforceable restrictions rather than crawling preferences.

Use firewall rules and rate limiting

Cloudflare’s Web Application Firewall (WAF) can help prevent unwanted requests from reaching your website in the first place. Security rules can identify common attack patterns, while custom rules and rate limiting can address excessive requests to particular pages or endpoints.

For example, if automated visitors are repeatedly accessing an expensive search function, a targeted rate limit may be more appropriate than blocking an entire crawler. Likewise, repeated attempts to access administrative areas can often be handled through security rules before they reach WordPress or Laravel.

These controls need to be configured carefully. Overly aggressive blocking can affect genuine visitors, search engines or legitimate integrations, particularly where multiple users share the same IP address.

Improve caching and application performance

Bot management shouldn’t be the only line of defence. A website that’s efficiently built and properly cached will generally cope better with automated requests, even when traffic increases unexpectedly.

For WordPress websites, this means checking that public pages are being served from an appropriate cache and identifying plugins or functionality that generate unnecessarily expensive requests. On Laravel applications, it may involve reviewing database queries, caching repeated operations and protecting resource-intensive endpoints.

It’s also worth examining how your website generates URLs. Search pages, product filters and pagination can sometimes create thousands of combinations that crawlers attempt to follow. Reducing unnecessary crawl paths can help prevent wasted requests without restricting access to valuable content.

In our experience, the most effective solution is often a combination of better caching, targeted security rules and sensible limits on expensive functionality, rather than simply blocking as many bots as possible.

Should you block AI crawlers altogether?

There’s no single answer that suits every website. Some businesses may be comfortable allowing their content to be accessed for AI model training, while others may prefer to restrict that use. Similarly, businesses that rely on content discovery may want to retain access for crawlers supporting AI-powered search.

It’s important to separate those decisions from website performance. A legitimate AI crawler isn’t necessarily causing a problem, and blocking it won’t help if the real issue is an aggressive scraper or automated attack.

We’d generally recommend reviewing which crawlers are visiting your website, understanding their purpose and measuring their actual impact before introducing restrictions. That allows you to protect your infrastructure without unnecessarily limiting how people discover your business online.

Malicious activity is a different matter. Automated attempts to exploit vulnerabilities, access restricted areas or overwhelm a website should be addressed through appropriate security measures rather than treated as normal crawling.

Keeping your website performing as bot traffic grows

AI crawlers have added another dimension to automated website traffic, but the underlying challenge isn’t new. Websites have always needed to handle search engines, scrapers, monitoring services and unwanted requests. What’s changing is the number of automated systems accessing public content and the different purposes behind those requests.

We’re seeing these issues more frequently across client websites, and the solution isn’t always as simple as blocking a particular bot. Understanding where requests originate, what they’re accessing and how much work they’re creating is essential before deciding what to do.

At Yellow Peach, we build, maintain and support WordPress websites and bespoke Laravel applications, with performance, security and infrastructure considered as part of the wider platform. Whether it’s investigating unusual traffic, improving caching or introducing more effective protection against unwanted requests, our focus is on keeping websites reliable for the people who actually use them.

If your website is experiencing unexplained slowdowns, unusual bot activity or increased hosting costs, get in touch. We can help identify what’s happening and recommend a practical way forward.

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?

Ready to push your platform?