Once you’ve identified what’s happening, there are several ways to reduce the impact. The most appropriate solution depends on whether you’re dealing with a legitimate crawler, an inefficient scraper or genuinely malicious activity.
Control crawler access with robots.txt
The robots.txt file allows website owners to communicate crawling preferences to automated services. For example, you might decide to allow a crawler that supports AI-powered search while restricting one associated with model training.
The following example disallows GPTBot while allowing OAI-SearchBot:
User-agent: GPTBot
Disallow: /User-agent: OAI-SearchBot Allow: /
This distinction is useful because blocking one type of AI crawler doesn’t necessarily mean excluding your website from every AI-related discovery service. However, robots.txt isn’t a security feature. It relies on crawlers voluntarily respecting your instructions, and poorly built scrapers or malicious bots may ignore it completely. For those visitors, you’ll need enforceable restrictions rather than crawling preferences.
Use firewall rules and rate limiting
Cloudflare’s Web Application Firewall (WAF) can help prevent unwanted requests from reaching your website in the first place. Security rules can identify common attack patterns, while custom rules and rate limiting can address excessive requests to particular pages or endpoints.
For example, if automated visitors are repeatedly accessing an expensive search function, a targeted rate limit may be more appropriate than blocking an entire crawler. Likewise, repeated attempts to access administrative areas can often be handled through security rules before they reach WordPress or Laravel.
These controls need to be configured carefully. Overly aggressive blocking can affect genuine visitors, search engines or legitimate integrations, particularly where multiple users share the same IP address.
Improve caching and application performance
Bot management shouldn’t be the only line of defence. A website that’s efficiently built and properly cached will generally cope better with automated requests, even when traffic increases unexpectedly.
For WordPress websites, this means checking that public pages are being served from an appropriate cache and identifying plugins or functionality that generate unnecessarily expensive requests. On Laravel applications, it may involve reviewing database queries, caching repeated operations and protecting resource-intensive endpoints.
It’s also worth examining how your website generates URLs. Search pages, product filters and pagination can sometimes create thousands of combinations that crawlers attempt to follow. Reducing unnecessary crawl paths can help prevent wasted requests without restricting access to valuable content.
In our experience, the most effective solution is often a combination of better caching, targeted security rules and sensible limits on expensive functionality, rather than simply blocking as many bots as possible.