Identify and Block AI Bots: Myra Keeps You Protected!

AI Bots Massively Increase Server Load

Such automated requests increase the server load and bandwidth usage of affected websites significantly and, in the worst case, can even lead to disruptions or outages. For example, in mid-March 2025, the Git hosting service SourceHut reported “disruptions due to aggressive LLM crawlers”. As with a DDoS attack, the servers were temporarily overloaded by the immense number of requests from AI crawlers.

OpenAI and other developers of AI models claim that their crawlers respect the site operators' settings in the robots.txt file, which defines which bots are allowed to search which areas of the domain. However, some AI bots apparently deliberately disregard these restrictions or even fake a different user agent to pretend to be a regular browser.

Block A-Based Bots and Crawlers Using WAF Rules

This is where the Myra WAF comes in, using behavioral analysis and fingerprinting to detect such deception attempts. Myra examines the trustworthiness of fingerprints using self-learning algorithms that analyze hundreds of billions of data points every day. Rate limiting and advanced scoring can be used to restrict or block requests. It is also possible to create specific WAF rules to block AI bots and crawlers.

With the Myra WAF, you can:

• Set up customized blocker rules for bots: Administrators can define a wide range of rules that target specific user agent data or other bot characteristics.

• Reliably identify bots: In combination with Myra Bot Management, the WAF enables unambiguous detection of bots via fingerprints, ensuring that bots using different IP addresses or user agents can also be blocked.

• Maintain granular control: With the WAF, you can block automated requests in a targeted manner and define exceptions, for example for trusted bots such as search engine crawlers.

Manage Collections of WAF Rules with Tags

The Myra WAF offers the option to create and manage entire collections of WAF rules using tags. This function makes it easier for users to apply their security policies efficiently and consistently across multiple domains.

Key benefits of the tag feature:

• Centralized management: Users can group multiple WAF rules under a tag, such as “AI bots”, to quickly and easily apply those rules to all their domains.

• Flexible customization: Tags allow administrators to make changes to a group of rules that are automatically applied to all associated domains.

• Time efficiency: Instead of configuring each domain individually, users can define and maintain global security policies using tags.

Example: A company could create a tag called “AI crawler protection” that contains rules for blocking certain AI crawlers. This tag is then applied to all of the company's domains – a simple and scalable solution for effective protection.

With the Myra Web Application Firewall (WAF), you get a powerful tool to defend against unwanted requests from AI bots and crawlers that consume your server resources and increase your operating costs.