A crawler can be genuine and still trigger an over-aggressive security rule. A legitimate search bot may request a missing URL without a Referer and receive an automated 30-day ban. Allowing its User-Agent in Apache does not help if the CrowdSec firewall bouncer drops its source IP first.
Identify actual blocked networks
Use the providers’ current, official machine-readable IP lists: OpenAI publishes separate ranges for GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic and Perplexity publish their own crawler ranges. IP ownership is stronger evidence than a User-Agent string, which is trivially spoofed. Never whitelist a large cloud ASN simply because some bot addresses are hosted there.
cscli decisions list -a --ip 203.0.113.10
ipset test crowdsec-blacklists-0 203.0.113.10
cscli alerts list -a
The documentation IP above is only an example. Investigate the Reason and Source fields: local scenarios, imported lists and central feed decisions may have different removal paths.
Remove decisions using cscli
Use cscli decisions delete --ip for a verified address, or cscli decisions delete --range ... --contained for an official prefix. The --contained flag matters: it selects decisions inside the specified CIDR instead of deleting unrelated values. Test on one known address before bulk operations.
cscli decisions delete --ip 203.0.113.10
# For a verified official prefix only:
cscli decisions delete --range 203.0.113.0/28 --contained
Prevent immediate re-bans
The real fix is to whitelist verified IP ranges in a CrowdSec parser, so new log events never reach triggering scenarios.
name: custom/verified-crawlers
description: "Verified crawler IP networks"
whitelist:
reason: "official crawler network"
cidr:
- "203.0.113.0/28" # DOCUMENTATION ONLY: replace with official prefix
Save the file in /etc/crowdsec/parsers/s02-enrich/, run crowdsec -t, and activate it during a controlled reload. Validate with cscli explain --log '...Apache access line...' --type apache2: the parser should report that the event was whitelisted. A one-off decision deletion is insufficient if the triggering rule remains unchanged.
Review the whitelist periodically because official cloud ranges change. Explicitly decide whether you want training crawlers, search crawlers and user-triggered retrieval clients to have the same access.