Automated HTTP clients often identify themselves as curl/, Wget/, python-requests/ or Go-http-client/. These signatures can be useful on a public site that has no API, but they are not evidence of an attack on their own. A trigger scenario produces an immediate ban, so start with a measured policy and exempt approved integrations.
Use parsed access logs only
A common false-positive cause is applying User-Agent rules to Apache error logs, where the User-Agent may be missing altogether. Restrict the filter to normalized access events.
type: trigger
name: custom/automated-http-clients
description: "Flag selected CLI HTTP clients"
filter: |
evt.Meta.log_type == 'http_access-log' &&
evt.Parsed.http_user_agent != nil &&
evt.Parsed.http_user_agent matches '(?i)^(curl/|wget/|python-requests/|python-urllib/|go-http-client/|aiohttp/|httpx/|apache-httpclient/|okhttp/|postmanruntime/)'
groupby: evt.Meta.source_ip
blackhole: 5m
labels:
remediation: true
service: http
behavior: "http:automated-client"
spoofable: 1
The anchored ^ pattern reduces accidental matches inside full browser strings. Add or remove clients based on your own logs; generic names such as Java/ or axios/ can match legitimate API consumers. Treat User-Agent as spoofable and consider rate thresholds, endpoints and response codes before enabling an immediate ban on a mixed-use site.
Test before activation
crowdsec -t
cscli explain --log '198.51.100.10 - - [13/Nov/2026:10:00:00 +0100] "GET / HTTP/1.1" 200 100 "-" "curl/8.0.1"' --type apache2
Documentation-only addresses are used in the example. Confirm that private monitoring addresses are excluded by the local parser whitelist, and that service checks do not use public addresses absent from your allowlist.
Reduce collateral damage
For many sites, a leaky rate-based scenario is safer than an unconditional trigger. Never add OpenAI, Anthropic or Perplexity crawler names to a ban regex unless you intentionally want to block those services. Do not rely on a declared User-Agent to verify their identity: check official source IP ranges and permitted access paths.