Take back control from unwanted AI crawlers.

A family of open-source tools for detecting, throttling, deceiving and blocking AI scrapers, built for creative, FOSS and documentation sites that never agreed to be training data. One defense strategy, three implementations: Python, Rust and .NET.

AI Scraping Defense logo

Three implementations, one defense

All three share the same roles (edge enforcement, escalation, tarpit, blocklists, operator tooling) and are licensed GPL-3.0. Pick the one that fits your stack.

ai-scraping-defense

PythonNginx + LuaIn development & testing

The original, microservice-based stack. Nginx and Lua at the edge, Python services behind it, containerized for Docker and Kubernetes.

  • Heuristics, ML model and optional LLM analysis
  • Tarpit API and AI Labyrinth honeypots
  • Community blocklist and federated peer sharing
  • Pay-per-crawl accounting and plugin API

ai-scraping-defense-rust

RustCargo workspaceIn development

A fast, explicit, service-based control plane. Tracks API and deployment parity with the Python original.

  • Escalation engine, tarpit, CAPTCHA, admin UI
  • Config recommender with incident alerts
  • Edge ops for WAF, CDN, TLS and blocklist sync
  • Cargo, Docker Compose, Kubernetes and Helm; tagged release bundles for Linux, Windows and macOS

ai-scraping-defense-iis

C# / .NETASP.NET CoreExperimental

A pure-.NET implementation for Windows Server and IIS shops, shipped as a single deployable for v1.

  • Redis-backed blocklist enforcement at the edge
  • Deterministic tarpit with synthetic pages
  • Authenticated operator API and dashboard
  • SQLite, PostgreSQL or SQL Server intake; webhook, Slack and SMTP alerts

Defense in depth

Cheap checks run first at the edge. Only suspicious traffic is escalated to heavier analysis, so legitimate visitors are not slowed down.

DetectHeuristics on user agents, headers, paths and request frequency flag likely bots.
EscalateSuspicious requests are scored by reputation data, a trained model and, optionally, an LLM.
DeceiveTarpits and endless honeypot pages waste the resources of confirmed scrapers.
BlockRedis-backed blocklists and adaptive per-IP rate limits shut persistent bots out.
ShareOptional community and peer blocklists spread what one site learns to others.

Pay-per-crawl support is included for sites that want to experiment with monetization. Whether bots may pay for access or stay blocked is entirely the operator's choice.

Which one should I use?

PythonRust.NET
Best forLinux hosts with Nginx; the most complete feature setPerformance-minded operators; native binariesWindows Server / IIS environments
ShapeNginx + Lua + Python microservicesMulti-service Rust workspaceSingle ASP.NET Core deployable
Deploy withDocker, KubernetesCargo, Docker Compose, Kubernetes, Helm, release bundlesIIS / ASP.NET Core host
StatusIn development and testingIn developmentExperimental release
LicenseGPL-3.0

Companion projects

MCP servers that give an AI agent explainable request-risk assessments. They work alongside the main stack or on their own.

request-guard-mcp

RustMCPv1.0.0

An MCP server that classifies inbound HTTP requests for risk and returns explainable bot and abuse signals, with optional enrichment. The AI Scraping Defense projects are optional clients. A companion agent skill is included.

request-guard-mcp-dotnet

C# / .NETMCP

A from-scratch .NET port of request-guard-mcp targeting the same 23 tools and behavior, checked against the Rust server by a differential test suite.

Related sites

Where to find more of this work.

requestguard.dev

The home of Request Guard MCP: a recorded example verdict, install guides for every platform, and optional agent instruction bundles.

Get started

Each repository documents its own setup. A quick taste of the Rust stack:

git clone https://github.com/rhamenator/ai-scraping-defense-rust
cd ai-scraping-defense-rust
cp config/sample.env .env
python scripts/configure_credentials.py
docker compose up --build

For the Python or .NET versions, start from the README in ai-scraping-defense or ai-scraping-defense-iis.

Heads up: most of these projects are still in development or testing. Review the configuration, test on a staging site, and keep an allowlist for the search engines and monitoring you want to keep.