ai-crawler-honeypot

This honeypot provides a transparent, public log of web crawler requests, allowing observation of how these agents interact with websites. Key features include: * Real-time logging of web crawler requests * Publicly accessible daily logs * Adherence to robots.txt standards tracking * Privacy-focused data handling (client IPs never stored) This service is designed to offer insights into web crawler behavior, showcasing whether agents follow standard web protocols like robots.txt or if they disregard them. It presents a clear overview of agent activity without compromising user privacy, using cryptographic hashing for data preservation. Built for webmasters, developers, and researchers, this tool helps understand the operational patterns of web agents. It serves as an educational resource to demonstrate the practical application of web crawling protocols and the impact of different crawler behaviors on a live site. Use it to monitor and analyze web agent interactions, providing valuable data for site optimization and security studies.