Drunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 2 days agoHow to combat large amounts of Ai scrapersmessage-squaremessage-square40linkfedilinkarrow-up10arrow-down10file-text
arrow-up10arrow-down1message-squareHow to combat large amounts of Ai scrapersDrunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 2 days agomessage-square40linkfedilinkfile-text
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
minus-squarefubarx@lemmy.worldlinkfedilinkEnglisharrow-up0·1 day agoIf nginx, here’s an open-source blocker/honeypot: https://github.com/raminf/RoboNope-nginx If you have it set up to be proxied or hosted by Cloudflare, they have their own solution: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
minus-squareDrunk & Root@sh.itjust.worksOPlinkfedilinkEnglisharrow-up0·15 hours agoill check robonope out seems promising
minus-squareikidd@lemmy.worldlinkfedilinkEnglisharrow-up0·edit-222 hours agoI wonder why that RoboNope doesn’t just make a fail2ban entry for anything that accesses a disallowed url and drop them entirely. Actually this look like it would do something similiar, then dumps them to fail2ban after the re-access the honeypot page too many times: https://petermolnar.net/article/anti-ai-nepenthes-fail2ban/
If nginx, here’s an open-source blocker/honeypot: https://github.com/raminf/RoboNope-nginx
If you have it set up to be proxied or hosted by Cloudflare, they have their own solution: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
ill check robonope out seems promising
I wonder why that RoboNope doesn’t just make a fail2ban entry for anything that accesses a disallowed url and drop them entirely.
Actually this look like it would do something similiar, then dumps them to fail2ban after the re-access the honeypot page too many times: https://petermolnar.net/article/anti-ai-nepenthes-fail2ban/