Drunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 3 days agoHow to combat large amounts of Ai scrapersmessage-squaremessage-square46linkfedilinkarrow-up10arrow-down10file-text
arrow-up10arrow-down1message-squareHow to combat large amounts of Ai scrapersDrunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 3 days agomessage-square46linkfedilinkfile-text
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
minus-squareSheldan@lemmy.worldlinkfedilinkEnglisharrow-up0·1 day agoSome of them are at least honest and have it as a user agent.
minus-squarekrakenfury@lemmy.sdf.orglinkfedilinkEnglisharrow-up0·1 day agoIs ignoring robots.txt considered “honest”?
Some of them are at least honest and have it as a user agent.
Is ignoring robots.txt considered “honest”?