moe90@feddit.nl to Technology@lemmy.mlEnglish · 1 month agoReddit is now blocking major search engines and AI bots — except the ones that paywww.theverge.comexternal-linkmessage-square71fedilinkarrow-up1379arrow-down15 cross-posted to: tech@pawb.socialinformatica@feddit.ittechnology@hexbear.net
arrow-up1374arrow-down1external-linkReddit is now blocking major search engines and AI bots — except the ones that paywww.theverge.commoe90@feddit.nl to Technology@lemmy.mlEnglish · 1 month agomessage-square71fedilink cross-posted to: tech@pawb.socialinformatica@feddit.ittechnology@hexbear.net
minus-squarecmnybolinkfedilinkEnglisharrow-up9·1 month agoSome Lemmy instances disallow indexing in robots.txt, however indexers can choose to ignore that and actually blocking them takes a lot more effort.
minus-squarebrucethemoose@lemmy.worldlinkfedilinkarrow-up8·1 month agoSome places on a “budget” like Ao3 just rate limit hard. I don’t like that solution at all though.
Some Lemmy instances disallow indexing in robots.txt, however indexers can choose to ignore that and actually blocking them takes a lot more effort.
Some places on a “budget” like Ao3 just rate limit hard.
I don’t like that solution at all though.