• unexposedhazard
    link
    fedilink
    arrow-up
    12
    ·
    4 months ago

    Yeah i dont think ignoring robots.txt is even illegal. They can ofcourse just block your crawlers IP but that would be a cat and mouse game that they would lose in the end.