Weekly thread to discuss whatever you’re working on, big or small, at work or in your free time.

  • dubbel
    link
    fedilink
    English
    arrow-up
    4
    ·
    4 months ago

    Private project, not really security related: Crawling robots.txts to gather some statistics on which bots people are most often excluding - weirdly I couldn’t find any recent/regularly updated stats on this.

      • dubbel
        link
        fedilink
        English
        arrow-up
        3
        ·
        3 months ago

        It started with a popular mastodon posts on how to block openai crawlers I think, and I’d like to know whether people are actually implementing it.

        • PaddleMaster@beehaw.org
          link
          fedilink
          English
          arrow-up
          1
          ·
          3 months ago

          That’s neat. I’m curious about this now. With “normal” search engines that have generally gone to shit, AI chat bots are on trend to give better results. If the robots.txt file is blocked from OpenAI, can I assume it hits other chatbots? And would that extend to Google/bing?