• wise_pancake@lemmy.ca
        link
        fedilink
        English
        arrow-up
        58
        ·
        edit-2
        10 months ago

        robots.txt is a file available in a standard location on web servers (example.com/robots.txt) which set guidelines for how scrapers should behave.

        That can range from saying “don’t bother indexing the login page” to “Googlebot go away”.

        IT’s also in the first paragraph of the article.

      • mrnarwall@lemmy.world
        link
        fedilink
        English
        arrow-up
        18
        arrow-down
        1
        ·
        10 months ago

        Robots.txt is a file that is is accessible as part of an http request. It’s a backend configuration file that sets rules for what automatically running web crawlers are allowed. It can set both who is and who isn’t allowed. Google is usually the most widely allowed domain for bots just because their crawler is how they find websites for search results. But it’s basically the honor system. You could write a scraper today that goes to websites that it is being told it doesn’t have permission to view this page, ignore it, and still get the information

        • Echo Dot@feddit.uk
          link
          fedilink
          English
          arrow-up
          5
          ·
          edit-2
          10 months ago

          I do not think it is even part of the HTTP protocol I think it’s just a pseudo add-on. It’s barely even a protocol it’s basically just a page that bots can look at with no really pre-agreed syntax.

          If you want to make a bot that doesn’t respect robots.txt you don’t even need to do anything complicated you just need to not include the requirement to look at the page. It’s not enforceable at all.