• loki@lemmy.ml
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    I can’t find any information about their search engine crawler. Isn’t it standard for search engines to label their crawlers or something?

    • darth_helmet@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      It’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion.

      So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.