• TychoOrdo@kbin.social
      link
      fedilink
      arrow-up
      2
      ·
      1 year ago

      Works on a lot of news sites. I think they let web crawlers get the full text in order to improve their search ranking, however it also lets in crawlers with other purposes like archiving.