I am trying to archive lefty reddit in the likely off chance that these subs get nuked. Any more subs to add to the list? Any advice on going about archiving it? What should I do with in bedded links?

  • Muad'DibberMA
    4•4 years ago

    Another possibility would just be using a tool like scrapy, creating a list of subreddits to scrape. I could test this out on a smaller subreddit sometime today. It wouldn’t be nice json, but it’d at least be a comprehensive and simple way to back it up.

    • @ImARabbit
      3•4 years ago

      I’m looking into scraping the json. Scrapy would be awful

      • Muad'DibberMA
        2•4 years ago

        Ya I’m trying it out, and its not going well @Closesniper@lemmygrad.ml . Reddit blocks all of these scrapers, esp on comments. They want you to use their json API, which is understandable.

        • @ClosesniperOP
          2•4 years ago

          yea same here i am getting a headache trying to get this to work, if we would want comments it would have to be manual. I am trying something so will update soon if I can get this program to work.

        • @ImARabbit
          2•4 years ago

          I’m able to get the banned subreddit data through the pushshift API in Python! I think we can recreate the banned subs

    • @ClosesniperOP
      2•4 years ago

      I will look into that option, and yes if you could test it out on some smaller subs that would be helpful