It’s not really in a convenient format anyway. You have to pay for the bandwith you use to download the archive, which I can understand because 32 TB is a lot of data, but you also have to download the whole of it.
A torrent would make more sense but again, that means they have to seed 32TB of data and have it on their computer somewhere. Either that or distribute it between selected seeders, and everyone can start seeding say 2TB of data at a time.
Still I’m downloading the text-only, 19gigs archive and will see what’s in there. I don’t think I’ll have the patience to go through all of it. Hope it doesn’t put me on any list lmao
What were the reasons they gave? I would guess speed, reliability, and overall size of the archive. My web host gives me only 100 gigs (which is plenty enough for a normal website).
Biggest reason is that it is convenient (more than buying a VPS and self hosting) and it’s very cheap (something like USD 0.0080 per GB (I am exaggerating upwards because I don’t remember the exact number)). The cost is also offset to the receiver a little bit so it’s easier on the uploader financially. Plus there is fault tolerance because the data is mirrored to two other physical locations.
I was curious because my employer also uses something similar (called MinIO) for hosting their datasets. But they also self host the MinIO clusters so I am still not sure what benefit it provides to them, especially since the data from this cluster is only accessed internally.
It’s not really in a convenient format anyway. You have to pay for the bandwith you use to download the archive, which I can understand because 32 TB is a lot of data, but you also have to download the whole of it.
A torrent would make more sense but again, that means they have to seed 32TB of data and have it on their computer somewhere. Either that or distribute it between selected seeders, and everyone can start seeding say 2TB of data at a time.
Still I’m downloading the text-only, 19gigs archive and will see what’s in there. I don’t think I’ll have the patience to go through all of it. Hope it doesn’t put me on any list lmao
Slightly off topic but why put it in S3 buckets instead of just hosting the file on a server? What are the advantages of it?
edit: nvm just watched a video about it
What were the reasons they gave? I would guess speed, reliability, and overall size of the archive. My web host gives me only 100 gigs (which is plenty enough for a normal website).
Biggest reason is that it is convenient (more than buying a VPS and self hosting) and it’s very cheap (something like USD 0.0080 per GB (I am exaggerating upwards because I don’t remember the exact number)). The cost is also offset to the receiver a little bit so it’s easier on the uploader financially. Plus there is fault tolerance because the data is mirrored to two other physical locations.
I was curious because my employer also uses something similar (called MinIO) for hosting their datasets. But they also self host the MinIO clusters so I am still not sure what benefit it provides to them, especially since the data from this cluster is only accessed internally.