This is very concerning. DOJ has stated explicitly that any CSAM was removed before releasing the files. Should I remove the magnet link to the merged Data Set 9 torrent?
I haven’t looked inside any of these sets myself. My primary goal has been to get the DOJ data distributed.
I second matrix
I will grab the first segment: -–startByte 49981423616 --endByte 60299411455 (9.61 GB)
Awesome! You’re a machine
Absolutely! By the way, I hadn’t thanked you yet for your massive effort here. Thank you very much for putting this all together. Also, love your username.
Do you think we could modify the script to use HTTP Range headers and download from the end of the file to the beginning? Or, perhaps we could work together and target different byte ranges?
You seem much better versed in this than I am to know what’s possible.
You might try merging with the set below to see if you’ve scraped files that aren’t in it?
/u/susadmin’s More Complete Data Set 9 (96.25 GB)
De-duplicated merger of (45.63 GB + 86.74 GB) versions
I bet you’ve grabbed a bunch of missing pieces from the puzzle.
I’ve been trying to achieve the same thing using aria2 with 8 concurrent download threads + cookies exported from my browser, the user agent from my browser, and a random retry interval between 5-30 seconds after each download failure.
But I think I’ve been blocked by the server.
My download attempts started to fail before I began using my browser’s user agent, so it’s difficult for me to know what exactly caused me to get blocked. The download was incredibly fast before things started breaking and could’ve finished within 30 minutes.
Does anyone know how long the apparent IP ban lasts?
Amazing, thank you. That was my thought, check hashes while merging the files to keep any copies that might have been modified by DOJ and discard duplicates even if the duplicates have different metadata, e.g. timestamps.
Thank you for this!
I’ve added all magnet links for sets 1-8 to the original post. Magnet links for 9-11 match OP. Magnet link for 12 is different, but we’ve identified that there are at least two versions. DOJ removed files before the second version was downloaded. OP contains the early version of data set 12.
When merging versions of Data Set 9, is there any risk of loss with simply using rsync --checksum to dump all files into one directory and merge the sets?
deleted by creator
deleted by creator
deleted by creator
deleted by creator
deleted by creator
deleted by creator
deleted by creator
deleted by creator
At first read that came off as callous, but I see your point. I had that thought as well regarding improving female birth control. Where’s the research into a hormone-free pill for women?
The flashing banner at the top says that it includes 101GB of Data Set 9. Unfortunately, I think they just grabbed the larger of the two torrents.