this post was submitted on 07 Jan 2025
1 points (100.0% liked)

It's A Digital Disease!

23 readers
1 users here now

This is a sub that aims at bringing data hoarders together to share their passion with like minded people.

founded 2 years ago
MODERATORS
 
The original post: /r/datahoarder by /u/FederalRub6835 on 2025-01-07 04:59:44.

What the title says. I've been trying to archive art of twitter accounts that are being deleted, and I can never get more than around 1200 images at a time. I can't find any way to continue the link search where I left off, and everything I've tried hasn't managed to get past this. I've tried going into the config settings and setting the task size above 1000, I've tried changing it to include replies. I tried going into the general settings and changing the delay between crawls to 2 seconds, but the only thing that did was slow down the rate it crawled links. It still stopped at 1293, which is only 1 more image than when I tried it without altering any settings at all.

Any advice for what to change the settings to on wfdownloader in order to stop it from getting capped? (I don't just need to know what settings to change, I also need exact values if you can provide them.)

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here