Robust Large-File Download

Purpose

Download multi-GB files reliably over flaky or throttled connections by combining three techniques:
1. HTTP Range chunking — split the file into chunks; any interruption only loses the current chunk, and the rest resumes.
2. Multi-connection parallelism — download several chunks at once to defeat per-connection bandwidth caps (a single connection often crawls at ~200 KB/s while the link is much faster).
3. Detached Scheduled Task — run the download under Windows Task Scheduler so closing WorkBuddy does not kill it.

When to use

• User reports a large download "keeps failing", stalls, or cannot resume.
• Downloading research datasets (Zenodo, OpenML, etc.) larger than ~1 GB.
• A single browser/curl download is too slow or unreliable.

Workflow

1. Verify Range support before writing anything:
   curl -s -r 0-0 -D hdr.txt -o /dev/null <URL> then inspect hdr.txt for accept-ranges: bytes and content-range: bytes 0-0/<TOTAL>. A 206 response confirms chunking; the total size comes from the header.
2. Check disk space: free space must exceed file size + one chunk.
3. Run the bundled downloader scripts/download_chunked.py — it auto-detects total size, downloads in 200 MB chunks, 4 parallel workers, resumes via .ok{i} markers, and optionally verifies MD5:
   python scripts/download_chunked.py --url <URL> --out <path> [--md5 <md5>] [--workers 4] [--chunk 209715200]
4. Run it detached so it survives app close — wrap in a Windows Scheduled Task:
   schtasks /create /tn <TaskName> /tr "\"<python>\" \"<script>\" --url <URL> --out <path>" /sc once /st 00:00 /f
   then schtasks /run /tn <TaskName>.
   - Do NOT use a .bat wrapper with non-ASCII paths (encoding garbles filenames). Pass python + script directly in /tr.
5. Monitor by tailing <out>.progress.log, or measure aggregate speed = sum of all <out>.cN temp sizes over a 30s window.
6. Resume is automatic: re-run the same command; completed chunks are skipped via markers.
7. Stop: schtasks /end /tn <TaskName> + taskkill /F /PID <pid>.

Key gotchas

• Per-connection throttle: if one connection crawls (~200 KB/s) but the link is faster, parallelism (4 workers) often yields 5–9× speedup. Measure sequential speed first, then switch to parallel if slow.
• Rate limits: some hosts (e.g. Zenodo) cap requests/min (~133). Large chunks (200 MB) keep request count low; sequential with 1s sleeps stays far under the cap.
• Resumability across restarts: never truncate the output on restart — open with O_RDWR|O_CREAT and os.pwrite each chunk at its own offset. Track completion with per-chunk marker files (not a single offset) so out-of-order parallel completion is safe.
• Safe-delete noise: a rm of an in-use temp file may emit trash-fallback errors — harmless; the next run overwrites/removes it.
• Proxy environments: curl honors http_proxy/https_proxy and Range through proxies; prefer a curl subprocess over urllib to avoid proxy surprises.

Reference

Bundled scripts/download_chunked.py is the complete, parameterized, production-used implementation.