Picks up where a wayback_save run left off: it re-tries
only the URLs whose captures failed or were still running, after first
checking whether the Wayback Machine already holds a copy of each. Use it
after a run that hit archive.org's limits, typically the next day.
Arguments
- output
The JSON Lines results file of an earlier
wayback_save()run (or ofrmbl wayback); results of the retry are appended to it.- include_gone
Logical, default
FALSE. Also capture again the pages whose failure says the page is gone.- lookup
Logical, default
TRUE. Look each failure up in the Wayback Machine (one lookup a second) before capturing again.- ...
Passed to
wayback_save():jobs,within, the keys,quiet,per_minute,max_captures.
Value
The whole run as it now stands, one row per URL (a
data frame of class rmbl_wayback_save), with the
attribute retry: how many failures there were, how
many a lookup found, how many were captured again, how many
were left as gone, and how many were left for tomorrow (a
daily limit hit today), as a one-row data frame with integer columns
failed, found_by_lookup, captured_again,
left_as_gone and left_for_tomorrow.
Details
Reads the results file a wayback_save run wrote
and tries again every URL whose latest result is
"failed" (or "submitted": a capture a stopped run
had started, which is polled rather than submitted again), so
nobody has to dig the failures out by hand. Each failed URL is
first looked up in the Wayback Machine, one lookup a second: a
capture made at any time (an earlier one of a page that has
since gone, or one made today by another run when the daily
limit refused this one) is recorded as "existing"
without capturing again. What is left is captured again with
wayback_save(), appending to the same file. A page its
source no longer has is not sent again unless
include_gone = TRUE: a new capture would only archive the error.
"Gone" means the recorded message contains archive.org's report of the
target server's reply ("target server cannot find", "returned 404",
"returned 410", "error:not-found" or "error:gone"); an HTTP 404 from
archive.org's own API does not count. A failure for a daily limit
recorded today (UTC) is not sent again until tomorrow.
Network: the lookups use the Wayback availability API and the new
captures use wayback_save(), which needs the archive.org keys.
When there is nothing left to send, no keys and no network are needed.
Each lookup that finds a capture is appended to output as an
"existing" line.
Errors: output not a single path to an existing file;
include_gone or lookup not TRUE or FALSE;
any error from wayback_save().
See also
wayback_save; rmbl wayback
--retry FILE.jsonl from the command line.
Examples
# Offline: a results file whose only failure is a page that is gone
# has nothing to send again, so no keys or network are used.
f <- tempfile(fileext = ".jsonl")
writeLines(c(
paste0('{"url":"https://example.org/","status":"saved",',
'"timestamp":"20260101120000",',
'"snapshot":"https://web.archive.org/web/20260101120000/https://example.org/",',
'"message":"","at":"2026-01-01T12:00:00Z"}'),
paste0('{"url":"https://example.org/old","status":"failed",',
'"message":"target server cannot find the page","at":"2026-01-01T12:00:00Z"}')
), f)
res <- wayback_retry(f, lookup = FALSE)
res
#> ── Wayback Machine captures ──────────────────────────────────────
#> 2 URLs: 1 saved, 0 existing, 1 failed
#>
#> status timestamp url
#> saved 20260101120000 https://example.org/
#> failed <NA> https://example.org/old
#> ──────────────────────────────────────────────────────────────────
attr(res, "retry")
#> failed found_by_lookup captured_again left_as_gone left_for_tomorrow
#> 1 1 0 0 1 0
try(wayback_retry(tempfile(fileext = ".jsonl")))
#> Error : no results file at /tmp/Rtmp1nz8V4/file1da831516dee.jsonl
# With failures to resend, keys and the network are needed.
if (FALSE) { # \dontrun{
res <- wayback_save(urls, output = "snapshots.jsonl")
# later, or the next day once the daily capture limits have reset:
res <- wayback_retry("snapshots.jsonl")
attr(res, "retry")
} # }