Retry a Cloud Storage Operation with Exponential Backoff

Wraps a function so that transient failures — dropped connections, expired tokens, HTTP 5xx from the storage backend — are retried with a randomised exponential backoff instead of failing the whole pipeline run.

Usage

with_storage_retry(f, max_times = 10, pause_cap = 300, quiet = FALSE)

Arguments

f

The function to wrap.

max_times

Maximum number of attempts. Default 10.

pause_cap

Longest pause between attempts, in seconds. Default 300.

quiet

Logical. If FALSE (default), each retry is logged at WARN level.

Value

A function with the same formals as f that retries on error.

Details

A thin wrapper over purrr::insistently() with purrr::rate_backoff(), centralising the retry policy that the country pipelines previously each maintained by hand.

See also

insistent_upload_cloud_file(), insistent_download_cloud_file()

Examples

if (FALSE) { # \dontrun{
insistent_get <- with_storage_retry(get_trips, max_times = 5)
} # }