“Sync the last year” sounds like one operation. On a phone, network interruption, process suspension, provider query limits, and partial server acceptance make it a long sequence of operations.
A reliable backfill is resumable by design.
Work in bounded windows#
Split history into small date ranges:
@dataclass(frozen=True)
class BackfillWindow:
start: date
end: date
def next_window(oldest_completed: date | None, today: date) -> BackfillWindow:
end = (oldest_completed - timedelta(days=1)) if oldest_completed else today
start = max(end - timedelta(days=13), today - timedelta(days=364))
return BackfillWindow(start=start, end=end)
A failed two-week window is cheap to retry. Progress becomes visible, and each request stays inside predictable payload and execution limits.
Store durable progress separately#
Track the oldest completed day, target boundary, last attempt, and status. Advance the cursor only after the server accepts the whole window.
Do not infer completion merely from some rows existing. A valid date may contain no samples, and partial writes may look complete.
Permit overlap#
The safest resume logic may resend the last completed boundary or overlap windows slightly. Idempotent server writes make overlap harmless and protect against ambiguous acknowledgements.
Distinguish progress from freshness#
Historical completion and live synchronization are different state. A user may finish a one-year backfill while today's incremental sync continues independently.
Make cancellation ordinary#
The application should be able to stop after any committed window and continue later. Avoid one transaction across the entire history and do not hold database locks while a device gathers data.
Expose progress in user-friendly terms, but keep exact cursors for engineering diagnostics. Test interruption before upload, during upload, after server commit but before acknowledgement, and after local cursor update.
Resumability is not an error-recovery add-on. It is the normal execution model for work that lasts longer than one reliable process lifetime.