Job Reconciliation
Job reconciliation is the step after submission where the recorded run state is compared with scheduler state, logs, checkpoints, and output artifacts. It prevents duplicate relaunches and stale “probably running” assumptions.
A reconciled job state can be represented as:
The job is not operationally closed until these pieces agree.
Why It Matters
- Long jobs often finish after the person or agent that launched them has stopped watching.
- A scheduler terminal state does not prove the output artifact is complete.
- A missing log line does not prove failure if the job wrote a valid checkpoint.
- Relaunching without reconciliation can duplicate expensive work.
Generic Reconciliation Flow
- Query scheduler state for the job or job array.
- Classify the terminal state: completed, failed, cancelled, timeout, out-of-memory, preempted, or unknown.
- Inspect the final log lines and resource summary.
- Check output artifacts and completion markers.
- Record the outcome in the run record.
- Decide whether to resume, retry, rerun with different resources, or stop.
Outcome Table
| Scheduler State | Artifact State | Interpretation |
|---|---|---|
| Completed | Complete | Close the run and record verification. |
| Completed | Missing or partial | Treat as artifact failure, not success. |
| Failed or timeout | Valid checkpoint | Resume if the protocol allows it. |
| Failed or timeout | No checkpoint | Diagnose before relaunch. |
| Unknown | Unknown | Do not report completion. Gather more evidence. |
Public Boundary
Public notes should describe the failure class and recovery lesson without publishing private job IDs, queue names, hostnames, account names, internal paths, user names, or unpublished results.
Checks
- Is the scheduler state terminal?
- Does the output have a completion marker?
- Is the checkpoint compatible with the current code and config?
- Did the run record capture the final state and recovery decision?
- Is the public summary sanitized?