...

Job restartability was introduced in PDI 5.0 to allow ETL developers to automatically skip over successfully executed parts of a job after a previously failed run.

Use Cases

Picture a hypothetical ETL workflow run nightly by a corporate headquarters.  The workflow executes a transformation to update the stores table first, then a transformation to pull down and update sales data for each store, next a transformation to update inventory counts based on sales, and finally, a report for low inventory so orders and shipments can be arranged.

If any of these steps fail, it is likely that the job could be restarted at that point rather than re-running any of the earlier steps again.  In this case, checkpointing would potentially provide a big benefit to the ETL maintainer and operations team.

On the other hand, if the steps that handle sales data and inventory counts are always sourcing "live" data, checkpointing might cause issues.  If the job failed during the inventory update due to a database table permission issue, and the sales data step was checkpointed, the subsequent run of the job might pull in new inventory data that is out of sync with the previously executed sales data.  The ETL designer should consider their ETL workflow and determine how checkpointing might interact to determine whether checkpoints can be used and after which steps.

Checkpoint log table

To keep track of the various execution attempts inside a single run of a job, we need a checkpoint log table.

...