Batch-Jobs in Cadence

Having these requirements I would just implement all the processing in an activity. This activity would:

  1. Paginate through results
  2. Heartbeat the progress (page token) back to Temporal
  3. In case of failure restart execution from the last heartbeated page token
  4. Process results locally, possibly using parallel threads
  5. In case of failure keep retrying locally

The workflow would invoke this activity with an appropriate retry policy to ensure that it is retried if its worker dies.