Workflow gets removed from temporal (can't find it by id after a while). Need help investigating/troubleshooting

Hello there, I have a problem with workflow getting removed from temporal history (basically I’m no longer able to find them in temporal admin panel after a while).

I have a workflow, which start a child workflow like that

  private fun sendAnnulmentReports(order: Order, participant: Participant): Promise<WorkflowExecution> {
    val annulmentReason = AnnulmentReason.fromCloseReason(closeParams.reason)
    val reports = reportActivity.createAnnulmentReports(, order.extension, annulmentReason, order.products.mapNotNull { product -> product.gtin }.toSet(), participant)
    val options = ChildWorkflowOptions.newBuilder()
    val annulmentReportsWorkflow = Workflow.newChildWorkflowStub(, options)
      annulmentReportsWorkflow::sendReports, reports, participant,
      AnnulmentReportWorkflowProps(workflowProps.orderWorkflowProps.timeToDeleteAnnulmentReports, workflowProps.orderWorkflowProps.shouldCheckReceipt)
    return Workflow.getWorkflowExecution(annulmentReportsWorkflow)

This workflow is long running and takes 30 days to complete, meanwhile the parent workflow proceeds and can finish while the child workflow is sleeping. I expect that if I set ParentClosyPolicy.ABANDON child workflow itsn’t affected by the parent completion/termination and proceeds as if it’s standalone. Am I wrong?

The first part of the child workflow id (before the semicolon) is the id of the parent workflow. When I try to search temporal admin panel for the parent workflow, I can’t find anything (not even complete or terminated workflow, just literally nothing)

This makes me wonder if workflows get removed from temporal after a certain amount of time after completion. What might possibly go on here?

When the child workflow wakes up it starts to fail with the following exception:

2022-10-11 11:46:03.822  WARN 1 --- [="default": 651] i.t.i.w.WorkflowWorker                   : Failure while reporting workflow progress to the server. If seen continuously the workflow might be stuck. WorkflowId=1f0e94d5-be8b-462c-b011-e45d290d4daa:58666d14-52c6-31dc-afb4-78145efcd55c, RunId=49e7e7f1-f6d1-4fbc-8ffe-407403183a37, startedEventId=20
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: invalid command sequence: [CompleteWorkflowExecution, RecordMarker], command CompleteWorkflowExecution must be the last command.
	at io.grpc.stub.ClientCalls.toStatusRuntimeException( ~[grpc-stub-1.48.1.jar:1.48.1]
	at io.grpc.stub.ClientCalls.getUnchecked( ~[grpc-stub-1.48.1.jar:1.48.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall( ~[grpc-stub-1.48.1.jar:1.48.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondWorkflowTaskCompleted( ~[temporal-serviceclient-1.16.0.jar:?]
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.lambda$sendReply$0( ~[temporal-sdk-1.16.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0( ~[temporal-serviceclient-1.16.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry( ~[temporal-serviceclient-1.16.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult( ~[temporal-serviceclient-1.16.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry( ~[temporal-serviceclient-1.16.0.jar:?]
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.sendReply( ~[temporal-sdk-1.16.0.jar:?]
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle( ~[temporal-sdk-1.16.0.jar:?]
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle( ~[temporal-sdk-1.16.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0( ~[temporal-sdk-1.16.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$ Source) [?:?]
	at Source) [?:?]

What might be the reason?

Hi @Nicholas_Neutron

you are right. As per documentation The child workflow will do nothing if you set ParentClosyPolicy.ABANDON, Workflows in Java | Temporal Documentation

Yes, the workflow history is removed after the retention period, please see What is a Retention Period? | Temporal Documentation, default retention period is 2 days.

you can check the retention period of your namespace

and update it

Could it be that because the parent workflow was removed as per retention period, my child workflow starts to fail with the provided above exception even thought I specified ParentClosePolicy.ABANDON on starting the child workflow?

@Nicholas_Neutron I don’t think so.

I think that might be related to what is mentioned in this post Exceptions from parallel activities causing Workflow restart infinitely - #11 by tihomir

Could you share the code of your child workflow?

Sure, here it is:

class AnnulmentReportsWorkflowImpl : AnnulmentReportsWorkflow {

  private val log = Workflow.getLogger(

  private lateinit var workflowProps: AnnulmentReportWorkflowProps

  private val docStorageActivity: DocStorageActivity = Workflow.newLocalActivityStub(,

  private val txActivity: TxActivity = Workflow.newLocalActivityStub(,

  private val registrarReportActivity = Workflow.newLocalActivityStub(

  private val reportActivity: ReportActivity = Workflow.newLocalActivityStub(,

  override fun sendReports(reports: List<AnnulmentReport>, participant: Participant, props: AnnulmentReportWorkflowProps) {"Annulment report workflow: {} {}", reports, props.shouldCheckReceipt)
    workflowProps = props
    val promises = { report -> Async.function { sendReport(report, participant) } }
    val deleteRequests = { report ->
      Async.procedure { reportActivity.deleteReport( }

  private fun sendReport(report: AnnulmentReport, participant: Participant) {
    val newReport = if (workflowProps.shouldCheckReceipt) {
      val saveAnnulmentReport = docStorageActivity.saveAnnulmentReport(report, participant)
      val txId = Workflow.randomUUID()
      val newReport = txActivity.createAnnulmentReportTransaction(txId, report, participant, Event(success = true, txEventType = TxEventType.ANNULMENT_CODES))
        .run { report.copy(txId = txId) }
        .also { reportActivity.setResultDocId(, txId) }
        .also { newReport -> txActivity.appendAnnulmentReportEvent(newReport, participant, Event(success = true, txEventType = TxEventType.ANNULMENT_CODES), saveAnnulmentReport) }
    } else {

    val (success, rejectionReason) = registrarReportActivity.sendAnnulmentReport(newReport, participant)

    val newStatus = if (success) ReportStatus.SENT else ReportStatus.REJECTED
    reportActivity.updateReportStatus(, ReportStatus.PENDING, newStatus, rejectionReason, removeSntins = true)

    if (workflowProps.shouldCheckReceipt) {
        newReport, participant,
          success = success,
          errorMessage = rejectionReason,
          txEventType = TxEventType.ANNULMENT_CODES_EMISSION_SEND

Can you try calling get() to wait for your local activities to complete before the workflow method finish?


1 Like

Yeah, I solved this like that just before I saw your answer:) Kinda slipped before my eyes! Thanks a lot for the help!