How do Import Workflow History from dump/Start a new workflow with history recorded in a dump?

Nicholas_Neutron · May 25, 2022, 12:20pm

Basically I exported workflow history and now I want to start a workflow with this history to fix some errors. How do I do that?

tihomir · May 25, 2022, 2:08pm

You can use WorkflowReplayer (samples for Go, Java) to replay the workflow history against your workflow impl.
WorkflowReplayer can be used for backwards compatibility testing as well as debugging production workflows.
If you set breakpoints for debugging make sure to set TEMPORAL_DEBUG env var to 1 or true (this disables the default deadlock detector during debugging).

I want to start a workflow with this history to fix some errors

If your workflow execution has completed (and hasnt reached the set namespace retention period) you can reset it to a specific WorkflowTaskStarted event. Note you can also reset via SDK apis (which SDK are you using?). Reset will replay the workflow history up to the specified reset point and then continue execution as a new workflow run.

Just to add you can get workflow histories via tctl, for example:
tctl wf show -w <wfid> -r <runid> --output_filename myhistory.json

Nicholas_Neutron · June 1, 2022, 7:51am

So I copied workflow history using this method and want to start it from tctl with command:

tctl workflow start --taskqueue alter.orders.queue --wt OrderWorkflow --input_file history.json

It fails to start with the following message:

Error: Failed to create workflow.
Error Details: context deadline exceeded
(‘export TEMPORAL_CLI_SHOW_STACKS=1’ to see stack traces)

Any advice?

Nicholas_Neutron · June 1, 2022, 7:52am

Here is the stacktrace:

Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
go.temporal.io/server/tools/cli.printError({0x1dafbf7, 0x1a}, {0x20b6580, 0xc000482258})
        /temporal/tools/cli/util.go:392 +0x22a
go.temporal.io/server/tools/cli.ErrorAndExit({0x1dafbf7, 0x24}, {0x20b6580, 0xc000482258})
        /temporal/tools/cli/util.go:403 +0x28
go.temporal.io/server/tools/cli.startWorkflowHelper.func1()
        /temporal/tools/cli/workflowCommands.go:229 +0x1a8
go.temporal.io/server/tools/cli.startWorkflowHelper(0xc0001b6c60, 0x0)
        /temporal/tools/cli/workflowCommands.go:266 +0x557
go.temporal.io/server/tools/cli.StartWorkflow(...)
        /temporal/tools/cli/workflowCommands.go:177
go.temporal.io/server/tools/cli.newWorkflowCommands.func3(0xc0001b6c60)
        /temporal/tools/cli/workflow.go:57 +0x1b
github.com/urfave/cli.HandleAction({0x19bbea0, 0x1e3f480}, 0x5)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:526 +0x50
github.com/urfave/cli.Command.Run({{0x1d838c0, 0x5}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x1dbbba0, 0x1e}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:173 +0x652
github.com/urfave/cli.(*App).RunAsSubcommand(0xc0006e6380, 0xc0001b69a0)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:405 +0x9ec
github.com/urfave/cli.Command.startApp({{0x1d8ab1b, 0x8}, {0x0, 0x0}, {0xc0006c35b0, 0x1, 0x1}, {0x1dad7fc, 0x19}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:372 +0x6e9
github.com/urfave/cli.Command.Run({{0x1d8ab1b, 0x8}, {0x0, 0x0}, {0xc0006c35b0, 0x1, 0x1}, {0x1dad7fc, 0x19}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:102 +0x808
github.com/urfave/cli.(*App).Run(0xc0006e6000, {0xc00012e000, 0x9, 0x9})
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:277 +0x705
main.main()
        /temporal/cmd/tools/cli/main.go:37 +0x33

tihomir · June 1, 2022, 1:54pm

So I copied workflow history using this method and want to start it from tctl with command:

tctl workflow start “input-file” flag does not take the whole history, its just the data in json format you wan to pass to your workflow execution as data input. In your case you are passing the entire workflow history json which is not the workflow data input.

To show this, you can use lets say the HelloActivity sample (comment the System.exit on line 185).

Run this sample (from ide or terminal or whatever, dont shut off the process to leave the worker running)
Crete a file lets say myinput.json which should just have value of “MyWorld”
Run tctl wf start --tq HelloActivityTaskQueue -w MyWorkflowId --wt GreetingWorkflow --if myinput.json
this should start your workflow with data input “MyWorld”.

Could you explain what you are trying to do? It seems you might want to reset a completed workflow exec? If you want to replay the history and debug it then use WorkflowReplayer as mentioned.

Nicholas_Neutron · June 2, 2022, 6:47am

Thank you! I’m using workflow replayer now - so far so good. The problem is that I wanted to introduce a change in my workflow and used versioning for that, tho I wrote copied code in DEFAULT_VERSION branch incorrectly and now all previous workflows fail with NonDeterministicException. I’m trying to fix this now to no avail:( This is the exception:

, queryType=__replay_only, args=Optional.empty, error=io.temporal.worker.NonDeterministicException: Failure handling event 22 of type ‘EVENT_TYPE_MARKER_RECORDED’ during replay. Local activity of type SaveClosedOrder is recorded in the history with id 2c727b1b-1af8-3b31-a012-5d394eb1834c but was not expected by the execution. {PreviousStartedEventId=16, workflowTaskStartedEventId=9223372036854775807, Currently Processing StartedEventId=16}

The thing is, although it states it wasn’t expected by the execution, it’s actually in the default branch.

Nicholas_Neutron · June 2, 2022, 6:48am

This is the code before change:

    if (workflowProps.shouldCheckReceipt) {
      val txId = params.txId ?: Workflow.randomUUID()

      // autoclosing - saving resultDocId to DB
      // when closed manually it's not needed since it was saved when participant made request
      if (params.txId == null) {
        orderActivity.setCloseOrderResultDocId(order.id, txId)
      }

      val closedOrder = ClosedOrder(
        id = Workflow.randomUUID(),
        txId = txId,
        orderId = order.id,
        extension = order.extension,
        reason = params.reason,
        reasonMessage = if (params.reason == CloseOrderReason.OTHER) params.reasonMessage else null,
        requestUrl = params.requestUrl
      )
      docStorageActivity.saveClosedOrder(closedOrder, participant)
      txActivity.createCloseOrderTransaction(txId, closedOrder, participant, Event(success = true, txEventType = TxEventType.CLOSE_ORDER))
        .also { txActivity.appendCloseOrderEvent(closedOrder, participant, Event(success = true, txEventType = TxEventType.CLOSE_ORDER)) }
    }

Nicholas_Neutron · June 2, 2022, 6:49am

This is the code after I made a change:

    if (workflowProps.shouldCheckReceipt) {
      val txId = params.txId ?: Workflow.randomUUID()

      val versionFix: Int = Workflow.getVersion("Hotfix: Fixed failing workflows", Workflow.DEFAULT_VERSION, 1)
      when (versionFix) {
        Workflow.DEFAULT_VERSION -> if (params.txId == null) {
          orderActivity.setCloseOrderResultDocId(order.id, txId)
        }
        1 -> Unit
      }

      val closedOrder = ClosedOrder(
        id = params.closeOrderId ?: Workflow.randomUUID(),
        txId = txId,
        orderId = order.id,
        extension = order.extension,
        reason = params.reason,
        reasonMessage = if (params.reason == CloseOrderReason.OTHER) params.reasonMessage else null,
        requestUrl = params.requestUrl
      )

      val version: Int = Workflow.getVersion("Synchronous close order receipt when manually closing order", Workflow.DEFAULT_VERSION, 1)
      when (version) {
        Workflow.DEFAULT_VERSION -> {
          docStorageActivity.saveClosedOrder(closedOrder, participant)
          txActivity.createCloseOrderTransaction(txId, closedOrder, participant, Event(success = true, txEventType = TxEventType.CLOSE_ORDER))
            .also { txActivity.appendCloseOrderEvent(closedOrder, participant, Event(success = true, txEventType = TxEventType.CLOSE_ORDER)) }
        }
        1 -> {
          if (params.txId == null) {
            // autoclosing - saving resultDocId to DB and creating new transaction
            // when closed manually it's not needed since it was saved when participant made request
            docStorageActivity.saveClosedOrder(closedOrder, participant)
            txActivity.createCloseOrderTransaction(txId, closedOrder, participant, Event(success = true, txEventType = TxEventType.CLOSE_ORDER))
              .also { txActivity.appendCloseOrderEvent(closedOrder, participant, Event(success = true, txEventType = TxEventType.CLOSE_ORDER)) }
            orderActivity.setCloseOrderResultDocId(order.id, txId)
          } else {
            txActivity.appendCloseOrderEvent(closedOrder, participant, Event(success = true, txEventType = TxEventType.CLOSE_ORDER))
          }
        }
      }
    }

Nicholas_Neutron · June 2, 2022, 6:53am

With the code provided it fails with this in debug mode (the other exception above in the normal run):

error=io.temporal.internal.replay.InternalWorkflowTaskException: Failure handling event 4 of type ‘EVENT_TYPE_WORKFLOW_TASK_COMPLETED’ during replay. {PreviousStartedEventId=3, workflowTaskStartedEventId=9223372036854775807, Currently Processing StartedEventId=3}

Whenever I try to change it starts to fail with the above, sometimes different exceptions as well(

Nicholas_Neutron · June 3, 2022, 2:45pm

Debugging with workflow replayer helped, we fixed this, thx!

Topic		Replies	Views
Is there an example of how you would replay a workflow in Java Community Support java-sdk	14	2645	July 21, 2023
Export/Import Workflow History from one temporal server to another Community Support go-sdk	1	291	July 3, 2023
Resuming workflow execution from json event history Community Support	3	308	June 4, 2023
Writing unit test for workflow that uses GetWorkflowHistory Community Support go-sdk , testing	1	100	February 28, 2024
WorkflowReplayer replayWorkflowExecution Community Support workflow-options	7	1458	October 21, 2021

How do Import Workflow History from dump/Start a new workflow with history recorded in a dump?

Related Topics