Temporal version = 1.0.0
Using MySQL DB 5.7
After running temporal in production for few months successfully, our team ran into a weird issue yesterday. Based on certain external factors, we trigger a workflow programmatically using java-sdk.
This call to start workflow failed with the stacktrace provided at the end of this message. When I attempted to run the same workflow using tctl, I got the same error back. Surprisingly, when I changed the workflowId to something else keeping the workflow_type same, it worked. For next workflow id, it failed again.
Restarting Temporal server resolved the issue for us, but the worry is that it can happen again, and we really don’t know why it happened in the first place. Why would temporal tell me that the default namespace doesn’t exist when I can see the namespace using tctl namespace list command?
Stacktrace
Error: Failed to run workflow.
Error Details: Namespace <id_of_default_namespace> does not exist.
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack(0xd, 0x0, 0x0)
/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x22
go.temporal.io/server/tools/cli.printError(0x1cf055d, 0x17, 0x20bfa80, 0xc0004aa5c0)
/temporal/tools/cli/util.go:531 +0x2ad
go.temporal.io/server/tools/cli.ErrorAndExit(0x1cf055d, 0x17, 0x20bfa80, 0xc0004aa5c0)
/temporal/tools/cli/util.go:542 +0x49
go.temporal.io/server/tools/cli.startWorkflowHelper.func2()
/temporal/tools/cli/workflowCommands.go:258 +0x687
go.temporal.io/server/tools/cli.startWorkflowHelper(0xc0000a7600, 0x1)
/temporal/tools/cli/workflowCommands.go:281 +0x639
go.temporal.io/server/tools/cli.RunWorkflow(...)
/temporal/tools/cli/workflowCommands.go:181
go.temporal.io/server/tools/cli.newWorkflowCommands.func4(0xc0000a7600)
/temporal/tools/cli/workflow.go:65 +0x30
github.com/urfave/cli.HandleAction(0x1963f00, 0x1d7ae08, 0xc0000a7600, 0xc0000a7600, 0x0)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/app.go:528 +0x7c
github.com/urfave/cli.Command.Run(0x1cd053e, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1d3da70, 0x38, 0x0, ...)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/command.go:174 +0x57a
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000463c00, 0xc0000a7340, 0x0, 0x0)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/app.go:407 +0x915
github.com/urfave/cli.Command.startApp(0x1cd6d48, 0x8, 0x0, 0x0, 0xc000749120, 0x1, 0x1, 0x1cf4e6e, 0x19, 0x0, ...)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/command.go:373 +0x845
github.com/urfave/cli.Command.Run(0x1cd6d48, 0x8, 0x0, 0x0, 0xc000749120, 0x1, 0x1, 0x1cf4e6e, 0x19, 0x0, ...)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/command.go:102 +0xa2b
github.com/urfave/cli.(*App).Run(0xc000463880, 0xc00011c0d0, 0xd, 0xd, 0x0, 0x0)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/app.go:279 +0x7c7
main.main()
/temporal/cmd/tools/cli/main.go:37 +0x4b