Hey folks! I’m trying to use the Java SDK’s WorkflowReplayer as a helper for determining if changes we make are Nondeterministic. While I have run into a few issues, I think it’s showing signs that it is working for catching some obvious cases.
Some workflows are showing as deterministic. Then I throw in a random extra activity and it fails as expected with Nondeterministic.
But we have two other failing workflows that share something in common and I’m wondering if you’re able to reproduce on your side. We have an activity that returns an Optional. For both workflows, it throws Nondeterministic every time at this juncture (both workflows use the same activity). We have some logic wrapped in try/finally and before doing the Optional::isPresent check, it always goes down to the finally. When I use the Debugger, I see the same. Is this a known issue for WorkflowReplayer and Optionals? I found this previous topic, my IDE is showing this same error when I use the Debugger, but it sounds like it’s because of an unrelated issue? Thanks!
The overall impact for us would be our inability to use WorkflowReplayer to determine deterministic constraints because there will always be false positives when Optionals are involved.
I just put together a watered down version of a workflow and I don’t get the same issue when it’s a simple Optional<String>. I will continue debugging but this seems isolated to the type that we’re using internally, Optional<InternalType>.
Feels good to at least isolate it further.
Indeed this was related to a deserializer issue with our internal type. The error in the Debugger was nested as something like Caused by: java.lang.IllegalArgumentException: Cannot find a deserializer for non-concrete Collection type [collection type; class com.google.common.collect.ImmutableSet, contains [simple type, class com.our.custom.Type. In other cases we also saw NullPointerException because without a proper deserializer, it was unable to translate null ---> Optional.empty.
The solution for us seems to be to override the DataConverter for the TestEnvironment.
// Override the TestEnvironment's DataConverter
final var testEnv = new TestWorkflowEnvironmentInternal(TestEnvironmentOptions.newBuilder()
.setWorkflowClientOptions(WorkflowClientOptions.newBuilder().setDataConverter(DefaultDataConverter.newDefaultInstance()
.withPayloadConverterOverrides(customJacksonConverter())).build())
.build());
final var worker = testEnv.newWorker("my-task-queue");
worker.registerWorkflowImplementationTypes(MyWorkflowImpl.class);
WorkflowReplayer.replayWorkflowExecutionFromResource("nondeterminism.json", worker);
// Once I figured out I was having issues with the DataConverter and Jackson, this section in the docs helped for Java-SDK
// https://docs.temporal.io/dev-guide/java/features#custom-payload-conversion
private static JacksonJsonPayloadConverter customJacksonConverter() {
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModule(new GuavaModule()); // I saw issues with this for our custom type
objectMapper.registerModule(new Jdk8Module());
objectMapper.registerModule(new JavaTimeModule());
// Add your custom logic here.
return new JacksonJsonPayloadConverter(objectMapper);
}
In summary, I think this was the core issue. It was hard to find because our try/finally swallowed the DataConverter exception. When I simplified the workflows to just the basics, the WorkflowReplayer + Debugger led the rest of the way. I love this tool. Hope we keep investing in it or whatever its successors may be!
P.S. – a lot of this work, we owe to the folks over at Convoy who published this article and inspired looking into the replayer. Really cool work folks! Thought I’d share in case anyone else was looking into this.