newExternalWorkflowStub throws InaccessibleObjectException

Community Support
1

Hello there!

I’m using jdk 17 (without module-info.java under project modules) and Spring Boot, doing configurations the old fashioned way, without using a starter. But with starter faced similiar exception.

I wanted to access an already existing workflow through newExternalWorkflowStub but ran into the following exception. I’ve tried passing arguments when starting the jar-file, which would potentially open the required package exports for unnamed modules. But it did not help.
Maybe there are ideas on how to deal with the problem? Or am I doing something wrong?

20-11-2022 16:52:23.584 [Workflow Executor taskQueue="discoveryQueue", namespace="default": 2] ERROR i.t.internal.worker.PollerOptions.lambda$build$0 - uncaught exception
java.lang.RuntimeException: Failure processing workflow task. WorkflowId=3a18b127-5ce8-4cdd-a407-56dc7fb7d2b2, RunId=ac2881b5-a050-430b-88fa-f723e9df116d, Attempt=10
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:327)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:188)
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:98)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event 6 of type 'EVENT_TYPE_WORKFLOW_TASK_STARTED' during execution. {WorkflowTaskStartedEventId=6, CurrentStartedEventId=6}
	at io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:257)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:236)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:208)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:208)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:192)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:147)
	at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:132)
	at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:97)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:336)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:246)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:188)
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
	... 3 common frames omitted
Caused by: java.lang.RuntimeException: WorkflowTask: failure executing SCHEDULED->WORKFLOW_TASK_STARTED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED]
	at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:152)
	at io.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:102)
	at io.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:68)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:277)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:234)
	... 13 common frames omitted
Caused by: io.temporal.common.converter.DataConverterException: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Invalid type definition for type `jdk.proxy2.$Proxy75`: Failed to construct BeanSerializer for [simple type, class jdk.proxy2.$Proxy75]: (java.lang.IllegalArgumentException) Failed to call `setAccess()` on Field 'h' (of class `java.lang.reflect.Proxy`) due to `java.lang.reflect.InaccessibleObjectException`, problem: Unable to make field protected java.lang.reflect.InvocationHandler java.lang.reflect.Proxy.h accessible: module java.base does not "opens java.lang.reflect" to unnamed module @6572421
	at io.temporal.common.converter.JacksonJsonPayloadConverter.toData(JacksonJsonPayloadConverter.java:85)
	at io.temporal.common.converter.DefaultDataConverter.toPayload(DefaultDataConverter.java:125)
	at io.temporal.common.converter.DefaultDataConverter.toPayloads(DefaultDataConverter.java:161)
	at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:281)
	at io.temporal.internal.sync.WorkflowExecuteRunnable.run(WorkflowExecuteRunnable.java:71)
	at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:116)
	at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102)
	at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:106)
	at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	... 3 common frames omitted
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Invalid type definition for type `jdk.proxy2.$Proxy75`: Failed to construct BeanSerializer for [simple type, class jdk.proxy2.$Proxy75]: (java.lang.IllegalArgumentException) Failed to call `setAccess()` on Field 'h' (of class `java.lang.reflect.Proxy`) due to `java.lang.reflect.InaccessibleObjectException`, problem: Unable to make field protected java.lang.reflect.InvocationHandler java.lang.reflect.Proxy.h accessible: module java.base does not "opens java.lang.reflect" to unnamed module @6572421
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:72)
	at com.fasterxml.jackson.databind.SerializerProvider.reportBadTypeDefinition(SerializerProvider.java:1272)
	at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.constructBeanOrAddOnSerializer(BeanSerializerFactory.java:474)
	at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.findBeanOrAddOnSerializer(BeanSerializerFactory.java:294)
	at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:239)
	at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:173)
	at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1495)
	at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1443)
	at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:544)
	at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:822)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
	at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4568)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3844)
	at io.temporal.common.converter.JacksonJsonPayloadConverter.toData(JacksonJsonPayloadConverter.java:77)
	... 13 common frames omitted
2

Are you passing the stub as an activity or workflow argument?

3

If I’m not mistaken, as workflow.
I wanted to get information about an already running DemandWorkflow as an object of the appropriate type from DiscoveryWorkflow which was intended as an access point to the contents of the queue.
I needed something like this to be able to communicate with an already running DemandWorfklow via signals in case of application restart. Perhaps there is another way to get the actual workflow as a typed object via id, but… couldn’t find it.

I am used the following approach:

DiscoveryWorkflow:

@WorkflowInterface
public interface WorkflowDiscovery {

    @WorkflowMethod
    <T> T get(Class<T> target, WorkflowExecution query);
}


public class WorkflowDiscoveryImpl implements WorkflowDiscovery {

    @Override
    public <T> T get(Class<T> target, WorkflowExecution serviceExecution) {
        return Workflow.newExternalWorkflowStub(target, serviceExecution);
    }
}

DiscoveryWorkflow worker registration:

    @Override
    public Worker register(WorkerFactory workerFactory) {
        Worker worker = workerFactory.newWorker(discoveryProperties.getQueue());
        worker.registerWorkflowImplementationTypes(WorkflowDiscoveryImpl.class);
        return worker;
    }

DiscoveryWorkflow stub creation:

@Component
public class WorkflowDiscoveryStub extends WorkflowStub<WorkflowDiscovery> {

    private final DiscoveryProperties discoveryProperties;

    public WorkflowDiscoveryStub(WorkflowClient workflowClient,
                                 DiscoveryProperties discoveryProperties) {
        super(workflowClient);
        this.discoveryProperties = discoveryProperties;
    }

    @Override
    public WorkflowDiscovery registerStub(String workflowID) {
        WorkflowOptions options = WorkflowOptions.newBuilder()
                .setWorkflowId(workflowID)
                .setTaskQueue(discoveryProperties.getQueue())
                .build();
        return super.workflowClient.newWorkflowStub(WorkflowDiscovery.class, options);
    }
}

DemandWorkflow:

@WorkflowInterface
public interface DemandWorkflow {

    @WorkflowMethod
    void create();

    @SignalMethod
    void edit();

    @SignalMethod
    void reject();

    @SignalMethod
    void release();
}

public class DemandWorkflowImpl implements DemandWorkflow {

    private boolean isClosed;

    private final DemandActivities activitiesStub;

    public DemandWorkflowImpl() {
        RetryOptions retryOptions = RetryOptions.newBuilder()
                .setMaximumAttempts(5)
                .build();
        ActivityOptions activityOptions = ActivityOptions.newBuilder()
                .setStartToCloseTimeout(Duration.ofMinutes(5))
                .setRetryOptions(retryOptions)
                .build();

        this.activitiesStub = Workflow.newActivityStub(DemandActivities.class, activityOptions);
    }

    @Override
    public void create() {
        activitiesStub.create();

        Workflow.await(() -> isClosed);
    }

    @Override
    public void edit() {
       activitiesStub.edit();
    }

    @Override
    public void reject() {
        activitiesStub.reject();

        isClosed = Boolean.TRUE;
    }

    @Override
    public void release() {
        activitiesStub.release();

        isClosed = Boolean.TRUE;
    }
}

DemandWorkflow worker registration:

    @Override
    public Worker register(WorkerFactory workerFactory) {
        Worker worker = workerFactory.newWorker(workflowProperties.getQueue());
        worker.registerWorkflowImplementationTypes(DemandWorkflowImpl.class);
        return worker;
    }

DemandWorkflow stub creation:

@Component
public class DemandWorkflowStub extends WorkflowStub<DemandWorkflow> {

    private final DemandProperties workflowProperties;

    public DemandWorkflowStub(WorkflowClient workflowClient,
                              DemandProperties workflowProperties) {
        super(workflowClient);
        this.workflowProperties = workflowProperties;
    }

    @Override
    public DemandWorkflow registerStub(String workflowID) {
        WorkflowOptions options = WorkflowOptions.newBuilder()
                .setWorkflowId(workflowID)
                .setTaskQueue(workflowProperties.getQueue())
                .build();

        return super.workflowClient.newWorkflowStub(DemandWorkflow.class, options);
    }
}

And… my test call attempt:

 @PostMapping("/edit")
    public ResponseEntity<String> editDemand(@RequestParam String workflowID) {
        ListWorkflowExecutionsRequest request = ListWorkflowExecutionsRequest.newBuilder()
                .setNamespace("default")
                .setQuery("WorkflowType='DemandWorkflow' and ExecutionStatus="
                        + WorkflowExecutionStatus.WORKFLOW_EXECUTION_STATUS_RUNNING_VALUE)
                .build();
        WorkflowExecution execution = serviceStubs.blockingStub()
                .listWorkflowExecutions(request)
                .getExecutionsList()
                .stream()
                .map(WorkflowExecutionInfo::getExecution)
                .filter(next -> workflowID.equals(next.getWorkflowId()))
                .findFirst()
                .orElseThrow();

        ExecutionDescription<WorkflowDiscovery> stub = discoveryStub.register();
        DemandWorkflow external = stub.getStub().get(DemandWorkflow.class, execution);

        external.edit();
        return new ResponseEntity<>(workflowID, HttpStatus.ACCEPTED);
    }
4

Some issues with the code:

  • The workflow interface cannot use generics.
  • Workflows and activities cannot use stubs as arguments or results.
  • Creating a Worker per request is not going to work. Share a single worker instance per process.
  • Using ListWorkflowExecution to start workflows is not going to scale.
5

Thanks a lot for the clarification! I think this answer can be counted as a solution. But I have a question which is based on the answer.

Therefore, if a workflow-method waits for a boolean condition to complete successfully, and the boolean condition becomes true via the workflow-signal, then… is that workflow destined to become dead weight when the application is restarted? Is there any way to get its state as an object that can be interacted with again?

We want to use Temporal as a helper to create an event-bus approach.

I’ll take an imaginary process. Suppose, if we consider the problem of collecting a shopping cart in an online store, then the following conditional actions are obtained:

  1. Create a shopping cart;
  2. Editing the cart;
  3. Placing an order;
  4. Payment;
  5. etc.

If you use Temporal, will it be correct to create a separate workflow for each event, and not a common one for the entire process? After all, if you create workflow for the entire process, where the completion is a user action (that is, a signal), then there is a problem with restoring worfklow in the application context if a restart occurs.

6

then there is a problem with restoring worfklow in the application context if a restart occurs

If I understand the question correctly, don’t think this is the case. Signals are recorded in workflow execution history, so on an internal worker replay, or a manual reset of this workflow to specific workflow task in history all the signals up to that point would be re-applied again putting your workflow in same state as before.

7

My case is not exactly like that.

Let’s say I’m using an infinite workflow for the process of processing a cart: creating, editing, and so on. The workflow completes successfully when the user calls a method on the controller - then an appropriate signal is sent to the workflow.
At the same time, from 1 to N such workflows can exist, but each has a unique id. If I’m not mistaken, this approach does not violate the concept of Temporal.
Accordingly, a situation may arise when the signal to close has not yet been received, but the application has already restarted for some reason.

But here comes the question. If there was a restart, how can I send a signal to a specific workflow by ID? All stub objects have been removed from the application context.
Theoretically, I can try to send a signal through newUntypedExternalWorkflowStub using the method name … but I would like strong typing, for which I use Java. Maybe I just couldn’t find a similar functionality in the documentation?

8

if there was a restart, how can I send a signal to a specific workflow by ID?

If you need typed way could do:

MyWorkflow wf = client.newWorkflowStub(MyWorkflow.class, "<wfid>");
// call signal handler on typed stub

(Note newUntypedExternalWorkflowStub is used inside workflow code, not client code)

9

Yes! This is exactly what I wanted to find.
The word"new" in method name was misleading - I did not even suspect that there was a call to an already existing workflow.

@tihomir , @maxim, thank you wery much! I will continue to implement the idea as planned before. So far, Temporal causes only positive emotions. I hope one day there will also be support for java-modularity to be able to use encapsulation to its fullest :slight_smile: