Running more than one instance with same workflowId in parallel

Community Support
,
1

I am running a DSL workflow in temporal. The DSL is constructed by the product UI and saved in db. Each DSL has an identifier (UUID) in the db. I thought of using DSL identifier as workflowId when starting workflow in temporal but I am not able to do so as temporal doesn’t allow running more than one workflow with same workflowId at the same time. Our use case is to run the same DSL workflow with different input arguments. So I am using a random UUID as workflowId while starting the workflows. Due to this I don’t have an association of DSL identifier to the temporal workflow execution. Is there any workaround here?

Another query related to workflowId.
If I am using my DSL identifier as workflowId in temporal, is it possible to query list of workflow executions which have same workflowId. I need this to show in our product UI.

Any help is much appreciated.

Thanks,
Swaroop

2

Hi @swaroop,

as temporal doesn’t allow running more than one workflow with same workflowId at the same time

Yes, that is due to workflow id reuse policy (default, if none is specified explicitly, is the first one “Allow Duplicate” in the list in that doc).

Is there any workaround here?

I have seem people use a combination of the workflow id defined in DSL and either a random number or a business value. The latter could be used if you let’s say want to start workflow executions for 100 unique orders, in that case for example the order id could be uses as part of the workflow id.

Regarding the query, yes you can use advanced search in the Web UI, or you can use with Java SDK ListWorkflowExecutions, for example:

    private static void listWorkflowExecutionsByQuery(String query, ByteString token) {
        ListWorkflowExecutionsRequest request;

        if(token == null) {
            request = ListWorkflowExecutionsRequest.newBuilder()
                    .setNamespace(client.getOptions().getNamespace())
                    .setQuery(query)
                    .setPageSize(1000)
                    .build();
        } else {
            request = ListWorkflowExecutionsRequest.newBuilder()
                    .setNamespace(client.getOptions().getNamespace())
                    .setQuery(query)
                    .setNextPageToken(token)
                    .setPageSize(1000)
                    .build();
        }
        ListWorkflowExecutionsResponse response =
                service.blockingStub().listWorkflowExecutions(request);
        for(WorkflowExecutionInfo info : response.getExecutionsList()) {
           // do what you need to do with info, like print
        }

        if(response.getNextPageToken() != null && response.getNextPageToken().size() > 0) {
            listWorkflowExecutionsByQuery(query, response.getNextPageToken());
        }
    }

Note that you need to enable Elasticsearch to use ListWorkflowExecutions.
The query can include whole-word wildcard searches with for example:

WorkflowId LIKE "%myidfromdsldefinition%"

where myidfromdsldefinition would match for example any
myidfromdsldefinition <randomuuid>

and WorkflowId is a pre-defined search attribute (no need for you to add it manually)

Hope this helps.

3

@tihomir Thank you for the response. I didn’t enable Elasticsearch in my temporal instance. Will enable it and try out your approach.
I remember that the max retention period for executions in temporal is 30 days. So post 30 days, is the data going get removed from ES also ?

4

Yes, default retention period is 2 days, min is 1 day and max is 30 days as you mentioned.

You can update retention via tctl:

tctl namespace update --retention 4

(where 4 is number of days)

or in Java SDK via UpdateNamespaceRequest, for example:

UpdateNamespaceRequest request =
        UpdateNamespaceRequest.newBuilder()
            .setNamespace(myNameSpace)
            .setConfig(
                NamespaceConfig.newBuilder().setWorkflowExecutionRetentionTtl(days).build())
            .build();
    UpdateNamespaceResponse response = service.blockingStub().updateNamespace(request);
   // ...

Yes, the retention period starts ticking only after the workflow execution completed, so you can have workflow that run for a very long period of time (much much longer than 30 days :slight_smile: ). Once the workflow execution completes retention period starts kicking in, and when retention period is reached both workflow history data and visibility data are removed.
You can use workflow archival feature to keep an archive of this data post the retention period. Note tho that this is an experimental feature still, but some people are using it, just fyi.

5

According to the archival doc, there is a command line utility to retrieve archival data using workflowId and runId. Is it possible to do the same from java sdk?

Archival feature doesn’t support visibility records, so I will not able to do list workflow executions query. I can only query the execution using workflowId and runId. If that is the case, I have to store the workflowId and runId in one of my application tables correct ?