Hi, I have two different workflows with one having 5 local activities and the other workflow with 10 local activities. When I browse through the history of each workflow on the UI, it shows the history entry as expected such as in the below screenshot.
Wondering if there is any option to turn off and avoid the history writes to Cassandra as these are local activities and we don’t wait or retry more than 5 seconds for a workflow. This way maybe the size and writes can be saved? Currently within a 8 hour period the size of the history_node table is reaching 2 TB at 800 workflows/sec.
Local activities have to be recorded as they are not reexecuted on replay. One possible optimization is to not include arguments into the marker event as they are there only for human consumption.
Set LocalActivityOptions.doNotIncludeArgumentsIntoMarker to true to not include them.
1 Like
Thank you, we will try this option.
Can you also please clarify when will the entries in history_node table get purged? I ran a 8 hour test starting yesterday 12 pm to 8 pm. The retention is 1 day (minimum available), and we expected the data storage to start going down today 12 pm, but not much changed . We also reduced the gc grace seconds on that table to 3 hours. Can you please confirm if our understanding is right?
Also, how are the deletes issued to history_node table? Do you mark the record with TTL or do you issue a delete command to Cassandra?
The history_node and other workflow data are purged after the retention period. The retention period for a workflow starts when it is closed.
We also reduced the gc grace seconds on that table to 3 hours. Can you please confirm if our understanding is right?
I think it is a very bad idea to reduce the gc grace seconds. It will lead to the appearance of zombie records and potentially corrupt data.
Try running manual compaction instead.
1 Like
Thanks @maxim this is helpful. Also I am trying to change the retention period for a namespace that already exists, but I cannot find documentation or figure out how to change the retention period just for a given namespace using the Java SDK.
NamespaceConfig nsConfig = UpdateNamespaceRequest.newBuilder().getConfigBuilder().setWorkflowExecutionRetentionTtl(Duration.newBuilder().setSeconds(86400).build()).build();
Also none of the workflow options or other setup seems to be taking namespace config as an input.
Deprecate does not delete the namespace so that we can freshly create the namespace again with the chosen retention period:
WorkflowServiceStubs workflowService = WorkflowServiceStubs.newInstance(workflowServiceStubsOptions);
DeprecateNamespaceRequest deleteRequest = DeprecateNamespaceRequest.newBuilder().setName(NAMESPACE).build();
workflowService.futureStub().deprecateNamespace(deleteRequest);
You can use tctl to update the retention period for an existing namespace, for example:
tctl namespace update --retention 2
where the retention period is set in days.
Namespace deletion is currently not supported, you can track this issue for updates.
1 Like
This is running in our company K8s cluster and we do not have access to login to run tctl command. Is there a way to do this same from Java SDK? @tihomir
Thanks.
But I cannot port forward to my laptop and installation of tctl is blocked in our K8s cluster. Is there a way I can do the same through Java SDK please?
I mean you can call that command using Java gRPC client to temporal:
WorkflowServiceStubs workflowService = WorkflowServiceStubs.newInstance(workflowServiceStubsOptions);
workflowService.blockingStub().updateNamespace(...);
1 Like
Here is a full example for record:
UpdateNamespaceRequest request =
UpdateNamespaceRequest.newBuilder()
.setNamespace(myNameSpace)
.setConfig(
NamespaceConfig.newBuilder().setWorkflowExecutionRetentionTtl(days).build())
.build();
UpdateNamespaceResponse response = service.blockingStub().updateNamespace(request);
where myNameSpace is a String name of the namespace you want to update (must set)
and days is of type com.google.protobuf.Duration
So to create it for like three days you can do:
com.google.protobuf.Duration days =
com.google.protobuf.Duration.newBuilder().setSeconds(3 * 86400).build();
You can inspect the response to see that retention is set to 3 days and you can also use tctl after to see if it indeed has been set with
tctl namespace desc
1 Like
Yes, I got it @tihomir that worked, appreciate your code snippet anyway.
Also, to make use of the “doNotIncludeArgumentsIntoMarker” on LocalActivityOptions, I was earlier using 0.29.1 Java SDK but I upgraded that to 1.0.8 which seems to be the version when it was introduced. However I get this error:
Exception in thread "main" java.lang.NoSuchMethodError: 'void com.google.common.base.Preconditions.checkArgument(boolean, java.lang.String, char, java.lang.Object)'
at io.grpc.Metadata$Key.validateName(Metadata.java:742)
at io.grpc.Metadata$Key.<init>(Metadata.java:750)
at io.grpc.Metadata$Key.<init>(Metadata.java:668)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
at io.grpc.Metadata$Key.of(Metadata.java:705)
at io.grpc.Metadata$Key.of(Metadata.java:701)
at io.temporal.serviceclient.WorkflowServiceStubsImpl.<clinit>(WorkflowServiceStubsImpl.java:51)
I see suggestions to upgrade guava but I see the version is already the latest in mvn dependencies:
com.google.guava:guava:jar:30.1.1-jre:compile
I have tried java sdk 1.0.0, 1.0.8, 1.2.0 but 0.29.1 is the only one that’s working for me. Have you seen this issue?
The worker is running on Temporal server 1.9.2 version.
@ramyamagham
tried with SDK 1.2.0 and setting setDoNotIncludeArgumentsIntoMarker on LocalActivityOptions
worked fine on Server versions:
1.10.5
and the just released 1.12.0
1 Like
I don’t think the error I am getting is specifically related to the setting “setDoNotIncludeArgumentsIntoMarker”.
The worker is not even starting up due to error in line
WorkflowServiceStubs workflowService = WorkflowServiceStubs.newInstance(workflowServiceStubsOptions);
Do I have to upgrade the temporal server to 1.10.5? Is our conclusion that it won’t work on 1.9.2?
There’s no compatibility chart available here or on temporalio website: Maven Central Repository Search
Tested with server version 1.9.2 and sdk version 1.2.0 and did not get the same problem. Maybe something else might be going on in your setup or workflows. I am testing just the UpdateNamespaceRequest and single activity execution.
1 Like
Thanks a lot for verifying that. I will check more on my end.
Have a nice day!
Just for the record if anybody else has the same issue, changing the guava to older version worked for us and excluding the one from Temporal SDK.
libraryDependencies += "com.google.guava" % "guava" % "20.0"
Would you file an issue to get the guava updated in the Temporal SDK?
So Temporal SDK is already using the latest “30.1.1-jre” which is what confused me yesterday when I googled for the error which suggested to upgrade to latest guava.
As @tihomir had verified it works for him, and from my application dependencies I could not find any conflicting jars or dependencies from other libraries. But I had to fall back to older version of guava 20.0.
So given this, I am not going to file a ticket. But thanks a ton for actively responding on this thread!