Using Compressed Input and Output

Community Support
1

The Workflow and Activity input and output are quite large for our scenario. With other parameters, we also exchange JSONs that are approximately 60KB. When we tried to execute 1000 concurrent Workflow per second, the Cassandra cluster became the bottleneck.

We tried to solve the problem by using the compressed objects (using GZIPInputStream/GZIPOutputStream) as input and output for Workflow and Activity. It solves the problem, but when we access the Temporal Web UI, it shows only the initial few characters of the Base64 encoded byte[]. Example:

"H4sIAAAAAAAAAO1d3Xbb.."

With this, it becomes difficult to know the input/output during debugging.

  • Is there any API that can return the entire input/output for a Workflow?
  • If not, can we retrieve it from Cassandra?

Once we get the entire byte[], we can write a utility to display decompressed content.

2

You can write a tctl plugin that would decompress the payloads. See cryptconverter example that uses it to show encrypted data in the UI.

3

Thanks, @maxim.

So far, to use compressed input/output for Workflows and Activities, we were using byte. We have changed that to use actual objects and using a custom DataConverter that uses GZIPInputStream/GZIPOutputStream for compressing/decompressing the input/output.

The code is much cleaner now. However, we are still trying to figure out how to get the original (uncompressed) input/output for debugging. I looked at cryptconverter. It looks like it will work only with tctl.

Since we have already written a DataConverter, can we use that with some Java-SDK API to get the uncompressed input/output? For example. can we use the DataConverter with WorkflowServiceStubs.getWorkflowExecutionHistory()?

4

However, we are still trying to figure out how to get the original (uncompressed) input/output for debugging. I looked at cryptconverter. It looks like it will work only with tctl.

It also works with UI by using tctl as a local proxy.

Since we have already written a DataConverter, can we use that with some Java-SDK API to get the uncompressed input/output? For example. can we use the DataConverter with WorkflowServiceStubs.getWorkflowExecutionHistory()?

The getWorkflowExecutionHistory is a gRPC API that is not using DataConverter directly. You have to call it yourself for each Payloads protobuf.