Payload converter invoked between Activity Tasks

Community Support
1

Hi everyone, new to Temporal here, trying to make sense of something that I am experiencing.

So, I’ll start with a bit of context. I’m creating a proof of concept for the specific scenario where Activities have to pass Large Payloads between them.
The idea is to use an external Object Store (like S3) and pass only the reference to the object between activities.

From the implementation point (JAVA SDK), the idea is to have a Payload Converter extending the Jackson one, and if the payload size is larger in bytes than a specific number, the payload would be transformed to have a different encoding (i.e. s3/blob) and other extra metadata. Smaller payloads would be delegated to the Jackson Payload converter.

There is a Payload Codec that would pick the Payloads with this specific encoding and push them to S3, dereferencing the Payload data at the same time. All the references to the Object in S3 are in the metadata.

Same thing during decoding. Object pulled from S3, Payload data copied into, and Payload converter instantiating the original POJO.

The workflow proof of concept is based on the simplest scenario I could think of right now, it is basically an Activity A that pushes to S3, and then reference is consumed by Activity B that pulls from S3 and does “something” with it.

About the setup:

  • java temporal-sdk 1.21.2
  • Temporal local dev server
  • Java 17

Sorry for the long context, now my question.

What I am experiencing and I haven’t been able to understand is:

  • After activity A finishes, the Payload Converter kicks in and then the Payload Codec, all running in an Activity Executor thread
  • What I am expecting to see after is Payload decoding, Payload Converting and Activity B code running again in an Activity Executor thread
  • What I am experiencing is a Workflow Task between the activities (my understanding that’s how the Temporal Server lets the workers know about picking the next Activity task) triggering the payload decoding and downloading the BLOB from S3 in a Workflow thread with name "activity completion callback", and right after that another Workflow thread triggering the payload encoding and trying to upload the same blob again.
  • After that, the expected Activity B payload decoding, converting and business logic triggers.

Could you please help me to understand why this Workflow Task would trigger the Payload encoding/decoding in this scenario? Looking at the raw Events, there is no information about the payload.

Thanks

2

That’s expected, the workflow code needs to run after Activity A completes in order to know what to do next (in this case, call Activity B). If you know the workflow code doesn’t use the Activity A retval, and just passes it to Activity B, and are trying to avoid the unnecessary decode/encode, I’m not sure what the best way to avoid it is— @tihomir ?