How to use async function in custom payload converter

Community Support
,
1

Hi Temporal! I am building a CustomPayloadConverter to handle data over 2MB. The specification is as follows.

  • The ToPayload function serializes the 2MB or larger data passed as an argument into a Protobuf message, stores it in shared storage, and passes the path to the data as a Protobuf message.
  • FromPayload function gets the file path from the received message, opens the file and returns a deserialized object

I would like to handle Promise in ToPayload and FromPayload since the functions that create and open files are async. How can I do this? Is it a good idea to use PayloadCodecs in these situations?

2

Hi. I’m pretty sure the API does not support this. An async function may fail, and in that case, your workflow and activity arguments/parameters can’t be serialized.

PayloadCodecs are lower level than PayloadConverters as they deal with bytes for compression and encryption purposes. So, they’re applied right before bytes are sent through the wire to the temporal server.

I think your use case fits more into an activity, that has built-in support for retries and timeouts to safely fetch your full payload. I would pass some URL to the full payload stored in some blob storage service, and then fetch it from an activity when needed.

1 Like
3

I would go with @rcomesan’s approach if there’s no need for these payloads to reach the workflow. Workflow tasks should be processed quickly and avoid additional network round trips if possible.

That being said, there are cases where the blob store and pointer approach is valid and for those cases you should use a PayloadCodec which is async.

1 Like
4

Thank you for your reply! Let me confirm that I understand your suggestion correctly. Do you mean that after the process of uploading the full payload, you pass the URL to the full payload as an argument to Activity or Workflow?

5

Yes. I mean, I don’t know what you’re trying to do but IMO 2MB is a pretty generous limit for the message payload. I guess most of your payloads should fit within these limits.

That said, you may have a few specific workflows/activities that require additional space. In such cases, I would include a URL to the resources that exceed this limit (e.g. video, image, large JSON, etc), instead of embedding it in the message payload itself. As you said, you’d need to upload this large resource to some blob storage first, and then include its URL in the workflow/activity payload (e.g. { video: “s3://mybucket/large_video.mp4” } when starting the workflow.

Also, you most likely don’t need this large resource in the workflow anyway. So, using a custom data converter would make you pay the cost of extra roundtrips to the blob storage service on every re-enter or retry of your workflow. Even if you no longer need it because the activity that uses it was already completed and its result is cached. Instead, passing around the URL is cheap. You can fetch the large resource in the activities that need it, process it, and temporal will cache the result.

Now, if you do need this data in the workflow, I would split it and run it in batches if that’s possible.

1 Like
6

Thank you for your careful explanation. I have never used Temporal and did not understand the intuition around it, so your opionion is very helpful. Thanks to @rcomesan and @bergundy both so much!

1 Like