Blob/External storage Example

Community Support
,
1

Hi,

I have seen in other topics that to avoid running into 2MB blob size limit issue, recommendation is to use external storage e.g. s3. Are there any examples available for this?

Thanks,
Jay.

2

We don’t have any samples for this that I am aware of. Which SDK are you using?

3

Hi @tihomir,

We’re using Go SDK.

Thanks,
Jay.

4

Thanks. Are you looking specifically how to write and retrieve data to/from s3 (this would be just your code so nothing Temporal specific), or are you trying to figure out how to handle errors if you go over the blob max?

5

So to give you some context.

  • We are looking at a DSL type implementation where we will have a generic workflow that will interpret the workflow definition from DSL.
  • Activities could be written by different teams, and they can specify the input/output data requirements for their activities.
  • We may need to pass data from one activity to other
  • Activities by different team may be executing in their own workers
  • Workflow will be executing in it’s in own worker
  • Workers will be running in K8S container

With the above in mind, we will have certain activities where we will exceed the 2Mb size for data that an activity may return. We want to make it easy for teams to write their own activities, so some questions we are trying to answer include:

  • Is it possible to make the read/write of data from external storage transparent to the person writing activity.
  • Do we apply this to all parameters even if they won’t exceed 2MB - disadvantages of this method - I assume performance could be one?
  • If we should do this selectively, how do we identify which parameters (if we assume that responsibility isn’t with the developer who writes activity)

So it’s more than just how to write and retrieve data to/from s3, it was more to see that code in context of Temporal and activity execution and understand what is/isn’t possible.

Thank you,
Jay.

6
  • Is it possible to make the read/write of data from external storage transparent to the person writing activity.

I think you should be able to use interceptors for this, see sample here.

7

Hi @tihomir,

Sorry for the delay in replying.

If we take the DSL example in samples: samples-go/dsl at main · temporalio/samples-go · GitHub, could you provide a little more insight into how the interceptors can be used to pass in the arguments from DSL using external storage?

I don’t need to see the actual code storing the parameter value in external storage, but I’m not fully clear on how exactly to write the interceptors? Things I’m not clear on include:

  • We have parameters defined in workflow that we are passing to activity: do we store this externally in WorkflowInboundInterceptor or WorkflowOutboundInterceptor? Do we instead pass some ID to the next activity - what does that look like?
  • In the activityInboundInterceptor, we would retrieve the parameters from the external storage based on ID, and call Next - what is the syntax to pass the activity parameters?
  • Similarly, in the activityOutboundInterceptor - we would again store the return parameters externally?
  • In the Temporal UI - what would the activity input/output show?

Thank you,
Jay.