Ways to pass large data from workflow to activity without affecting history size

Community Support
1

Say I’ve a big DSL Graph which the temporal workflow executes. I want to push the state of this to S3 every time the graph makes progress.

  1. Do it in the workflow: Once the activity returns (eg. result = await execute_activity(...) update the state in memory, serialize it (around 2MBs) and call boto3_s3.put_object.
    a. Problem: Workflow doesn’t allow this as it breaks determinism on replay.
    b. Possible Solution: run the S3 upload under with temporalio.workflow.unsafe.sandbox_unrestricted(): if temporalio.workflow.unsafe.is_replaying() is False, otherwise NoOp.
  2. Do it in the activity: Workflow updates the state, serializes it and passes to an activity (local or otherwise). Activity is free to do whatever it wants.
    a. Problem: Prohibitive history size. Since anything workflow passes to an activity is recorded in history, this will grow to a prohibitive size soon.
  3. Do it in the activity but use a query: Workflow triggers an activity, which instead of receiving the state as input, issues a query to the workflow. The workflow then calculates the state (serializes it) and passes to the activity. Activity can do whatever it wants with it.
    a. Problem: Feels a bit roundabout and there’s a limit to gRPC response size (from the query)? If this say was 10 MB it probably wouldn’t work? (Not sure). Any other drawback (I’m not very familiar with queries)?

Which is the preferred option in this case? Or are there any other options?

2

hmm, I just discovered another option - local activities, which seems the best for this?

So continuing, option 4 would be:

  1. Calculate and serialize big data in workflow code, use with temporalio.workflow.unsafe.sandbox_unrestricted(): to open a local file say at /tmp/big-data and write to it. Do this only if temporalio.workflow.unsafe.is_replaying() == False. Then execute_local_activity(...) to start an activity that reads from this location and does whatever it wants (push to S3 etc). This seems better than using 3rd party libs like boto3 etc in workflow since that’s not even working just now. I’ve to turn the sandbox completely off workflow.defn(sandboxed=False) for boto3.client('s3') to work for eg. otherwise with just with temporalio.workflow.unsafe.sandbox_unrestricted(): I get errors like:

message: “metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases”

which I suppose is due to a temporal bug (in python sdk).

I’m assuming here that I understood correctly that a local activity will always be run on the same container/machine where the workflow task which created the file is running on.