# Payload Storage and type safety

**URL:** <https://community.temporal.io/t/payload-storage-and-type-safety/17412>\
**Category:** Community Support\
**Created:** [May 13, 2025, 9:32am UTC](https://community.temporal.io/t/payload-storage-and-type-safety/17412 "2025-05-13T09:32:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dontpanic](https://avatars.discourse-cdn.com/v4/letter/d/a9a28c/32.png) [@dontpanic](https://community.temporal.io/u/dontpanic)\
**Post date:** [May 13, 2025, 9:32am UTC](https://community.temporal.io/t/payload-storage-and-type-safety/17412/1 "2025-05-13T09:32:32Z")

</div>

Hello everyone 🙂, another topic on handling large payload.

What I found when exploring this forum and Slack is:

- A **codec server** transparently handles large payload, but may have a **large impact on performance** when replaying workflows ([source](https://community.temporal.io/t/codec-for-handling-large-payloads-encounters-a-script-execution-timeout-after-5000ms/12126/4));
- Recommended approach is to **upload payload to a blob storage and pass around keys**.

I chose to explore the latter option, and came to the following conclusion:

Let’s say I have to these message definitions:

```c
message Payload {
  google.protobuf.Any data = 1;
}

// Reference to a Payload stored in a blob storage
message PayloadRef {
  string id = 1;
  uint64 content_size_bytes = 2;
}

message Image {
  bytes data = 1; // Can be very large
  string description = 2;
}

```

## Option 1: automatically handle payloads with wrapper around Activities

Much like [this post](https://temporalio.slack.com/archives/CTT84RS0P/p1713727340389459?thread_ts=1713642283.429269&cid=CTT84RS0P), the idea is to “intercept” an Activity input and output, serialize them, send them to the blob storage and return a key (or equivalent).

```python
def to_payload(obj: Message) -> PayloadRef:
    # 1) Build `Payload` from `obj`
    # 2) Upload it, and return the received `PayloadRef`

def from_payload(payload_ref: PayloadRef, message_type: T) -> T:
    # Do the opposite of `to_payload`

@activity.defn
def first_activity() -> PayloadRef:
	image: Image = do_activity_stuff()
	return to_payload(image)

@activity.defn
def second_activity(payload_ref: PayloadRef) -> None:
	image = from_payload(payload, Image)

```

**Pros** :

- Can be implemented generically, once and for all for any Activity, making the devs’ life easier

**Cons** :

- **We lose all type safety** : I really don’t like manipulating keys as inputs/outputs, because it is like (are almost like) manipulating void pointers in the sense that keys act like pointers to data of arbitrary type.
- It is difficult to do any operation on Activities inputs/outputs in the workflow code (filtering, merging…)

## Option 2: manually handle large payloads

To alleviate Option 1’s cons, I imagined the following:

```c
message Image {
  bytes data = 1; // Can be very large
  string description = 2;
}

message ImageRef {
  PayloadRef payload_ref = 1;
  string description = 2;
}

message ActivityInput {
  repeated ImageRef images = 1;
}

```

With these, I can implement the following functions:

```python
def as_ref(image: Image) -> ImageRef:
    # 1) Serialize `image` into bytes
    # 2) Upload these bytes, get a `PayloadRef` back
    # 3) Return an `ImageRef` with the received `PayloadRef`
    # and `image`'s description

def deref(image_ref: ImageRef) -> Image:
    # Do the opposite of `as_ref`

```

**Pros** :

- “Type safety”
- Possibility of manipulating data outside of Activities (e.g.: reading an image’s description)

**Cons** :

- For each potential large object `LargeObject`, we have to:
  - define `LargeObjectRef`
  - implement `as_ref(LargeObject) -> LargeObjectRef`
  - implement `deref(LargeObjectRef) -> LargeObject`.

## Conclusion

Each of these options’ cons are hard to ignore.

I prefer Option 2, but long-term maintenance is a bit frightening. It could be alleviated with code gen for `as_ref` and `deref` though.

What are your thoughts? What solutions have been found out there? Thank you very much 🙂

---

<div class="post-metadata">

**Author:** ![dontpanic](https://avatars.discourse-cdn.com/v4/letter/d/a9a28c/32.png) [@dontpanic](https://community.temporal.io/u/dontpanic)\
**Post date:** [May 14, 2025, 8:30pm UTC](https://community.temporal.io/t/payload-storage-and-type-safety/17412/2 "2025-05-14T20:30:34Z")

</div>

Huh, after some time outside I realized that I might have overthought it…

Something much simpler like this could also work:

```c
message BytesRef {
  PayloadRef payload_ref = 1;
}

message Image {
  BytesRef data_ref = 1;
  string description = 2;
}

// Another example:
message Document {
  BytesRef content = 1;
  string title = 2;
  string author = 3;
}

```

in which case:

- I would only have to implement `deref` and `as_ref` for `BytesRef` (and also `StringRef`);
- It would be the responsibility of the application developers to load the actual data from the blob storage if they want to perform operations on it…
- … but they can still write some logic on `Image` without having to load it.