Temporal for small-scale plugin architecture

Recently found Temporal, and congrats on a great product! I am interested in resiliency aspects, orchestration / workflows, saga’s, and possibly to provide an actor system. A lot of the docs focus on scalability requirements for large-scale (‘centralized’) platforms.

Use case

I am eyeing Temporal for a decentralized FOSS application that will be part of the Fediverse. There will be many server instances that communicate with other instances and interop with other Fediverse applications using ActivityPub and ActivityStreams (actor-based protocol using JSON message exchange between inboxes and outboxes over HTTP).

A server typically hosts a small community of users (smaller is better) or is even self-hosted by a single person. Each server is a platform, where service modules (plugins) can be downloaded and dynamically added at runtime (was thinking of using Hashicorp go-plugin here). The core service would only provide basic features, with service modules providing (building blocks of) actual applications. Think similar to how Nextcloud platform works.

Prior to finding Temporal I was looking into a domain-driven design (DDD) with service modules corresponding to a Bounded Context or sub-domain and designed according to Ports & Adapters + CQRS/ES. Optional in this design was to have Proto.Actor actor model and have actors in the Application layer to load and cache domain aggregates.

Default install would be for small-scale use, and having a SQLite database. The service modules would run on the same server, so not a true microservices design (though communication uses gRPC). Scaling up and out would be needed in rare cases, and then Postgres is available as db (a large-scale service instance would have 50-100k users).

Considerations

Trying to wrap my head how Temporal might fit, I see a lot of matches.

  • CQRS write-side would be covered with workflow + activities packaged in service modules.
  • Actor model is possible with Activities in the Application layer that invoke domain aggregates in Domain layer.
  • Found a project in the works that deals with SQLite persistence for Temporal data.

Some open questions and things I ponder about:

  • With all the focus on scalability and complex microservices architecture Temporal may be overkill.
  • I don’t have realtime (e.g. gaming) low-latency requirements. Temporal might be at the core of entire architecture.
  • But with that do I have enough control of how data is persisted, or maybe I don’t need to care about that any more.
  • How would I go about implementing the read-side of CQRS using Temporal (Kafka suggested elsewhere probably overkill too)?
  • I saw that Temporal can be imported / embedded. How involved is that, and will you continue to support that option?
  • Would it be complex to replace Elasticsearch with something else (thinking of lieu [1] )?

Curious about any thoughts on this architecture…

[1] lieu community search: https://github.com/cblgh/lieu (cannot include more links)

  • With all the focus on scalability and complex microservices architecture Temporal may be overkill.

This is what most users care about. But Temporal programming is a good fit for some monolithic applications that require reliability as well.

  • I don’t have realtime (e.g. gaming) low-latency requirements. Temporal might be at the core of entire architecture.

Good fit here.

  • But with that do I have enough control of how data is persisted, or maybe I don’t need to care about that any more.

It depends on the data. Temporal persists data that is related to the lifecycle of objects it manages. It is not built to store historical data for example.

  • How would I go about implementing the read-side of CQRS using Temporal (Kafka suggested elsewhere probably overkill too)?

Use query workflow feature of Temporal.

  • I saw that Temporal can be imported / embedded. How involved is that, and will you continue to support that option?

We want to support extending Temporal without the need to fork the main repository. So we certainly going to support this option.

  • Would it be complex to replace Elasticsearch with something else (thinking of lieu [1] )?

It would be a pretty significant project. I think we are going to add support for custom attributes on top of SQL databases sooner.

1 Like

Thanks a lot for your quick response @maxim!

Ah, that wasn’t obvious to me yet, and I’d have to look deeper into. I may still have to provide my own persistence layer then, both on the write- as well as on the read-side. OTOH maybe if, say, I have a UserAccount domain aggregate that is controlled and accessed via an actor-like Workflow in the Application layer, then this workflow could be active for as long as the user account exists?

  • Domain layer has no dependencies (inversion of control), guarantee valid (domain) state, and invocations raise events.
  • Application layer has business logic which are mostly Workflows, Child Workflows + Activities.
  • Domain events and domain errors trigger Signals in the Application layer to the running workflows.
  • (Maybe) There’s no need for Repository interfaces in Domain layer, and DB impls in Infra layer. Temporal persistence is used.

The above may make no sense, not work. Shows my noobness on the paradigm shift that Temporal seems to offer. A safe architecture design would be to have the ports & adapters, DDD, CQRS/ES as I intended, but having Temporal only provide Saga capability and handle some clearly long-running tasks.

Yes, but the read-site of CQRS would offer denormalized views on the data. But I think you mean to say this is the way to get at workflow state for implementing the read-side, which then has its own way of dealing with persistence.

With imported / embedded I was referring to the Production deployment page stating “The Temporal Server is a Go application which you can import or run as a binary.” and then in my Go code calling s := temporal.NewServer() to fully embed the server instead of having it separately deployed (e.g. via Docker, etc.). I was not referring to forking and extending, just invoke functionality as-is but have the server code fully encapsulated in my own application. Is that possible, or did I misinterpret?

@aschrijver This might be what you looking for (at small scale) → GitHub - DataDog/temporalite: An experimental distribution of Temporal that runs as a single process Combine it with litestream for simpler data backup/restore → https://litestream.io/

2 Likes

Thank you Michael, an interesting link. Litestream is a great project. I found it some time ago when it featured on Hacker News.

Example of Temporalite with Litestream; tested with Digital Ocean Spaces as S3 backing → GitHub - leowmjw/temporalite at docker-composelite

3 Likes

Is Temporalite with Litestream suitable for small-scale production use cases now? In December, it was one of the ambitious goals of Temporalite to integrate with Litestream to run on standalone servers. Just checking if this was currently advised against, or just experimental and not fully tested. Thanks!

2 Likes

Do you have a link to the temporalitestream integration plans?

Given that

Litestream aims to provide a balance between durability and operational complexity by batching changes into one-second windows and asynchronously backing those changes up to external storage. This improves write performance at the expense of having a small window of data loss in the event of a catastrophic failure.

Does this tradeoff make sense for temporalite?