Elixir SDK Support In The Roadmap?

Hey All,

Just wondering if anyone knows of plans for an ELixir SDK? I see this: How much work is it to write an SDK? that hasn’t had any movement for some time.

Seems like the only language missing in this ectosystem, and I think it would be great to see elixir supoprted here. Additionally, I’m seeing other similar projects like inngest where they are taking a crack at elixir support.

I think the warning that this is likely a 6 month plus task maybe put a lot of people off just beginning this, is there any docs anywhere that might help with building out a bare bones api for this?

Also, there is already a protobuff generated elixir project here temporalio | Hex which looks like a good start.

Chris

1 Like

I am finalizing work on the Erlang SDK (unofficial), initial release should be available in Sep-Oct. Elixir SDK as a wrapper on top of Erlang SDK is planned for Q4. All dates are tentative.

I would say that 6 months is a very optimistic schedule assumption for a full-feature Temporal SDK development, even if it would be based on a Core SDK.

1 Like

Great news, thanks for taking up the challenge. Let me know if you need anything to test things out in the future

1 Like

@andrzej-mag - This is very neat! Elixir/Erlang should fit Temporal’s workflow model well (maybe more than any other runtime). Feel free to ask any questions in the #sdk-core Slack channel, we’d be happy to clarify anything confusing about SDK internals.

2 Likes

Hey @andrzej-mag did you make any progress in this endeavor?

If so, do you have a public GitHub repo to see the state of play?

One thought, while it’s amazing to have a fully fledged feature complete version, I think there’s a lot of value in just a partial solution.

For example just bring able to bundle up Erlang/elixir activities, with limited functionality, and have workflows defined in other languages may still be valuable.

Chris

Hi Chris,
Project is advancing steadily however it is not ready for publishing yet. Tentative project schedule was provided in my previous answer.

As you mentioned Temporal activities, quick preview of current WIP Erlang SDK activities API implementation:

-module(temporal_sdk_activity).

-export([
    await/0,
    await/1,
    cancel/1,
    complete/1,
    fail/3,
    heartbeat/0,

    is_cancel_requested/0,
    elapsed_time/0,
    remaining_time/0,

    get_data/0,
    get_failure_info/0,
    get_last_heartbeat/0,
    set_data/1,
    set_failure_info/1
]).

-include("temporal_sdk_proto.hrl").

-type task() :: ?TEMPORAL_SPEC:'temporal.api.workflowservice.v1.PollActivityTaskQueueResponse'().
-export_type([task/0]).

-type data() :: term().
-export_type([data/0]).

-type last_heartbeat() :: term().
-export_type([last_heartbeat/0]).

-type context() ::
    #{
        cluster := temporal_sdk_cluster:cluster_name(),
        executor_pid := pid(),
        otel_ctx := otel_ctx:t(),
        worker_opts := temporal_sdk_worker:opts(),
        task := task(),
        started_at := SystemTime :: integer(),
        is_cancel_requested := boolean(),
        task_timeout := erlang:timeout(),
        data := data(),
        last_heartbeat := last_heartbeat()
    }.
-export_type([context/0]).

-type fail_reason() :: {Message :: term(), Source :: term(), Stacktrace :: term()}.
-export_type([fail_reason/0]).

-type handle_event_result() :: term().
-export_type([handle_event_result/0]).

%% -------------------------------------------------------------------------------------------------
%% behaviour

-type cancel_action() :: {cancel, CanceledDetails :: temporal_sdk:term_to_payloads()}.
-type complete_action() :: {complete, Result :: temporal_sdk:term_to_payloads()}.
-type fail_action() ::
    {fail, Reason :: fail_reason()}
    | {fail, Reason :: fail_reason(), FailureInfo :: temporal_sdk_worker:failure_info()}.
-type terminate_action() :: cancel_action() | complete_action() | fail_action().
-type heartbeat_action() :: heartbeat | {heartbeat, Heartbeat :: temporal_sdk:term_to_payloads()}.

-callback execute(Context :: context(), Args :: temporal_sdk:term_from_payloads()) ->
    Result :: temporal_sdk:term_to_payloads().

-callback handle_heartbeat(Context :: context()) ->
    terminate_action() | heartbeat_action().

-callback handle_cancel(Context :: context()) ->
    terminate_action() | ignore.

-callback handle_call(Context :: context(), From :: gen_statem:from(), Request :: term()) ->
    handle_event_result() | ignore.

-callback handle_cast(Context :: context(), Request :: term()) ->
    handle_event_result() | ignore.

-callback handle_info(Context :: context(), Info :: term()) ->
    handle_event_result() | ignore.

-optional_callbacks([
    handle_heartbeat/1,
    handle_cancel/1,
    handle_call/3,
    handle_cast/2,
    handle_info/2
]).

%% -------------------------------------------------------------------------------------------------
%% public

-spec await() -> handle_event_result() | no_return().
await() ->
    await(infinity).

-spec await(Timeout :: erlang:timeout()) -> handle_event_result() | no_return() | timeout.
await(Timeout) ->
% implementation follows...
1 Like

Will love to know if there is any progress on this :slight_smile:

Project is moving forward, (full) SDK implementation is almost complete. Few minor items like Nexus or documentation are still missing - I need some more time.

Activities API mentioned above is now cleaner and simpler:

-module(temporal_sdk_activity).

-export([
    await_data/1,
    await_data/2,

    complete/1,
    cancel/1,
    fail/3,

    heartbeat/0,
    heartbeat/1,

    is_cancel_requested/0,
    is_activity_paused/0,
    elapsed_time/0,
    elapsed_time/1,
    remaining_time/0,
    remaining_time/1,

    get_data/0,
    get_failure_info/0,
    get_last_heartbeat/0,
    set_data/1,
    set_failure_info/1
]).

-type context() ::
    #{
        cluster := temporal_sdk_cluster:cluster_name(),
        executor_pid := pid(),
        otel_ctx := otel_ctx:t(),
        task := temporal_sdk_api_activity_task:task(),
        worker_opts := temporal_sdk_worker:opts(),
        started_at := SystemTime :: integer(),
        task_timeout := erlang:timeout()
    }.
-export_type([context/0]).

-type handler_context() ::
    #{
        data := data(),
        is_cancel_requested := boolean(),
        is_activity_paused := boolean(),
        last_heartbeat := heartbeat(),
        elapsed_time := non_neg_integer(),
        remaining_time := erlang:timeout()
    }.
-export_type([handler_context/0]).

-type data() :: term().
-export_type([data/0]).

-type heartbeat() :: temporal_sdk:term_to_payloads().
-export_type([heartbeat/0]).

%% -------------------------------------------------------------------------------------------------
%% behaviour

-type complete_action() :: {complete, Result :: temporal_sdk:term_to_payloads()}.
-type cancel_action() :: {cancel, CanceledDetails :: temporal_sdk:term_to_payloads()}.
-type fail_action() ::
    {fail, {
        Source :: temporal_sdk:serializable(),
        Message :: temporal_sdk:serializable(),
        Stacktrace :: temporal_sdk:serializable()
    }}.
-type terminate_action() :: cancel_action() | complete_action() | fail_action().
-type heartbeat_action() :: heartbeat | {heartbeat, Heartbeat :: heartbeat()}.
-type data_action() :: {data, NewData :: data()}.

-callback execute(Context :: context(), Args :: temporal_sdk:term_from_payloads()) ->
    Result :: temporal_sdk:term_to_payloads().

-callback handle_heartbeat(HandlerContext :: handler_context()) ->
    terminate_action() | heartbeat_action().

-callback handle_cancel(HandlerContext :: handler_context()) ->
    terminate_action() | ignore.

-callback handle_message(HandlerContext :: handler_context(), Message :: term()) ->
    terminate_action() | data_action() | ignore.

-optional_callbacks([
    handle_heartbeat/1,
    handle_cancel/1,
    handle_message/2
]).

Hi! Thank you for your work! Any chance you can give an approximate schedule? Or may be make the repository public, so, we can track progress / start using it as it is / help with something?

Hi,

My current best estimate is 2-3 months to RC1 release.

I have considered releasing a partial implementation many times. Erlang SDK introduces a number of unique, interrelated core features and concepts that were improved and have matured in recent months. Such new additions need to be supported with solid documentation and examples which are not ready yet. As I mentioned before some other smaller pieces like Nexus are still work in progress. In my opinion publishing incomplete and undocumented complex software with frequently changing APIs would cause a lot of confusion, and this is the main reason for not releasing this yet.

3 Likes

Good work @andrzej-mag, being a long time since I thought of this and ended up using various other flavors of the sdk, but an Erlang and elixir flavor would be more then welcome.

Honestly, I think you should just make what you have public …at this stage nobody is expecting anything other than constant breaking changes and a half dine solution… and you can always disable any GitHub issues etc.

But not too push, I’m glad you taken up they mantel in this either way. Keep up they good work.

I understand your arguments about publishing an incomplete SDK. I am not quite ready for this yet, a little more time is needed. Project is taking much longer than expected due to unplanned implementation of new concepts. Erlang offers a number of unique features and capabilities and it took a lot of time to incorporate them into the SDK.

1 Like

@andrzej-mag What’s the current progress with the SDK? :slight_smile:

Progress report on Erlang/Elixir SDK since March:

  1. Workflow replay (from json) functionality has been implemented.

  2. Almost complete rewrite of the workflow executor. Rewrite was necessary to fix multiple race conditions issues discovered in workflow parallel executions.

  3. As of today nearly 1000 workflow replay tests have been added as EUnit tests. A large number of issues identified during EUnit replay tests implementation were fixed.

  4. During workflow executor rewrite I found a way to improve workflow replay performance. After this optimization, SDK can process ~20k activities/second during replay. This translates to replaying a workflow with a 50k-event history in less than one second.

  5. Initial basic Nexus implementation has been completed.

I didn’t plan to work on points 1 to 4 in my March schedule; however, they consumed most of my time. Accordingly, I need to push back the RC1 tentative release date to ~October.

3 Likes

Surely Erlang/Elixir is the best language overall in many ways for Temporal. I am so excited about this project. I wish we could see a public github repo. I’m sure many others would love to try to help.

In my opinion, Erlang is currently the best choice for developing complex software requiring parallelism and distributed computing. I was able to introduce multiple optimizations and new features to the native Erlang SDK implementation.

The main reasons I haven’t published SDK yet are:

  • large number of features are still missing at current stage which makes SDK practically impossible for a normal user to use,
  • frequent and substantial SDK interface changes as Temporal and OTP semantics reconciliation is difficult,
  • lack of any documentation,
  • limited number of tests which may result in large number of bugs. In recent months I have improved this by adding quite a lot of EUnit replay tests - I really hope that workflow executor can finally be considered stable after huge amount of work that has been invested.

Another complicating factor is my plan to release SDK with a source-available license that requires a monthly subscription. This should ensure that future SDK maintenance, development, and user support are secured.
The project has been my full-time focus for a long time already, and I am working hard to release it as soon as possible.

Thanks for these details. I can see why we all have to keep waiting for your work to reach a high quality stage. Cheers, Rudi

1 Like

Hey, @andrzej-mag, I looked at your GitHub profile and noticed you just published the initial version of the Erlang SDK at GitHub - andrzej-mag/temporal_sdk: Temporal Erlang SDK. Thanks for your hard work, I’m sure many people in this forum will find it invaluable.

Would you like to say a few words to the community on how to approach this project?

Temporal Erlang SDK initial pre-release is available on hex.pm:

temporal_sdk_samples repository provides a few samples showing basic SDK usage.
Additional code examples can be extracted from the EUnit tests in the test_replay directory.

Current Temporal Erlang SDK project status: work in progress.

High level list of the available major Temporal features with their current SDK implementation status:

  • activities: fully implemented,
  • workflows: few commands pending (cancel_external_workflow & signal_external_workflow),
  • runtime and dynamic task workers: observability is WIP,
  • dynamic task rate limiter: implemented,
  • sync/async gRPC client created specifically for this project is available.

Erlang SDK provides multiple unique features which will be announced as soon as they are finalized and documented. One of such features is a single workflow execution per Erlang cluster instead of a workflow executions caching mechanism used by other SDKs, which should reduce operating costs.

Current SDK TODO list is attached to the project repository. Current project priorities:

  1. Add Elixir SDK syntactic wrapper.
  2. Finalize/improve/fix few priority SDK internals.
  3. Add top priority pending Temporal WF commands and services.
  4. Add most important parts of the SDK documentation.
  5. Close remaining items from the TODO list.

After Elixir SDK part is introduced project will be renamed to: Temporal Erlang and Elixir SDK.

1 Like

Is this for real?

The software monthly subscription fee is €100 (plus VAT/tax where applicable) per production application that uses this SDK as a dependency.