Elixir SDK Support In The Roadmap?

Hey All,

Just wondering if anyone knows of plans for an ELixir SDK? I see this: How much work is it to write an SDK? that hasn’t had any movement for some time.

Seems like the only language missing in this ectosystem, and I think it would be great to see elixir supoprted here. Additionally, I’m seeing other similar projects like inngest where they are taking a crack at elixir support.

I think the warning that this is likely a 6 month plus task maybe put a lot of people off just beginning this, is there any docs anywhere that might help with building out a bare bones api for this?

Also, there is already a protobuff generated elixir project here temporalio | Hex which looks like a good start.

Chris

I am finalizing work on the Erlang SDK (unofficial), initial release should be available in Sep-Oct. Elixir SDK as a wrapper on top of Erlang SDK is planned for Q4. All dates are tentative.

I would say that 6 months is a very optimistic schedule assumption for a full-feature Temporal SDK development, even if it would be based on a Core SDK.

1 Like

Great news, thanks for taking up the challenge. Let me know if you need anything to test things out in the future

1 Like

@andrzej-mag - This is very neat! Elixir/Erlang should fit Temporal’s workflow model well (maybe more than any other runtime). Feel free to ask any questions in the #sdk-core Slack channel, we’d be happy to clarify anything confusing about SDK internals.

2 Likes

Hey @andrzej-mag did you make any progress in this endeavor?

If so, do you have a public GitHub repo to see the state of play?

One thought, while it’s amazing to have a fully fledged feature complete version, I think there’s a lot of value in just a partial solution.

For example just bring able to bundle up Erlang/elixir activities, with limited functionality, and have workflows defined in other languages may still be valuable.

Chris

Hi Chris,
Project is advancing steadily however it is not ready for publishing yet. Tentative project schedule was provided in my previous answer.

As you mentioned Temporal activities, quick preview of current WIP Erlang SDK activities API implementation:

-module(temporal_sdk_activity).

-export([
    await/0,
    await/1,
    cancel/1,
    complete/1,
    fail/3,
    heartbeat/0,

    is_cancel_requested/0,
    elapsed_time/0,
    remaining_time/0,

    get_data/0,
    get_failure_info/0,
    get_last_heartbeat/0,
    set_data/1,
    set_failure_info/1
]).

-include("temporal_sdk_proto.hrl").

-type task() :: ?TEMPORAL_SPEC:'temporal.api.workflowservice.v1.PollActivityTaskQueueResponse'().
-export_type([task/0]).

-type data() :: term().
-export_type([data/0]).

-type last_heartbeat() :: term().
-export_type([last_heartbeat/0]).

-type context() ::
    #{
        cluster := temporal_sdk_cluster:cluster_name(),
        executor_pid := pid(),
        otel_ctx := otel_ctx:t(),
        worker_opts := temporal_sdk_worker:opts(),
        task := task(),
        started_at := SystemTime :: integer(),
        is_cancel_requested := boolean(),
        task_timeout := erlang:timeout(),
        data := data(),
        last_heartbeat := last_heartbeat()
    }.
-export_type([context/0]).

-type fail_reason() :: {Message :: term(), Source :: term(), Stacktrace :: term()}.
-export_type([fail_reason/0]).

-type handle_event_result() :: term().
-export_type([handle_event_result/0]).

%% -------------------------------------------------------------------------------------------------
%% behaviour

-type cancel_action() :: {cancel, CanceledDetails :: temporal_sdk:term_to_payloads()}.
-type complete_action() :: {complete, Result :: temporal_sdk:term_to_payloads()}.
-type fail_action() ::
    {fail, Reason :: fail_reason()}
    | {fail, Reason :: fail_reason(), FailureInfo :: temporal_sdk_worker:failure_info()}.
-type terminate_action() :: cancel_action() | complete_action() | fail_action().
-type heartbeat_action() :: heartbeat | {heartbeat, Heartbeat :: temporal_sdk:term_to_payloads()}.

-callback execute(Context :: context(), Args :: temporal_sdk:term_from_payloads()) ->
    Result :: temporal_sdk:term_to_payloads().

-callback handle_heartbeat(Context :: context()) ->
    terminate_action() | heartbeat_action().

-callback handle_cancel(Context :: context()) ->
    terminate_action() | ignore.

-callback handle_call(Context :: context(), From :: gen_statem:from(), Request :: term()) ->
    handle_event_result() | ignore.

-callback handle_cast(Context :: context(), Request :: term()) ->
    handle_event_result() | ignore.

-callback handle_info(Context :: context(), Info :: term()) ->
    handle_event_result() | ignore.

-optional_callbacks([
    handle_heartbeat/1,
    handle_cancel/1,
    handle_call/3,
    handle_cast/2,
    handle_info/2
]).

%% -------------------------------------------------------------------------------------------------
%% public

-spec await() -> handle_event_result() | no_return().
await() ->
    await(infinity).

-spec await(Timeout :: erlang:timeout()) -> handle_event_result() | no_return() | timeout.
await(Timeout) ->
% implementation follows...
1 Like

Will love to know if there is any progress on this :slight_smile:

Project is moving forward, (full) SDK implementation is almost complete. Few minor items like Nexus or documentation are still missing - I need some more time.

Activities API mentioned above is now cleaner and simpler:

-module(temporal_sdk_activity).

-export([
    await_data/1,
    await_data/2,

    complete/1,
    cancel/1,
    fail/3,

    heartbeat/0,
    heartbeat/1,

    is_cancel_requested/0,
    is_activity_paused/0,
    elapsed_time/0,
    elapsed_time/1,
    remaining_time/0,
    remaining_time/1,

    get_data/0,
    get_failure_info/0,
    get_last_heartbeat/0,
    set_data/1,
    set_failure_info/1
]).

-type context() ::
    #{
        cluster := temporal_sdk_cluster:cluster_name(),
        executor_pid := pid(),
        otel_ctx := otel_ctx:t(),
        task := temporal_sdk_api_activity_task:task(),
        worker_opts := temporal_sdk_worker:opts(),
        started_at := SystemTime :: integer(),
        task_timeout := erlang:timeout()
    }.
-export_type([context/0]).

-type handler_context() ::
    #{
        data := data(),
        is_cancel_requested := boolean(),
        is_activity_paused := boolean(),
        last_heartbeat := heartbeat(),
        elapsed_time := non_neg_integer(),
        remaining_time := erlang:timeout()
    }.
-export_type([handler_context/0]).

-type data() :: term().
-export_type([data/0]).

-type heartbeat() :: temporal_sdk:term_to_payloads().
-export_type([heartbeat/0]).

%% -------------------------------------------------------------------------------------------------
%% behaviour

-type complete_action() :: {complete, Result :: temporal_sdk:term_to_payloads()}.
-type cancel_action() :: {cancel, CanceledDetails :: temporal_sdk:term_to_payloads()}.
-type fail_action() ::
    {fail, {
        Source :: temporal_sdk:serializable(),
        Message :: temporal_sdk:serializable(),
        Stacktrace :: temporal_sdk:serializable()
    }}.
-type terminate_action() :: cancel_action() | complete_action() | fail_action().
-type heartbeat_action() :: heartbeat | {heartbeat, Heartbeat :: heartbeat()}.
-type data_action() :: {data, NewData :: data()}.

-callback execute(Context :: context(), Args :: temporal_sdk:term_from_payloads()) ->
    Result :: temporal_sdk:term_to_payloads().

-callback handle_heartbeat(HandlerContext :: handler_context()) ->
    terminate_action() | heartbeat_action().

-callback handle_cancel(HandlerContext :: handler_context()) ->
    terminate_action() | ignore.

-callback handle_message(HandlerContext :: handler_context(), Message :: term()) ->
    terminate_action() | data_action() | ignore.

-optional_callbacks([
    handle_heartbeat/1,
    handle_cancel/1,
    handle_message/2
]).

Hi! Thank you for your work! Any chance you can give an approximate schedule? Or may be make the repository public, so, we can track progress / start using it as it is / help with something?

Hi,

My current best estimate is 2-3 months to RC1 release.

I have considered releasing a partial implementation many times. Erlang SDK introduces a number of unique, interrelated core features and concepts that were improved and have matured in recent months. Such new additions need to be supported with solid documentation and examples which are not ready yet. As I mentioned before some other smaller pieces like Nexus are still work in progress. In my opinion publishing incomplete and undocumented complex software with frequently changing APIs would cause a lot of confusion, and this is the main reason for not releasing this yet.

3 Likes

Good work @andrzej-mag, being a long time since I thought of this and ended up using various other flavors of the sdk, but an Erlang and elixir flavor would be more then welcome.

Honestly, I think you should just make what you have public …at this stage nobody is expecting anything other than constant breaking changes and a half dine solution… and you can always disable any GitHub issues etc.

But not too push, I’m glad you taken up they mantel in this either way. Keep up they good work.

I understand your arguments about publishing an incomplete SDK. I am not quite ready for this yet, a little more time is needed. Project is taking much longer than expected due to unplanned implementation of new concepts. Erlang offers a number of unique features and capabilities and it took a lot of time to incorporate them into the SDK.

1 Like

@andrzej-mag What’s the current progress with the SDK? :slight_smile:

Progress report on Erlang/Elixir SDK since March:

  1. Workflow replay (from json) functionality has been implemented.

  2. Almost complete rewrite of the workflow executor. Rewrite was necessary to fix multiple race conditions issues discovered in workflow parallel executions.

  3. As of today nearly 1000 workflow replay tests have been added as EUnit tests. A large number of issues identified during EUnit replay tests implementation were fixed.

  4. During workflow executor rewrite I found a way to improve workflow replay performance. After this optimization, SDK can process ~20k activities/second during replay. This translates to replaying a workflow with a 50k-event history in less than one second.

  5. Initial basic Nexus implementation has been completed.

I didn’t plan to work on points 1 to 4 in my March schedule; however, they consumed most of my time. Accordingly, I need to push back the RC1 tentative release date to ~October.

3 Likes

Surely Erlang/Elixir is the best language overall in many ways for Temporal. I am so excited about this project. I wish we could see a public github repo. I’m sure many others would love to try to help.

In my opinion, Erlang is currently the best choice for developing complex software requiring parallelism and distributed computing. I was able to introduce multiple optimizations and new features to the native Erlang SDK implementation.

The main reasons I haven’t published SDK yet are:

  • large number of features are still missing at current stage which makes SDK practically impossible for a normal user to use,
  • frequent and substantial SDK interface changes as Temporal and OTP semantics reconciliation is difficult,
  • lack of any documentation,
  • limited number of tests which may result in large number of bugs. In recent months I have improved this by adding quite a lot of EUnit replay tests - I really hope that workflow executor can finally be considered stable after huge amount of work that has been invested.

Another complicating factor is my plan to release SDK with a source-available license that requires a monthly subscription. This should ensure that future SDK maintenance, development, and user support are secured.
The project has been my full-time focus for a long time already, and I am working hard to release it as soon as possible.

Thanks for these details. I can see why we all have to keep waiting for your work to reach a high quality stage. Cheers, Rudi

1 Like