# Invoking coroutine in Activity (Python SDK)

**URL:** <https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127>\
**Category:** Community Support\
**Tags:** python-sdk\
**Created:** [December 4, 2020, 9:33am UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127 "2020-12-04T09:33:35Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chen\_Wan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chen_wan/32/358_2.png) [@Chen\_Wan](https://community.temporal.io/u/Chen_Wan)\
**Post date:** [December 4, 2020, 9:33am UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/1 "2020-12-04T09:33:35Z")

</div>

I have a newbie question about python-sdk usage regarding invoking coroutines inside an activity method.

My first attempt was to declare the activity method as `async`so I can use `await`. That approach failed with a coroutine-cannot-be-serialized-to-JSON exception.

In the second approach I put the coroutine call in a separate local function and invoked it from the activity method with `asyncio.run(fn())`. That seemed to work. The code looks like this:

```
 class MyActivity:
   @activity_method(....)
   def my_func():
       pass
   class MyActivityImpl(MyActivity):
   def my_func():
       async def fn():
           .....
           await async_http_client.init_session()
           
       asyncio.run(fn())

```

1. Is there a canonical way to invoke co-routines in an activity method?
2. While using `asyncio.run(fn())` works, it’s not ideal in my case. The coroutines I used maintains a connection pool and it’s more efficient to initialize it once and keep it around, and re-use it. The asyncio.run would terminate the event-loop at the end, which makes the resources inaccessible. I would like to keep the resources accessible for the duration of a workflow, but not sure how to accomplish that.

---

<div class="post-metadata">

**Author:** ![ryland](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ryland/32/11_2.png) [@ryland](https://community.temporal.io/u/ryland)\
**Post date:** [December 4, 2020, 5:16pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/2 "2020-12-04T17:16:55Z")

</div>

Hey, I will have to add @firdaus because I’m genuinely not sure about Python.

---

<div class="post-metadata">

**Author:** ![firdaus](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/firdaus/32/48_2.png) [@firdaus](https://community.temporal.io/u/firdaus)\
**Post date:** [December 4, 2020, 11:02pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/3 "2020-12-04T23:02:04Z")

</div>

In the temporal sdk , the @activity\_method implementation itself can be `async def` so you can `await` directly within it.

In fact, I might remove support for non async activity method implementations all together forcing all to be `async def`.

---

<div class="post-metadata">

**Author:** ![Chen\_Wan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chen_wan/32/358_2.png) [@Chen\_Wan](https://community.temporal.io/u/Chen_Wan)\
**Post date:** [December 10, 2020, 3:16am UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/4 "2020-12-10T03:16:05Z")

</div>

Hi @firdaus. You’re absolutely correct that the Temporal Python SDK’s @activity\_method indeed supports `async def`. I re-checked my code and realized the problem was with Cadence’s Python SDK. After some code cleanup and migrated everything to Temporal, the `await` calls work as expected. My apologies for the confusing question; it’s definitely a rookie mistake on my end.

Now, regarding my question about keeping resources available for the activities for the duration of the workflow - my use case requires activities updating records in a Cassandra db and the Cassandra driver client recommends setting up one connection session for the app’s runtime to avoid expensive reconnects to the Cassandra nodes. I was thinking that the worker might be a good place to house the Cassandra client. However, I haven’t figured out a way to access the client from the activities. For prototyping purposes, I am reinitializing the client in the activity method, which is fine for small numbers of workflows, but it won’t likely to scale. Do you have any recommendations on how to handle resources that need to live for the duration of the workflow? Or perhaps there are other ways to accomplish it and I’m going about it in the wrong way?

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [December 10, 2020, 6:47pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/5 "2020-12-10T18:47:51Z")

</div>

Can you pass your dependencies to the activity object before registering it with the worker?

---

<div class="post-metadata">

**Author:** ![Chen\_Wan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chen_wan/32/358_2.png) [@Chen\_Wan](https://community.temporal.io/u/Chen_Wan)\
**Post date:** [December 10, 2020, 11:24pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/6 "2020-12-10T23:24:59Z")

</div>

Hi @maxim. For data structures, passing them to the activity object before registering works. The issue is with the asyncio based clients/libraries (e.g, cassandra, pg & amqp clients) which require an event loop. If I initialize those clients before registering the activity object, the clients are tied to the main thread’s event loop. After activity registration, I believe the Python SDK would start a new thread and run an activity event loop to execute activities. Then calling the clients in the activities would result in an error (`RuntimeError: Event loop is closed`).

My current workaround is to have an init activity method for initialization of all the asyncio based clients, and calls the init activity at the beginning of the workflow. That way the initialization runs on the activity thread.

> async def my\_workflow(self, param):  
> await self.activities.init()  
> await self.activities.dosomething(param)  
> await self.activities.close()

This approach is inefficient because the clients are re-initialized every workflow execution. Ideally, the clients should initialize only once for the whole lifetime of the worker.

I was wondering if the SDK can take the approach of using the main asyncio event loop in the activity thread. I have seen a couple of examples demonstrating that technique. Personally, I haven’t tried it so not sure if it’d really work.

- [Python asyncio event loop in a separate thread · GitHub](https://gist.github.com/dmfigol/3e7d5b84a16d076df02baa9f53271058)
- [linw1995 - Run Asyncio Event Loop in another thread](https://linw1995.com/en/blog/Run-Asyncio-Event-Loop-in-another-thread/)

---

<div class="post-metadata">

**Author:** ![firdaus](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/firdaus/32/48_2.png) [@firdaus](https://community.temporal.io/u/firdaus)\
**Post date:** [December 13, 2020, 3:54pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/7 "2020-12-13T15:54:12Z")

</div>

Correct, the Python SDK starts a new thread and event loop per activity worker so you cannot do initialization of await-able objects in the constructor because they would be attached to the “main” event loop rather than the activity thread’s event loop.

A few possible solutions :

1.) Within the activity method, you can check whether the resources are initialized and initialize them if they aren’t. Since, the activity instance is a singleton, future calls to the activity will reuse the resources e.g.

```auto
class MyActivities:
   def __init__ (self):
       self.connection = None

   def my_activity_function(self):
       if not self.connection:
            self.connection = create_connection(...........)

```

2.) Run workflow and activities in the main event loop as you propose – this is theoretically possible, I think I opted against this at first because I wanted it to be possible to write non async activity methods but I think that requirement might be a thing of the past – the idea here is that if the activity thread owns the event loop then it’s free to block as it wishes — it won’t prevent other code from running.

3.) Modify the Python SDK to invoke a constructor-like method of the activity instance when the activity thread is started e.g. `post_init()` or `init_activity_instance()` or `init_async()` or `on_attached()` — I wouldn’t mind making this change to the library if I can settle on the method name to use.

4.) Introduce a method in the Worker that allows us to pass the arguments to the activity class’s constructor and have the Worker instantiate the activity class on the activity worker’s thread e.g.

```auto
worker.register_activity_class(MyActivities, arg1, arg2, arg3).

```

Then the Python SDK will invoke the constructor `MyActivities(arg1, arg2, arg3)`  
on the activity thread.

@Chen_Wan let me know what you think.

* * *

I’m actually struggling with an almost identical issue. I’m implementing `DataConverter` and the way it is done in the Java library is that the `DataConverter` is passed to WorkflowClient which is then passed to the Worker.

So the problem here is that in the Python SDK, WorkflowClient eagerly initializes the WorkflowServiceStub so we cannot pass the WorkflowServiceStub from WorkflowClient to the activity and workflow threads (at the moment there is no relationship between WorkflowClient and Workers).

Thinking out loud here, 2.) would actually solve both problems with the trade off that activity methods would have to be `async` — and using promises and threads we can actually work around that limitation as well and implement a layer on top of this that supports non-async activities.

One problem with using a single event loop is that badly written activities could starve the the workflow method from executing and meeting its execution time requirements.

Edit Dec 14: 2.) Is looking like the best option to me.

---

<div class="post-metadata">

**Author:** ![Chen\_Wan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chen_wan/32/358_2.png) [@Chen\_Wan](https://community.temporal.io/u/Chen_Wan)\
**Post date:** [December 15, 2020, 4:03pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/8 "2020-12-15T16:03:25Z")

</div>

Hi @firdaus. Thanks for the detailed analysis.  
Also, thinking out loud:

1. How do other SDKs (Java, Go) handle this scenario? Perhaps they don’t have the same problem because they don’t deal with the event loop. As long as the resource/client is thread-safe, they can be passed around between threads?
2. Option 2 makes sense. With this design, does the activity even need to be running on a separate thread? You’re right that a badly written activity can block too long and cause a problem.
3. Option 3 makes sense too. As long as I can initialize the resources on the activity thread at startup and performs clean up when the activity thread goes away.

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [December 15, 2020, 4:28pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/9 "2020-12-15T16:28:07Z")

</div>

> 1. How do other SDKs (Java, Go) handle this scenario? Perhaps they don’t have the same problem because they don’t deal with the event loop. As long as the resource/client is thread-safe, they can be passed around between threads?

Java and Go use threads to execute activities and don’t employ any event loops. So any shared resources are expected to be thread safe.

---

<div class="post-metadata">

**Author:** ![firdaus](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/firdaus/32/48_2.png) [@firdaus](https://community.temporal.io/u/firdaus)\
**Post date:** [December 15, 2020, 4:44pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/10 "2020-12-15T16:44:37Z")

</div>

> 1. How do other SDKs (Java, Go) handle this scenario? Perhaps they don’t have the same problem because they don’t deal with the event loop. As long as the resource/client is thread-safe, they can be passed around between threads?

When I ported the Python client from Cadence to Temporal the biggest difference was that the Thrift/Tchannel client uses blocking IO whereas the GRPC client I chose (python-betterproto) uses asyncio. So when I was doing this I was thinking about how to map the initial design from cadence-python to “asyncio” and my idea was to stick to having a thread per worker and then just introduce a separate event loop per thread.

So now I’m running into the limitations of that solution (both the issue you’re experiencing and my DataConverter issue) and I think it’s time to settle on something more long term which I believe is Option 2 — which in retrospect is probably what I should have done initially.

I’ll start working on this on my end and I’ll let you know when it’s done - maybe a couple of weeks.

---

<div class="post-metadata">

**Author:** ![Chen\_Wan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chen_wan/32/358_2.png) [@Chen\_Wan](https://community.temporal.io/u/Chen_Wan)\
**Post date:** [December 16, 2020, 8:41pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/11 "2020-12-16T20:41:47Z")

</div>

@maxim & @firdaus, thanks again for your quick replies. Your timely and concise responses are much appreciated.

@firdaus, with option 2, I’d assume that if I put `time.sleep(10)` in the activity code, it will block the whole worker for 10 seconds and even the workflow execution would be blocked. I guess it’s the trade-off of using asyncio’s cooperative model that the developers should be aware of.

---

<div class="post-metadata">

**Author:** ![firdaus](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/firdaus/32/48_2.png) [@firdaus](https://community.temporal.io/u/firdaus)\
**Post date:** [December 17, 2020, 12:02am UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/12 "2020-12-17T00:02:51Z")

</div>

Yup that is correct, you need to use `asyncio.sleep()` in activities.

In the future, we can consider an API for writing activities that use blocking IO.

---

<div class="post-metadata">

**Author:** ![Chen\_Wan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chen_wan/32/358_2.png) [@Chen\_Wan](https://community.temporal.io/u/Chen_Wan)\
**Post date:** [December 18, 2020, 5:39pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/13 "2020-12-18T17:39:51Z")

</div>

@firdaus. Understood, thanks.  
I look forward to the new changes 🙂

---

<div class="post-metadata">

**Author:** ![Chen\_Wan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chen_wan/32/358_2.png) [@Chen\_Wan](https://community.temporal.io/u/Chen_Wan)\
**Post date:** [January 20, 2021, 2:41am UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/14 "2021-01-20T02:41:31Z")

</div>

Hi @firdaus,  
Is it possible to provide an update on the progress of the refactoring (option 2)? Thanks.

---

<div class="post-metadata">

**Author:** ![firdaus](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/firdaus/32/48_2.png) [@firdaus](https://community.temporal.io/u/firdaus)\
**Post date:** [January 24, 2021, 9:50pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/15 "2021-01-24T21:50:38Z")

</div>

Hello, @Chen_Wan not much progress at the moment. I’m trying to carve out some free afternoons for this but it’s the usual beginning of the year craze for me at the moment.

---

<div class="post-metadata">

**Author:** ![Chen\_Wan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chen_wan/32/358_2.png) [@Chen\_Wan](https://community.temporal.io/u/Chen_Wan)\
**Post date:** [January 24, 2021, 11:24pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/16 "2021-01-24T23:24:12Z")

</div>

@firdaus. I completely understand. Your work and efforts are highly appreciated.  
If there is anything that I can help with (e.g, testing), please don’t hesitate to let me know.

---

<div class="post-metadata">

**Author:** ![firdaus](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/firdaus/32/48_2.png) [@firdaus](https://community.temporal.io/u/firdaus)\
**Post date:** [January 30, 2021, 12:10pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/17 "2021-01-30T12:10:40Z")

</div>

Could you try the latest commit.

---

<div class="post-metadata">

**Author:** ![firdaus](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/firdaus/32/48_2.png) [@firdaus](https://community.temporal.io/u/firdaus)\
**Post date:** [January 30, 2021, 1:55pm UTC](https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127/18 "2021-01-30T13:55:36Z")

</div>

@Chen_Wan Have a look at the HelloWorld example in the README. The contract for establishing a connection changed a bit.