Workflow and streaming event subscription & Make wait_condition retryable

Community Support
1

Hi, below is our use case

  1. we want to use our workflow to create some tickets, and the workflow need to listen to updates of those tickets, once we receive enough approvals from those tickets, the workflow can move forward
  2. listen to updates means subscribe to live stream events (e.g kafka consumer)

we have 2 implementation plan, can you give us advices on it

  1. we can create a web service that actively listens to the streaming events and send signal to workflows, then the workflow can use the wait_condition to receive signal from this service;
  2. The alternative solution is making the workflow subscribe to the streaming event directly

the second question is what if the stream event is down, the workflow therefore miss some events and we don’t have alternative ways to load those missing events

would it be ok if we make wait_condition retryable to solve this issue

  1. the wait_condition will timeout after waiting for a week
  2. before retry, the workflow will send a message to engineer to manually look into the issue
  3. the wait_condition will be retry
  4. engineer look into the issue and realize this is due to missing streaming events, engineer can manual send a signal to let wait_condition pass

if we want to implement this, should we

  1. create a separate child workflow to just make the wait_condition retryable without need to rerun the entire parent workflow
  2. and we can set timeout for wait_condition to make the child workflow fail if it waits for too long
  3. if the child workflow error out due to timeout, we use the retry policy to rerun the child workflow
2

Why do you need a webservice or Kafka? What component is notifying about ticket state changes? Can it call Temporal signal API directly?

Can you have an activity that checks for missing events? Then, you can wait for new events and, in parallel, run that activity periodically to perform the check.

reate a separate child workflow to just make the wait_condition retryable without need to rerun the entire parent workflow

This doesn’t require a child workflow. You can write a loop inside the parent to implement this logic.

3

Thanks for the quick response!

  1. the ticket is created by the company internal infra; and the only way to get update is via subscribing to the live stream (very similar to kafka), and the internal infra wouldn’t send signal to workflow directly; and if the live stream is down, we can’t fetch the missed events. at this case, we will need human intervene (though this should be very rare case)
  • our workaround is to create a separate web service (implementation plan 1) to subscribe to the live stream and then send a signal to our workflows
  • alternative workaround is to let workflow subscribe to the livestream directly, but the ticket would be open for 1-2 weeks, we will have 50 workflow has open live stream connection, we think this might be a bad practice
  1. how do we loop the wait_condition without making it a child workflow?
    approval_received = 0
while approval_received < 2:
      workflow.wait_condition(approval_received<2, timeout=2weeks)

my understanding is if the timeout of wait_condition is reached, the entire workflow will fail right? and we have to retry the entire workflow right, what if we just want to

  1. send a message to engineer to investigate
  2. retry workflow.wait_condition
  3. engineer can manual send a signal to make workflow.wait_condition pass.
4

if the timeout of wait_condition is reached, the entire workflow will fail right?

It will only fail if you don’t handle the TimeoutError. You can handle it using try-catch and continue in the loop.

1 Like