Using TestWorkflowEnvironment for testing for an hour with around 350K workflow instances

Community Support
,
1

Hello Team,

I have a requirement to simulate the system with around 350K workflow’s overall and run time for an hour. I would like to prefer the TestWorkflowEnvironment for end to end testing as its all in memory. But considering a lot of data getting generated, would the the memory be impacted or all the historical workflow’s and other data getting persisted are going to be deleted as a clean up process, to reduce the memory usage.

Request to shed light on the memory usage and approach to cleanup the historical workflow’s.

2

Hi Kartik :wave:,

I’m Connor from the Temporal team, we have been working with some of the other users at Verizon and would be happy to help as you work through this testing.

If you’d like feel free to schedule 15-30 mins and we can help you out to get started.

Scheduling availability: Calendly - Connor Lewis

4

Hi Kartik, I know this is the Discourse community - but I’d highly recommend hopping into the slack community (it is much more vibrant and our developer success teams are much more active in there and can assist you much better).

5

Hi, just wanted to add in case there is some confusion after reading this forum thread.

We do not recommend slack over forum, it is users choice and in most cases if you have a
good question (which you all do :slight_smile: ) we would prefer if you ask it on the forum as here we can preserve the answer forever
which also benefits other Temporal users. We do and have been doing community support on both
forum here and on slack. From support perspective both are equal.

Hope that helps. Thanks!

6

Sure thank you @tihomir and @connor

7

Hello @tihomir ,

I tried using the TestWorkflowEnvrionment, to perform a simulation, just to speed up the process, instead of running the workflows with dev server. The test execution created 56250 workflow execution instances, 123750 signals for a period of 15 min, in c5d.2xlarge aws ec2 instance. Observed that the CPU is under utilized and hence less throughput overall. I am checking the other aspects of the application though, meanwhile tried exposing the metrics using below config to observe temporal metrics.

public void init() {

        PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);

        StatsReporter reporter = new MicrometerClientStatsReporter(registry);

        Scope scope = new RootScopeBuilder()
                .reporter(reporter)
                .reportEvery(com.uber.m3.util.Duration.ofSeconds(2));

        TestEnvironmentOptions testEnvOptions = TestEnvironmentOptions.newBuilder()
                .setWorkflowServiceStubsOptions(
                        WorkflowServiceStubsOptions.newBuilder()
                                .setMetricsScope(scope)
                                .build()
                )
                .build();

        testEnvironment = TestWorkflowEnvironment.newInstance(testEnvOptions);
    }

Currently using springboot and hence could enable /actuator/prometheus endpoint, but the temporal metrics are not visible at all.

Is there any config missing or metrics are not captured in TestWorkflowEnvironment.