Hi,
I have an issue with PHP while trying to make some load tests. I have a standard setup with Roardunner workers (exactly 4 of them). Temporal runs from auto-setup docker container, with mysql backend and with elastic.
My load testing happens in the following way:
- I have a script which inserts 10000 workflows called FORK (a parent workflow which creates exactly one child workflow)
- the child workflows are empty (I have some code for them to run an activity, but since I found some troubles while testing, I’m trying to find a minimum working example)
- while the script from 1) is executing, I’m running 4 temporal workers (wrapped in symfony but that shouldn’t be an issue)
There are no timers in any of the workflows, they should execute immediately. What I’m observing are two things:
- the workflows processing from the workers sometimes hangs for up to two minutes or even more, no idea why, I’m seeing hundreds of incomplete workflows in the UI and the workers do nothing
- only one of the configured workers seems to do some processing, the other 3 are waiting/idling
When watching the CPU utilisation in containers, when the worker is working as expected, it takes a normal amopunt of CPU and the temporal container (temporal is deployed from the auto setup) takes up to 2 CPU cores… When the worker is idling, the temporal container CPU runs on 6% … From my point of view it seems that the temporal worker is waiting for something but I have no idea what it can be, since the UI shows many running workflows and none of them is being scheduled…
This is the Fork workflow code:
final class DefaultForkingWorkflow implements ForkingWorkflow
{
/**
* @return Generator<int, CancellationScopeInterface, mixed, void>
*/
// @todo remove namespace dependency https://github.com/temporalio/sdk-php/issues/104
public function fork(Event $event, HandlerDefinitions $definitions, string $namespace)
{
//yield
Workflow::asyncDetached(static function () use ($event, $definitions, $namespace) {
$promises = [];
foreach ($definitions as $definition) {
$options = ChildWorkflowOptions::new()
->withWorkflowTaskTimeout(CarbonInterval::seconds(10))
->withParentClosePolicy(ParentClosePolicy::POLICY_ABANDON)
->withNamespace($namespace)
->withSearchAttributes([
SearchAttributes::ATTR_EVENT_ID => $event->getMetadata(Id::class)->toString(),
SearchAttributes::ATTR_HANDLER_ID => $definition->getServiceId(),
SearchAttributes::ATTR_HANDLER_METHOD => $definition->getHandlerMethod(),
]);
$handlerWorkflow = Workflow::newChildWorkflowStub(
HandlerWorkflow::class,
$options
);
$promises[] = Workflow::asyncDetached(
static fn () => /*yield*/ $handlerWorkflow->handle($event, $definition)
);
}
//return yield
Promise::all($promises);
});
}
}
I have also tried with yielding version (not sure what the difference is because they behave quite similarly):
final class DefaultForkingWorkflow implements ForkingWorkflow
{
/**
* @return Generator<int, CancellationScopeInterface, mixed, void>
*/
// @todo remove namespace dependency https://github.com/temporalio/sdk-php/issues/104
public function fork(Event $event, HandlerDefinitions $definitions, string $namespace)
{
yield Workflow::asyncDetached(static function () use ($event, $definitions, $namespace) {
$promises = [];
foreach ($definitions as $definition) {
$options = ChildWorkflowOptions::new()
->withWorkflowTaskTimeout(CarbonInterval::seconds(10))
->withParentClosePolicy(ParentClosePolicy::POLICY_ABANDON)
->withNamespace($namespace)
->withSearchAttributes([
SearchAttributes::ATTR_EVENT_ID => $event->getMetadata(Id::class)->toString(),
SearchAttributes::ATTR_HANDLER_ID => $definition->getServiceId(),
SearchAttributes::ATTR_HANDLER_METHOD => $definition->getHandlerMethod(),
]);
$handlerWorkflow = Workflow::newChildWorkflowStub(
HandlerWorkflow::class,
$options
);
$promises[] = Workflow::asyncDetached(
static fn () => yield $handlerWorkflow->handle($event, $definition)
);
}
return yield Promise::all($promises);
});
}
}
This is my .rr.yaml:
rpc:
listen: tcp://127.0.0.1:6001
server:
command: "/srv/www/src/LoadTest/bin/console event-bus:temporal-worker:run"
temporal:
address: ${TEMPORAL_CLI_ADDRESS}
namespace: ${TEMPORAL_NAMESPACE}
activities:
num_workers: 4
allocate_timeout: 10s
destroy_timeout: 20s
logs:
mode: production
level: debug
output: stderr
channels:
temporal:
level: debug
informer:
mode: production
All the testing is executed locally in docker containers.
I’m not really sure what the problem might be, could someone help me please?
Thank you.
