Error during Spring boot app start

Hi everyone! Workflow works fine locally on Win11/Spring Boot 3/JDK 21.

Spring Boot v3.5.3, Spring v6.2.8

Temporal Server version - at least in frontend I see 2.36.0
Temporal SDK version (tag the SDK used) - 1.30.1

DevOps provided self-hosted Temporal server recently. After filling configuration and reaching balancer I got an error:

2025-08-18T16:51:16.301+05:00 ERROR 15568 — [ncp-collector-api] [           main] i.t.internal.worker.SyncWorkflowWorker   : [BUG] Unexpected exception during shutdown

java.util.concurrent.CompletionException: io.grpc.StatusRuntimeException: UNKNOWN: HTTP status code 405invalid content-type: application/json; charset=UTF-8headers: Metadata(:status=405,date=Mon, 18 Aug 2025 11:51:16 GMT,content-type=application/json; charset=UTF-8,allow=OPTIONS, GET,vary=Accept-Encoding,vary=Origin,x-content-type-options=nosniff,x-frame-options=SAMEORIGIN,x-xss-protection=1; mode=block,strict-transport-security=max-age=15724800; includeSubDomains,content-length=33)DATA-----------------------------{“message”:“Method Not Allowed”}

DATA-----------------------------

yml:

  temporal:
    namespace: billing-dev
    connection:
      target: dev-temporal.kz
    workersAutoDiscovery:
      packages:
        - com.mtelecom.ncp.collector.orchestration.workflow
        - com.mtelecom.ncp.collector.orchestration.activity

WorkflowServiceStubs config:

@Configuration
public class TemporalConfig {

    @Value("${spring.temporal.connection.target}")
    private String server;

    @Bean
    public WorkflowServiceStubs workflowServiceStubs(MyJwtTokenSupplier tokenSupplier) {
        ManagedChannel channel = NettyChannelBuilder.forAddress(server, 443)
                .useTransportSecurity()
                .build();
        WorkflowServiceStubsOptions options = WorkflowServiceStubsOptions.newBuilder()
                .addGrpcMetadataProvider(new AuthorizationGrpcMetadataProvider(tokenSupplier))
                .setChannel(channel)
                .build();

        return WorkflowServiceStubs.newServiceStubs(options);
    }

worker:

@Component
@RequiredArgsConstructor
@Slf4j
public class CollectorPaymentWorker {

    private final WorkerConfigurationProperties workerConfigurationProperties;
    private final WorkflowClient workflowClient;
    private final WorkflowLogService workflowLogService;
    private final ObjectMapper objectMapper;

    public void start(CollectorNewPaymentMessage collectorNewPaymentMessage) {
        CollectorPaymentWorkflow workflow = workflowClient.newWorkflowStub(CollectorPaymentWorkflow.class, WorkflowOptions.newBuilder()
                .setTaskQueue(workerConfigurationProperties.getTaskQueue())
                .setWorkflowId(workerConfigurationProperties.getId())
                .build());

        WorkflowExecution workflowExecution = WorkflowClient.start(workflow::handleCollectorPayment, collectorNewPaymentMessage,
                workerConfigurationProperties);

trigger:

@KafkaListener(topics = "#{'${kafka.topics.collector-new-payment}'.split('\\\\ ')}", groupId = "ncp-api")
public void listenForCollectorPayments(String message) {
    log.info("Received Kafka message \"{}\" for topic \"{}\"", message, topic);
    try {
        CollectorNewPaymentMessage collectorNewPaymentMessage = mapper.readValue(message, CollectorNewPaymentMessage.class);
        collectorPaymentWorker.start(collectorNewPaymentMessage);
    } catch (JsonProcessingException e) {
        log.error("ERROR: ", e);
    }
}

token (valid, works through Postman when calling REST API):

@Service
public class TokenServiceImpl implements TokenService {

@Override
public String getJwtToken() {
    return "Bearer eyJhbGciOiJSUzI1NiIs_________________-iXCQGKHqmWTg9seKfLVWS6wo50DJstQ";
}

}

log:

2025-08-18T16:51:00.198+05:00  INFO 15568 --- [ncp-collector-api] [           main] c.m.n.c.NcpCollectorApiApplication       : Starting NcpCollectorApiApplication using Java 21.0.7 with PID 15568 (F:\dev\prj\ncp-collector-api\target\classes started by zhassulan.tokbayev in F:\dev\prj\ncp-collector-api)
2025-08-18T16:51:00.199+05:00 DEBUG 15568 --- [ncp-collector-api] [           main] c.m.n.c.NcpCollectorApiApplication       : Running with Spring Boot v3.5.3, Spring v6.2.8
2025-08-18T16:51:00.199+05:00  INFO 15568 --- [ncp-collector-api] [           main] c.m.n.c.NcpCollectorApiApplication       : The following 1 profile is active: "dev"
2025-08-18T16:51:01.945+05:00  INFO 15568 --- [ncp-collector-api] [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data R2DBC repositories in DEFAULT mode.
2025-08-18T16:51:01.960+05:00  INFO 15568 --- [ncp-collector-api] [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 5 ms. Found 0 R2DBC repository interfaces.
2025-08-18T16:51:02.364+05:00  INFO 15568 --- [ncp-collector-api] [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data R2DBC repositories in DEFAULT mode.
2025-08-18T16:51:02.512+05:00  INFO 15568 --- [ncp-collector-api] [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 145 ms. Found 1 R2DBC repository interface.
2025-08-18T16:51:04.442+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.WorkflowServiceStubsImpl           : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=dev-temporal.kz:443}}
2025-08-18T16:51:05.598+05:00  INFO 15568 --- [ncp-collector-api] [           main] org.flywaydb.core.FlywayExecutor         : Database: jdbc:oracle:thin:@10.200.203.140:1540:tst2 (Oracle 19.21)
2025-08-18T16:51:05.874+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.f.core.internal.command.DbValidate     : Successfully validated 2 migrations (execution time 00:00.043s)
2025-08-18T16:51:06.141+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.f.core.internal.command.DbMigrate      : Current version of schema "NCP_APP": 1.1.0
2025-08-18T16:51:06.153+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.f.core.internal.command.DbMigrate      : Schema "NCP_APP" is up to date. No migration necessary.
2025-08-18T16:51:06.486+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 14 endpoints beneath base path '/actuator'
2025-08-18T16:51:06.607+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.WorkflowServiceStubsImpl           : Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=3, target=dev-temporal.kz}}
2025-08-18T16:51:06.661+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.b.a.template.WorkersTemplate       : Creating a worker with default settings for a task queue 'ncpCollectorTaskQueue' caused by an auto-discovered workflow class class com.mtelecom.ncp.collector.orchestration.workflow.CollectorPaymentWorkflowImpl
2025-08-18T16:51:06.699+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.b.a.template.WorkersTemplate       : Registering auto-discovered workflow class class com.mtelecom.ncp.collector.orchestration.workflow.CollectorPaymentWorkflowImpl on a worker  with a task queue 'ncpCollectorTaskQueue'
2025-08-18T16:51:06.704+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.b.a.template.WorkersTemplate       : Registering auto-discovered activity bean 'authActivitiesImpl' of class class com.mtelecom.ncp.collector.orchestration.activity.AuthActivitiesImpl on a worker  with a task queue 'ncpCollectorTaskQueue'
2025-08-18T16:51:06.705+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.b.a.template.WorkersTemplate       : Registering auto-discovered activity bean 'dealerProActivitiesImpl' of class class com.mtelecom.ncp.collector.orchestration.activity.DealerProActivitiesImpl on a worker  with a task queue 'ncpCollectorTaskQueue'
2025-08-18T16:51:06.705+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.b.a.template.WorkersTemplate       : Registering auto-discovered activity bean 'ncpActivitiesImpl' of class class com.mtelecom.ncp.collector.orchestration.activity.NcpActivitiesImpl on a worker  with a task queue 'ncpCollectorTaskQueue'
2025-08-18T16:51:06.705+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.b.a.template.WorkersTemplate       : Registering auto-discovered activity bean 'oneCActivitiesImpl' of class class com.mtelecom.ncp.collector.orchestration.activity.OneCActivitiesImpl on a worker  with a task queue 'ncpCollectorTaskQueue'
2025-08-18T16:51:06.705+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.b.a.template.WorkersTemplate       : Registering auto-discovered activity bean 'subscriberActivitiesImpl' of class class com.mtelecom.ncp.collector.orchestration.activity.SubscriberActivitiesImpl on a worker  with a task queue 'ncpCollectorTaskQueue'
2025-08-18T16:51:07.391+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8083 (http)
2025-08-18T16:51:07.436+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.a.k.clients.consumer.ConsumerConfig    : ConsumerConfig values: 
	allow.auto.create.topics = true
	auto.commit.interval.ms = 5000
	auto.include.jmx.reporter = true
	auto.offset.reset = latest
	bootstrap.servers = [inh-kafka-test.kz:9092]
	check.crcs = true
	client.dns.lookup = use_all_dns_ips
	client.id = consumer-ncp-api-1
	client.rack = 
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = false
	enable.metrics.push = true
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = ncp-api
	group.instance.id = null
	group.protocol = classic
	group.remote.assignor = null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	internal.throw.on.fetch.stable.offset.unsupported = false
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metadata.recovery.strategy = none
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retry.backoff.max.ms = 1000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.connect.timeout.ms = null
	sasl.login.read.timeout.ms = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.login.retry.backoff.max.ms = 10000
	sasl.login.retry.backoff.ms = 100
	sasl.mechanism = GSSAPI
	sasl.oauthbearer.clock.skew.seconds = 30
	sasl.oauthbearer.expected.audience = null
	sasl.oauthbearer.expected.issuer = null
	sasl.oauthbearer.header.urlencode = false
	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
	sasl.oauthbearer.jwks.endpoint.url = null
	sasl.oauthbearer.scope.claim.name = scope
	sasl.oauthbearer.sub.claim.name = sub
	sasl.oauthbearer.token.endpoint.url = null
	security.protocol = PLAINTEXT
	security.providers = null
	send.buffer.bytes = 131072
	session.timeout.ms = 45000
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

2025-08-18T16:51:07.474+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.a.k.c.t.i.KafkaMetricsCollector        : initializing Kafka metrics collector
2025-08-18T16:51:07.578+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.9.1
2025-08-18T16:51:07.579+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: f745dfdcee2b9851
2025-08-18T16:51:07.579+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1755517867577
2025-08-18T16:51:07.589+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.a.k.c.c.i.ClassicKafkaConsumer         : [Consumer clientId=consumer-ncp-api-1, groupId=ncp-api] Subscribed to topic(s): ncp-collector-new-payment
2025-08-18T16:51:07.612+05:00  INFO 15568 --- [ncp-collector-api] [           main] c.m.n.c.NcpCollectorApiApplication       : Started NcpCollectorApiApplication in 8.057 seconds (process running for 10.452)
2025-08-18T16:51:08.130+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] org.apache.kafka.clients.Metadata        : [Consumer clientId=consumer-ncp-api-1, groupId=ncp-api] Cluster ID: bJFBdxx5QdmhfydaZo0eDA
2025-08-18T16:51:11.182+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] k.c.c.i.ConsumerRebalanceListenerInvoker : [Consumer clientId=consumer-ncp-api-1, groupId=ncp-api] Adding newly assigned partitions: ncp-collector-new-payment-0
2025-08-18T16:51:11.195+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerUtils        : Setting offset for partition ncp-collector-new-payment-0 to the committed offset FetchPosition{offset=120, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.250.5.158:9092 (id: 1 rack: rack0)], epoch=0}}
2025-08-18T16:51:11.196+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : ncp-api: partitions assigned: [ncp-collector-new-payment-0]
2025-08-18T16:51:13.969+05:00 ERROR 15568 --- [ncp-collector-api] [           main] o.s.boot.SpringApplication               : Application run failed

io.grpc.StatusRuntimeException: UNKNOWN: HTTP status code 405
invalid content-type: application/json; charset=UTF-8
headers: Metadata(:status=405,date=Mon, 18 Aug 2025 11:51:11 GMT,content-type=application/json; charset=UTF-8,allow=OPTIONS, GET,vary=Accept-Encoding,vary=Origin,x-content-type-options=nosniff,x-frame-options=SAMEORIGIN,x-xss-protection=1; mode=block,strict-transport-security=max-age=15724800; includeSubDomains,content-length=33)
DATA-----------------------------
{"message":"Method Not Allowed"}

DATA-----------------------------

	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getSystemInfo(WorkflowServiceGrpc.java:6376)
	at io.temporal.serviceclient.SystemInfoInterceptor.getServerCapabilitiesOrThrow(SystemInfoInterceptor.java:112)
	at io.temporal.serviceclient.SystemInfoInterceptor.lambda$getServerCapabilitiesWithRetryOrThrow$0(SystemInfoInterceptor.java:98)
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:49)
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:40)
	at io.temporal.serviceclient.SystemInfoInterceptor.getServerCapabilitiesWithRetryOrThrow(SystemInfoInterceptor.java:97)
	at io.temporal.serviceclient.ChannelManager.lambda$getServerCapabilities$3(ChannelManager.java:324)
	at io.temporal.internal.worker.WorkflowPollTask.<init>(WorkflowPollTask.java:78)
	at io.temporal.internal.worker.WorkflowWorker.start(WorkflowWorker.java:172)
	at io.temporal.internal.worker.SyncWorkflowWorker.start(SyncWorkflowWorker.java:164)
	at io.temporal.worker.Worker.start(Worker.java:408)
	at io.temporal.worker.WorkerFactory.start(WorkerFactory.java:204)
	at io.temporal.spring.boot.autoconfigure.RootNamespaceAutoConfiguration$WorkerFactoryStarter.onApplicationEvent(RootNamespaceAutoConfiguration.java:205)
	at io.temporal.spring.boot.autoconfigure.RootNamespaceAutoConfiguration$WorkerFactoryStarter.onApplicationEvent(RootNamespaceAutoConfiguration.java:195)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:454)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:387)
	at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109)
	at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
	at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
	at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:332)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1361)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1350)
	at com.mtelecom.ncp.collector.NcpCollectorApiApplication.main(NcpCollectorApiApplication.java:13)

2025-08-18T16:51:13.977+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] k.c.c.i.ConsumerRebalanceListenerInvoker : [Consumer clientId=consumer-ncp-api-1, groupId=ncp-api] Revoke previously assigned partitions ncp-collector-new-payment-0
2025-08-18T16:51:13.978+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : ncp-api: partitions revoked: [ncp-collector-new-payment-0]
2025-08-18T16:51:13.979+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.a.k.c.c.i.ClassicKafkaConsumer         : [Consumer clientId=consumer-ncp-api-1, groupId=ncp-api] Unsubscribed all topics or patterns and assigned partitions
2025-08-18T16:51:14.249+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.apache.kafka.common.metrics.Metrics    : Metrics scheduler closed
2025-08-18T16:51:14.250+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.apache.kafka.common.metrics.Metrics    : Closing reporter org.apache.kafka.common.metrics.JmxReporter
2025-08-18T16:51:14.250+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.apache.kafka.common.metrics.Metrics    : Closing reporter org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter
2025-08-18T16:51:14.250+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.apache.kafka.common.metrics.Metrics    : Metrics reporters closed
2025-08-18T16:51:14.255+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.a.kafka.common.utils.AppInfoParser     : App info kafka.consumer for consumer-ncp-api-1 unregistered
2025-08-18T16:51:14.256+05:00  INFO 15568 --- [ncp-collector-api] [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : ncp-api: Consumer stopped
2025-08-18T16:51:14.258+05:00  INFO 15568 --- [ncp-collector-api] [           main] o.s.b.w.embedded.netty.GracefulShutdown  : Commencing graceful shutdown. Waiting for active requests to complete
2025-08-18T16:51:14.260+05:00  INFO 15568 --- [ncp-collector-api] [ netty-shutdown] o.s.b.w.embedded.netty.GracefulShutdown  : Graceful shutdown complete
2025-08-18T16:51:16.286+05:00  INFO 15568 --- [ncp-collector-api] [           main] io.temporal.worker.WorkerFactory         : shutdown: WorkerFactory{identity=15568@ALA-D-480281}
2025-08-18T16:51:16.301+05:00 ERROR 15568 --- [ncp-collector-api] [           main] i.t.internal.worker.SyncWorkflowWorker   : [BUG] Unexpected exception during shutdown

java.util.concurrent.CompletionException: io.grpc.StatusRuntimeException: UNKNOWN: HTTP status code 405
invalid content-type: application/json; charset=UTF-8
headers: Metadata(:status=405,date=Mon, 18 Aug 2025 11:51:16 GMT,content-type=application/json; charset=UTF-8,allow=OPTIONS, GET,vary=Accept-Encoding,vary=Origin,x-content-type-options=nosniff,x-frame-options=SAMEORIGIN,x-xss-protection=1; mode=block,strict-transport-security=max-age=15724800; includeSubDomains,content-length=33)
DATA-----------------------------
{"message":"Method Not Allowed"}

DATA-----------------------------

	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1194)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341)
	at io.temporal.internal.worker.WorkflowWorker.shutdown(WorkflowWorker.java:206)
	at io.temporal.internal.worker.SyncWorkflowWorker.shutdown(SyncWorkflowWorker.java:175)
	at io.temporal.worker.Worker.shutdown(Worker.java:417)
	at io.temporal.worker.WorkerFactory.lambda$shutdownInternal$2(WorkerFactory.java:282)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1787)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at io.temporal.worker.WorkerFactory.shutdownInternal(WorkerFactory.java:283)
	at io.temporal.worker.WorkerFactory.shutdown(WorkerFactory.java:256)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:316)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:249)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:798)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:748)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1474)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:707)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1467)
	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1219)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1180)
	at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.doClose(ReactiveWebServerApplicationContext.java:155)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1126)
	at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:814)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:336)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1361)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1350)
	at com.mtelecom.ncp.collector.NcpCollectorApiApplication.main(NcpCollectorApiApplication.java:13)
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: HTTP status code 405
invalid content-type: application/json; charset=UTF-8
headers: Metadata(:status=405,date=Mon, 18 Aug 2025 11:51:16 GMT,content-type=application/json; charset=UTF-8,allow=OPTIONS, GET,vary=Accept-Encoding,vary=Origin,x-content-type-options=nosniff,x-frame-options=SAMEORIGIN,x-xss-protection=1; mode=block,strict-transport-security=max-age=15724800; includeSubDomains,content-length=33)
DATA-----------------------------
{"message":"Method Not Allowed"}

DATA-----------------------------

	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getSystemInfo(WorkflowServiceGrpc.java:6376)
	at io.temporal.serviceclient.SystemInfoInterceptor.getServerCapabilitiesOrThrow(SystemInfoInterceptor.java:112)
	at io.temporal.serviceclient.SystemInfoInterceptor$1.start(SystemInfoInterceptor.java:66)
	at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32)
	at io.temporal.serviceclient.GrpcMetadataProviderInterceptor$HeaderAttachingClientCall.start(GrpcMetadataProviderInterceptor.java:52)
	at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:335)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:311)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:224)
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceFutureStub.shutdownWorker(WorkflowServiceGrpc.java:7523)
	at io.temporal.internal.worker.WorkflowWorker.lambda$shutdown$2(WorkflowWorker.java:212)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
	... 32 common frames omitted

2025-08-18T16:51:16.305+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.WorkflowServiceStubsImpl           : shutdown
2025-08-18T16:51:16.308+05:00  INFO 15568 --- [ncp-collector-api] [           main] i.t.s.WorkflowServiceStubsImpl           : shutdown

Process finished with exit code 1

Does anybody face that error? Thank you!

More Logs - Spring Boot Log Trace Level

Workflow works fine locally on Win11/Spring Boot 3/JDK 21.

are you setting token supplier locally too?

do you see same issue if you override service stubs options as shown in sample here?

can you use same credentials to connect to your service using cli?

No, I added token supplier after successful local tests. Now I’m trying to use remote Temporal dev server. Admin gave me Keycloak account to get auth token. Overriding stubs gave me another error:

 [io/temporal/spring/boot/autoconfigure/RootNamespaceAutoConfiguration.class]: Unsatisfied dependency expressed through method 'rootNamespaceTemplate' parameter 1: Error creating bean with name 'temporalWorkflowServiceStubs' defined in class path resource [io/temporal/spring/boot/autoconfigure/ServiceStubsAutoConfiguration.class]: Failed to instantiate [io.temporal.serviceclient.WorkflowServiceStubs]: Factory method 'workflowServiceStubsTemplate' threw exception with message: Only one of the 'target' or 'channel' options can be set at a time

I’m not sure that local distributive can help me in this case. I can call REST API using creds.

Looks like I used wrong API endpoint.