Unable to run temporal workflow from jar

Shean · November 30, 2022, 9:24am

I am using temporal for running workflows, and I could successfully run it in Intellij. I have created a jar with my app using mvn package -f ./pom.xml. and running the below cmd from terminal java -jar ./target/temporal-jdk-test-1.0-SNAPSHOT.jar. errors below appears:

Exception in thread "main" io.grpc.StatusRuntimeException: UNKNOWN
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
        at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getSystemInfo(WorkflowServiceGrpc.java:4139)
        at io.temporal.serviceclient.SystemInfoInterceptor.getServerCapabilitiesOrThrow(SystemInfoInterceptor.java:95)
        at io.temporal.serviceclient.ChannelManager.lambda$getServerCapabilities$3(ChannelManager.java:330)
        at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60)
        at io.temporal.serviceclient.ChannelManager.connect(ChannelManager.java:297)
        at io.temporal.serviceclient.WorkflowServiceStubsImpl.connect(WorkflowServiceStubsImpl.java:161)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.temporal.internal.WorkflowThreadMarker.lambda$protectFromWorkflowThread$1(WorkflowThreadMarker.java:83)
        at jdk.proxy1/jdk.proxy1.$Proxy0.connect(Unknown Source)
        at io.temporal.worker.WorkerFactory.start(WorkerFactory.java:210)
        at org.example.App.main(App.java:84)
Caused by: java.nio.channels.UnsupportedAddressTypeException
        at java.base/sun.nio.ch.Net.checkAddress(Net.java:146)
        at java.base/sun.nio.ch.Net.checkAddress(Net.java:157)
        at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816)
        at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839)
        at io.grpc.netty.shaded.io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91)
        at io.grpc.netty.shaded.io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        at io.grpc.netty.shaded.io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88)
        at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:322)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:248)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
        at io.grpc.netty.shaded.io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:54)
        at io.grpc.netty.shaded.io.grpc.netty.WriteBufferingAndExceptionHandler.connect(WriteBufferingAndExceptionHandler.java:157)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.access$1000(AbstractChannelHandlerContext.java:61)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext$9.run(AbstractChannelHandlerContext.java:538)
        at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)

My main function (A hello world):

package org.example;

import io.temporal.activity.ActivityInterface;
import io.temporal.activity.ActivityMethod;
import io.temporal.activity.ActivityOptions;
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.client.WorkflowOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerFactory;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;
import java.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class App {

    static final String TASK_QUEUE = "HelloActivityTaskQueue";

    static final String WORKFLOW_ID = "HelloActivityWorkflow";

    @WorkflowInterface
    public interface GreetingWorkflow {

        @WorkflowMethod
        String getGreeting(String name);
    }

    @ActivityInterface
    public interface GreetingActivities {

        @ActivityMethod(name = "greet")
        String composeGreeting(String greeting, String name);
    }

    public static class GreetingWorkflowImpl implements GreetingWorkflow {
        private final GreetingActivities activities =
                Workflow.newActivityStub(
                        GreetingActivities.class,
                        ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofSeconds(2)).build());

        @Override
        public String getGreeting(String name) {
            return activities.composeGreeting("Hello", name);
        }
    }

    static class GreetingActivitiesImpl implements GreetingActivities {
        private static final Logger log = LoggerFactory.getLogger(GreetingActivitiesImpl.class);

        @Override
        public String composeGreeting(String greeting, String name) {
            log.info("Composing greeting...");
            return greeting + " " + name + "!";
        }
    }

    public static void main(String[] args) {

        WorkflowServiceStubs service =
                WorkflowServiceStubs.newLocalServiceStubs();

        WorkflowClient client =
                WorkflowClient.newInstance(
                        service, WorkflowClientOptions.newBuilder().build());

        WorkerFactory factory = WorkerFactory.newInstance(client);

        Worker worker = factory.newWorker(TASK_QUEUE);

        worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class);

        worker.registerActivitiesImplementations(new GreetingActivitiesImpl());

        factory.start();

        /* ======================== line between listener and worker ======================== */

        GreetingWorkflow workflow =
                client.newWorkflowStub(
                        GreetingWorkflow.class,
                        WorkflowOptions.newBuilder()
                                .setWorkflowId(WORKFLOW_ID)
                                .setTaskQueue(TASK_QUEUE)
                                .build());

        String greeting = workflow.getGreeting("World");

        System.out.println(greeting);
        System.exit(0);
    }
}

my maven file is:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>org.example</groupId>
  <artifactId>temporal-jdk-test</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>temporal-jdk-test</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>io.temporal</groupId>
      <artifactId>temporal-sdk</artifactId>
      <version>1.17.0</version>
    </dependency>
    <dependency>
      <groupId>io.temporal</groupId>
      <artifactId>temporal-opentracing</artifactId>
      <version>1.17.0</version>
    </dependency>
    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>1.2.11</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.2.2</version>
        <configuration>
          <createDependencyReducedPom>false</createDependencyReducedPom>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <!-- Build an executable JAR -->
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <classpathPrefix>lib/</classpathPrefix>
              <mainClass>org.example.App</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.6.2</version>
      </plugin>
    </plugins>
  </build>
</project>

If I want to deploy pods in pods in production, is this the right way to package file in jar (using Dockerfile to do this) and run them directly?

Shean · November 30, 2022, 9:28am

I dig into jar and found out that in this public static InetSocketAddress checkAddress(SocketAddress sa) , the SocketAddress will become /127.0.0.1:7723 (my original addr is 127.0.0.1:7723, a / is added automatically…). Then the validation check failed… I still don’t know how to solve it.

iftekhar09 · November 30, 2022, 10:49am

I am facing the same issue. @Shean - were u able to fix the issue ?

Shean · November 30, 2022, 10:59am

no…i’m still trying to solve it… just curious how other java-sdk users solve this… seems like a common issue.

antonio.perez · November 30, 2022, 11:25am

Hi @Shean @iftekhar09

I am facing the same issue, however I manage to build and run the jar as a spring-boot application,

<build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                        <configuration>
                            <classifier>spring-boot</classifier>
                            <mainClass>
                                org.example.App
                            </mainClass>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

I will continue looking into this

Shean · November 30, 2022, 11:45am

Thanks for the reply! This could work. Thank you so much!
I just got some other findings: if I added ~/.m2/repository/io/grpc/grpc-core/1.50.2/grpc-core-1.50.2.jar to the classpath, it could also successfully run, like java -cp ./target/temporal-jdk-test-1.0-SNAPSHOT.jar:~/.m2/repository/io/grpc/grpc-core/1.50.2/grpc-core-1.50.2.jar org.example.App
I think this issue may be related to grpc framework Temporal imported…

iftekhar09 · November 30, 2022, 4:26pm

Hi, I am now getting the below error when I deployed my app to the Kubernetes cluster.

Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.50.2.jar!/:1.50.2]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.50.2.jar!/:1.50.2]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.50.2.jar!/:1.50.2]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getSystemInfo(WorkflowServiceGrpc.java:4139) ~[temporal-serviceclient-1.17.0.jar!/:na]
	at io.temporal.serviceclient.SystemInfoInterceptor.getServerCapabilitiesOrThrow(SystemInfoInterceptor.java:95) ~[temporal-serviceclient-1.17.0.jar!/:na]
	at io.temporal.serviceclient.ChannelManager.lambda$getServerCapabilities$3(ChannelManager.java:330) ~[temporal-serviceclient-1.17.0.jar!/:na]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar!/:na]
	at io.temporal.serviceclient.ChannelManager.connect(ChannelManager.java:297) ~[temporal-serviceclient-1.17.0.jar!/:na]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.connect(WorkflowServiceStubsImpl.java:161) ~[temporal-serviceclient-1.17.0.jar!/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
	at io.temporal.internal.WorkflowThreadMarker.lambda$protectFromWorkflowThread$1(WorkflowThreadMarker.java:83) ~[temporal-serviceclient-1.17.0.jar!/:na]
	at jdk.proxy2/jdk.proxy2.$Proxy57.connect(Unknown Source) ~[na:na]
	at io.temporal.worker.WorkerFactory.start(WorkerFactory.java:210) ~[temporal-sdk-1.17.0.jar!/:na]
	at com.hok.furlenco.workflow.refundStatusSync.RefundStatusSyncSaga.createWorkFlow(RefundStatusSyncSaga.java:41) ~[classes!/:na]
	at com.hok.furlenco.workflow.refundStatusSync.RefundStatusSyncSaga.main(RefundStatusSyncSaga.java:17) ~[classes!/:na]
	at com.hok.furlenco.workflow.HokTemporalWorkflowsApplication.lambda$CommandLineRunnerBean$0(HokTemporalWorkflowsApplication.java:21) ~[classes!/:na]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:767) ~[spring-boot-3.0.0.jar!/:3.0.0]
	... 13 common frames omitted
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /127.0.0.1:7233
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
	at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:321) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:477) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[grpc-netty-shaded-1.50.2.jar!/:1.50.2]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

@antonio.perez / @Shean - did u guys also face same issue ?

antonio.perez · December 1, 2022, 11:46am

@iftekhar09

It looks like your worker is trying to connect to 127.0.0.1:7233, is your temporal cluster running here?

If not you can set the target endpoint when you create the WorkflowServiceStubs,

    WorkflowServiceStubs service =
        WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder()
                .setTarget("...")  <--- here
                .build());

iftekhar09 · December 1, 2022, 11:57am

@antonio.perez - Thanks! i tried the same and it started to work

spikhalskiy · December 1, 2022, 4:09pm

Shading is not the simplest or best way to package java applications, because of how invasive it is. Shade only if there is no other way.

If you are using application servers to run your workers, prefer building and using EAR containers that package and distribure dependencies inside.
Instead, you can use maven/gradle to collect all runtime dependencies of your app during the build into a directory and distribute this dependencies directory with your application jar. You will need to specify it in java classpath during start. Gradle equivalent to Maven's "copy-dependencies"? - Stack Overflow.

Bonus point looking at the stacktrace: This exception is coming from the repackaged grpc/netty internals, so excluding all transitive “io.grpc:*” dependencies that Temporal artifacts bring and replacing them with an added direct dependency on io.grpc:grpc-netty-shaded may help. This will also help to just reduce the number of dependencies.

Shean · December 2, 2022, 11:24pm

Thanks for the reply @spikhalskiy ! i’m not so clear about your 2nd advice. You mean I could package all dependencies in another jar (let’s call it dep.jar) and run my app with classpath of dep.jar right? What is the difference between this method and package all dependency into my app jar?

spikhalskiy · December 2, 2022, 11:48pm

No, jar can’t contain dependency jars inside, it’s one of the pros of EAR containers.

What I mean is distribute the directory with jars of dependencies together with your application jar and specify them in the java classpath during start. Typically it’s done by packing your app into a .zip that has an application jar and a lib directory in the root.
You have two options to define the classpath:

Put MANIFEST.MF file into your application jar specifying that there is a directory with dependencies next to the main application jar. This will allow you to launch your app using only the application jar itself, like you do it now. Maven-jar and maven-dependency-plugin plugins will take care of both collecting all dependencies into the directory and generation of the MANIFEST.MF for you if you configure them properly.
Directly specify the classpath to java command that will include an application jar and the directory with jars of dependencies. See “Option 1” in How to run a jar file from command line.

Shean · December 4, 2022, 7:10am

Got it! Thanks a lot for your reply! I now get deeper understanding of jar and its running method. I think taking “springboot” package is convenient and efficient for me now.