How to add custom tags in existing Temporal Metric using java sdk

Community Support
, ,
1

Hello Experts,

How can i add custom tags in the temporal existing Metrcis. For example everytime when my workflow execute i want to pass my tags like regionId,ProductId as per the parameter passed by caller method. I have followed the approach of How to enable Metrics Support using JAVA SDK to invoke the metrics and following is the code base we are deploying in the HelloActivity.java sample .

MetrcisName: workflow_success
in an Existing Temporal Metric: workflow_success, i want to add these 2 parameters for which i have done the following code in the HelloActivity.java

StatsReporter reporter = new MicrometerClientStatsReporter((new SimpleMeterRegistry()));

	Map<String, String> mp = new HashMap<>();
	mp.put("prodCode", "ODE");
	mp.put("regionId", "US_WEST_2");

	Scope scope = new RootScopeBuilder().reporter(reporter).tags(mp)
			.reportEvery(com.uber.m3.util.Duration.ofSeconds(10));

	Counter reqCounter = scope.counter("workflow_success");
	reqCounter.inc(1);

	WorkflowServiceStubsOptions options = WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build();
	WorkflowServiceStubs stubs = WorkflowServiceStubs.newInstance(options);
	WorkflowClient client = WorkflowClient.newInstance(stubs);

Please find the attached file

HelloActivity
HelloActivity1570×884 63.7 KB

2

you already seem to have your custom tags

Map<String, String> mp = new HashMap<>();
	mp.put("prodCode", "ODE");
	mp.put("regionId", "US_WEST_2");

are you not seeing those tags in your metric server (say prometheus or something else) ?

3

No @madhu we are not able to get this on the metrics

4

Have you tried assigning tags to the Scope and not the reporter?

Scope scope = new RootScopeBuilder().reporter(reporter)
			.reportEvery(com.uber.m3.util.Duration.ofSeconds(10)).tagged(mp);
5

Yes we used following code, is there something wrong here?
Map<String, String> mp = new HashMap<>();
mp.put(“prodCode”, “ODE”);
mp.put(“regionId”, “US_WEST_2”);

Scope scope = new RootScopeBuilder().reporter(reporter).tags(mp)
.reportEvery(com.uber.m3.util.Duration.ofSeconds(10));

Now my workflow is success, so I am looking new tags in temporal “workflow_success” metrics . But in Prometheus I cannot see my custom tags.

6

I don’t know what is wrong with your code. Have you tried the alternative I proposed?

7

you mean to say inplace of tags() use tagged() correct.

8

Yes, as it applies to the Scope instead of ScopeBuilder.

10

Hi maxim,

As suggested by you we used tagged to scope but still we cannot see it on the prometheus or metrics.

public static void main(String[] args) {
		StatsReporter reporter = new MicrometerClientStatsReporter((new SimpleMeterRegistry()));
		Map<String, String> mp = new HashMap<>();
		mp.put("prodCode", "ODE");
		mp.put("regionId", "US_WEST_2");

	
	
	Scope scope = new RootScopeBuilder().reporter(reporter).reportEvery(com.uber.m3.util.Duration.ofSeconds(10)).tagged(mp);
	reporter.reportCounter("workflow_success", mp, 1);
	Counter reqCounter = scope.counter("workflow_success");
	reporter.reportCounter("workflow_success_1", mp, 1);
	reqCounter.inc(1);

	WorkflowServiceStubsOptions options = WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build();
	WorkflowServiceStubs stubs = WorkflowServiceStubs.newInstance(options);
	WorkflowClient client = WorkflowClient.newInstance(stubs);
	WorkerFactory factory = WorkerFactory.newInstance(client);
	Worker worker = factory.newWorker(TASK_QUEUE);
	worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class);
	worker.registerActivitiesImplementations(new GreetingActivitiesImpl());
	factory.start();
	GreetingWorkflow workflow = client.newWorkflowStub(GreetingWorkflow.class,WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).build());
	String greeting = workflow.getGreeting("World");
	System.out.println(greeting);
	System.exit(0);
}
11

Do you see the tags added by the SDK? For example namespace and workflow type?

12

Yes i can see the default namespace appearing in the Workflow Success by Namespace Metric. I have registered this namespace

GrafanaNameSpace
GrafanaNameSpace1898×1001 67.4 KB

13

Hello Maxim,

In which Metric can we see the custom tags passed in the code which i have attached. I am running HelloActivity and my namespace is default.

We have checked the following Metrics on Prometheus -
1> workflow_success
2> temporal_DescribeNamespace_temporal_latency_count
3>temporal_PollActivityTaskQueue_temporal_latency_count

14

It looks like your infra concatenates tags into the metric name. The actual metric name is temporal_latency_count with tag operation name equal to POllActivityTaskQueue. Here is the relevant unit test.

15

Have you tried using PrometheusMeterRegistry?

16

Hey folks, we found a few issues on our side with the latest Java SDK and proper integration with Prometheus. We are investigating this and will get back to you. In the interim, it is probably not worth your time troubleshooting this issue on your end until we get back to you with an update.

17

Thanks for the update.

We are trying to pass the custom tags so that we can view them on temporal Metrics. Any idea on which Metric panel will it be appearing and how. Plus can we get any documentation / explanation on the metrics which are getting generated. We have already added the code in this thread but unfortunately we dont see any change in the Metrics.

TIA

18

Hi @maxim and @manu ,

We have used both SimpleMeterRegistry and PrometheusMeterRegistry. Please find the below analysis.

when we firstly used the following code using SimpleMeterRegistry in HelloActivity example, the code was success but we were not able to find out the tags in prometheus metrics

StatsReporter reporter = new MicrometerClientStatsReporter((new SimpleMeterRegistry()));
Map<String, String> mp = new HashMap<>();
mp.put("prodCode", "ODE");
mp.put("regionId", "US_WEST_2");

Then when we change to PrometheusMeterRegistry with following code

      StatsReporter reporter = new MicrometerClientStatsReporter(new 
      PrometheusMeterRegistry(PrometheusConfig.DEFAULT));
	Map<String, String> mp = new HashMap<>();
    mp.put("prodCode", "ODE");
    mp.put("regionId", "US_WEST_2");

Exception: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'temporal_request_latency_seconds' containing tag keys [Namespace, Operation, TaskQueue, WorkflowType, prodCode, regionId]. The meter you are attempting to register has keys [Operation, prodCode, regionId].

we are getting exception which means that userDefined extra tags which we passed in above has been considered but we cannot find the same metrics “temporal_request_latency_seconds” in prometheus server . So looks like something is missing .

We are getting pattern name like “temporal_DescribeNamespace_temporal_request_latency_count” in prometheus which means somehow “temporal_DescribeNamespace_” is getting appended

We are getting the following exception when we are trying to run the sample with PrometheusMeterRegistry

Aug 26, 2020 9:11:30 PM io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed@45837160
java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named ‘temporal_request_latency_seconds’ containing tag keys [Namespace, Operation, TaskQueue, WorkflowType, prodCode, regionId]. The meter you are attempting to register has keys [Operation, prodCode, regionId].
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$16(PrometheusMeterRegistry.java:419)
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1877)
at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:403)
at io.micrometer.prometheus.PrometheusMeterRegistry.newTimer(PrometheusMeterRegistry.java:197)
at io.micrometer.core.instrument.MeterRegistry.lambda$timer$2(MeterRegistry.java:308)
at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:612)
at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:566)
at io.micrometer.core.instrument.MeterRegistry.timer(MeterRegistry.java:306)
at io.micrometer.core.instrument.Timer$Builder.register(Timer.java:539)
at io.micrometer.core.instrument.MeterRegistry.timer(MeterRegistry.java:433)
at io.temporal.common.reporter.MicrometerClientStatsReporter.reportTimer(MicrometerClientStatsReporter.java:69)

19

Thanks a lot for looking into this. We did confirm the reported problem on our end and are going to fix it as soon as we can. We’ll update this post. You can also submit a github issue with your findings for easier tracking of the progress.

1 Like
20

Following issue has been raised for the same Custom Tags addition in existing Metric using java sdk is not working · Issue #695 · temporalio/temporal · GitHub

1 Like
21

Meanwhile, consider using work-around suggested in the https://github.com/temporalio/sdk-java/issues/200