How to add custom tags in existing Temporal Metric using java sdk

techFool · August 24, 2020, 8:30am

Hello Experts,

How can i add custom tags in the temporal existing Metrcis. For example everytime when my workflow execute i want to pass my tags like regionId,ProductId as per the parameter passed by caller method. I have followed the approach of How to enable Metrics Support using JAVA SDK to invoke the metrics and following is the code base we are deploying in the HelloActivity.java sample .

MetrcisName: workflow_success
in an Existing Temporal Metric: workflow_success, i want to add these 2 parameters for which i have done the following code in the HelloActivity.java

StatsReporter reporter = new MicrometerClientStatsReporter((new SimpleMeterRegistry()));

	Map<String, String> mp = new HashMap<>();
	mp.put("prodCode", "ODE");
	mp.put("regionId", "US_WEST_2");

	Scope scope = new RootScopeBuilder().reporter(reporter).tags(mp)
			.reportEvery(com.uber.m3.util.Duration.ofSeconds(10));

	Counter reqCounter = scope.counter("workflow_success");
	reqCounter.inc(1);

	WorkflowServiceStubsOptions options = WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build();
	WorkflowServiceStubs stubs = WorkflowServiceStubs.newInstance(options);
	WorkflowClient client = WorkflowClient.newInstance(stubs);

Please find the attached file

madhu · August 24, 2020, 10:00am

you already seem to have your custom tags

Map<String, String> mp = new HashMap<>();
	mp.put("prodCode", "ODE");
	mp.put("regionId", "US_WEST_2");

are you not seeing those tags in your metric server (say prometheus or something else) ?

techFool · August 24, 2020, 10:15am

No @madhu we are not able to get this on the metrics

maxim · August 24, 2020, 3:26pm

Have you tried assigning tags to the Scope and not the reporter?

Scope scope = new RootScopeBuilder().reporter(reporter)
			.reportEvery(com.uber.m3.util.Duration.ofSeconds(10)).tagged(mp);

Atul_Parihar · August 24, 2020, 5:26pm

Yes we used following code, is there something wrong here?
Map<String, String> mp = new HashMap<>();
mp.put(“prodCode”, “ODE”);
mp.put(“regionId”, “US_WEST_2”);

Scope scope = new RootScopeBuilder().reporter(reporter).tags(mp)
.reportEvery(com.uber.m3.util.Duration.ofSeconds(10));

Now my workflow is success, so I am looking new tags in temporal “workflow_success” metrics . But in Prometheus I cannot see my custom tags.

maxim · August 24, 2020, 5:28pm

I don’t know what is wrong with your code. Have you tried the alternative I proposed?

Atul_Parihar · August 24, 2020, 5:29pm

you mean to say inplace of tags() use tagged() correct.

maxim · August 24, 2020, 5:30pm

Yes, as it applies to the Scope instead of ScopeBuilder.

techFool · August 25, 2020, 1:52am

Hi maxim,

As suggested by you we used tagged to scope but still we cannot see it on the prometheus or metrics.

public static void main(String[] args) {
		StatsReporter reporter = new MicrometerClientStatsReporter((new SimpleMeterRegistry()));
		Map<String, String> mp = new HashMap<>();
		mp.put("prodCode", "ODE");
		mp.put("regionId", "US_WEST_2");

	
	
	Scope scope = new RootScopeBuilder().reporter(reporter).reportEvery(com.uber.m3.util.Duration.ofSeconds(10)).tagged(mp);
	reporter.reportCounter("workflow_success", mp, 1);
	Counter reqCounter = scope.counter("workflow_success");
	reporter.reportCounter("workflow_success_1", mp, 1);
	reqCounter.inc(1);

	WorkflowServiceStubsOptions options = WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build();
	WorkflowServiceStubs stubs = WorkflowServiceStubs.newInstance(options);
	WorkflowClient client = WorkflowClient.newInstance(stubs);
	WorkerFactory factory = WorkerFactory.newInstance(client);
	Worker worker = factory.newWorker(TASK_QUEUE);
	worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class);
	worker.registerActivitiesImplementations(new GreetingActivitiesImpl());
	factory.start();
	GreetingWorkflow workflow = client.newWorkflowStub(GreetingWorkflow.class,WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).build());
	String greeting = workflow.getGreeting("World");
	System.out.println(greeting);
	System.exit(0);
}

maxim · August 25, 2020, 2:40am

Do you see the tags added by the SDK? For example namespace and workflow type?

techFool · August 25, 2020, 3:22am

Yes i can see the default namespace appearing in the Workflow Success by Namespace Metric. I have registered this namespace

techFool · August 25, 2020, 4:39am

Hello Maxim,

In which Metric can we see the custom tags passed in the code which i have attached. I am running HelloActivity and my namespace is default.

We have checked the following Metrics on Prometheus -
1> workflow_success
2> temporal_DescribeNamespace_temporal_latency_count
3>temporal_PollActivityTaskQueue_temporal_latency_count

maxim · August 25, 2020, 4:49am

It looks like your infra concatenates tags into the metric name. The actual metric name is temporal_latency_count with tag operation name equal to POllActivityTaskQueue. Here is the relevant unit test.

maxim · August 25, 2020, 3:29pm

Have you tried using PrometheusMeterRegistry?

manu · August 26, 2020, 5:20am

Hey folks, we found a few issues on our side with the latest Java SDK and proper integration with Prometheus. We are investigating this and will get back to you. In the interim, it is probably not worth your time troubleshooting this issue on your end until we get back to you with an update.

techFool · August 26, 2020, 9:52am

Thanks for the update.

We are trying to pass the custom tags so that we can view them on temporal Metrics. Any idea on which Metric panel will it be appearing and how. Plus can we get any documentation / explanation on the metrics which are getting generated. We have already added the code in this thread but unfortunately we dont see any change in the Metrics.

TIA

techFool · August 26, 2020, 4:17pm

Hi @maxim and @manu ,

We have used both SimpleMeterRegistry and PrometheusMeterRegistry. Please find the below analysis.

when we firstly used the following code using SimpleMeterRegistry in HelloActivity example, the code was success but we were not able to find out the tags in prometheus metrics

StatsReporter reporter = new MicrometerClientStatsReporter((new SimpleMeterRegistry()));
Map<String, String> mp = new HashMap<>();
mp.put("prodCode", "ODE");
mp.put("regionId", "US_WEST_2");

Then when we change to PrometheusMeterRegistry with following code

      StatsReporter reporter = new MicrometerClientStatsReporter(new 
      PrometheusMeterRegistry(PrometheusConfig.DEFAULT));
	Map<String, String> mp = new HashMap<>();
    mp.put("prodCode", "ODE");
    mp.put("regionId", "US_WEST_2");

Exception: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'temporal_request_latency_seconds' containing tag keys [Namespace, Operation, TaskQueue, WorkflowType, prodCode, regionId]. The meter you are attempting to register has keys [Operation, prodCode, regionId].

we are getting exception which means that userDefined extra tags which we passed in above has been considered but we cannot find the same metrics “temporal_request_latency_seconds” in prometheus server . So looks like something is missing .

We are getting pattern name like “temporal_DescribeNamespace_temporal_request_latency_count” in prometheus which means somehow “temporal_DescribeNamespace_” is getting appended

We are getting the following exception when we are trying to run the sample with PrometheusMeterRegistry

Aug 26, 2020 9:11:30 PM io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed@45837160
java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named ‘temporal_request_latency_seconds’ containing tag keys [Namespace, Operation, TaskQueue, WorkflowType, prodCode, regionId]. The meter you are attempting to register has keys [Operation, prodCode, regionId].
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$16(PrometheusMeterRegistry.java:419)
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1877)
at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:403)
at io.micrometer.prometheus.PrometheusMeterRegistry.newTimer(PrometheusMeterRegistry.java:197)
at io.micrometer.core.instrument.MeterRegistry.lambda$timer$2(MeterRegistry.java:308)
at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:612)
at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:566)
at io.micrometer.core.instrument.MeterRegistry.timer(MeterRegistry.java:306)
at io.micrometer.core.instrument.Timer$Builder.register(Timer.java:539)
at io.micrometer.core.instrument.MeterRegistry.timer(MeterRegistry.java:433)
at io.temporal.common.reporter.MicrometerClientStatsReporter.reportTimer(MicrometerClientStatsReporter.java:69)

maxim · August 26, 2020, 10:10pm

Thanks a lot for looking into this. We did confirm the reported problem on our end and are going to fix it as soon as we can. We’ll update this post. You can also submit a github issue with your findings for easier tracking of the progress.

techFool · August 27, 2020, 4:18am

Following issue has been raised for the same Custom Tags addition in existing Metric using java sdk is not working · Issue #695 · temporalio/temporal · GitHub

Vitaly · September 30, 2020, 6:45pm

Meanwhile, consider using work-around suggested in the https://github.com/temporalio/sdk-java/issues/200

Topic		Replies	Views
Custom tags to be added to out of box sdk metrics Community Support	6	897	January 19, 2021
Add custom tags to common metrics emitted by temporal Community Support go-sdk , metrics	0	455	December 21, 2023
Update custom tag value of built-in metrics Community Support java-sdk	7	1074	March 30, 2023
Attaching custom tags workflow metrics Community Support prometheus , metrics	8	2708	August 26, 2020
Adding tags to temporal metrics Community Support go-sdk , metrics , interceptor	5	789	September 9, 2024

How to add custom tags in existing Temporal Metric using java sdk

Related topics