Better monitoring of what workflows are starting

Community Support
1

Hey Team,

We haven’t made the switch over to temporal yet (still running Cadence 0.13.0), however I feel this question applies to both projects.

We are seeing some sawtooth CPU consumption within our shared cadence instance each night.

image
image1619×343 62.1 KB

What are ways to properly instrument the cadence golang client to determine which workflows are starting at this time? Reviewing the cadence golang client today, I believe we are missing the specification of MetricsScope so that we can harvest this information from the cadence metrics endpoints.

// NewWorker will start the cadence workers for the current application & taskList
func (c *Cadence) NewWorker(ctx context.Context, appName string, taskList string, options ...WorkerOption) error {
	workerOptions := worker.Options{
		MetricsScope:                   tally.NewTestScope(appName, map[string]string{}),
		BackgroundActivityContext:      ctx,
		NonDeterministicWorkflowPolicy: worker.NonDeterministicWorkflowPolicyFailWorkflow,
	}
	for _, o := range options {
		o(&workerOptions)
	}

	// will create a logger that is powered by google cloud logging. If unable to create the logger
	// the default logger provided by cadence will be used
	logger, err := newLogger(ctx, appName)
	if logger != nil && err == nil {
		workerOptions.Logger = logger
	}

	// taskList identifies set of client workflows, activities, and workers.
	// It could be your group or client or application name.
	w := worker.New(
		c.service,
		appName,
		taskList,
		workerOptions,
	)
	err = w.Start()
	if err != nil {
		return err
	}
	c.workers = append(c.workers, w)
	return nil
}

Or are we just not harvesting the correct metrics from cadence?

2

For reference, the DB is a google cloud mysql instance:

image
image774×170 3.43 KB

3

Answering my own comment here, make sure to attach a metrics reporter to get this insight:

statter, err := statsd.NewClientWithConfig(&statsd.ClientConfig{
	Address:       "your address",
	Prefix:        "your prefix",
	UseBuffered:   true,
	FlushInterval: 100 * time.Millisecond,
	FlushBytes:    1440,
})
if err != nil {
	return tally.NoopScope, ioutil.NopCloser(bytes.NewBuffer([]byte{})), err
}

reporter := tallystatsd.NewReporter(statter, tallystatsd.Options{
	SampleRate: 1.0,
})

scope, closer := tally.NewRootScope(tally.ScopeOptions{
	Prefix: appName,
	Tags: map[string]string{},
	Reporter:        reporter,
	SanitizeOptions: &sanitizeOptions,
}, time.Second)
2 Likes