Issues running docker-compose for mysql in local environment

Community Support
1

Hello,

We are running docker-compose up with the mysql config ( https://github.com/temporalio/temporal/blob/master/config/development_mysql.yaml ) for development tests in our local machine. Although it is running with success, the statsd volume is growing very fast in disk usage ( 150Mb per minute ) which is causing some issues in development tests. Tried to remove statsd from docker-compose but seems that the service receives an error :

{"level":"fatal","ts":"2020-07-09T13:21:25.620Z","msg":"error creating statsd client","error":"lookup statsd on 127.0.0.11:53: no such host","logging-call-at":"metrics.go:114","stacktrace":"github.com/temporalio/temporal/common/log/loggerimpl.(*loggerImpl).Fatal\n\t/temporal/common/log/loggerimpl/logger.go:144\ngithub.com/temporalio/temporal/common/service/config.(*Metrics).newStatsdScope\n\t/temporal/common/service/config/metrics.go:114\ngithub.com/temporalio/temporal/common/service/config.(*Metrics).NewScope\n\t/temporal/common/service/config/metrics.go:81\ngithub.com/temporalio/temporal/cmd/server/temporal.(*server).startService\n\t/temporal/cmd/server/temporal/server.go:135\ngithub.com/temporalio/temporal/cmd/server/temporal.(*server).Start\n\t/temporal/cmd/server/temporal/server.go:85\ngithub.com/temporalio/temporal/cmd/server/temporal.startHandler\n\t/temporal/cmd/server/temporal/temporal.go:91\ngithub.com/temporalio/temporal/cmd/server/temporal.BuildCLI.func1\n\t/temporal/cmd/server/temporal/temporal.go:211\ngithub.com/urfave/cli.HandleAction\n\t/go/pkg/mod/github.com/urfave/cli@v1.22.4/app.go:528\ngithub.com/urfave/cli.Command.Run\n\t/go/pkg/mod/github.com/urfave/cli@v1.22.4/command.go:174\ngithub.com/urfave/cli.(*App).Run\n\t/go/pkg/mod/github.com/urfave/cli@v1.22.4/app.go:279\nmain.main\n\t/temporal/cmd/server/main.go:38\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}
temporal_temporal_1 exited with code 1

Any way to overcome the fast volume disk space growth?

Also noticed websockets errors in the Web UI using the docker-compose above.

image
image1526×90 16.2 KB

I believe the Stack-Trace Tab and Query tab is not loading because of the websockets?

Thanks for all the help,

1 Like
2

yes i too see issues, but mostly related to disc size i have about 50gb allocated to my docker, and after running temporal for 20 to 30 days, i see disc all most full. is this because this is development and logging are verbose, or is it general behaviour of temporal where one need a big disc space. possibly because of huge amount of data to maintain states? Last time this happened, i forgot to verify which docker volume took most space :frowning:

1 Like
4

Hi Pedro,

Thank you for playing with docker-compose and for reporting this.

I tried removing statsd from our docker-compose-mysql and it seemed to work in my test –

Will this do what you want, or are there other problems with this?

Thank you!
Mark.

5

Hello Mark,

Thanks for checking this.

If statsd is removed from docker-compose, when opening the web UI we get this :

image
image2020×666 38.4 KB

Starting a dummy test workflow doesn’t work either.

If statsd is present all work well in Web UI ( except for the non related Websockets issue ) and i can start workflows.

2 Likes
6

This looks like a bug. Running an errand at the moment, will investigate this afternoon. (Feel free to create an issue in the meantime, or I will do this when I am back at my desk.)

1 Like
7

Hi Pedro thank you for finding and reporting this.

We are tracking removing statsd from docker-compose files in
https://github.com/temporalio/temporal/issues/319

What is the urgency of this issue for your use case? (“need a fix, like, today?” or a few days, or with the production release, …)?

Thank you,
Mark.

8

Hello @markmark,

Thanks again for watching this.

We are at the beginning of developing a Proof of Concept for our USE CASE with Temporal, and goal is to end of July have a functional test Solution. Although not a blocking issue ( we are currently stopping and clearing the statsd volume and staring again several times during the day ), would be great to have this sorted in a few days if possible.

Thanks again,
Pedro Almeida

2 Likes