Gocql: no hosts available in the pool

All of the sudden our cadence server stopped connecting to the Cassandra db. We checked the cadence logs and we got the following message:

gocql: no hosts available in the pool

Everything went back to normality once we restarted the cadence server. The cassandra DB seemed to have been working normally during this event, Right now we are kind of clueless to know what occurred and how we can prevent it in the future. We have the suspicion that probably had to do with some connectivity issue, but nothing sure.

Does this sound familiar to some of you? Is there something we can do to keep this from happening in the future?

This has something to do with gocql & Cassandra

NOTE:

  1. cadence 0.15.0 is using an really old version of gocql:
    https://github.com/uber/cadence/blob/v0.15.0/go.mod#L25

  2. later, 0.15.1 is using a newer version, maybe you want to take a try
    https://github.com/uber/cadence/blob/v0.15.1/go.mod#L24