A few months ago, I joined a software team that’s making use of Kubernetes as a platform to provide elastically scalable software services. It’s Docker + more magic.
One of the tenets espoused by some folks building container deployments is that a container should only have the software on it required to run a single process / microservice. The particular software system I’m deploying is a private Docker registry, but to deliver the service, there are currently 4 Docker containers in play. (I actually think that to solve a problem I’m having, I’ll need to add two more.)
One of the containers wasn’t behaving nicely for me today. The container provides a web user interface and needs to communicate with a database provided by another container to store and retrieve its data. But the logs were saying that the user interface container couldn’t log into the database.
The first two options that came to mind:
– the hostname wasn’t accessible to the ui container
– the port wasn’t open on the database container
Turns out, it was neither of those: it was a badly shared database password. But in trying to prove or disprove those options, I ran into a challenging problem. See, the UI container _only_ had the binary for the UI process on it. Which meant: it didn’t have ssh, telnet, curl, wget, netstat, or any of the various other options I came up with to verify network connectivity between the containers. How to get my answer? Google + Stack Exchange to the rescue!
In recent versions of bash (which thankfully the golang image did provide), bash will let you access TCP and UDP ports directly, ala
cat < /dev/tcp/databasehost/3306
Running the above from the ui container got me a garbled prompt received from the database container, which proved that the hostname resolved AND the port was open, all in one command.
Kudos to Peter Mortenson and his answer on superuser.com for helping me crack the bug. Tuesday’s bug (courtesy of the Monday holiday) will be to see if Beego.Run() doesn’t crash if I manage to map in the app.conf file not otherwise being provided through my Kubernetes replication controller descriptor. Not good to have applications just stop without an error message!!