Docker, Compose, Go

It has been a month since I set chrsm.org up again. Seems like a good time to write something.

At $DAYJOB, we have a monorepo (monorepos are SO hot right now) consisting of several different services - monoliths, microservices, things that are highly interdependent and some that are decoupled.

Not everything is written in the same language and we don’t expect everyone to have a local environment set up with exact versions of tools that we use, so utilities like Docker help ensure that all developers have the same tools… at least, that is what we strive for.

For a while I’ve worked off in the equivalent of a nuclear bunker on the Go side. I write a lot of Go for personal projects, so I have my local environment set up to accommodate that. Naturally, this leaks into my work: of course I have gofmt, gometalinter, staticcheck, dep, etc installed locally. If you’re reviewing my code, here are the steps to follow:

dep ensure
go test -v ./…
…

Oh, wait - you don’t have Go installed? Well, I guess you’re SoL, but you can trust me!

…

But..

I’ve shipped bad code to production
I’ve missed obvious bugs
I’m bad at writing the typical Go “doc comment” above types and functions

The truth is, I don’t trust myself, so I try to write tests where possible. I auto-format code on save. I appreciate in-depth reviews because they can catch issues that I may have missed.

But without the proper tools available, it becomes much more difficult for the reviewer to actually verify that your work does what it says on the tin.

So how do we fix that?

Goals

The ideal state of the repository is that every developer has the same set of tools available. At any time, they should be able to run a command and have their results in stdout as usual. It would also be quite nice if we could have some type of web-app “hot reload” equivalent for these services.

What we’re looking at implementing:

formatting
linting
testing
building
rebuild-on-change

Making it work

In order to support rebuild-on-change (“hot reload”), we can’t ADD or COPY files in the Dockerfile - the changes won’t be reflected there. Our only choice is to use a VOLUME.

In this case, we’ve made the entire /go path a volume rather than just the company.net subdirectory. Our Go code is stored in a subfolder go in our repository, leaving us with a hierarchy that works well for this:

/
    go/
		bin/
		src/
		pkg/
    otherapp/
    README
    ...

The contents of go/bin and go/pkg are simply gitignored.

I don’t think this is ideal. While I do prefer to work outside of Docker, this means we can’t go get various tools as a part of the Docker build process. The snippet of our Dockerfile below highlights this issue - it doesn’t do anything at all.

FROM golang:alpine

VOLUME /go
WORKDIR $GOPATH/src/company.net

CMD [ "/bin/true" ]

Instead, we rely on various docker-compose commands - only after dependencies have been installed; in order, it’s a bit dumb.

# first install dependencies
docker-compose run go sh -c "go get -v -u github.com/alecthomas/gometalinter"
# then do useful things..
docker-compose run go sh -c "gometalinter ./..."

Outside of Docker, I simply run export GOPATH=/path/to/src/go - so it works well in the sense that it is a real GOPATH locally, but creates baggage for other people. If it was baked in to the base image, all they’d need to do is pull it down or build it themselves, leaving much less confusion when things don’t work out of the box.

The proper way to handle this issue would be to pass only /go/src/company.net to the container as a volume, but since I didn’t end up doing that…

Leaving this issue aside for now, we’ll hook this up to Compose via docker-compose.yml.

version: '2'
services:
  go:
    build:
      context: ./go
      dockerfile: src/company.net/Dockerfile
    image: company/gobase:local
    volumes:
      - ./go:/go

Any services that are based on the Go code simply extend this “go” service within the compose file:

echoserver:
    extends: gobase
    build:
      context: ./go
      dockerfile: src/company.net/cmd/echoserver/Dockerfile
    image: company/echoserver:local
    environment:
      - SOMETHING_NECESSARY=xyz
    links:
      - some-dependency

Making it easy

Ship a shell script that has various commands implemented for other devs to make it simple. Note that I did not write this shell script; we already had something for all current projects, I simply added these helpers to it - credit goes to Nathan Wong.

#!/bin/bash

CMD=$1

case $CMD in
	deps)
		echo docker-compose run go sh -c "go get -u -v github.com/golang/dep/cmd/dep ; ..."
		;;
	fmt) 
		echo docker-compose run go sh -c "gofmt ./..."
		;;
	lint)
		echo docker-compose run go sh -c "gometalinter ./..."
		;;
	test)
		echo docker-compose run go sh -c "go test -v ./..."
		;;
	*)
		echo "$CMD not found"
		exit 1
		;;
esac

These helpers make it dead-simple for anyone to test/lint/etc our Go code: just run script.sh lint and presto, output! Again, annoying side effect that script.sh deps is required first so that gometalinter is present :-)

Rebuild on change

But wait, there’s more (TM). We haven’t yet hooked anything up to rebuild the code on change. There are a number of tools that watch files on the filesystem for changes. There are Go-specific “watchers” that will run tests, lint, and all that fancy stuff but I wanted to keep this simple, especially since installed dependencies pollute the local repository at this time. I decided just to use inotify.

In order to use it, it needs to be installed in the base image, so our Dockerfile is slightly different now:

FROM golang:alpine
RUN apk add --no-cache inotify-tools
VOLUME /go
WORKDIR $GOPATH/src/company.net
CMD [ "/bin/true" ]

I quickly rigged up a script that rebuilds a specific service after changes to the source, kills the old one and runs the new one. Because the entrypoint of the Dockerfile will be this script, killing the old service doesn’t cause it to quit, either.

#!/bin/sh

WATCH="$1"
CDIR="$2"
EXEC="$3"
INCLUDE="(\.go)"

wait_kill(){
	prog=$1
	while pkill -SIGINT $prog; do
		echo "."
		sleep 1
	done
}

if [ -z "$WATCH" ] || [ -z "$CDIR" ]; then
	echo "Invalid command; specify watchdir, cdir and exec."
	echo "Example: $0 /go/src/company.net /go/src/company.net/cmd/echoserver echoserver"
	exit 1;
fi

cd $CDIR
go build -v
./$EXEC &

inotifywait -m -r \
	--include $INCLUDE \
	-e close_write $WATCH \
	--format %T \
	--timefmt %M:%S \
	| \
	while read time
	do
		wait_kill $EXEC
		go build -v
		./$EXEC &
	done

In order to use this script, it’s added to the Compose file under the base “go” section:

version: '2'
services:
  go:
    build:
      context: ./go
      dockerfile: src/company.net/Dockerfile
    image: company/go:local
    volumes:
      - ./go:/go
      - ./path/to/script.sh:/usr/bin/rebuild.sh

Each service needs its own Dockerfile at this point. echoserver as an example would look something like this:

FROM company/go:local

ENTRYPOINT ["sh", "-c", "rebuild.sh /go/src/company.net /go/src/company.net/cmd/echoserver echoserver"]

Done!

That’s it. Everyone on the team now has the same tools available to them whether they care to install Go or not.

The biggest improvements to be made here: don’t pass the entire /go tree to the container and instead only pass /go/src/company.net. You’ll be able to use go get in the Dockerfile to add dependencies, thereby eliminating the need to have a “install deps” step before using linters or any other tools you integrate with.