Very often we find ourselves changing parameters of scripts or googling what was the right combination of bash commands. What’s the best way to remember those? Let’s take for instance running a machine learning project in a docker container. We all love reproducibility, and we should all do reproducible science. Because you know, otherwise it´s not science.

Docker abstraction

To build a consistent runtime environment we can use tools like python virtual environments, anaconda, or Docker. It is relevant for all cases, but for simplicity, we’ll focus on Docker. Building a container is rather straightforward if we are in the root directory of our project.

docker build . -t my-ml-project

We can also add -f NonStandardNamedDockerfile in case we have several dockerfiles in a multi-step build process. And if we want to run a python evaluation file from inside, my memory pointed to something like:

docker run -it my-ml-project python module/evaluate.py

Though I just forgot to add my visualization which is supported by matplotlib Agg backend on port 8000, so we also need -p 8000:8000. Oh, and the data, of course. It needs to be mounted on a shared volume -v `pwd`/data:/data. That’s running rather slow, because we forgot to add the nvidia runtime --gpus all to use the GPUs.

So in total, the command may look somewhat similar to:

docker run -it --gpus all -v `pwd`/data:/data -p 8000:8000 \
		my-ml-project python module/evaluate.py

It’s not necessarily hard to build up the command and then use ‘Ctrl+R’ to find it in bash history, but what happens in 2 months when you rediscover your old forgotten work, as your paper needs more tuning because of Reviewer#2?

We can always store those commands in bash files, and run those as proxies for the commands. However, quickly we’ll reach a point where we have half a dozen bash scripts in the core of your folder. It feels there should be a better way. Enter Makefiles.

Makefiles

For those of us coming from a C/C++ background, it wouldn’t come to us as a surprise that Makefiles are a very capable tool of abstracting processes of building things. But how are they relevant to our task of “obliviate”-ing ¹ our docker and parameterized commands from memory? As a recap, their basic structure is as follows:

target: dependencies
	command

In our case, we can run dependencies as a sequence of targets (e.g. format/lint code, then run), but it may obfuscate the initial examples, so we’ll forget about this parameter for now.

We can start building our first targets in a plain Makefile file. Let’s start with building:

build:
	docker build . -t my-ml-project

then in our command line, we can simply run make build, which would initiate the build command. Adding evaluate results in

build:
	docker build . -t my-ml-project

evaluate:
	docker run -it --gpus all -v `pwd`/data:/data -p 8000:8000 \
	my-ml-project python module/evaluate.py

You start to see the premises of this. We can quickly build additional commands for running the model (our CMD entry in the Dockerfile), entering to a bash terminal, etc. This would generate a lot of replication of our parameters. So, let’s abstract them into a variable - DOCKER_PARAMS=--gpus all -v `pwd`/data:/data -p 8000:8000 my-ml-project.

This way we can build our set of commands for replicating what we once were using, as follows:

DOCKER_PARAMS=--gpus all -v `pwd`/data:/data -p 8000:8000 my-ml-project

build:
	docker build . -t my-ml-project

run:
	docker run -t ${DOCKER_PARAMS}

# Train our model on the data, saves a model X and outputs stats.
train:
	docker run -t ${DOCKER_PARAMS} python module/train.py
	cat data/stats

# Shows evaluation script output in a web figure
evaluate:
	docker run -it ${DOCKER_PARAMS} python module/evaluate.py

# Evaluate baseline 1
evaluate-baseline1:
	docker run -it ${DOCKER_PARAMS} python module/evaluate_baseline1.py

test:
	docker run -it ${DOCKER_PARAMS} python module/tests/test.py

It is much easier to remember what make train would do. We can even add descriptions! This creates a single file that stores the standard commands we can use to interact with our code.

What’s next?

This little Makefile can also store our commands for lint-ing, formatting and checking, to even serving and deploying our code. To not miss those parts, subscribe to my newsletter!

Bonus round

In some instances, we are sharing different infrastructure between our home setup, lab machine or HPC farm. Sadly, working on multiple projects, on multiple locations can lead to some differences in our host software stack. Nobody wants to update base software mid-deployment…

A common issue may be that different version of the host software may have different flags - e.g. nvidia docker runntime can be specified as --runtime=nvidia for docker below $19$. Makefile does not explicitly support numerical comparisons. So a helpful workaround, in the preface of your Makefile, is as follows:

# Set nvidia runtime based on docker version
DOCKER_VERSION:=$(shell docker --version | cut -f1 -d. | tr -dc '0-9')
DOCKER_GR_19:=$(shell echo $(DOCKER_VERSION)\>=19 | bc )
ifeq ($(DOCKER_GR_19),1)
	NVIDIA_RUNTIME = --gpus all
else
	NVIDIA_RUNTIME = --runtime=nvidia
endif

test:
	echo ${NVIDIA_RUNTIME}

The first line extracts the docker major version from the shell, the second compares it with our target of 19. So that variable allows us to use the if-else statement to define the needed nvidia runtime command. You can test the output with make test.

Docker compose

I want to make sure that people know, that in situations where the above will be used predominantly to manage docker containers, it is a much better idea to rely on docker-compose. In those cases, by writing up a docker-compose.yml the start/stop of multiple containers, volumes and networks can be easily automated by docker-compose up/down. That is a much cleaner and reproducible solution.

The extra 20%

There has been a lot of ideas and suggestions around ways to improve the above. And in the spirit of 80/20, here are a few memorable extract on how to improve on the above, increasing the complexity, but making sure you are not hit by edge cases.

Here are a few comments in no particular order:

• the dependencies/targets that we so quickly overruled above have the quirk (inherited by the fact that Makefiles are made for compiling projects), in which if a file or a folder exists with the same name, nothing is executed. pathslog recommends fixing this by using .PHONY targets by prefacing our Makefile with:

.PHONY: $(MAKECMDGOALS)

• The is a much more in depth write-up here.

• Alternative tools may include:

  • justfile - my personal favorite
  • pydoit
  • Fire
  • run
  • bash

• Speaking of bash, radarsat1 shows an alternative implementation using bash. It does however lack autocomplete (your favorite tab-tab) and parallelism.

#!/bin/bash

cmd1() {
    echo running cmd1..
}

cmd2() {
    echo running cmd2..
}

for f in `declare -F`; do
    if [ "$1" = $f ]; then $1; exit; fi
done
echo Unknown command $1
exit 1

At the end of the day, the complexity of the project will dictate how much or little of these 20% will need to be examined.

For more informative recap and suggestions do subscribe to the newsletter!