introducing witchery: tools for building distroless images with alpine

As I noted in my last blog, I have been working on a set of tools which enable the building of so-called “distroless” images based on Alpine.  These tools have now evolved to a point where they are usable for testing in lab environments, thus I am happy to announce the witchery project.

For the uninitiated, a “distroless” image is one which contains only the application and its dependencies.  This has some desirable qualities: since the image is only the application and its immediate dependencies, there is less attack surface to worry about.  For example, a simple hello-world application built with witchery clocks in at 619kB, while that same hello-world application deployed on alpine:3.14 clocks in at 5.6MB.  There are also drawbacks: a distroless image typically does not include a package manager, so there is generally no ability to add new packages to a distroless image.

As for why it’s called witchery: we are using Alpine’s package manager in new ways to perform truly deep magic.  The basic idea behind witchery is that you use it to stuff your application into an .apk file, and then use apk to install only that .apk and its dependencies into a rootfs: no alpine-base, no apk-tools, no busybox (though witchery allows you to install those things if you want them).

Deploying an an example application with witchery

For those who want to see the source code without commentary, you can find the Dockerfile for this example on the witchery GitHub repo.  For everyone else, I am going to try to break down what each part is doing, so that you can hopefully understand how it all fits together. We will be looking at the Dockerfile in the hello-world example.

The first thing the reader will likely notice is that Docker images built with witchery are done in three stages.  First, you build the application itself, then you use witchery to build what will become the final image, and finally, you copy that image over to a blank filesystem.

FROM alpine:3.14 AS build WORKDIR /root COPY . . RUN apk add –no-cache build-base && gcc -o hello-world hello-world.c

The first stage to build the application is hopefully self explanatory, and is aptly named build.  We fetch the alpine:3.14 image from Dockerhub, then install a compiler (build-base) and finally use gcc to build the application.

The second stage has a few steps to it, that I will split up so that its easier to follow along.

FROM kaniini/witchery:latest AS witchery

First, we fetch the kaniini/witchery:latest image, and name it witchery.  This image contains alpine-sdk, which is needed to make packages, and the witchery tools which drive the alpine-sdk tools, such as abuild.

RUN adduser -D builder && addgroup builder abuild USER builder WORKDIR /home/builder

Anybody who is familiar with abuild will tell you that it cannot be used as root.  Accordingly, we create a user for running abuild, and add it to the abuild group.  We then tell Docker that we want to run commands as this new user, and do so from its home directory.

COPY –from=build /root/hello-world . RUN mkdir -p payloadfs/app && mv hello-world payloadfs/app/hello-world RUN abuild-keygen -na && fakeroot witchery-buildapk -n payload payloadfs/ payloadout/

The next step is to package our application.  The first step in doing so involves copying the application from our build stage.  We ultimately want the application to wind up in /app/hello-world, so we make a directory for the package filesystem, then move the application into place.  Finally, we generate a signing key for the package, and then generate a signed .apk for the application named payload.

At this point, we have a signed .apk package containing our application, but how do we actually build the image?  Well, just as we drove abuild with witchery-buildapk to build the .apk package and sign it, we will have apk build the image for us.  But first, we need to switch back to being root:

USER root WORKDIR /root

Now that we are root again, we can generate the image.  But first, we need to add the signing key we generated in the earlier step to apk’s trusted keys.  To do that, we simply copy it from the builder user’s home directory.

RUN cp /home/builder/.abuild/*.pub /etc/apk/keys

And finally, we build the image.  Witchery contains a helper tool, witchery-compose that makes doing this with apk really easy.

RUN witchery-compose -p ~builder/payloadout/payload*.apk -k /etc/apk/keys -X http://dl-cdn.alpinelinux.org/alpine/v3.14/main /root/outimg/

In this case, we want witchery-compose to grab the application package from ~builder/payloadout/payload*.apk.  We use a wildcard there because we don’t know the full filename of the generated package.  There are options that can be passed to witchery-buildapk to allow you to control all parts of the .apk package’s filename, so you don’t necessarily have to do this.  We also want witchery-compose to use the system’s trusted keys for validating signatures, and we want to pull dependencies from an Alpine mirror.

Once witchery-compose finishes, you will have a full image in /root/outimg.  The final step is to copy that to a new blank image.

FROM scratch CMD ["/app/hello-world"] COPY –from=witchery /root/outimg/ .

And that’s all there is to it!

Things left to do

There are still a lot of things left to do.  For example, we might want to implement layers that users can build from when deploying their apps, like one containing s6 for example.  We also don’t have a great answer for applications written in things like Python yet, so far this only works well for programs that are compiled in the traditional sense.

But its a starting point none the less.  I’ll be writing more about witchery over the coming months as the tools evolve into something even more powerful.  This is only the beginning.