Building a security response team in Alpine

Starting this past month, thanks to the generous support of Google and the Linux Foundation, instead of working on the usual Alpine-related consulting work that I do, I’ve had the privilege of working on various initiatives in Alpine relating to security that we’ve needed to tackle for a long time.  Some things are purely technical, others involve formulating policy, planning and recruiting volunteers to help with the security effort.

For example, my work to replace poorly maintained software with better replacements is a purely technical security-related effort, while building a security response team has social aspects as well as designing and building tools for the team to use.  Our security issue tracker has gone live and is presently being tested by the community, and with that work we’re already off to a great start at an organized security response.

If you didn’t know what Alpine Linux is already, it is a popular Linux system with over a billion installations on Docker alone.  By building on efficient building blocks, such as the musl C library and busybox, Alpine maintains a slim installation image size while also providing the conveniences of a general-purpose Linux distribution.  As a result, Alpine has been deployed as the base of many Docker images, has been ported to hundreds of devices as the basis of postmarketOS, has been used to build hundreds of appliances with LinuxKit and has been deployed everywhere from 5G networks to solar farms and oil rigs thanks to the work done by Zededa with Project EVE.  With all of this growth in the few years, it’s important to rethink a lot of things in the distribution including our approach to security.

Why having a security team matters

You might be wondering why distributions need a dedicated security team.  The answer is simple: having a dedicated security team helps to ensure that issues are fixed more quickly, and that the dissemination of security fix information is consistent.  The lack of consistent security fix information has been one of the major concerns for some potential users when evaluating Alpine for their projects, so improving upon this with a proper security team will hopefully assuage those concerns.

The other main benefit to having a dedicated security team is that the team can be a helpful resource for maintainers who need help figuring out how to fix their packages.  Sometimes figuring out which commits need to be backported in order to fix a vulnerability is tricky, because upstreams do not necessarily disclose what commits fix a problem.  Having more eyeballs that can be pinged to look at an issue is helpful here.

How Alpine has historically done security response

In the past, an anonymous user working under a pseudonym reported vulnerabilities to package maintainers by hand as she learned about them.  While this has worked out fairly well for us in the past, this has understandably not scaled very well as Alpine’s package count has grown.  Although the security tracker is not yet integrated into Gitlab, it will soon provide a familiar experience for anyone who has worked issues reported by the anonymous contributor.

Imagine my surprise when I found out that the account opening issues against packages with security vulnerabilities was not a bot, but instead an actual person doing the work by hand!  Needless to say, in 2021, it is time to come up with a better system, which is why I am working on starting an actual security team, and building tools for the security team to use like the security tracker.

One of the main initiatives I have been working on the past few months has been getting Rust on all of Alpine’s release architectures.  At the moment, we are still missing s390x, mips64 and riscv64 (under consideration as a release architecture beginning with 3.15).  While I was not able to complete this work in the 3.14 release cycle, we have established a great working relationship with the Rust team, and have been working on detangling the musl port in Rust.  Once the musl port is detangled in Rust, it will be trivial to bring up support on new architectures we add in the future (such as loongarch, for example), and bootstrap.sh can already build a Rust cross compiler for the targets we support today.

Another initiative is replacing poorly maintained software with modern replacements.  I have a lot of plans in this area, for example, replacing GNU gettext with gettext-tiny, which will make container images smaller and allow Alpine to use musl’s native gettext routines which are considerably more robust than GNU libintl.  Similarly, I am looking into replacing GNU ncurses with netbsd’s implementation for the same reasons.  I am also planning to refactor apk-tools’ cryptographic code to allow using BearSSL as a backend instead of OpenSSL’s libcrypto.

We also want to harden Alpine’s supply chain from supply-chain attacks.  This work is largely going to be focused on two areas: integrating support into abuild for verifying signatures on downloaded source code and rebooting the reproducible builds effort.  The things we need to do to support reproducible builds however is a blog post in and of itself, so that will be coming soon as we are able to move onto working on that.

Getting involved

If working on these kinds of problems interests you, I’d love to get you on board and working on these initiatives.  Feel free to ping me in IRC, or ping me in Alpine’s gitlab.  There is a literal mountain of security-related work to be done in Alpine, so the possibilities are basically endless.

Also, if you work for a company that is interested in funding work on Alpine, there are many initiatives that could be more quickly implemented with funding, such as Laurent Bercot’s planned OpenRC replacement.