Bits related to Alpine Security Initiatives in September

The past month has been quite busy as we prepare to wrap up major security-related initiatives for the Alpine 3.15 release. Some progress has been made on long-term initiatives as well.

OpenSSL 3 migration

As I noted in my last status update, we began the process to migrate the distribution to using OpenSSL 3. As a result of this, we have found and mitigated a few interesting bugs, for example, ca-certificates-bundle was being pulled into the base system as a side effect rather than intentionally, despite apk-tools explicitly needing it for validating the TLS certificates used by Fastly for our CDN.

Migrating to OpenSSL 3 has not been without its share of difficulties however, as I noted in a blog post earlier in the month discussing some of these difficulties. I hope to be able to fully finish the OpenSSL 3 migration during the Alpine 3.16 development cycle as the last remaining packages such as mariadb and php make releases which support the new API. One other major issue needing to be addressed is updating wpa_supplicant and hostap to use the new OpenSSL APIs, but WPA requires the use of RC4 which has been moved to the legacy provider, so this will require programmatic loading of the legacy OpenSSL provider. Accordingly, we moved it back to OpenSSL 1.1 for now until upstream releases an update to address these problems.

OpenSSL 3 also introduces some unintended regressions. Specifically, a bug was reported against apk-tools where using apk --allow-untrusted would result in a crash. After some debugging work, I was able to reduce the issue to a simple reproducer: the EVP_md_null digest family was never updated to be compatible with the new OpenSSL 3 provider APIs, and so attempting to use it results in a crash, as the prerequisite function pointers never get set up on the EVP_MD_CTX context. This means that apk-tools is still using OpenSSL 1.1 for now, despite otherwise working with OpenSSL 3.

Coordinating the OpenSSL 3 migration consumed a lot of my time in September, for example, I spent a few days investigating OpenSSL 3 regressions on CPUs which claim to be Pentium-compatible but actually lack support for the lock cmpxchg8b instruction, and CPUs which claim to be Pentium 3-compatible, but lack the CMOV instructions. This investigation was quite frustrating, but also interesting, as the Vortex86DX3+ looks like a nice x86 CPU that is not vulnerable to Meltdown or Spectre due to not supporting speculation.

Rust in Alpine main for 3.16

As I noted in my previous blog about the OpenSSL 3 transition, we had to migrate Ansible from main to community due to Rust presently being in the community repository. This is largely because main has a general policy that is similar to other distributions: once we cut a new release series, we generally don’t do upgrades of packages in main, instead preferring to backport security and reliability fixes, unless we are certain they won’t cause regressions. Normally, this is a good thing, as new versions of software frequently bring ABI/API changes that are not backwards compatible, and upstream developers sometimes forget to update their SONAME versions to reflect those changes.

Distributions traditionally have to provide a maintenance lifecycle which is several years long without breaking their users’ environments, and so tend to be conservative in what post-release updates are made. Alpine takes a more “hybrid” approach and thus has more flexibility, but we still prefer to err on the side of caution. In the case of Rust, this meant that we wanted a working relationship that allowed us to have a high level of confidence in the Rust toolchains we were delivering to our users.

After almost a year of ongoing discussion, and a Rust MCP, we have come up with a plan in concert with Rust upstream which allows us to deliver production-quality Rust toolchains for Alpine users, and keep them continuously updated in Alpine releases. I expect the Alpine TSC to approve this for Alpine 3.16 and later. And yes, for other languages we are willing to offer a similar proposal, if there is a proven track record of maintaining backward compatibility and portability. Please feel free to reach out.

The more interesting development is that this allows for using components written in Rust for the base system in the future. While we do not plan to start evaluating these options until the 3.18 release cycle at the earliest, this does open a path to enabling rustls and hyper to replace OpenSSL and libfetch in apk-tools at some point, which could potentially be an interesting development. It also opens a path for components of apk-tools to eventually be written in Rust as well.

Distroless images

Another project I started recently is Witchery, a build system for generating distroless images using Alpine components. This allows a user to easily build a distroless image for their application by leveraging apk-tools to do the work. Distroless images are interesting from a security perspective as they contain less moving parts, in most cases, a distroless image built with Witchery will only contain musl, some data files in /etc and your application. By avoiding other components like busybox and apk-tools, the attack surface of an image is reduced, as there is nothing available outside your application payload for an attacker to use.

There is still a lot of work to do on Witchery, as it is presently in the proof of concept stage, and I plan on doing a second blog post about it soon. I believe that there is intrinsic value to deploying applications built against musl from a security point of view over glibc, as there is much more hardening in musl.

Acknowledgement

My activities relating to Alpine security work are presently sponsored by Google and the Linux Foundation. Without their support, I would not be able to work on security full time in Alpine, so thanks!