The long-term consequences of maintainers’ actions

OpenSSL 3 has entered Alpine, and we have been switching software to use it over the past week.  While OpenSSL 1.1 is not going anywhere any time soon, it will eventually leave the distribution, once it no longer has any dependents.  I mostly bring this up because it highlights a few examples of maintainers not thinking about the big picture, let me explain.

First, the good news: in distribution-wide rebuilds, we already know that the overwhelming majority of packages in Alpine build just fine with OpenSSL 3, when individually built against it.  Roughly 85% of main builds just fine with OpenSSL 3, and 89% of community builds with it.  The rebuild effort is off to a good start.

Major upgrades to OpenSSL are not without their fallout, however.  In many cases, we cannot upgrade packages to use OpenSSL 3 because they have dependencies which themselves cannot yet be built with OpenSSL 3.  So, that 15% of main ultimately translates to 30-40% of main once you take into account dependencies like curl, which builds just fine with OpenSSL 3, but has hundreds of dependents, some of which don’t.

A major example of this is mariadb.  It has been known that OpenSSL 3 was on the horizon for over 4 years now, and that the OpenSSL 3 release would remove support for the classical OpenSSL programming approach of touching random internals.  However, they are just now beginning to update their OpenSSL support to use the modern APIs.  Because of this, we wound up having to downgrade dozens of packages which would otherwise have supported OpenSSL 3 just fine, because the maintainers of those packages did their part and followed the OpenSSL deprecation warnings as they showed up in OpenSSL releases.  MariaDB is a highly profitable company, who do business with the overwhelming majority of the Fortune 500 companies.  But yet, when OpenSSL 3 releases started to be cut, they weren’t ready, and despite having years of warning they’re still not, which accordingly limits what packages can get the OpenSSL 3 upgrade as a result.

Another casualty will be Ansible: we have already moved it to community.  You are probably wondering why Ansible, a software package which does not use OpenSSL at all, would be a casualty, so please let me explain.  Ansible uses paramiko for its SSH client, which is a great SSH library for Python, and is a totally solid decision to make.  However, paramiko uses cryptography for its cryptographic functions, again a totally solid decision to make, cryptography is a great library for developers to use.

For distributions, however, the story is different: cryptography moved to using Rust, because they wanted to leverage all of the static analysis capabilities built into the language.  This, too, is a reasonable decision, from a development perspective.  From the ecosystem perspective, however, it is problematic, as the Rust ecosystem is still rapidly evolving, and so we cannot support a single branch of the Rust compiler for an entire 2 year lifecycle, which means it exists in community.  Our solution, historically, has been to hold cryptography at the latest version that did not require Rust to build.  However, that version is not compatible with OpenSSL 3, and so it will eventually need to be upgraded to a new version which is.  And so, since cryptography has to move to community, so does paramiko and Ansible.

The ideology of moving fast and breaking things, while tolerated in the technology industry, does not translate to success in the world at large.  Outside of technology, the world prefers stability: the reason why banks still buy mainframes, and still use z/OS, is because the technology works and can be depended upon.  Similarly, the engine controller in cars, and medical devices like pacemakers and insulin pumps, are running on C.  They don’t run on C because C is a good language with all the latest features, they run on C because the risks and mitigations for issues in C programs are well-understood and documented as part of MISRA C.

Distributions exist to provide a similar set of stability and reliability guarantees.  If we cannot provide a long-term support lifecycle for a piece of technology your software depends on, then we are forced to provide a shorter support lifecycle for your software as well.  For some, that is fine, but I think many will be disappointed to see that they haven’t fully gotten OpenSSL 3, or that Ansible has had to be moved to community.

Efficient service isolation on Alpine with VRFs

Over the weekend, a reader of my blog contacted me basically asking about firewalls.  Firewalls themselves are boring in my opinion, so let’s talk about something Alpine can do that, as far as I know, no other distribution can easily do out of the box yet: service isolation using the base networking stack itself instead of netfilter.

A note on netfilter

Linux comes with a powerful network filtering framework, called netfilter. In most cases, netfilter is performant enough to be used for packet filtering, and the newer nftables project provides a mostly user friendly interface to the system, by comparison to the older iptables and ip6tables commands.

However, when trying to isolate individual services, a firewall is sometimes more complicated to use, especially when it concerns services which connect outbound. Netfilter does provide hooks that allow for matching via cgroup or PID, but these are complicated to use and carry a significant performance penalty.

seccomp and network namespaces

Two other frequently cited options are to use seccomp and network namespaces for this task. Seccomp is a natural solution to consider, but again has significant overhead, since all syscalls must be audited by the attached seccomp handler. Although seccomp handlers are eBPF programs and may be JIT compiled, the performance cost isn’t zero.

Similarly, one may find network namespaces to be a solution here. And indeed, network namespaces are a very powerful tool. But because of the flexibility afforded, network namespaces also require a lot of effort to set up. Importantly though, network namespaces allow for the use of an alternate routing table, one that can be restricted to say, the management LAN.

Introducing VRFs

Any network engineer with experience will surely be aware of VRFs. The VRF name is an acronym which stands for virtual routing and forwarding. On a router, these are interfaces that, when packets are forwarded to them, use an alternative routing table for finding the next destination. In that way, they are similar to Linux’s network namespaces, but are a lot simpler.

Thanks to the work of Cumulus Networks, Linux gained support for VRF interfaces in Linux 4.3. And since Alpine 3.13, we have supported managing VRFs and binding services to them, primarily for the purpose of low-cost service isolation. Let’s look at an example.

Setting up the VRF

On our example server, we will have a management LAN of 10.20.30.0/24. A gateway will exist at 10.20.30.1 as expected. The server itself will have an IP of 10.20.30.40. We will a single VRF, in conjunction with the system’s default route table.

Installing the needed tools

By default, Alpine comes with Busybox’s iproute2 implementation. While good for basic networking use cases, it is recommended to install the real iproute2 for production servers. To use VRFs, you will need to install the real iproute2, using apk add iproute2-minimal, which will cause the corresponding ifupdown-ng modules to be installed as well.

Configuring /etc/network/interfaces

We will assume the server’s ethernet port is the venerable eth0 interface in Alpine. First, we will want to set up the interface itself and it’s default route. If you’ve used the Alpine installer, this part should already be done, but we will include the configuration snippet for those following along.

auto eth0
iface eth0
  address 10.20.30.40/24
  gateway 10.20.30.1

The next step is to configure a VRF. In this case, we want to limit the network to just the management LAN, 10.20.30.0/24. At the moment, ifupdown-ng does not support configuring interface-specific routes out of the box, but it’s coming in the next version. Accordingly, we will use iproute2 directly with a post-up directive.

auto vrf-management
iface vrf-management
  requires eth0
  vrf-table 1
  pre-up ip -4 rule add pref 32765 table local
  pre-up ip -6 rule add pref 32765 table local
  pre-up ip -4 rule del pref 0
  pre-up ip -6 rule del pref 0
  pre-up ip -4 rule add pref 2000 l3mdev unreachable
  post-up ip route add 10.20.30.0/24 dev eth0 table 1

This does four things: first it creates the management VRF, vrf-management using the second kernel route table (each network namespace may have up to 4,096 routing tables).  It also asserts that the eth0 interface must be present and configured before the VRF is configured.  Next, it removes the default route lookup rules and moves them so that the VRFs will be checked first.  Finally, it then adds a route defining that the management LAN can be accessed through eth0. This allows egress packets to make their way back to clients on the management LAN.

In future versions of ifupdown-ng, the routing rule setup will be handled automatically.

Verifying the VRF works as expected

Once a VRF is configured, you can use the ip vrf exec command to run a program in the specified VRF context. In our case, the management VRF lacks a default route, so we should be able to observe a failure trying to ping hosts outside the management VRF, using ip vrf exec vrf-management ping 8.8.8.8 for example:

localhost:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=1.287 ms
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.287/1.287/1.287 ms
localhost:~# ip vrf exec vrf-management ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Network unreachable

Success!  Now we know that using ip vrf exec, we can launch services with an alternate routing table.

Integration with OpenRC

Alpine presently uses OpenRC as its service manager, although we plan to switch to s6-rc in the future.  Our branch of OpenRC has support for using VRFs, for the declarative units.  In any of those files, you just add vrf=vrf-management, to the appropriate /etc/conf.d file, for example /etc/conf.d/sshd for the SSH daemon.

For services which have not been converted to use the declarative format, you will need to patch them to use ip vrf exec by hand.  In most cases, all you should need to do is use ${RC_VRF_EXEC} in the appropriate place.

As for performance, this is much more efficient than depending on netfilter, although the setup process is not as clean as I’d like it to be.

introducing witchery: tools for building distroless images with alpine

As I noted in my last blog, I have been working on a set of tools which enable the building of so-called “distroless” images based on Alpine.  These tools have now evolved to a point where they are usable for testing in lab environments, thus I am happy to announce the witchery project.

For the uninitiated, a “distroless” image is one which contains only the application and its dependencies.  This has some desirable qualities: since the image is only the application and its immediate dependencies, there is less attack surface to worry about.  For example, a simple hello-world application built with witchery clocks in at 619kB, while that same hello-world application deployed on alpine:3.14 clocks in at 5.6MB.  There are also drawbacks: a distroless image typically does not include a package manager, so there is generally no ability to add new packages to a distroless image.

As for why it’s called witchery: we are using Alpine’s package manager in new ways to perform truly deep magic.  The basic idea behind witchery is that you use it to stuff your application into an .apk file, and then use apk to install only that .apk and its dependencies into a rootfs: no alpine-base, no apk-tools, no busybox (though witchery allows you to install those things if you want them).

Deploying an an example application with witchery

For those who want to see the source code without commentary, you can find the Dockerfile for this example on the witchery GitHub repo.  For everyone else, I am going to try to break down what each part is doing, so that you can hopefully understand how it all fits together. We will be looking at the Dockerfile in the hello-world example.

The first thing the reader will likely notice is that Docker images built with witchery are done in three stages.  First, you build the application itself, then you use witchery to build what will become the final image, and finally, you copy that image over to a blank filesystem.

FROM alpine:3.14 AS build
WORKDIR /root
COPY . .
RUN apk add --no-cache build-base && gcc -o hello-world hello-world.c

The first stage to build the application is hopefully self explanatory, and is aptly named build.  We fetch the alpine:3.14 image from Dockerhub, then install a compiler (build-base) and finally use gcc to build the application.

The second stage has a few steps to it, that I will split up so that its easier to follow along.

FROM kaniini/witchery:latest AS witchery

First, we fetch the kaniini/witchery:latest image, and name it witchery.  This image contains alpine-sdk, which is needed to make packages, and the witchery tools which drive the alpine-sdk tools, such as abuild.

RUN adduser -D builder && addgroup builder abuild
USER builder
WORKDIR /home/builder

Anybody who is familiar with abuild will tell you that it cannot be used as root.  Accordingly, we create a user for running abuild, and add it to the abuild group.  We then tell Docker that we want to run commands as this new user, and do so from its home directory.

COPY --from=build /root/hello-world .
RUN mkdir -p payloadfs/app && mv hello-world payloadfs/app/hello-world
RUN abuild-keygen -na && fakeroot witchery-buildapk -n payload payloadfs/ payloadout/

The next step is to package our application.  The first step in doing so involves copying the application from our build stage.  We ultimately want the application to wind up in /app/hello-world, so we make a directory for the package filesystem, then move the application into place.  Finally, we generate a signing key for the package, and then generate a signed .apk for the application named payload.

At this point, we have a signed .apk package containing our application, but how do we actually build the image?  Well, just as we drove abuild with witchery-buildapk to build the .apk package and sign it, we will have apk build the image for us.  But first, we need to switch back to being root:

USER root
WORKDIR /root

Now that we are root again, we can generate the image.  But first, we need to add the signing key we generated in the earlier step to apk‘s trusted keys.  To do that, we simply copy it from the builder user’s home directory.

RUN cp /home/builder/.abuild/*.pub /etc/apk/keys

And finally, we build the image.  Witchery contains a helper tool, witchery-compose that makes doing this with apk really easy.

RUN witchery-compose -p ~builder/payloadout/payload*.apk -k /etc/apk/keys -X http://dl-cdn.alpinelinux.org/alpine/v3.14/main /root/outimg/

In this case, we want witchery-compose to grab the application package from ~builder/payloadout/payload*.apk.  We use a wildcard there because we don’t know the full filename of the generated package.  There are options that can be passed to witchery-buildapk to allow you to control all parts of the .apk package’s filename, so you don’t necessarily have to do this.  We also want witchery-compose to use the system’s trusted keys for validating signatures, and we want to pull dependencies from an Alpine mirror.

Once witchery-compose finishes, you will have a full image in /root/outimg.  The final step is to copy that to a new blank image.

FROM scratch
CMD ["/app/hello-world"]
COPY --from=witchery /root/outimg/ .

And that’s all there is to it!

Things left to do

There are still a lot of things left to do.  For example, we might want to implement layers that users can build from when deploying their apps, like one containing s6 for example.  We also don’t have a great answer for applications written in things like Python yet, so far this only works well for programs that are compiled in the traditional sense.

But its a starting point none the less.  I’ll be writing more about witchery over the coming months as the tools evolve into something even more powerful.  This is only the beginning.

Bits relating to Alpine security initiatives in August

As always, the primary focus of my work in Alpine is related to security, either through non-maintainer updates to address CVEs, new initiatives for hardening Alpine, maintenance of critical security-related packages or working with other projects to improve our workflows with better information sharing.  Here are some updates on that, which are slightly delayed because of the long weekend.

sudo deprecation

One of the key things we discussed in the last update was our plan to deprecate sudo, by moving it to community.  sudo exists in a similar situation as firejail: it allows for some interesting use cases, but the security track record is not very good.  Additionally, the maintenance lifecycle for a release branch of sudo is very short, which makes it difficult to provide long-term support for any given version.

As such, the security team proposed to the Technical Steering Committee that we should deprecate sudo and move to an alternative implementation such as doas.  This required some work, namely, doas needed to gain support for configuration directories.  I wrote a patch for doas which provides support for configuration directories, and last week, pushed a doas package which includes this patch with some migration scripts.

At this point, basically everything which depended on sudo for technical reasons has been moved over to using doas.  We are just waiting for the cloud-init maintainer to finish testing their support for doas.  Once that is done, sudo will be moved to community.

OpenSSL 3.0

OpenSSL 3.0 was released today.  It is my intention to migrate Alpine to using it where possible.  As OpenSSL 3.0 will require a major rebuild, after talking with Timo, we will be coordinating this migration plan with the Technical Steering Committee.  Switching to OpenSSL 3.0 should not be as invasive as the OpenSSL 1.0 to 1.1 migration, as they did not change the APIs that much, and will give us the benefit of finally being free of that damn GPL-incompatible OpenSSL license, as OpenSSL 3 was relicensed to use the Apache 2.0 license.

I have already done some test rebuilds which covered much of the aports package set, and did not see much fallout so far.  Even packages which use the more lowlevel APIs, such as those in libcrypto compiled without any major problems.

A nice effect of the license change is that we should be able to drop dependencies on less audited TLS libraries, like GNU TLS, as many programs are licensed under GPL and therefore not compatible with the original OpenSSL license.

Reproducible Builds

We are starting to take steps towards reproducible packages.  The main blocker on that issue was determining what to do about storing the build metadata, so that a build environment can be recreated precisely.  To that extent, I have a patch to abuild which records all of the details exactly.  A rebuilder can then simply install the pinned packages with apk add --virtual.

We will need some way to archive historically built packages for the verification process.  Right now, the archive only ships current packages for each branch.  I am thinking about building something with ZFS or similar which snapshots the archive on a daily basis, but suggestions are welcome if anyone knows of better approaches.

Once these two things are addressed, we need to also add support for attempting rebuilds to the rebuilderd project.  In general, we should be able to model our support based on the support implemented for Arch Linux.

I am expecting to make significant progress on getting the .BUILDINFO file support merged into abuild and support for rebuilderd over the next month.  kpcyrd has been quite helpful in showing us how Arch has tackled reproducibility, and we have applied some lessons from that already to Alpine.

If you’re interested in this project, feel free to join #alpine-reproducible on irc.oftc.net.

secfixes-tracker

I am working on overhauling the JSON-LD documents which are presently generated by the secfixes-tracker application, so that they are more aligned with what the UVI vocabulary will look like.  At the same time, the UVI group have largely endorsed the use of Google’s new OSV format for use cases that do not require linked data.

Accordingly, I am writing a Python library which translates UVI to OSV and vice versa.  This is possible to do without much issues because UVI is intended to be a superset of OSV.

However, we need to request two mime-types for OSV documents and UVI JSON-LD documents.  In the meantime, the secfixes tracker will support querying with the .osv+json extension for querying our security tracker in the OSV format.

Anybody with experience requesting mime-types from IANA is encouraged to provide advice on how to do it most efficiently.

Best practices for Alpine installations

A couple of weeks ago, a kerfuffle happened where Adoptium planned to ship builds of OpenJDK with a third-party glibc package for Alpine.  Mixing libc on any system is a bad idea and has not necessarily obvious security implications.

As one option intended to discourage the practice of mixing musl and glibc on the same system, I proposed installing a conflict with the glibc package as part of the musl packaging.  We asked the Technical Steering Committee for guidance on this plan, and ultimately the TSC decided to solve this with documentation.

Therefore, I plan to work with the docs team to document practices to avoid (such as mixing libc implementations and release branches) to ensure Alpine systems remain secure and reliable.

Distroless, for Alpine

Google’s Distroless project provides tooling to allow users to build containers that include only the runtime dependencies to support an application.  This has some nice security advantages, because images have less components available to attack.  There has been some interest in building the same thing for Alpine, so that users can take advantage of the musl C library, while also having the security advantages of distroless.

It turns out that apk is capable of easily building a tool like this.  I already have a proof of concept, and I plan on expanding that into a more fully featured tool over the next week.  I also plan to do a deep dive into how that tool works once the initial version is released.

Acknowledgement

My activities relating to Alpine security work are presently sponsored by Google and the Linux Foundation. Without their support, I would not be able to work on security full time in Alpine, so thanks!

I drove 1700 miles for a Blåhaj last weekend and it was worth it

My grandmother has Alzheimer’s and has recently had to move into an assisted living facility. You’ve probably seen bits and pieces outlining my frustration with that process on Twitter over the past year or so. Anyway, I try to visit her once or twice a month, as time permits.

But what does that have to do with blåhaj, and what is a blåhaj, anyway? To answer your question literally, blåhaj is Swedish for “blue shark.” But to be more precise, it’s a popular shark stuffed animal design produced by IKEA. As a stuffed animal sommelier, I’ve been wanting one for a hot minute.

Anyway, visiting grandmother was on the way to the St. Louis IKEA. So, I figured, I would visit her, then buy a blåhaj, and go home, right? Well, it was not meant to be: after visiting with my grandmother, we went to the IKEA in St. Louis, and they had sold out. This had happened a few times before, but this time I decided I wasn’t going to take no for an answer. A blåhaj was to be acquired at any cost.

This led us to continue our journey onto Chicago, as the IKEA website indicated they had 20 or so in stock. 20 blåhaj for the entire Chicagoland area? We figured that, indeed, odds were good that a blåhaj could be acquired. Unfortunately, it still wasn’t meant to be: by the time we got to Chicago, the website indicated zero stock.

So we kept going, onward to Minneapolis. At the very least, we could see one of the great monuments to laissez-faire capitalism, the Mall of America, and the historic George Floyd Square, which was frankly more my speed and also quite moving. But again, our attempt to get a blåhaj was unsuccessful – the IKEA at the Mall of America was out of stock.

Our search for blåhaj wasn’t finished yet: there were two options, Winnipeg had nearly 100 in stock, and the Kansas City location had 53. A decision had to be made. We looked at the border crossing requirements for entering Canada and found out that if you present your CDC vaccination card, you could enter Canada without any problems. So, we flipped a coin: do we go six hours north, or six hours south?

Ultimately, we decided to go to the Kansas City location, as we wanted to start heading back towards home. It turns out that Kansas City is only about six hours away from Minneapolis, so we were able to make it to the Kansas City IKEA about an hour before it closed. Finally, a success: a blåhaj was acquired. And that’s when my truck started pissing oil, but that’s a story for another day.

Does blåhaj live up to or exceed my expectations?

Absolutely! As far as stuffed animals go, blåhaj is quite premium, and available for a bargain at only 20 dollars for the large one. The quality is quite comparable to high-end plush brands like Jellycat and Aurora. It is also very soft, unlike some of the other IKEA stuffed animals.

Some people asked about sleeping with a blåhaj. Jellycat stuffed animals, for example, are explicitly designed for spooning, which is essential for a side sleeper. The blåhaj is definitely not, but due to its softness you can use it as a body pillow in various ways, such as to support your head or back.

The shell of the blåhaj is made out of a soft micro plush material very similar to the material used on second generation Jellycat bashful designs (which is different than the yarn-like material used on the first generation bashfuls). All stitching is done using inside seems, so the construction is quite robust. It should last for years, even with constant abuse.

All in all, a fun trip, for a fun blåhaj, though maybe I wouldn’t drive 1700 miles round trip again for one.

How networks of consent can fix social platforms

Social platforms are powerful tools which allow a user to communicate with their friends and family. They also allow for activists to organize and manage political movements. Unfortunately, they also allow for users to harass other users and the mitigations available for that harassment are generally lacking.

By implementing networks of consent using the techniques presented, centralized, federated and distributed social networking platforms alike can build effective mitigations against harassment. Not all techniques will be required, depending on the design and implementation of a given platform.

What does consent mean here?

In this case, consent does not have any special technical meaning. It means the same thing as it does in real life: you (or your agent) is allowing or disallowing an action concerning you in some way.

As computers are incapable of inferring whether consent is given, the user records a statement affirming their consent if granted. Otherwise, the lack of a consent statement must be taken to mean that consent was not granted for an action. How this affirmation is recorded is platform specific.

In technical terms, we refer to these affirmations of consent as an object capability. Many things in this world are already built on object capabilities, for example Mach’s port system and cryptographic assets are forms of object capabilities.

How object capabilities can be used

In a monolithic system, you don’t really need real object capabilities, as they access grants can simply be recorded in the backend, and enforced transparently.

In a federated or distributed system, there are a few techniques that can be used to represent and invoke object capabilities. For example, an object capability might be represented by a key pair. In this case, the capability is invoked by signing the request with that key. Alternatively, capability URLs are another popular option, popularized by the Second Life Grid.

In a distributed system, simply having an opaque pointer to a given set of rights (and demonstrating possession of it) is sufficient, as the actor invoking the capability will invoke it directly with all relevant participants. This works because all participants are able to directly verify the validity of the capability as they witnessed its issuance to begin with.

However, in a federated system, you also need a way to provide proof that the invocation of a capability was accepted. This is usually implemented in the form of a signed proof statement created by the participant which issued the capability to begin with. Other more exotic schemes exist, but for the purpose of explaining everything this should suffice.

Building networks of consent with object capabilities

Now that we understand the basic concepts behind object capabilities, we can use them to model what a social network built from the ground up with a consent-oriented design would look like.

It should be noted that the user may configure her user agent to automatically consent to any incoming action, but this is an implementation detail. The presence of a consent framework at the network level does not imply the requirement for a user to manage whether consent is granted, it just allows for the necessary decision points to exist.

An example on a monolithic network

Let’s say that Alice wants to reply to her friend Bob’s post on Tooter, an imaginary proprietary social network that is focused on microblogging. In a proprietary network, Alice composes her reply, and then sends it to the network. The network then asks Bob’s user agent to approve or disapprove the reply. Bob’s user agent can choose to automatically accept the reply because Alice and Bob are already friends.

Now, let’s say that Karen98734762153 wants to reply to Bob’s post as well. Karen98734762153 has no prior relationship with Bob, but because Bob’s user agent is asked to make the decision, it can present the message to Bob to accept or reject. As Karen98734762153 is wanting to suggest the use of apple-flavored horse paste as a possible prophylactic for COVID, he chooses to reject the post, and Karen98734762153 is not granted any new privileges.

The same example in a distributed system

On a proprietary network, all of this can be implemented transparently to the end user. But can it be implemented in a distributed system? In this case, we assume a simple peer to peer network like Scuttlebutt. How would this work there?

As noted before, we can use object capabilities here. In this case, both Alice and Karen98734762153 would send their replies to Bob. Alice would reference her pre-existing relationship, and Karen98734762153 would not reference anything. Bob would commit Alice’s reply to the Scuttlebutt ledger and distribute that commitment to the pub-servers he is subscribed to, and ignore Karen98734762153’s reply.

The same example in a federated system

As seen, in a distributed system with a distributed ledger, where all objects are signed, this approach can work. Federated systems are a lot trickier to get right in regards to trust relationships, but it can be done here too. In this case, we introduce proof objects to demonstrate the acceptance of a capability. We will refer to these proofs as endorsements.

To this end, both Alice and Karen98734762153 send their replies to Bob, like before. Bob’s user agent then makes the decision to accept or reject the replies. In this example, Bob would add the reply to his local copy of replies, and then at a minimum send an endorsement back to Alice. Either Alice, Bob or both would then distribute that endorsement to a list of interested subscribers, who could verify the validity of the endorsement.

While other instances may choose to accept replies without an endorsement, they can also choose to reject them, or to give endorsed replies special status in their user interface. As there is not a unified consensus mechanism in federated networks, that is all that can be done. But it’s still pretty good.

The application of these ideas to other scenarios is left as an exercise for the reader.

I am planning to move to Europe

I have been considering a move to Europe since the 2018 midterm election, though a combination of friends being persuasive and the COVID-19 pandemic put a damper on those plans.  Accordingly, I have tried my best to give Biden and the democrats an opportunity to show even the most basic modicum of progress on putting the country on a different path.  I did my part, I held my nose and as I was told, voted blue no matter who, despite a total lack of enthusiasm for the candidates.  But honestly, I can’t do this anymore.

Yesterday, Texas’ SB 8 went into force of law.  This is a law which incentivizes citizens to bring litigation against anybody involved in administering an abortion to anybody who is six weeks pregnant or later, by issuing a $10,000 reward paid by the state for such litigation if successful.

As I do not have a uterus, I am unable to get pregnant, and so you might wonder why I care about this so much.  It is simple: the right to bodily autonomy should matter to everyone.  However, there is a second reason as well.  The US Supreme Court, when requested to review the law, determined there was no constitutionality issue with it, because of the financial incentive system implemented by the law.

This means that the legislators in Texas have found a vulnerability in our legal system, one which will be surely abused to institute other draconian policies driven by the religious right.  The people who are implementing this strategy will do their best to outlaw any form of contraception.  They will try their best to outlaw queer relationships.  And, most relevant to me, they will use this vulnerability against trans people to make even our basic existence illegal.

My confidence in the continued viability for my safety in the US began to wane earlier this summer, when a transphobic activist invented false allegations that a transgender woman flashed her genitalia at other women at Wi Spa in Los Angeles.  These allegations resulted in violent demonstrations by white supremacists.  Don’t worry, the message was received.

This isn’t strictly about safety, however.  I also recognize that leaving the US is a selfish choice, and that I have a lot of privilege that others may not have.  But the thing is, I’ve worked my tail off to get where I am, on my own.

As a skilled tax-paying worker, I believe the US has an obligation to respect me as a person, if they wish for me to retain my patronage.  In other words, they should be competing for myself and other skilled labor to remain.  Instead, the lack of any tangible action to bring an end to Trumpism and lack of any legislative resistance to trans people being the next stop in the culture war, shows that I am not wanted here, and so I will move somewhere else, where I will be safe and respected as a person.

And so I plan to move to the Netherlands within the next year.  You see, as I primarily work as a contractor, I have a business vehicle.  The Dutch American Friendship Treaty allows me to very easily acquire a permanent Schengen visa, and after some time, citizenship, simply by moving my business vehicle to the Netherlands.  Additionally, I already have many friends in Europe, so making new friends is not something I have to worry a lot about.

I’m not sure how moving my business vehicle to Europe will affect any current engagements, so in the interest of redundancy, I will be putting together a CV package and putting together a website for my business vehicle within the next few weeks.  If you have a European company that could benefit from my unique experiences and insight, feel free to reach out.  Similarly, I’ll probably be looking for some sort of housing arrangements in the next few months, so suggestions in that area are welcome too.

there is no such thing as a “glibc based alpine image”

For whatever reason, the alpine-glibc project is apparently being used in production.  Worse yet, some are led to believe that Alpine officially supports or at least approves of its usage.  For the reasons I am about to outline, we don’t.  I have also proposed an update to Alpine which will block the installation of the glibc packages produced by the alpine-glibc project, and have referred acceptance of that update to the TSC to determine if we actually want to put our foot down or not.  I have additionally suggested that the TSC may wish to have the Alpine Council reach out to the alpine-glibc project to find a solution which appropriately communicates that the project is not supported in any way by Alpine.  It should be hopefully clear that there is no such thing as a “glibc based alpine image” because Alpine does not use glibc, it uses musl.

Update: the TSC has decided that it is better to approach this problem as a documentation issue.  We will therefore try to identify common scenarios, including using the glibc package, that cause stability issues to Alpine and document them as scenarios that should ideally be avoided.

What the alpine-glibc project actually does

The alpine-glibc project attempts to package the GNU C library (glibc) in such a way that it can be used on Alpine transparently.  However, it is conceptually flawed, because it uses system libraries where available, which have been compiled against the musl C library.  Combining code built for musl with code built for glibc is like trying to run Windows programs on OS/2: both understand .EXE files to some extent, but they are otherwise very different.

But why are they different?  They are both libraries designed to run ELF binaries, after all.  The answer is due to differences in the application binary interface, also known as an ABI.  Specifically, glibc supports and heavily uses a backwards compatibility technique called symbol versioning, and musl does not support it at all.

How symbol versioning works

Binary programs, such as those compiled against musl or glibc, have something called a symbol table.  The symbol table contains a list of symbols needed from the system libraries, for example the C library functions like printf are known as symbols.  When a binary program is run, it is not executed directly by the kernel: instead, a special program known as an ELF interpreter is loaded, which sets up the mapping from symbols in the symbol table to the actual locations where those symbols exist.  That mapping is known as the global object table.

On a system with symbol versioning, additional data in the symbol table designates what version of a symbol is actually wanted.  For example, when you request printf on a glibc system, you might actually wind up requesting printf@GLIBC_2_34 or some other kind of versioned symbol.  This allows newer programs to prefer the newer printf function, while older programs can reference an older version of the implementation.  That allows for low-cost backwards compatibility: all you have to do is keep around the old versions of the routines until you decide to drop support in the ABI for them.

Why mixing these two worlds is bad

However, if you combine a world expecting symbol versioning and one which does not, you wind up with undefined behavior.  For very simple programs, it appears to work, but for more complicated programs, you will wind up with strange behavior and possible crashes, as the global object table references routines with different behavior than expected by the program.  For example, a program expecting a C99 compliant printf routine will get one on musl if it asks for printf.  But a program expecting a C99 compliant printf routine on glibc will ask for printf@GLIBC_2_12 or similar.

The symbol versioning problem spreads to the system libraries too: on Alpine, libraries don’t provide versioned symbols: instead, you get the latest version of each symbol.  But if a glibc program is expecting foo to be an older routine without the semantics of the current implementation of foo, then it will either crash or do something weird.

This has security impacts: the lack of consistency for whether versioned symbols are actually supported by the system basically turns any interaction with versioned symbols into what is called a weird machine.  This means that an attacker possibly controls more attack surface than they would in a situation where the system ran either pure glibc or pure musl.

Alternatives to alpine-glibc

As alpine-glibc is primarily discussed in the context of containers, we will keep this conversation largely focused on that.  There are a few options if you want a small container to run a binary blob linked against glibc in a container that are far better than using alpine-glibc.  For example, you can use Google’s distroless tools, which will build a Debian-based container with only the application and its runtime dependencies, which allows for a fairly small container.  You can also try to use the gcompat package, which emulates the GNU C library ABI in the same way that WINE emulates the Windows ABI.

But whatever you do, you shouldn’t use the alpine-glibc project to do this.  You will wind up with something completely broken.

a tail of two bunnies

As many people know, I collect stuffed animals.  Accordingly, I get a lot of questions about what to look for in a quality stuffed animal which will last a long time.  While there are a lot of factors to consider when evaluating a design, I hope the two examples I present here in contrast to each other will help most people get the basic idea.

the basic things to look for

A stuffed animal is basically a set of fabric patches sewn together around some stuffing material.  Therefore, the primary mode of failure for a stuffed animal is when one or more seams suffers a tear or rip in its stitching.  A trained eye can look at a design and determine both the likelihood of failure and the most vulnerable seams, even in a high quality stuffed animal.

There are two basic ways to sew together a stuffed animal: the fabric patches can be sewn together to form inward-facing seams, or they can be sewn together to form outward-facing seams.  Generally, the stuffed animals that have inward facing seams have more robust construction.  This means that if you can easily see the seam lines that the quality is likely to be low.  Similarly, if eyes and other accessories are sewn in along a main seam line, they become points of vulnerability in the design.

Materials also matter: if the purpose of the stuffed animal is to be placed on a bed, or in a crib, it should be made out of fire-retardant materials.  Higher quality stuffed animals will use polyester fill with a wool-polyester blend for the outside, while lower quality stuffed animals may use materials like cotton.  In the event of a fire, polyester can potentially melt onto skin, but materials like cotton will burn much more vigorously than polyester (which is fire retardant).

Finally, it is important to verify that the stuffed animal has been certified to a well-known safety standard.  Look for compliance with the European Union’s EN71 safety standard or the ASTM F963 standard.  Do not buy any stuffed animal made by a company which is not compliant with these standards.  Stuffed animals bought off maker-oriented websites like Etsy will most likely not be certified, in these cases, you may wish to verify with the maker that they are familiar with the EN71 and ASTM F963 standards and have designed around those standards.

a good example: the jellycat bashful bunny

A jellycat bashful bunny, cream colored, size: really big. it is approximately 4 feet tall.

One of my favorite bunny designs is the Jellycat Bashful Bunny.  I have several of them, ranging from small to the largest size available.

This is what I would consider to be a high quality design.  While the seam line along his tummy is visible, it is a very small seam line, which is indicative that the stitching is inward-facing.  There are no other visible seam lines.  Cared for properly, this stuffed animal will last a very long time.

a bad example: build a bear’s pawlette

Jumbo Pawlette, from build a bear. This variant is 3 feet tall.

A few people have asked me about Build a Bear’s Pawlette design recently, as it looks very similar to the Jellycat Bashful Bunny.  I don’t think it is a very good design.

To start with, you can see that there are 21 separate panels stitched together: 4 for the ears, 3 for the head, 4 for the arms, 2 for the tummy, 2 for the back, 4 for the legs, and 2 for the feet.  The seam lines are very visible, which indicates that there is a high likelihood that the stitching is outward rather than inward.  That makes sense, because it’s a lot easier to stitch up a stuffed animal in store that way.  Additionally, you can see that the eyes are anchored to the seam lines that make up the face, which means detachment of the eyes is a likely possibility as a failure mode.

Build a Bear has some good designs that are robustly constructed, but Pawlette is not one of them.  I would avoid that one.

Hopefully this is helpful to somebody, at the very least, I can link people to this post now when they ask about this stuff.

free software does not come with any guarantees of support

This evening, I stumbled upon a Twitter post by an account which tracks features being added to GitHub:

To be absolutely clear, this is a terrible idea.  Free software maintainers already have to deal with a subset of users who believe they are automatically entitled to support and, in some cases, SLAs from the maintainer.

Thankfully, the license I tend to use these days for my software makes it very clear that no support is provided:

This software is provided ‘as is’ and without any warranty, express or implied. In no event shall the authors be liable for any damages arising from the use of this software.

However, it’s not only my license which does this.  So does the GPL, in all caps none the less:

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

Other free software licenses, such as the BSD license, and Apache 2.0 license also have a similar clause.  While maintainers might offer to provide support for free, or perhaps offer a support agreement for a fee, free software is ultimately what you make of it, for better or worse.

By integrating anti-features such as direct messages into software development forges like the proposed GitHub feature, the developers of these forges will inevitably be responsible for an onslaught of abusive messages directed at free software maintainers, which ultimately will serve to act as a Denial of Service on our time.  And, most likely, this abuse will be targeted more frequently at maintainers who are women, or people of color.

I urge the developers of software development forges to respect the labor donated to the commons by free software maintainers, by developing features which respect their agency, rather than features that will ultimately lead abusive users to believe maintainers are available at their beck and call.

Otherwise, the developers of these forges may find projects going to a platform that does respect them.