A tale of two envsubst implementations

Yesterday, Dermot Bradley brought up in IRC that gettext-tiny’s lack of an envsubst utility could be a potential problem, as many Alpine users use it to generate configuration from templates.  So I decided to look into writing a replacement, as the tool did not seem that complex.  That rewrite is now available on GitHub, and is already in Alpine testing for experimental use.

What envsubst does

The envsubst utility is designed to take a set of strings as input and replace variables in them, in the same way that shells do variable substitution.  Additionally, the variables that will be substituted can be restricted to a defined set, which is nice for reliability purposes.

Because it provides a simple way to perform substitutions in a file without having to mess with sed and other similar utilities, it is seen as a helpful tool for building configuration files from templates: you just install the cmd:envsubst provider with apk and perform the substitutions.

Unfortunately though, GNU envsubst is quite deficient in terms of functionality and interface.

Good tool design is important

When building a tool like envsubst, it is important to think about how it will be used.  One of the things that is really important is making sure a tool is satisfying to use: a tool which has non-obvious behavior or implies functionality that is not actually there is a badly designed tool.  Sadly, while sussing out a list of requirements for my replacement envsubst tool, I found that GNU envsubst has several deficiencies that are quite disappointing.

GNU envsubst does not actually implement POSIX variable substitution like a shell would

In POSIX, variable substitution is more than simply replacing a variable with the value it is defined to.  In GNU envsubst, the documentation speaks of shell variables, and then outlines the $FOO and ${FOO} formats for representing those variables.  The latter format implies that POSIX variable substitution is supported, but it’s not.

In a POSIX-conformant shell, you can do:

% FOO=“abc_123” % echo ${FOO%_*} abc

Unfortunately, this isn’t supported by GNU envsubst:

% FOO=“abc_123” envsubst $FOO abc_123 ${FOO} abc_123 ${FOO%_*} ${FOO%_*}

It’s not yet supported by my implementation either, but it’s on the list of things to do.

Defining a restricted set of environment variables is bizzare

GNU envsubst describes taking an optional [SHELL-FORMAT] parameter.  The way this feature is implemented is truly bizzare, as seen below:

% envsubst -h Usage: envsubst [OPTION] [SHELL-FORMAT] … Operation mode:  -v, –variables             output the variables occurring in SHELL-FORMAT … % FOO=“abc123” BAR=“xyz456” envsubst FOO $FOO $FOO % FOO=“abc123” envsubst -v FOO % FOO=“abc123” envsubst -v \$FOO FOO % FOO=“abc123” BAR=“xyz456” envsubst \$FOO $FOO abc123 $BAR $BAR % FOO=“abc123” BAR=“xyz456” envsubst \$FOO \$BAR envsubst: too many arguments % FOO=“abc123” BAR=“xyz456” envsubst \$FOO,\$BAR $FOO abc123 $BAR xyz456 $BAZ $BAZ % envsubst -v envsubst: missing arguments %

As discussed above, [SHELL-FORMAT] is a very strange thing to call this, because it is not really a shell variable substitution format at all.

Then there’s the matter of requiring variable names to be provided in this shell-like variable format.  That requirement gives a shell script author the ability to easily break their script by accident, for example:

% echo ‘Your home directory is $HOME’ | envsubst $HOME Your home directory is $HOME

Because you forgot to escape $HOME as \$HOME, the substitution list was empty:

% echo ‘Your home directory is $HOME’ | envsubst \$HOME Your home directory is /home/kaniini

The correct way to handle this would be to accept HOME without having to describe it as a variable.  That approach is supported by my implementation:

% echo ‘Your home directory is $HOME’ | ~/.local/bin/envsubst HOME Your home directory is /home/kaniini

Then there’s the matter of not supporting multiple variables in the traditional UNIX style (as separate tokens).  Being forced to use a comma on top of using a variable sigil for this is just bizzare and makes the tool absolutely unpleasant to use with this feature.  For example, this is how you’re supposed to add two variables to the substitution list in GNU envsubst:

% echo ‘User $USER with home directory $HOME’ | envsubst \$USER,\$HOME User kaniini with home directory /home/kaniini

While my implementation supports doing it that way, it also supports the more natural UNIX way:

% echo ‘User $USER with home directory $HOME’ | ~/.local/bin/envsubst USER HOME User kaniini with home directory /home/kaniini

This is common with GNU software

This isn’t just about GNU envsubst.  A lot of other GNU software is equally broken.  Even the GNU C library has design deficiencies which are similarly frustrating.  The reason why I wish to replace GNU software in Alpine is because in many cases, it is defective by design.  Whether the design defects are caused by apathy, or they’re caused by politics, it doesn’t matter.  The end result is the same, we get defective software.  I want better security and better reliability, which means we need better tools.

We can talk about the FSF political issue, and many are debating that at length.  But the larger picture is that the tools made by the GNU project are, for the most part, clunky and unpleasant to use.  That’s the real issue that needs solving.