In my line of work as a system administrator, I naturally have to maintain and manage several hosts at once. Add to that the machines I set up and use myself. There’s my main desktop PC for development and administration, my Dell laptop that acts as a VPN gateway and “backup” workstation when I am working or playing on Windows, my Lenovo tablet/laptop that I take with me when travelling, my HP ProLiant home server for storage and Asterisk, and my root server on which this site and several other things (ZNC, mail) are hosted. Each one of those runs Gentoo, a flexible source-based meta-distribution - and my operating system of choice. egress emerged out of a need for centralising certain configuration files required by it.

Configuration hell

Gentoo’s package manager - portage - albeit a bit slow, is one of the best PMs out there. Being a source-based distribution, there is a lot you can tweak when compiling and installing packages in Gentoo: USE flags - settings passed to the program’s ./configure routine - for example. Or ACCEPT_KEYWORDS - masking settings that determine whether portage selects a stable or unstable (“masked”) package. You can also allow or disallow certain licenses through ACCEPT_LICENSE. To make portage aware of your preferences, you can use the configuration files in /etc/portage, for example:

/etc/portage/package.use/vim
app-editors/vim cscope -lua -perl -python
app-editors/vim-core minimal
/etc/portage/package.accept_keywords/htop
sys-process/htop ~amd64

This would set certain USE flags for the vim suite and “unmask” htop - allowing the latest unstable version to be installed. As you can see, creating a single file for each program is quite inconvenient. For minimal installations or small servers, a single package.use or package.accept_license file suffices, but what about more complex setups and hosts? I used to just use one big file for those as well. That’s fine until you have to maintain several Gentoo hosts. Most of the time, you want changes you made on one of the hosts to propagate to all other hosts - or a certain subset thereof. “Right,” you think, “let’s just use rsync and be done with it”. Well, what if you have one host that needs vim without minimal? Or has issues with the newest version of htop? The next step would then be to split the configuration file into smaller parts - each one “tuned” for a specific host. This will get very messy without some sort of file management.

Enter egress

So, wouldn’t it be better if you could somehow set up a centralised repository for portage configuration? You could diff changes, move and edit files with ease, and - most importantly - deploy setups lightning-fast. Clone the repo, copy the files, emerge -tuUD @world, done. Managing the repository will be git’s task, but how do you get the files - and only the files for that specific host - to /etc/portage?

This is what egress does (amongst other things). It provides an interface between the repository and your portage configuration files. It’s fast, small (egress is under 150 LOC and only needs app-admin/stow and app-admin/sudo - everything else is already in @system), VCS-agnostic and easy to use. Its counterpart - ingress - manages edits, searches and listings. Together with a VCS, they form a complete suite for the maintenance, management and deployment of portage’s configuration files.

egress: Core concepts

Let’s start by describing the main concepts behind this whole system. We have to talk about what hosts are, which directory structure egress uses, how it interfaces with the directory structure in ‘/etc/portage` and what it considers to be a “profile”. Keep in mind that this is a very in-depth description of egress’ features. In order to use it you, only have to know about profiles and the repository structure.

Hosts

This part is quite straight-forward. A host is simply a workstation on which egress was set up. Internally, the script uses the hostname command to retrieve the name.

The “compiled” list of configuration files for one specific host is kept in /etc/egress, and is thus owned by root. egress keeps this directory in sync with the repository. GNU stow creates and manages symlinks pointing from /etc/portage to /etc/egress. This structure ensures that you don’t accidentally delete or edit important files in the portage directory. The program simply does not touch any file in it.

$ ls -l /etc/portage
[..]
package.accept_keywords -> ../egress/portage/package.accept_keywords
package.license -> ../egress/portage/package.license
[..]

stow creates symlinks as needed every time you synchronise your configuration files. Once you’re up and running, you won’t even notice it anymore.

Profiles

Profiles consist of a set of portage configuration files. You could create a server profile, for example, or a desktop profile. Every host has a set of profiles. When syncing, egress will sync every profile you specified for that host to the egress directory.

There are two special profiles, host and temp:

  1. The host profile contains host-specific configuration files (so that you don’t need to create one profile per host)

  2. The temp profile is ignored by egress. Use this to introduce temporary changes you don’t want synced to the repository.

Using profiles

A file named profiles in each host directory (see here) contains profiles for that host separated by a newline:

host/tux/profiles
core
desktop
haskell

With this setup, egress will sync the core, desktop, haskell and the host profile “tux” (kept in host/tux/portage) to the egress directory.

Directory structure - repository

You keep all configuration files (every host, every profile, etc.) in a local, user-owned directory. This is called the “repository”. This directory is usually managed by your favourite VCS (I like git). For egress to work, you need at least two directories here:

profiles

A folder profiles, with at least one profile directory (like profiles/server). Each profile contains a directory portage, with further directories and files adhering to the structure described in portage(5). Naming is up to you, but the files have to end in .<profile>:

$ ls -l profiles/server/portage/package.use/
00-sys.server
30-net.server

host

This folder houses the host-specific profile and each host’s profile configuration. Again, naming is up to you, but files in this directory have to end in .host.

$ ls -l host/tux/
portage/
profiles
$ ls -l host/tux/portage/package.accept_keywords
05-sys.host
65-misc.host

List of supported files and directories

Right now, egress can sync:

  • portage/package.*

  • portage/*.conf - like make.conf and repos.conf

  • portage/sets

  • portage/env

  • portage/savedconfig

Setting up and using egress

Setting up egress is simple, the hard part is converting your current configuration setup into a format egress understands. This means creating profiles, setting up hosts and organising your USE flags/keywords. Once you have that, install egress (either with the provided Makefile or the ebuild in my private overlay), copy config.template to /etc/egress/config and set up your preferences there.

Since you already converted your configuration files and moved them to the repository, you can delete the old config in /etc/portage/ - but have a backup ready. With egress installed, run egress push, and you’re all set. Should stow warn you about potential conflicts (don’t worry, it will never delete or touch anything it doesn’t own), you will have to resolve them yourself. The first push will look similar to this:

$ egress push
Pushing configuration files for tux from repo to /etc/egress.
Continue? [Y/n] y
cd+++++++++ portage/
>f+++++++++ portage/make.conf
>f+++++++++ portage/repos.conf
cd+++++++++ portage/env/
>f+++++++++ portage/env/debug
cd+++++++++ portage/package.accept_keywords/
>f+++++++++ portage/package.accept_keywords/20-dev.core
>f+++++++++ portage/package.accept_keywords/30-net.desk
>f+++++++++ portage/package.accept_keywords/50-media.desk
>f+++++++++ portage/package.accept_keywords/60-misc.host
[..]
>f+++++++++ portage/package.use/60-misc.core
>f+++++++++ portage/package.use/90-deps.desk
[..]
cd+++++++++ portage/sets/
>f+++++++++ portage/sets/tex
Updating symlinks...
LINK: env => ../egress/portage/env
LINK: sets => ../egress/portage/sets
LINK: package.license => ../egress/portage/package.license
LINK: savedconfig => ../egress/portage/savedconfig
LINK: package.use => ../egress/portage/package.use
LINK: make.conf => ../egress/portage/make.conf
LINK: package.accept_keywords => ../egress/portage/package.accept_keywords
LINK: repos.conf => ../egress/portage/repos.conf
LINK: package.mask => ../egress/portage/package.mask
Done.

As you can see, “tux” uses the core and desk profile. There is also one file (60-misc.host) that is host-specific. This particular host also keeps make.conf and repos.conf in the repository, along with a “tex” set, and debug settings in portage/env.

If you want to make changes to your portage configuration, simply open the relevant configuration file in /etc/portage with your favourite editor - no changes in workflow here. Then, run egress pull to sync these changes to the repository:

$ egress pull
Pulling configuration files for tux from /etc/egress to repo.
Continue? [Y/n] y
Pulling profile: core
Pulling profile: desk
>f.st...... portage/package.use/50-media.desk
Pulling host-specific profile
.d..t...... portage/
Pulling env.
Pulling sets.
Pulling savedconfig.
Pulling make.conf
Pulling repos.conf
Done.

From there you can diff, manage and deploy the changes with a VCS:

$ git diff
diff --git a/profiles/desk/portage/package.use/50-media.desk b/profiles/desk/portage/package.use/50-media.desk
index e8de9e5..f932b18 100644
--- a/profiles/desk/portage/package.use/50-media.desk
+++ b/profiles/desk/portage/package.use/50-media.desk
@@ -1,11 +1,10 @@
 media-gfx/sxiv savedconfig

-media-sound/ncmpcpp outputs taglib
+media-sound/ncmpcpp outputs taglib curl
 media-sound/mpd inotify
 media-sound/abcde cdparanoia replaygain -aac -vorbis -id3tag -lame
-media-sound/audacity libsamplerate ladspa

 media-video/ffmpeg threads opus openssl
-media-video/mpv -dvd -enca -encode -iconv -jpeg -lcms -lua -mpg123 -xscreensaver -xv -luajit
+media-video/mpv dvd bluray -enca -encode -iconv -jpeg -lcms -lua -mpg123 -xv -luajit

 media-fonts/terminus-font -pcf-unicode-only

Using ingress to edit files

If you end up with a lot of configuration files in /etc/portage/package.*, editing and keeping track of specific settings can be quite hard. ingress tries to alleviate this task by providing an interactive interface between your favourite editor and the files in question. Say you want to edit the USE flags for your web browser. Those are defined in /etc/portage/package.use/30-net.desk. Instead of entering the path yourself (or searching for it with one of those nifty fuzzy finders), you can simply use ingress:

$ ingress e net
1) /etc/portage/package.accept_keywords/30-net.desk
2) /etc/portage/package.use/30-net.core
3) /etc/portage/package.use/30-net.desk
4) [create]
#? 3
Editing /etc/portage/package.use/30-net.desk

After selecting the right file, ingress will call sudoedit. This will automatically launch the editor defined in the SUDO_EDITOR, VISUAL or EDITOR environment variables. If you don’t want to use sudoedit, you can disable it in /etc/egress/config; ingress will then call sudo $EDITOR instead.

You can also edit multiple files at once:

$ ingress e -m accept_keywords
Select multiple files. Type 'q' when you're done.
1) /etc/portage/package.accept_keywords/20-dev.core
2) /etc/portage/package.accept_keywords/30-net.desk
3) /etc/portage/package.accept_keywords/50-media.desk
4) /etc/portage/package.accept_keywords/60-misc.core
5) /etc/portage/package.accept_keywords/60-misc.desk
6) /etc/portage/package.accept_keywords/70-games.desk
7) /etc/portage/package.accept_keywords/90-deps.desk
8) [create]
#? (multiple) 4
#? (multiple) 5
#? (multiple) q
Editing 2 files:
/etc/portage/package.accept_keywords/60-misc.core
/etc/portage/package.accept_keywords/60-misc.desk

In this example, we did not specify a file name, but a package.* specifier. We can even use both. Here, ingress finds one exact match and edits it right away:

$ ingress e license games
Editing /etc/portage/package.license/70-games.desk

The script can also list files in the portage configuration directory (ingress list unmask), search for files containing a certain pattern (ingress grep vim) and print a “compiled” output of all the settings for one package.* specifier:

$ ingress view mask
# vim:ft=gentoo-package-mask:

# -- /etc/portage/package.mask/00-sys.core --
sys-apps/systemd
sys-apps/dbus

sys-auth/polkit
sys-auth/consolekit

sys-fs/udev

# -- /etc/portage/package.mask/10-x11.core --
# Pulls in a lot of stuff I don't need.
x11-misc/xdg-utils

For all of this to work, you don’t even need a special setup. ingress works even if you don’t use egress, as it looks only in /etc/portage/.

Finally

One year ago when I started this project it was merely a small wrapper around two rsync operations. Now, after one year of usage and numerous changes (thanks to c_14 for feedback & ideas), I finally feel like releasing a 0.1 version. Please report any bugs on the github page. Feel free to send pull requests and ideas.