In my line of work as a system administrator, I naturally have to maintain and manage several hosts at once. Add to that the machines I set up and use myself. There’s my main desktop PC for development and administration, my Dell laptop that acts as a VPN gateway and “backup” workstation when I am working or playing on Windows, my Lenovo tablet/laptop that I take with me when travelling, my HP ProLiant home server for storage and Asterisk, and my root server on which this site and several other things (ZNC, mail) are hosted. Each one of those runs Gentoo, a flexible source-based meta-distribution - and my operating system of choice. egress emerged out of a need for centralising certain configuration files required by it.
Gentoo’s package manager - portage - albeit a bit slow, is one of the best PMs out there. Being a source-based distribution, there is a lot you can tweak when compiling and installing packages in Gentoo: USE flags - settings passed to the program’s ./configure routine - for example. Or ACCEPT_KEYWORDS - masking settings that determine whether portage selects a stable or unstable (“masked”) package. You can also allow or disallow certain licenses through ACCEPT_LICENSE. To make portage aware of your preferences, you can use the configuration files in /etc/portage, for example:
app-editors/vim cscope -lua -perl -python app-editors/vim-core minimal
This would set certain USE flags for the vim suite and “unmask” htop - allowing the latest unstable version to be installed. As you can see, creating a single file for each program is quite inconvenient. For minimal installations or small servers, a single package.use or package.accept_license file suffices, but what about more complex setups and hosts? I used to just use one big file for those as well. That’s fine until you have to maintain several Gentoo hosts. Most of the time, you want changes you made on one of the hosts to propagate to all other hosts - or a certain subset thereof. “Right,” you think, “let’s just use rsync and be done with it”. Well, what if you have one host that needs vim without minimal? Or has issues with the newest version of htop? The next step would then be to split the configuration file into smaller parts - each one “tuned” for a specific host. This will get very messy without some sort of file management.
So, wouldn’t it be better if you could somehow set up a centralised repository for portage configuration? You could diff changes, move and edit files with ease, and - most importantly - deploy setups lightning-fast. Clone the repo, copy the files, emerge -tuUD @world, done. Managing the repository will be git’s task, but how do you get the files - and only the files for that specific host - to /etc/portage?
This is what egress does (amongst other things). It provides an interface between the repository and your portage configuration files. It’s fast, small (egress is under 150 LOC and only needs app-admin/stow and app-admin/sudo - everything else is already in @system), VCS-agnostic and easy to use. Its counterpart - ingress - manages edits, searches and listings. Together with a VCS, they form a complete suite for the maintenance, management and deployment of portage’s configuration files.
egress: Core concepts
Let’s start by describing the main concepts behind this whole system. We have to talk about what hosts are, which directory structure egress uses, how it interfaces with the directory structure in ‘/etc/portage` and what it considers to be a “profile”. Keep in mind that this is a very in-depth description of egress’ features. In order to use it you, only have to know about profiles and the repository structure.
This part is quite straight-forward. A host is simply a workstation on which egress was set up. Internally, the script uses the hostname command to retrieve the name.
Directory structure - egress , symlinks & stow
The “compiled” list of configuration files for one specific host is kept in /etc/egress, and is thus owned by root. egress keeps this directory in sync with the repository. GNU stow creates and manages symlinks pointing from /etc/portage to /etc/egress. This structure ensures that you don’t accidentally delete or edit important files in the portage directory. The program simply does not touch any file in it.
[..] package.accept_keywords -> ../egress/portage/package.accept_keywords package.license -> ../egress/portage/package.license [..]
stow creates symlinks as needed every time you synchronise your configuration files. Once you’re up and running, you won’t even notice it anymore.
Profiles consist of a set of portage configuration files. You could create a server profile, for example, or a desktop profile. Every host has a set of profiles. When syncing, egress will sync every profile you specified for that host to the egress directory.
There are two special profiles, host and temp:
The host profile contains host-specific configuration files (so that you don’t need to create one profile per host)
The temp profile is ignored by egress. Use this to introduce temporary changes you don’t want synced to the repository.
A file named profiles in each host directory (see here) contains profiles for that host separated by a newline:
core desktop haskell
With this setup, egress will sync the core, desktop, haskell and the host profile “tux” (kept in host/tux/portage) to the egress directory.
Directory structure - repository
You keep all configuration files (every host, every profile, etc.) in a local, user-owned directory. This is called the “repository”. This directory is usually managed by your favourite VCS (I like git). For egress to work, you need at least two directories here:
A folder profiles, with at least one profile directory (like profiles/server). Each profile contains a directory portage, with further directories and files adhering to the structure described in portage(5). Naming is up to you, but the files have to end in .<profile>:
This folder houses the host-specific profile and each host’s profile configuration. Again, naming is up to you, but files in this directory have to end in .host.
List of supported files and directories
Right now, egress can sync:
portage/*.conf - like make.conf and repos.conf
Setting up and using egress
Setting up egress is simple, the hard part is converting your current configuration setup into a format egress understands. This means creating profiles, setting up hosts and organising your USE flags/keywords. Once you have that, install egress (either with the provided Makefile or the ebuild in my private overlay), copy config.template to /etc/egress/config and set up your preferences there.
Since you already converted your configuration files and moved them to the repository, you can delete the old config in /etc/portage/ - but have a backup ready. With egress installed, run egress push, and you’re all set. Should stow warn you about potential conflicts (don’t worry, it will never delete or touch anything it doesn’t own), you will have to resolve them yourself. The first push will look similar to this:
Pushing configuration files for tux from repo to /etc/egress. Continue? [Y/n] y cd+++++++++ portage/ >f+++++++++ portage/make.conf >f+++++++++ portage/repos.conf cd+++++++++ portage/env/ >f+++++++++ portage/env/debug cd+++++++++ portage/package.accept_keywords/ >f+++++++++ portage/package.accept_keywords/20-dev.core >f+++++++++ portage/package.accept_keywords/30-net.desk >f+++++++++ portage/package.accept_keywords/50-media.desk >f+++++++++ portage/package.accept_keywords/60-misc.host [..] >f+++++++++ portage/package.use/60-misc.core >f+++++++++ portage/package.use/90-deps.desk [..] cd+++++++++ portage/sets/ >f+++++++++ portage/sets/tex Updating symlinks... LINK: env => ../egress/portage/env LINK: sets => ../egress/portage/sets LINK: package.license => ../egress/portage/package.license LINK: savedconfig => ../egress/portage/savedconfig LINK: package.use => ../egress/portage/package.use LINK: make.conf => ../egress/portage/make.conf LINK: package.accept_keywords => ../egress/portage/package.accept_keywords LINK: repos.conf => ../egress/portage/repos.conf LINK: package.mask => ../egress/portage/package.mask Done.
As you can see, “tux” uses the core and desk profile. There is also one file (60-misc.host) that is host-specific. This particular host also keeps make.conf and repos.conf in the repository, along with a “tex” set, and debug settings in portage/env.
If you want to make changes to your portage configuration, simply open the relevant configuration file in /etc/portage with your favourite editor - no changes in workflow here. Then, run egress pull to sync these changes to the repository:
Pulling configuration files for tux from /etc/egress to repo. Continue? [Y/n] y Pulling profile: core Pulling profile: desk >f.st...... portage/package.use/50-media.desk Pulling host-specific profile .d..t...... portage/ Pulling env. Pulling sets. Pulling savedconfig. Pulling make.conf Pulling repos.conf Done.
From there you can diff, manage and deploy the changes with a VCS:
diff --git a/profiles/desk/portage/package.use/50-media.desk b/profiles/desk/portage/package.use/50-media.desk index e8de9e5..f932b18 100644 --- a/profiles/desk/portage/package.use/50-media.desk +++ b/profiles/desk/portage/package.use/50-media.desk @@ -1,11 +1,10 @@ media-gfx/sxiv savedconfig -media-sound/ncmpcpp outputs taglib +media-sound/ncmpcpp outputs taglib curl media-sound/mpd inotify media-sound/abcde cdparanoia replaygain -aac -vorbis -id3tag -lame -media-sound/audacity libsamplerate ladspa media-video/ffmpeg threads opus openssl -media-video/mpv -dvd -enca -encode -iconv -jpeg -lcms -lua -mpg123 -xscreensaver -xv -luajit +media-video/mpv dvd bluray -enca -encode -iconv -jpeg -lcms -lua -mpg123 -xv -luajit media-fonts/terminus-font -pcf-unicode-only
Using ingress to edit files
If you end up with a lot of configuration files in /etc/portage/package.*, editing and keeping track of specific settings can be quite hard. ingress tries to alleviate this task by providing an interactive interface between your favourite editor and the files in question. Say you want to edit the USE flags for your web browser. Those are defined in /etc/portage/package.use/30-net.desk. Instead of entering the path yourself (or searching for it with one of those nifty fuzzy finders), you can simply use ingress:
1) /etc/portage/package.accept_keywords/30-net.desk 2) /etc/portage/package.use/30-net.core 3) /etc/portage/package.use/30-net.desk 4) [create] #? 3 Editing /etc/portage/package.use/30-net.desk
After selecting the right file, ingress will call sudoedit. This will automatically launch the editor defined in the SUDO_EDITOR, VISUAL or EDITOR environment variables. If you don’t want to use sudoedit, you can disable it in /etc/egress/config; ingress will then call sudo $EDITOR instead.
You can also edit multiple files at once:
Select multiple files. Type 'q' when you're done. 1) /etc/portage/package.accept_keywords/20-dev.core 2) /etc/portage/package.accept_keywords/30-net.desk 3) /etc/portage/package.accept_keywords/50-media.desk 4) /etc/portage/package.accept_keywords/60-misc.core 5) /etc/portage/package.accept_keywords/60-misc.desk 6) /etc/portage/package.accept_keywords/70-games.desk 7) /etc/portage/package.accept_keywords/90-deps.desk 8) [create] #? (multiple) 4 #? (multiple) 5 #? (multiple) q Editing 2 files: /etc/portage/package.accept_keywords/60-misc.core /etc/portage/package.accept_keywords/60-misc.desk
In this example, we did not specify a file name, but a package.* specifier. We can even use both. Here, ingress finds one exact match and edits it right away:
The script can also list files in the portage configuration directory (ingress list unmask), search for files containing a certain pattern (ingress grep vim) and print a “compiled” output of all the settings for one package.* specifier:
# vim:ft=gentoo-package-mask: # -- /etc/portage/package.mask/00-sys.core -- sys-apps/systemd sys-apps/dbus sys-auth/polkit sys-auth/consolekit sys-fs/udev # -- /etc/portage/package.mask/10-x11.core -- # Pulls in a lot of stuff I don't need. x11-misc/xdg-utils
For all of this to work, you don’t even need a special setup. ingress works even if you don’t use egress, as it looks only in /etc/portage/.
One year ago when I started this project it was merely a small wrapper around two rsync operations. Now, after one year of usage and numerous changes (thanks to c_14 for feedback & ideas), I finally feel like releasing a 0.1 version. Please report any bugs on the github page. Feel free to send pull requests and ideas.