To understand this document

This document assumes you are quite familiar with the standard UNIX™ tools make(1) and the shell sh(1). The example commands are as obvious as I could make them; for example, date and hostname are taken for granted.

In the document, examples are blocked in a non-white background, and any shell command you should type is prefixed with a dollar sign ($):

$ echo "Like this"
Like this

If an example is an expanded version of the last, the new text is emphasized as new. Parameters you should replace are always coded as keyword, where the "keyword" is a mnemonic for what you are expected to substitute. All of this markup comes from a common style sheet that I use in all my HTML documents. When you install your copy of the source code you can tweak that style sheet to make it more like your local site's pages.

Automated configuration management

We quote the word same above because in some cases the files and commands sent to the remote host are filtered or processed on the way to compensate for differences between the peer targets, or the targets and the authoritative host.

Getting started with msrc

$ msrc -V
msrc: $Id: msrc.m,v 1.72 2009/09/09 13:50:07 ksb Exp $
msrc:  makefile code: $Id: make.m,v 1.32 2008/11/07 21:31:53 ksb Exp $
msrc: hxmd as: hxmd
msrc: auto MAP suffix: .host
msrc: make recipe file: ...
msrc: make hook: __msrc
msrc: default utility: make
msrc: temporary directory template: mtfcXXXXXX

Now you are ready to try your hand at automating a configuration with the tools and tactics this structure offers. If you've used the older distrib-based structure, we'll talk about conversion at the end of this document.

I'm also assuming that you have at least 2 hosts. 1 to act a the authoritative source, and each other to act as target hosts. Most commands I suggest here are to be run on the authoritative host, if they are to be run on a target host, a secure shell (ssh) command is displayed before the example session.

Setup tasks

Your ssh configuration

Start an ssh-agent and add your keys to it. If you are an admin, think about adding pam_ssh to your global PAM configuration, which starts the agent for you on login.

Upgrade rdist (aka rdist6)

After you build a new rdist and rdistd, you should install them on all the hosts you want to manage. These programs do not have to be in the standard place, in case you can't write there, but the less-buggy versions are required to get recursion to work. If you don't install the environment patch, some commands will prompt you for your pass-phrase (or password), as $SSH_AUTH_SOCK may be the lucky variable dropped from the environment.

If you have rdist version 4 installed on your host and msrc finds that one, it just plain won't work. See below later (not now) to see how to wire-in your path to rdistd and rdist to avoid this.

Build a list of the target hosts

$ vi /tmp/my.cf
target1
target2
...
targetN

I put the file in /tmp because I need an absolute path to it for the session; you can use any path that is easy to type (or put the path in a shell variable).

Optional speed improvement for ssh

You can either turn off X forwarding in your .ssh/config, unset $DISPLAY, or you might install a simple script I call ssh-x:

#!/bin/sh
# $Id: ssh-x,v 1.1 1999/03/06 17:04:42 ksb Exp $
# We need this because rdist won't exec ssh with -x, we must for
# a trip to a host with no [working] "xauth" program. -- ksb
exec /usr/local/bin/ssh -x "$@"

We only need this file installed on the authoritative host, the client host(s) could get it if they need to send data out farther into your network, but we don't talk about recursion or reflection in this quick-start guide.

If you don't want to do that yet, just explicitly unset $DISPLAY in your shell to speed these commands: forwarding your X display for each command takes more time than the command, usually.

$ unset DISPLAY

You are good to go

$ xapply -f 'ssh -n %1 date' /tmp/my.cf

Turn your list of hosts into a real configuration file

Locate programs msrc needs

The configuration file (/tmp/my.cf) we are building can have assignment lines as well as lines with hostnames on them. Each assignment takes effect from the line it is on until either the end of the file or the next line which reassigns the same macro. Some macros are used by msrc as an interface to your process; most are left with no predefined meaning. For the time being, you need to know 4 macros with predefined meanings: RSH_PATH, RDIST_PATH, RDISTD_PATH, and HOST.

You should almost never assign a value to HOST. This is the iterator macro that msrc assigns to help you know the current target host for any update process. You assign a value to HOST by listing hostnames in the configuration file. Let's be more specific in our file; we'll tell msrc that we mean column 1 to be the hostname (which is the default):

$ vi /tmp/my.cf
%HOST
target1
target2
...

We could add additional per-host values by adding more macro names to the header line, which always starts with a percent (%).

At this point, the xapply trick we used to visit each host doesn't work. The issue is xapply doesn't know to skip the header line, so it tries to visit a host named "%HOST". We could use a filter (like grep) to remove the line, but we shant need to soon. We'll eventually use the efmd program to extract data from our configuration files (see below).

Next, you may insert assignments for the program_PATH macros.

$ vi /tmp/my.cf

RSH_PATH="/usr/local/bin/ssh-x"
%HOST
RDISTD_PATH="/opt/admin/bin/rdist6"
target1
target2
...
RDISTD_PATH="/usr/bin/rdist6"
target6
...

$ vi /tmp/my.cf
RDIST_PATH="/usr/local/bin/rdist6 >/dev/null"
...

Remember, even if you didn't install a special version of ssh you must specify the path to it here, as rdist is not cool enough to search $PATH for the path to the transport program option.

Next, test that what you've done so far is sane

$ hxmd -P1 -d M -C /tmp/my.cf 'ssh -n HOST date'
time-stamp from target1
time-stamp from target2
...

If you have a poorly formed configuration file, hxmd outputs a descriptive error message for you. That's because of the -d M specification, in normal operation it is not an error to omit columns. The -P1 specification slows hxmd down to a single thread to make it easier to debug the list. If you have more than 3 hosts, you can run that same command under -P4 to see it go much much faster.

$ hxmd -P4 -d M -C /tmp/my.cf 'ssh -n HOST date'

If you stopped here and just used hxmd to run commands across your hosts, you would be ahead of the game. Command lines tend to get really long for hxmd; while it is a very powerful tool, the weight of the command-lines drive you to put lots of scripts around it. Believe me when I say the better path is to move up to msrc to turn all those little scripts into make recipe files. Really.

And, in the long run, you might want to be able to pull changes to a host; if you depend on hxmd you'll never make that leap. One can argue about pull versus push updates, but it is hard to argue that only being able to do 1 of the 2 is better than being able to do either.

Build your first authoritative source directory

The main purpose of msrc is to automate running the same command with the same data files and parameters on lots of hosts. The real power comes from making contextual changes to those data files and parameters based on the target host's needs.

We'll start with a first directory. I'm assuming that you are not doing this with a class of people all on the same server; if you are replace "first" with your login name and "test1" with your uid or something like that:

$ mkdir /tmp/first
$ cd /tmp/first
$ vi Makefile
INTO=/tmp/test1

all:
	echo `hostname` : `pwd`

Test the make recipe file to be sure you got a tab before the echo command, not 8 spaces:

$ make all
host10.example.com : /tmp/first

Now, lets send that request to every host in our configuration file:

$ msrc -C /tmp/my.cf -- make -s all

rdist output for host4
host4.example.com : /tmp/test1
rdist output for host2
host2.example.com : /tmp/test1
rdist output for host3
host3.example.com : /tmp/test1
rdist output for host1
host1.example.com : /tmp/test1
rdist output for host5
host5.example.com : /tmp/test1
...

If it didn't work, look on the target hosts for missing /tmp/test1 directories; if they never showed up then rdistd is spelled wrong or not installed. Using the -d SPXM options to msrc should help you debug this process later. Now it might only make any confusion worse.

To help you debug, you can apply 2 helpful options: -P1 and -E HOST=host6.example.com. The parallel factor reduces the number of processes to something you can trace, and the selection specification limits the effect to the host that doesn't work. N.B. I'm not including the rdist output in the rest of the example output.

You might also remove the silent option from the make option specification if your issue is with the remote recipe.

What this means

If you packed more files into the authoritative source directory, they would be sent to the remote hosts along with the make recipe file. For example, you could send shell scripts, or source files for programs, or data extracts to compare the state of a host to a known good state.

Trust me, the uses are endless. As an example, let's cleanup after ourselves:

$ msrc -C /tmp/my.cf -- rm -r /tmp/test1

A more stable playing field

To fix little nits like this, you can build a shell script in a common place (like /opt/example/lib/local.defs) that msrc sources into the remote shell before it runs your remote command. I happen to put this file in the same place distrib wanted it, but you should not be encumbered by my mistakes. But remember that the super user might source this file, so put it someplace mortal logins cannot update.

The msrc macro you need to set is ENTRY_DEFS. Before you set that in your configuration file, you should use msrc to push the file you want to every host in your domain (or xapply a ssh command to do the same).

Later, we'll see how to send a different file to some hosts to compensate for vendor or location differences. That is really the whole point of this tool. After you've put an entry defs file out just add the macro assignment to the top of your configuration file:

$ vi /tmp/my.cf
ENTRY_DEFS="/opt/example/lib/entry.defs"
%HOST
host1.example.com ...

Then use

$ msrc -C /tmp/my.cf hostname \; printenv PATH

Note that you don't have to use make as your remote command, but the recipe file is a key part of the data-flow here. We really need it to set INTO -- in the next section we'll use it to set more parameters for msrc.

You know enough to be dangerous

We are going to skip that step for now. For the time being limit the number of hosts in your configuration file to a set of test hosts. After the next section, we'll see how to limit the scope of a spell.

Before getting more power, you should see how to get less. There are more uses for a make recipe file than my master source directory trick, so it would be clever if we could put a ward in a recipe to tell msrc to leave it alone. We do that by setting only the INTO macro to more than a single word.

$ grep INTO Makefile
INTO=_This is not an msrc directory, use the depthcharge tool here.

$ msrc
msrc: Makefile: This is not an msrc directory, use the depthcharge tool here.

$ msrc
msrc: Makefile: "INTO" should have exactly one value, not 11

How to expand your new powers

I believe the UNIX maxim that the strongest solution comes from a set of parts organized into an exact fit for each specific problem. The best of these tool kits are made of parts that always work well with each other, in most any combination. I know this works better than a single large tool you can't describe in terms of `basic parts'. You can see this in the wrapper design of this system, and in my use of m4.

Using msrc effectively requires an understanding of the basic parts and how it puts them together.

What else msrc reads from the recipe file

In the manual page, I describe in some detail, under "MACRO SELECTION", how each macro is defined from the data gathered from make and from the list of files in the current directory; we'll not repeat that here.

What you need to grasp is that you can control the disposition of each file in the directory by listing the filename in 1 of the macros below:

IGNORE -- files that are not sent to the target host

You don't need to send a README or TODO file with notes to a human reader to the remote host, just tell msrc to ignore it.

MAP -- files processed though m4 in-flight

Any file that needs to be macro expanded per host is included in this list. Usually, the file is given a name that ends in .host to tell msrc (and any human reader) that the file is intended to be macro processed.

HXINCLUDE -- an option list for hxmd

This is for later. Files that end in .hxmd are read as options that msrc should include when it builds the hxmd machine to execute the requested commands. We'll use this later to put safety latches on really dangerous spells, and to enable recursion.

SEND -- files sent to the target host as-is

This is what we used to send the recipe file in the example, and it is the default for files that don't have .host or .hxmd on the end of their name. Put any file you want to copy exactly in this list to prevent it from being macro expanded or given to hxmd as options.

To your existing recipe file, add the lines:

$ vi Makefile
INTO=/tmp/test1
MAP=sent2.host

all: ...

Next, create sent2.host with

$ echo "target host HOST" >sent2.host

Then, test the spell with this command:

$ msrc -C /tmp/my.cf -- cat sent2
target host target1
target host target2
...

This doesn't look all that powerful by itself. Soon, we'll see that we can substitute far more than just the hostname -- and that creates a lot more synergy.

Careful processing of files with m4

To make m4 a little more friendly, you should follow these rules:

Always quote text you don't want expanded

Start every file with an open quote (`) and end it with a line that just says end quotes, delete to newline ('dnl).

Distinguish macros from common text

If the target contents is mostly lower-case use upper-case macro names, if the target text is mostly upper-case, do the opposite.

Limit changequote's scope

Always leave quotes the way POSIX says: with the default open and close single quotes for most of your work. When you include a file set the quotes to the default. When you end a file, always end in the default quotes. I almost always end each marked-up file with 'dnl, which helps reinforce this.

Avoid the use of patsubst and regexp

Not every version of m4 supports them, and some that do don't do them correctly.

For example, we should re-quote our sent2 file to only expose the HOST macro to expansion:

$ vi sent2.host
`target host 'HOST`
'dnl

This way, common words like "include" or "divert" don't do crazy things to the payload.

Using make to build files to send

In the output of msrc's -V option, there is a line that says "__msrc" is the "make hook". Note that the spelling includes 2 underscores (_ _) followed by the name of the program. Given that nobody specified a command-line option to change that string (-m prereq), we can assume that's the target msrc is going to update to fetch macro values from our recipe.

By adding a dependency to the make hook, we force an update of that file before msrc starts sending files to any target hosts. For example, let's add a file named now to the recipe:

$ vi Makefile
...
all:
	@echo `hostname` : `pwd`


now:
	date >$@

# hook our files into msrc
__msrc: now

$ msrc -C /tmp/my.cf cat sent2
target host target1
...
$ cat now
Thu Oct 06 12:24:00 CDT 2011

We can use that file from m4 to include a timestamp in the sent2 file.

$ vi sent2.host
`target host 'HOST`
	"now" was last updated on 'include(`now')dnl
`'dnl

Rerun the test twice to see that the timestamps don't change between runs unless we remove the now file. When you do that it forces msrc to rebuild now with the recipe in Makefile.

Leverage in the control recipe file

The best way to do that is to use a make macro, like GEN to hold the name of all the generated files, then install a target clean to remove those files on request. This is a lot better, in that the list of generated files is clear to the human reader, and adding a new file is easier.

Here is our example updated with that tactic:

$ vi Makefile
INTO=/tmp/test1
MAP=sent2.host
GEN= now

all:
	@echo `hostname` : `pwd`

clean: FRC
	rm -f ${GEN}

now:
	date >$@

FRC:

__msrc: ${GEN}

At this point, we can introduce the -d switch to show how all this works. We can run msrc with no target configuration files or hosts and ask it the disposition of each file and macro:

$ msrc -d PX
SEND: [generated]
	Makefile
	now
INTO: [make]
	/tmp/test1
IGNORE: [generated]
	sent2
MAP: [make]
	sent2.host
SUBDIR: [empty]
HXINCLUDE: [empty]
MODE: [generated]
	auto
msrc: masquerade hxmd as our name
msrc: msrc -d PX -DMSRC_MODE=remote -F0 ...

Notice that msrc doesn't extract the value of GEN. It doesn't know about all of the recipe macros, just the ones it needs to drive the distribution process.

Later you can link the clean target into the end of the msrc processing in a parallel manner.

Link to the local revision control facility

$ vi Makefile
INTO=/tmp/test1
MAP=sent2.host
GEN= now
SOURCE=Makefile send2.host

all:
	@echo `hostname` : `pwd`

clean: FRC
	rm -f ${GEN}

now:
	date >$@

source: ${SOURCE} ${GEN}

${SOURCE}:
	co -q $@

FRC:

__msrc: source

A few control recipe pointers

And the whole point is to make tough problems easier to solve.

Assure that each target in GEN may be built in any order (use make's depends to create any prerequisites, don't just depend on the order in GEN). Use the command make source to check our work, and make clean to remove the temporary files. I didn't put a command in to do any other revision control operations: those are for you to work out under your site policy.

The name FRC is short for "force" and is a common convention to `force' a target to always happen. Never make a file called FRC. This is just 1 reason why make's -t option is evil.

Why 2 recipe files are better than 1

There is a clever way to move the logic for the target host into another file. If we create a Makefile.host file in this directory, we can use the mapping feature to replace the control recipe file output from m4 run over that file. In effect, we'll ignore the control recipe file (Makefile) and send the remote recipe file (Makefile.host) as the new recipe file on the target hosts.

Let's build a sample:

$ vi Makefile.host
`# remote recipe file

all:
	sed -e "s/^/'HOST`: /" now
'dnl

To fix the control recipe (in Makefile), we must remove the definition of MAP because msrc builds the correct one (which now includes Makefile.host).

$ vi Makefile
INTO=/tmp/test1
# MAP from msrc
GEN= now

all:
	echo "update with msrc" 1>&2
...

Check the disposition of the files with msrc -d XP:

$ msrc -d XP
...
SEND: [generated]
	Makefile
	now
INTO: [make]
	/tmp/test1
IGNORE: [generated]
	sent2
	Makefile
MAP: [generated]
	sent2.host
	Makefile.host
SUBDIR: [empty]
...

Now let's test the new split control/payload recipe files:

$ msrc -C /tmp/my.cf make all
sed -e "s/^/target1.example.com: /" now
target1.example.com: Thu Oct 06 12:24:00 CDT 2011
sed -e "s/^/target2.example.com: /" now
target2.example.com: Thu Oct 06 12:24:00 CDT 2011
...

Next, we can load Makefile.host with any logic we need on the target host without mixing in any of the control logic from the Makefile. Add whatever operational targets you need to the file and any additional scripts or data files you might need to muster to the control directory. Often, helper scripts allow the target recipe file to be much simpler, and easier to markup under m4.

Another way to split the control from the target recipe, without the m4 markup, is to name the control recipe file Msrc.mk and the target logic Makefile. I don't usually do that, as I use the markup in the target recipe file in almost every spell I cast. I think you can work out an example on your own. The clue is that msrc prefers Msrc.mk to all other control recipe names.

A few target recipe pointers

'changequote([,])dnl
[Q='
G=`
]changequote(`,')dnl
`...

action:
	sed -e ${Q}s/^[a-z]*[0-9]*\\.//${Q} ...

You can use mk's marked lines in either the control or target recipe files to embed complex commands that you don't want to type over and over. See the HTML documents on mk. For example, a marked line in the control recipe might select the correct configuration file:

$ mk -mAuto Msrc.mk
	msrc -C /opt/example/lib/my.cf -- make all

Less obvious uses of an entry definitions file

But there is another use for that hook. You can use it to turn off any commands run by msrc: just put an exit command in the file. When the update script sources the file (with .) on the target host, the remote shell terminates. To make it clear to the driver, you should send a message to stderr before you bail-out, and exit with EX_UNAVAILABLE (usually 69) or some code that means something under your site policy.

While we are on the topic of remote host access, we should mention that ssh's authorized_keys file can limit access, but that's not the right way to limit msrc's actions. Use an unprivileged login and op to limit what the delivery login can do. See the op HTML docs.

Limit your impact with hxmd specifications

In our /tmp/my.cf, we should add something useful about each target host. An example might be the location (data center, room, rack) or the serial number, or the OS currently loaded on it, or the list of applications that it is expected to run. Less useful would be information we should get from DNS (the IP, mail routing, host info, or any reverse IP mapping) since we can use a program to gather that information with the control recipe file.

Think about that, then read more about hxmd's host specification options in it's HTML documents. Recall while you are reading that (down to "How do we process each node?"), that we are going to do some examples here -- so just get the main ideas.

Use more than 1 configuration file

When the segmentation of the nodes means something to the people running operations it helps a lot. If it is just a fiction some bureaucrat made up, it will cause more problems than it solves. Keep the groupings large enough to matter, and simple enough to describe to a new hire.

It also helps if they are useful in more than a single context. And we don't need an example, as we always specify -C in each command. (You could change that with configuration in /usr/local/lib/msrc/msrc.hxmd.)

Use host selection options

Think about some simple command-line rules you might want to apply. For example, if a host has a list of services it provides, you might want to select all the hosts that provide "dns". Given that we added SERVICES to as an attribute for hosts with services as an underscore separated list (viz. "service1_service2_service3"), we can code an m4 guard macro "HAS_SERVICE" in the configuration as

$ vi /tmp/my.cf
# guard to check for a given service (use under -G)
HAS_SERVICE=`ifelse(-1,index(`_'SERVICES`_',`_'$1`_'),,HOST)'
...
%HOST			SERVICES
target1.example.com	one_three
target2.example.com	one_two_three
...

To select just "target2.example.com", use a selection guard like:

$ hxmd -C/tmp/my.cf -G "HAS_SERVICE(two)" 'echo HOST'

$ msrc -C/tmp/my.cf -G "HAS_SERVICE(two)" -- make all

Notice that the HAS_SERVICE definition doesn't know anything about the service list except that it is separated with underscores (_). That means you can use it for every service as long as your site policy doesn't force an underscore into a service name. If you need to change the separator, I might move to exclamation mark (!), but never comma or a shell meta character, as they are error prone.

Location, location, location

I would put it under /usr/msrc/etc/resolv.conf. I put the generator in a directory named for the target location so I can always find it.

To me, that also implies a value of INTO. Since msrc changes the string "/msrc/" into "/src/" to get /usr/src/etc/resolv.conf. Even though this conflicts with some system source files (which most production hosts don't install anyway), I would still use it. If you like this policy, you don't have to set INTO; otherwise, you should always set INTO (as I do). Then, you can force your destination directories under /usr/src/local/ or some other name-space that only the local wise guys know and love.

I hard code INTO and set it with a -y yoke on the command-line when I push to my own directory; thereby avoiding any issues with msrc picking the target directory, since home directory paths are not always identical across hosts (or could be NFS mounted and would race on update). Pushing to a system account (like the superuser) with a relative INTO creates junk under slash or /root. And pushing to a machine where your home directory is not under the same absolute path usually fails.

If you want to work in your home directory, you can build both a src and msrc directory. By putting your master structures under your own home you should be pretty safe, as long as you always work as yourself on both sides. If you need to become another login on the target host you should use op (or sudo).

The pull version of msrc

For any `pull system' we need to fetch a directory structure to the target host with something like rsync or with a remote filesystem mount (aka over NFS), or with a processor that builds our specific target directory. Then, request a "local update" with msrc's -l option, or with the boot-strap program mmsrc, or by execution of the update directory created a priori.

In any case, this only works if you plan carefully for it in every spell.

Requisite planning polled updates

First we need to be able to build all the requisite files on any target. That also implies that when you can't, each control recipe fails (exit's non-zero); otherwise, a local update process will continue with garbage data or missing files.

That clause assumes lots of things, among them:

The target host's name has a common spelling

That is to say, the target host's name in the configuration file is a name we can spell without some extra knowledge about the file.

The authoritative host doesn't have special knowledge (viz. data, facts)

When either the host itself or some special login on the host is granted special access to requisite data (viz. via a firewall rule, permissions to a local data repository or some other special privilege escalation rule), then the target host with a copy of the spell is not going to be able to instance it.

The spell we are using is self-contained

Relative references to sibling directories will break, unless we clone that much more of the filesystem. This is a problem because msrc explicitly allows references to paths outside of the source directory to send files to the target host with the /./ hack or the similar ../ hack.

Common tools are installed on every host

When the tools used to build the configured version of the spell are specially installed on the authoritative host then the target hosts don't have the gear they need.

Second, you won't have the configuration files on the target host, unless you've installed them and kept them up-to-date (or projected them to the target host with the directory). Assuming you know what name the authoritative host called this host, you still need to know which configuration files to specify; the spell can encode them in some hxmd options file (see a reference in the main page). This can actually be the hardest part of the process and is one of the main uses of the -o option and the HXINCLUDE macro.

Lastly, you won't get to use ssh to change logins. If you depend on an authorized_keys file to allow a substitute user in the process, you'll be sad. That stops my spells a lot, and is easy to work around with op by putting "op make ..." as the command, rather than just "make ...". To be more explicit, you might need "op -u login make ...". Consider the op rule below, which allows anyone in groups "source", "root" or "wheel" to run make as any login that is a member of group "source":

make	/usr/bin/make $* ;
	groups=^wheel$,^root$,^source$
	%u@g=^source$
	uid=%u initgroups=%u
	environment=.*
	# Set a umask here
	# Limit $* here, as needed

Even with all those issues, the news is not so dire: we can get simple structures to work in 3 ways, described in the next 3 sections.

Pull the authoritative directory to the target host then use msrc -l

To use our test directory locally:

$ ssh target4
target4$ rsync -arSH control:/tmp/first /tmp/shadow
target4$ rsync -a control:/tmp/my.cf /tmp/shadow/my.cf
target4$ cd /tmp/shadow
target4$ msrc -l -C ./my.cf -E HOST=target4.example.com -- make all
target4$ # leave for the next example

target4$ msrc -l -C ./my.cf -E HOST=target4.example.com -- /bin/sh -c '$SHELL -i </dev/tty >/dev/tty'
target4% pwd
/tmp/mtfcyo5AL9/lsEJyWFc
target4% exit
target4$ ls -als /tmp/mtfcyo5AL9/lsEJyWFc
ls: /tmp/mtfcyo5AL9/lsEJyWFc: No such file or directory
target4$ exit

This interactive shell trick is also quite useful for debugging very complex configurations. You'll need it someday, I'm sure. Add the -d SPXMR switches to show how it works once, at least. In that shell look at the output of ptree, then feel the awe.

Pull the sources and use mmsrc

We could use mkdtemp and -y to force the local file cache into a random, but fixed location. Or if we can write to the default INTO, we can just use that. But how would we know the value of INTO? It turns out that mmsrc has code in it to consult a recipe file for the value of a macro.

$ ssh target5
target5$ rsync -arSH control:/tmp/first /tmp/shadow
target5$ rsync -a control:/tmp/my.cf /tmp/shadow/my.cf
target5$ cd /tmp/shadow
target5$ mmsrc -b INTO
/tmp/test1
target5$ # leave for the next step

If we may write to the directory INTO specifies, we are good to go:

target5$ pwd
/tmp/shadow
target5$ mmsrc -C ./my.cf -E HOST=target5.example.com -- make all

If we may not write to that directory, we can force a yoke option to change the directory to a temporary space we build:

target5$ pwd
/tmp/shadow
target5$ TD=`mktemp -d /tmp/$$pickXXXXXX`
target5$ mmsrc -C ./my.cf -y INTO=$TD -E HOST=target5.example.com -- make all
target5$ rm -rf $TD
target5$ exit

If we had to build the directory, we should also remove it (the last command above). In the next section we can use that fact to our advantage.

This is handy to boot-strap the system because it doesn't require any trust relationship via ssh that we might not have (or be able to generate) on any arbitrary host. It also has a clever feature wherein the HOST macro is specially processed to allow the specification of that on the command-line to try to pick the right name from the configuration files.

This is exactly what mpull does, see the HTML documentation.

Create custom sources on the authoritative host as a per-client service

We could package the newly created directory for a given target host, either on request from the host, or as part of another move to production system. Assume that a request for a directory came from a client (via some network service or the other). We can build a directory for the client, fill it as above, then send the client an archive of the customized files.

In this example, I use rsync to send the new directory back to the client ($CLIENT):

$ cd /tmp/thing
$ : ${RD:=`mmsrc -b INTO`}
$ TD=`mktemp -d /tmp/$$pushXXXXXX`
$ mmsrc -C /tmp/my.cf -y INTO=$TD -E HOST=$CLIENT -- :
$ rsync -arSH $TD/ $CLIENT:$RD
$ rm -rf $TD

The shell spell to set the remote directory above is phrased the way I meant it; the colon command is one of the more useful shell built-in commands, even if few people understand why.

This is exactly what the msrcmux RFC 1078 service does. See the HTML documentation.

The last option is way too obvious

To make it slightly more secure, you could offer a private key that was only good for a short time, then revoke the key (or offer the key over a forwarded ssh-agent). This effectively reverses the sense of the trust relationship, as the client provides temporary access to itself, and the authoritative allows some access from every target host.

But that's too easy.

How to describe msrc to your management.

When we had a few servers, we kept them all up-to-date by coping files from one to the other by hand. We did a lot of hand edits to those files, and sometimes lost important changes.

Later on we got smart and started pushing files from the main file server to the others; that way we got the latest version on each host, not just what ever was installed on the last machine we thought we updated. We built scripts to run our update process in a for loop for each host (or used apply). That was a fine way to make all our hosts look exactly the same, but it didn't really automate files that were unique per host.

Then, we built helper scripts to automate that configuration process, and started running them on a schedule. Sometimes, we even pulled a script from the server with rsync to get the right version to run. That also kept the client host in control of the update clock, which was both good and bad.

We found that that wasn't working for complex tasks, so we pushed more complex scripts over to each remote host to build the correct files. But each process was unique and nobody knows how all of them work. When Bill left, we realized that he was still doing most of his work by hand, so we had to figure out how to do his stuff all over again.

We tried a single common data source for some parts of the system (viz. NFS, NIS, RADIUS, and the like), but they are not solving the issues for local applications and the really hard parts. And we ran into some security issues, as you know.

None of these solutions gave us a common context that everyone could use to get the work done.

We just tried ksb's msrc structure, and we think we can use it to automate lots of our configuration and build processes in a common way we can explain to everyone and teach to new hires. It's kind of technical, but everyone should be able to run every process we automate without having to learn the internal details of each one.

Some of us will have to codify some "site policy" that we all follow, and get buy-in from everyone. We'll have to do that anyway to be SOX and PCI-DSS compliant, so it is not a bad thing, really.

Process engineering rather than patches on patches

In terms of payload, we often need more than just a set of common files; most of the time we need some specific information about the target host. We use this information to produce the correct configuration for that host before we even get there to do the installation or update recipe. This separation makes the process easier to understand and control, so when we do need to gather specifics on the target host, we can apply msrc again, just plain old make, or any local automation that gets the job done. Having a structure to get us started doesn't limit us -- it frees us to do the more complex design tasks without messing endlessly with the common details.

Processes engineered with msrc are automated, repeatable, faster, and scale better than ad hoc implementations. They can be used in a push or pull direction with some care and planning. They shouldn't encode hostnames or IP addresses in the logic of the process, but may in the resulting configuration files.

Follow up reading

Another missing article here is "reflection". This is when a seed host slaves a pool to help push a large number of updates out to many hosts in parallel. For that to work perfectly, you need the next tool-chain (which is not released yet).

But I do offer some other follow-up items below.

Conversion from distrib-based master source

Before you convert a directory you need to check for advanced uses of the @file@ notation in the Distfile. If more than just 1 reference to Make.host (the old name for Makefile.host) exists you have to fix it in a more detailed way, possibly with a local mode spell, which is not covered in this document (see the main page).

In each `plain' directory, you have to do these steps:

$ cd /usr/src/local/...
$ rcsclean Distfile Make.host Makefile
$ mv -i RCS/Make.host,v RCS/Makefile.host,v
$ co -u Makefile.host
$ co -l Makefile
$ TFILE=/tmp/tmp$$.$((RANDOM%100))
cat <<\! >$TFILE
# The next file in the editor is the Makefile for this directory
# make it look like a master source version 2008 control Makefile.
# msrc support
INTO=/usr/src/local/...

GEN=
SOURCE=	Makefile Makefile.host ....

clean: FRC
	rm -f ${GEN}

source: ${SOURCE} ${GEN}

${SOURCE}:
	co -q $@

FRC:

# msrc patch to get up-to-date and checked-out sources
__msrc: source
!
$ vi $TFILE Makefile
$ echo "test the conversion and 'ci -u Makefile'"

That works most of the time. If it works for you, and site policy allows, you can remove the unused revision control file for Distfile.

Extract text data from configuration files via efmd

Use it in-place of any hxmd invocation that starts with echo for a huge gain in speed. See the HTML document on it for details.

Only unique elements

In the context of msrc, we sometimes must created a list of hosts to process that includes duplicates, but we only really want to process each host (or file, or task) once. The oue filter starts working on tasks immediately, where sort -u waits to read the entire list before sending any data to the next process in the pipeline. This may create a long pause before any tasks start.

Using oue rather than sort -u is a huge win for a long running process engineered under msrc when the processing (or start-up) includes a unique filter requirement. Also, the code to generate the list of tasks can usually be made simpler if it doesn't have to keep track of the candidates it has already produced.

Thanks for reading all this

$Id: qstart.html,v 1.35 2012/06/28 17:37:37 ksb Exp $