To understand this document

The master source system is based on many levers. Each lever allows small changes that cascade through the structure to create required output in the features beyond, which in turn, mayhap, will create more ripples. Since automating the correct levers solves configuration management issue much more quickly that picking the wrong levers, we present an overview of the key levers in this document.

Points of leverage in msrc

This document explains and indexes some of the subtle points of leverage available in the master source tool-chain.

Msrc's program name and $MSRC_LIB

The first leverage point for any of my tools is the name the program is passed in argv[0], and msrc is no exception. Msrc looks for a file based on the name it was called under $MSRC_LIB named basename.hxmd. This is could beused to set global options for msrc to pass to hxmd. But it almost never is actually instanced, so the default installation doesn't even build the empty support directory. It is almost always used by shell wrappers that have their own library directory (which they specify as below).

For example, if we set $MSRC_LIB to a directory we can write in (it defaults to /usr/local/lib/msrc), then create a file msrc.hxmd in that directory, which adds options to all the enclosed hxmd instances. In this example I'll force an uptime command after the last host is updated: 

$ mkdir /tmp/ksb
$ export MSRC_LIB=/tmp/ksb
$ echo "-K 'uptime'" >$MSRC_LIB/msrc.hxmd
$ cd /usr/msrc/local/bin/oue
$ msrc -Csite.cf -G `hostname` :
 ...
sulaco.example.com: updating of sulaco.example.com finished
10:41AM  up 144 days, 17:50, 55 users, load averages: 0.08, 0.02, 0.01
$ unset MSRC_LIB

Note that the existence of any HXINCLUDE file from the controlling make recipe will suppress the path search for a default file. (Thus this trick doesn't work in the source to msrc itself.)

So that gives you 2 intercept points: the value of $MSRC_LIB and the name you provide to msrc. The contents of the file provide many hooks, as any option to hxmd could be included. You might change the program name via a symbolic link, or in some shells you can explicitly set it (viz. zsh).

Environment variables in the argv[0].hxmd file

The code that installs those hxmd options may read run-time values from the environment. This is sometimes used to pass a command or configuration file down to cache processes. 
# Install the Fuze hook
-D
$FuzeMe
Then pass the define you want set via the variable hook you installed: 
$ FuzeMe="INIT_CMD=uptime" msrc ...
This tactic allows comments in the file, use them to describe what you are actually intending. And where the value is set in the environment. Also note that other hxmd options, like -j and -C allow other features to be remote controlled.

That lever allows the injection of m4 defines at the hxmd level, additional options and configuration files, as well as xapply and xclate option injection.

On the other-hand it fails when the environment is not set. It is best to present use this from a structure that sets a default value for all required variables. Also you should include the current (or default) value of MSRC_LIB on the end of your value to allow nested application of the hook.

M4 Hooks

These are covered in detail in the msrc manual page. But we should explain here why one might change some of them, and how.

The hooks generally come from attribute macros defined in any of the configuration files specified under -C, -X, or -Z or by an explicit definition under -D.

HXMD_PHASE and -j
Use -j to include a file of m4 markup, then use the fixed values of HXMD_PHASE to unpack only the macros you need for the context you are in. This is best done by include'ing a file detailing what you need, rather than by nested ifelse logic. See the description in the hxmd HTML page.
SSH
This is almost always set to the path to a script that runs ssh with options to suppress X11 forwarding. Sometimes other performance options are enabled (e.g -c might specify arcfour). But it could also be used to log out-bound requests, for example.
ENTRY_DEFS
This is a great hook. If you want to keep the authoritative host from pushing (updating) a host, you can either change the path to a file that just contains an exit command, or update the default file to contain a (conditional) exit. Since the file is sourced into the remote shell, any command the terminates the shell (or renders it lame, like setting -n) renders the update powerless. This is often used to "freeze" a machine for testing or critical production.
Putting a set -x in the file traces the update actions at the shell level.
INCLUDE_CMD(mode)
This often overlooked hook allows a complete rewrite of the remote update script. If you don't like my version, just insert your own, then exit before mine starts (or divert(-1)).
m4wrap
Catch the end of the update script with this m4 built-in. This allows some clever hooks, for example, to remove remote files with an ssh back the target host.

Make macros

The best hook for any of these is to set the macro to a command prefixed with a semicolon (;). That forces the output of the command to be the value msrc uses a prefix of "echo", followed by the macro it wants, and a shell markup to redirect the output to an already open file descriptor (like 1>&7): 
echo ${MAP} 1>&7
So if we set MAP like this: 
# pick a file at random to send today
MAP= ; random-file *.host
Then make runs the shell command: 
echo ; random-file *.host 1>&7
Since the stdout of the make process is /dev/null the blank line is ignored, and the output from the random-file command is read as the value from MAP.

A few macros make recipe macro hooks.

INTO=_multi-word message
This is clearly documented as a way to prevent a package build recipe from being mistaken for an msrc control recipe. But it might also output other hints about how to use the recipe.
INTO=.
Setting this to dot (.) forces the remote target directory to the home directory of the remote login.
SEND
Adding files that end in .host to this forces them to be send without m4 processing, which means they can be processed on the remote host.
MAP
Force a directory into the list to use it as a Cache directory. Cache directories are explained well in several places.

Clever use of options to hxmd

Some options are best used as a lever for other options. For example the include path option (-I) might change the path to a file specified under -j or one included by a MAP'd file's m4 include markup. Changing from a "test" include directory to a "production" one might change many included files by cascaded inclusion. This limits mismatched part-test/part-production files from mistakenly being built (or deployed). A special recipe is intentionally created to make any chimeras.

It is possible to use -j to provide a FIFO to m4, which will be opened once for each host processed. This allows a few hooks (for example rate-limits, unique content per-host), but doesn't really tell you which host is getting which content. Also note that the -j files are used for guard processes and cleanup work, so you'd have to handle those as well (one might use HXMD_PHASE). Never be tempted to reverse-engineer the target host via lsof, use a cache directory for such things. The cache recipe knows a lot more context, so it is clearly safer.

Prerequisite or MAP'd file

The home directory pull spell needs a perl script the deletes the first line from a non-text file. This script is generated by a very short command: i.e. s2p -e 1d.

There are 2 ways we could construct that script:

with a recipe __mmsrc prerequisite
Me might code a recipe like: 
GEN= 1d

...
1d:
	s2p -e 1d >$@
...
source: ${SOURCE} ${GEN}

# msrc hooks below
__msrc: source
with a MAP file 1d.host
Which addes 1 line (at most) to the recipe file: 
MAP=1d.host
to which we must add a file 1d.host that looks like: 
dnl $Id: ...
syscmd(`s2p -e 1d')`
'dnl

The difference between these two is subtle, and may lead to unintended service failures.

In the first case the recipe file needs to update a file in the master directory before it provides a push or pull service. That might be impossible for the effective uid/gid of the update process. For example the msrcmux service may run as an unprivileged user.

With the MAP the s2p is run once for each target host, but the output is sent to a temporary file owned by the invoker, which always works. The only likely failures would be permissions on $TMPDIR or a full filesystem.

Compilation of code

One of the best uses for msrc is the compilation and deployment of compiled code (C or the like). Many of the levers and hooks you need to make compile-time tunes are easy with this layering.

Layer 2 product builds

The layer 2 package specification file (ITO.spec) comments (via mk)
These comments allow you to set several options for level2s. See manpage for level2s(8l) for details.
The layer 2 package control recipe (Msrc.mk, or Makefile) comments (via mk)
The comments of a control recipe may contain mk markup to automate common usage of the spells contained within. This is largely an issue of local site policy: I can't tell you what operations you'll want to automate -- but I can tell how to do it. See mk(1l), and the mk HTML document.
The layer 2 package control recipe's macros and targets
This might set an HXINCLUDE to change its behavior. It used the __msrc hook to force prerequisites into the master-side build process. It can use a __cleanup target to cleanup any local cached data as well.
Any macros msrc fills in for you
If you add files to a directory msrc picks which macro gets the new file. The . and +++ markup allows you to pick which macro gets unclaimed resources. See the HTML explanation.
Any cache directories built per-host
These could have anything you can build with make presented to any following step. There is nearly unlimited power in this step. For example reaching out to each target host to recover some current-state to mix that into the new state.
For example we could base the new resolv.conf on the current one. Or automation might check ntpdc's sysinfo for a target host before we update ntpd.conf.
The platform recipe (Makefile.host) while being processed by m4
The Makefile.host file might force some -DHAVE_feature or -DNEED_resource macros based on HOSTTYPE or other values in site.cf. for each host.
The Makefile.host file might include library options, or add/replace source files using m4 logic.
Another common meme is to add -D'HOSTTYPE` to the C compiler flags. This allows cpp markup to leverage the definition of the host's type. Nearly all ksb's tools use this rather than auto-configuration, for good or bad.
The platform recipe (having become Makefile) on the remote host
The recipe passes down all compiler options produced by the m4 markup above (viz. -D'HOSTTYPE` becomes -DDARWIN). This clues machine.h to set the apropos default values any HAVE_feature macros (or the like).
Dependencies included by m4 markup that force a command to build emulation code on the target host (see below).
Any autoconf spell to make config.h
This and machine.h are the only places where the host type or operating system version is directly examined, on the target host. By converting the host type into a set of property macros we may consolidate to more common code. Few of my tools use autoconfig. (Presently only mmsrc, to get us started and scdpn just to show I know how.)
Any machine.h or config.h file sets default property macros.
A machine.h file looks for any already defined HAVE_feature macros and any clues set in common include files to refine the picture of the target host. This usually only sets more macros. Sometimes it forced the inclusion of specific #include files.
Below #include "machine.h" in a C source file
We may conditionally include a header file (<strings.h> vs <string.h>). These are often conditional on a property macro set by machine.h or autoconf's config.h.
Any external declarations for any emulation code might be #if'd in based on any property macros set above.
Next a section of optional data declarations (e.g. struct fsent FSNew;) might be #if'd into the file. These might be declared as a macro type to unify the code even more.
In the C source files (e.g. main.c)
At the actual usage of the facility we call through a macro or through some emulation code to unify the alternatives as much as possible.
In another file (machine.c) any platform specific emulation code is instanced.
This allows the consolidation of the emulation code to a single file, or a few files. Some of the emulation code is pulled from the explode repository, so we can use it over and over again (for example the emulation for strcasecmp or strlcpy).
How we get the explode spell
We might pull generic emulation code from explode's repository with a trigger from the Makefile.host m4 logic, or by creating an explicit dependency on the emulation file in the recipe via m4 markup. (When building the platform recipe.)
Using mkcmd to reuse code and options
At this point mkcmd does path searches for executables we need to get us back to the shell level at run-time.
We also use mkcmd to incorporate options from other programs. For example mmsrc imports very large portions of hxmd and msrc into itself.
Load against local libraries
When emulation code is so common that you need it for many applications just build a shared library and load against it.
Like every other configuration management task this is a loop. Always refine your local base of libraries, options, explode and mkcmd modules, machine.h logic, autoconf parts, recipe files, cache directories, and rpm spec files until you have the correct balance between efficiency and robustness/fragility.

The best lever you have is the shell

Since msrc and all the related tools are largely configured with shell code and hxmd format files (which have be generated with shell or perl code quite easily), the best friend you have is your shell skills. You don't have to learn another specially-built language to use this structure.

Learning to use m4 to markup files is required to customize sendmail, autoconfig, syslog and many facilities.

The make recipe driver is used by countless packages.

Lastly the columnar configuration file format required by hxmd (and therefore msrc), efmd, and distrib is not hard to use; while is expresses detailed information about a large number of instances in a format that is both human and machine friendly. (I find it much easier than some crazy XML scheme.)

Return to the main page. 

$Id: levers.html,v 1.5 2012/07/03 22:57:54 ksb Exp $