The pundits tool-chain from the NPC Guild (

These tools are the ones ksb uses to keep his hosts up-to-date, to monitor configurations, resources, and track usage and escalation. Once you have them installed you'll miss them on every other host. I promise. See the quick start page for some information and examples.

Installing these tools is a minor pain, the first time, then it gets really easy. That's because one of the key aspects of the tool-chain is that they are used to build, install and upgrade themselves on peer hosts. The whole purpose of these tools is to help you manage hundreds or thousands of hosts with as few as 5 people (24x7) or 3 people for business hours. We do this by sending the correct configuration to hosts (via msrc and install), by letting mortal users manage their own processes (via op and installus), and with proactive close-the-loop tests (like hostlint).

To get started you should download the level2s script and the current msrc_base-2.29.tgz and install_base-8.30.tgz into /tmp on your favorite test host. These 2 base packages and the helper script let you boot-strap the tools on a single host. From there you can follow the quick-start HTML document to learn the chains and quickly install the same tools on all your hosts. At least that's the idea.

See the README file in msrc_base for more help at this point.

On a RedHat system with rpmbuild installed (and build-essentials) you should then su to the superuser (so you can write in /usr/src/redhat). From /tmp run:

./level2s rpm msrc_base-2.29.tgz
lots of output

That should build and install all the msrc tools (xapply, xclate, ptbw, hxmd, msrc, mmsrc). And leave you with an RPM to install on any like hosts (ls /usr/src/redhat/RPMS/*/msrc_base*). If it broke for want of some tools like "bison" you need to install the prereqs first.

On other platforms you need six lines of shell code to get it going: They are in green text in the sample output from a session where I boot-strap this (Ubuntu 8.04) host. Here are the 6 lines I used running as myself:

mkdir /tmp/slash /tmp/_src
EXPLODE="-I /tmp/slash/usr/local/lib/explode"
MKCMD="-I /tmp/slash/usr/local/lib/mkcmd:/tmp/slash/usr/local/lib/mkcmd/type"
cd msrc_base-2.29
make HOSTTYPE=LINUX HOSTOS=80400 INTO=/tmp/ksb_src DESTDIR=/tmp/slash boot
# then just to check that it worked:
mkcmd -V
explode -V
msrc -V
mmsrc -V

If that looks OK to you then you can install them into the real system (not /tmp/slash) as root:

make HOSTTYPE=LINUX HOSTOS=80400 INTO=/tmp/ksb_src DESTDIR=" boot

Alternatively there is a Makefile target if you don't have (can't find) the level2s script, or you don't want to build this as the superuser.

tar zxf msrc_base-2.29.tgz msrc_base-2.29/Makefile msrc_base-2.29/ITO.spec
make -f msrc_base/Makefile rpm-boot
rm -rf msrc_base-2.29
(then you must add the RPM files geneated as root, so that really doesn't help much).

Online documentation

If you have the unpacked source for any of these packages you can run this command (as root) to install the manual pages for each tool:
 find . -name \*.man -print | xapply -fx 'mk -mInstall' -

level2s build script

version: level2s-4.40
   12 -rw-r--r-- 1 ksb ksb 10343 2009-10-19 12:20 level2s-4.40.tgz
MD5 561e6a7e1057f81c1ab92b3ce2c43e83
level2s changes (from 4.39)

msrc 2010, Release 29 Production 14 Aug 2010

File version: msrc_base-2.29.tgz which includes mmsrc, xapply, xclate, ptbw, hxmd, and msrc (optionally the dmz and posse scripts):
  1240 -rw-r--r-- 1 ksb ksb 1264778 2010-08-14 12:15 dl/msrc_base-2.29.tgz
MD5 bc122c13f107d7fa29d2cae7b3c9639e msrc_base-2.29.tgz
msrc_base-2.29 changes (from 2.28) msrc_base-2.28 changes (from 2.25)

The next package we need to install is install_base

File version: install_base-8.30.tgz includes install, installus, instck, mk, op, purge, and vinst:
 372 -rw-r--r-- 1 ksb ksb 373663 2010-08-13 13:00 dl/2010/install_base-8.30.tgz
MD5 db6f9c8af12785368ee43c03e0a3888e  dl/2010/install_base-8.30.tgz
This package has the configuration management tools we use to manage hosts. Everything needs them that comes after. Build the rpm with level2s like you did for msrc_base. (Shown above.) That installs all the file update tools (local/bin/install, purge, instck, installus, vinst, mk, op). And leave you with an RPM to install on any other hosts you might have:
./level2s rpm install_base-8.30.tgz
lots of output

If you don't have RPM support you can build the package with mmsrc: Since you have msrc installed use:

cd /tmp/ksb
tar zxf install_base-8.30.tgz
cd install_base-8.26
mmsrc -C/usr/local/lib/hxmd/ -y INTO=/tmp/_src/install-x make install

If you have just installed msrc_base in /tmp/slash for testing you can put install_base in there too:

cd /tmp/ksb
tar zxf install_base-8.30.tgz
EXPLODE="-I /tmp/slash/usr/local/lib/explode"
MKCMD="-I /tmp/slash/usr/local/lib/mkcmd:/tmp/slash/usr/local/lib/mkcmd/type"
cd install_base-8.30
mmsrc -DHOST=`hostname` -C/tmp/slash/usr/local/lib/hxmd/ -y INTO=/tmp/_src DESTDIR=/tmp/slash make install clean

If that went OK you should be able to run:

op -V
to see op's version. But the -l output should tell you that no rules are installed in "". You'll have to build your own rule-base. There is a prototype in the top level of the install_base package. If you didn't get op installed it is because you didn't run the install as the superuser (and you didn't get installus either).

Changes from install_base 8.26 to 8.28 (29 Mar 2010)

As far as I know there are no open issues with any elements of this package. I have a pty addition to op(1) that I may never export.


At this point you need to install some packages that I use, if you haven't installed them:
For any tcpmux services (which I use a lot)
perl-ExtUtils-MakeMaker and perl-devel and h2phperl
pdksh or ksh (or the crazy zsh link might work)
We expect perl-5.8 or better, for scripts and PEG samplers.
libpcap and libpcap-devel

The plain level2 packages should be installed in about this order

Assuming you installed the to base packages you may install these tools Do them the same way, with "./level2s rpm package", or by unpacking each compressed tar file, then running mmsrc in the newly created directory:
mmsrc -DHOST=`hostname`  make install
Note that is tries to put the platform source in /usr/src/local, if that's a problem use this instead:
msrc -l -E HOST=`hostname`  make install
Which specifies a mkdtemp directory for the platform source. To install PEG support plus hostlint, netlint:

File Entombing version 3.11

File entomb_base-3.11.tgz includes preend, unrm, entomb, libtomb.a, untmp, and rmfile
 96 -rw-r--r-- 1 ksb ksb 91225 2010-08-16 09:27 dl/2010/entomb_base-3.11.tgz
MD5 3e07da241354489db0b08fb1aac12929  dl/2010/entomb_base-3.11.tgz
This package has the basis for installing file recovery for mortals that accidentally remove a file they wanted. This might save you from running restore 7 days a week. You can add entombing on FreeBSD with this spell:

The console server (my version)

The guys at have done a great job on there version, but I still don't like programs that reimplement users and groups. So I use mine.

See the INSTALL file in the archive. Most people don't have trouble with this one.

Spares from before others included a version

Don't install these, unless you really need them: (viz. don't have one in the base system).

These tar archives all assume a login with the uid 810. This convention makes it easier to build them as a non-root login (later). For now you can ignore that feature. Later you can make a login ("source") with that uid and use "op" to build programs as that user, as not to let the superuser run a recipe to destroy you build host.

View the unpacked sources

See the top level directory or a [dead] mirror on my cable moden.
This file is located at