To understand this document

This document assumes you are quite familiar with the standard UNIX™ tools from section 1 date, sed, tr, cat, and are very familiar with the shell itself, that is sh(1).

In addition, you need a working understanding of the common I/O file descriptors, and how they are manipulated by the shell (stdin, stdout, stderr). And you should be able to quote parameter lists using double-quotes or single-quotes -- and know when to use each.

What is xclate?

The xclate wrapper collates multiple output streams such that the output from each task is complete before any other's output begins. This does not mean that the program's execution is sequenced -- just the output. The tasks may run in parallel or even in reverse order, but the output is never mixed together.

Intermingled results from multiple parallel processes create confusion. For example, the error output from "cat foo" and "make bar" mixed-up might look like: 

make: cat: don't know how to make foo:  bar.  Stop
No such file or directory

After working on large scale parallel processing for a time, I noticed that the main restriction I had left was the serialization of output. I had to write the output from all the processes to individual files, then cat those to assemble a report I could understand. While all that was going on, I couldn't even examine the output from the finished tasks.

I decided to code a solution that would show me the output from the finished tasks as soon as they terminated. The best way I knew how to do it was with a wrapper.

Building the wrapper

Briefly, a wrapper is a two-part program: a part that holds a socket open to service clients, and a part that services clients. The difference between this and a typical "client/server" model is that there are usually many "servers" running on a host for different logins, programs, and functions all at the same time -- and there may by many clients connecting to each of those in parallel.

The through-put provided to 1 instance is not bound in any way to the actions of another (given sufficient resources). For example, peer wrappers to not share the same listen(2) queue, or the even the same socket options or accept filters. This reduces 1 bottle-neck in the client/server model, and removes some of the need for "well known ports".

To speed the implementation more, we use local-domain (unix-domain) sockets. This removes the overhead of TCP/IP for communications, making inter-process communications exactly as fast as a pipe. There is a lot more information about wrappers in the wrapw documentation, which I've not release yet.

Using the wrapper

The master instance of the wrapper and the client mode are almost always the same program, and xclate is no exception. Select the master instance with the -m option; the client mode is the default case.

The usage for a master instance of xclate is:

xclate -m [-dQrsv] [-depth] [-H hr] [-i input] [-L cap] [-N notify] [-O output] [-T title] [-u unix] [-W widow] [utility]

That shows that the command "xclate -m" is a legal invocation of the program, and it is. When you run that command, a new shell is created which has access to the new master process's resources. Here is an example showing the shell history number resets to 1 for the new shell, the process tree, and the socket that that instance opened (which I found with the version output from xclate): 

wrkstn /home/ksb 85  xclate -m
wrkstn /home/ksb 1  ptree $$
117   /usr/sbin/sshd
  8758  sshd: ksb@ttyp2
    8761  -ksh
      99919 xclate -m
        99920 ksh
          99984 ptree 99920

wrkstn /home/ksb 2  xclate -V
xclate: $Id: xclate.m,v 2.53 2008/09/11 17:55:36 ksb ...
xclate: environment prefix "xcl"
xclate: environment tags: "link", "list", "d", "1"
xclate: protocol version 0.7
xclate: safe directory template: xclXXXXXX
xclate:  1 /tmp/xcloDQ2vJ/1 [target]

wrkstn /home/ksb 3  ls -lals /tmp/xcloDQ2vJ/1
0 srwxr-xr-x  1 ksb  wheel  0 Sep 20 15:20 /tmp/xcloDQ2vJ/1=

wrkstn /home/ksb 4  exit
wrkstn /home/sac1/ksb 86  ...

While an interactive shell is a possible utility, it is probably not the best choice. It a lot easier to use in a script, and an interactive shell doesn't usually require the services on an xclate master instance. Usually, the structure built for an xclate master instance is a script or a tool that knows about xclate.

But we will come back to the shell example to show some of the features off later. For the time being, we need to know how to start a client program. The options for a client instance look like:

xclate [-Qsvw] [-IEDY] [-depth] [-H hr] [-L cap] [-T title] [-u unix] [xid] [client]

There are 3 important points in this option specification:

[xid] [client]
Each client specifies 2 positional parameters: the xid name and the client process. It is all too common the forget the xid; I do it all the time.
it is not shown that master instance must enclose a client instance
A master instance must enclose a client instance, like any other wrapper. But unlike other wrappers the primary use-case is via xapply which knows how to create both the master instance and the client instances, see the xapply HTML document.
[-Qsvw] [-IEDY]
There are 2 different bundles of single letter switches. This is meant to indicate that the second switches turn on other features inside the program, and may subtly change the behavior of the client. In effect providing any of -I, -E, -D, or -Y implies a major change in how xclate processes the client request.

It would be a lot of typing, but one could collate output with xclate from a command prompt. It is much more likely to be part of a shell script, or even designed into a large-scale parallel application. That's why it is an integral part of xapply and hxmd, see the hxmd HTML document.

How useful is that?

All this talk about collated output doesn't look like a super feature by itself. To see where this would be helpful, think about a large process tree, with many parallel workers, all of which might want access to a single output-stream. For this example, let's say the output goes to a interactive less.

How would we present the output to the reader? Which task should output first (first started or first finished)? How shall we label the output "parts" so they will be recognizable sections for the reader. We might also like to handle nested groupings (as chapters in a book, with sections in chapters, or even paragraphs in those sections).

Output which is not properly directed to a collated sections should be diverted to the end, or trapped someplace. And it would be pretty neat if we could keep a running tally of the sections still open, or finished.

Xclate provides those features, and a few more.

Integration with ksb tools

Since I coded xclate it stands to reason that I'd use it. Oh, and I do!

With xapply

The key application that uses this is xapply, which drives many parallel tasks, and might want the output from each task to be continuous. Another example would be a "make world" on a FreeBSD host. If we could put a "|xclate $component" on the end of each major component, we could make the output of the build a lot easier to read, while making it much more parallel.

In fact, xclate must be installed before xapply is built. On a related note, hxmd uses options to xclate directly, so it won't run without xclate either.

The shell fragment below starts a collated stream over an xapply with 3 tasks in parallel. Each task stalls a bit between the output of a sequence of lines, for a total of about 6 seconds (1+2+1+2+0). We ask for 9 iterations ('first' ... 'last'). That makes the whole xapply command take about 18 seconds (9*6 / 3), but the output is not intermixed. 

$ time xclate -m xapply -P3 'for i in 1 2 1 2 0 ; \
	 do date +"%1 %%c"; sleep $i; done |xclate %1' \
		first t1 t2 t3 t4 t5 t6 t7 last
... (output of the data line) ...
   19.09s real     0.25s user     0.46s system

The extra second overhead is the time for xapply to fork all the processes and build all the pipes, and also a small constant term due to code that compensates for a race condition in a select(2) loop.

A shorthand for this is: 

$ time xapply -m -P3 'for i in 1 2 1 2 0 ; \
	do date +"%1 %%c"; sleep $i; done' \
		first t1 t2 t3 t4 t5 t6 t7 last
...
   19.08s real     0.21s user     0.48s system
which almost completely hides the xclate usage from the script writer. That code is what makes xapply a wrapper: it wraps itself automatically in an xclate.

That shorthand does limit the command-line options which can be passed directly to xclate. For example, to pass the '-T "start %x"' option to the xclate-filter, we must leverage an environment variable, like this (see the xclate manual page under ENVIRONMENT): 

$ XCLATE_1='-T "start %x"' ; export XCLATE_1
$ time xapply ...

With hxmd and msrc

Since hxmd uses xapply and always passes the -m switch down, it also uses xclate. The collated stream under hxmd is really useful, as hxmd output without collation is extremely jumbled.

This is such a benefit that there is no command-line option to turn off the use of xclate. Now think about how much I like command-line options and wonder "What's gotten into him?"

The msrc remote build tool uses hxmd to build the wrapper machine it needs, so it uses xclate in exactly the same way as hxmd.

With ptbw

The parallel token broker (ptbw) is used to limit the number of parallel tasks inside a collated stream, but not really used to produce collated output. But it has a subtle link to xclate, none the less.

Any ptbw instance nests its UNIX domain communications socket inside the tightest enclosing xclate directory. This is an attempt to reduce the load on the filesystem be reducing the number of meta updates for the creation and deletion of temporary directories. Also see the ptbw HTML document.

With distrib

Quite often, calls to distrib were wrapped in an xapply -m, which is where hxmd came from.

Hxmd builds the full xapply stack with detailed knowledge and intent. It does a better job of constructing the stack than any shell programmed could by creating pipes that the shell cannot (easily) express. Well, ksh can with a co-process. 

$Id: xclate.html,v 2.22 2012/03/21 16:15:04 ksb Exp $