What I learned trying to log wall

soda and the wall

Sometime around 1993 I discovered through one channel or another that a certain multiuser machine at Berkeley, soda.csua.berkeley.edu, was creating daily logs of all the messages written to all terminals via the "wall" (write all) command. You could retrieve the day's log using the old finger protocol: finger [email protected]. Soda was populated by what seemed (to me at the time, anyway) to be a talkative cabal of wise and clever UNIX hackers, and I learned a fair amount by scrolling through a daily dump of their banter. Times have changed, ports have been closed, and there's little trace of the soda cabal online today (and what little there is hasn't aged well).

When tilde launched, I nearly fell out of my chair. A shared multiuser unix box? With hundreds of users? Online? Using wall? I logged in and watched the messages scroll by. All pouring into /dev/null for all the good it was going to do me. I had to bring back wall logging.

wall: it's so easy

wall is probably one of the simplest utilities in the UNIX toolbox. You can find the source of a widely used version in the util-linux package (apt-get source util-linux if you want to take a look). It's about 8K, most of it licensing and comments. The copyright is '88-'93, and the last modification-- native language support-- dates to 1999.

All wall does is read text from standard input (or a file), open up every terminal on the system that has a user logged into it, and write the text out to that terminal. (All mesg y and mesg n do is change the group write permission on your terminal; it will just silently fail to write to your terminal if it doesn't have sufficient permissions.) It can do this because it has the setguid permission for the tty group; this means that when wall is run by any user, it acts as if that user was a member of the tty group. Since every terminal on the system belongs to the tty group, wall has the power to write to every terminal that has the group write permissions bit set (which is set by default, unless you're run mesg n).

There's nothing particularly secret about wall messages-- quite the opposite. So there shouldn't be any reason I can't, as a regular user with no special permissions, create a small program that listens to the messages wall is sending out and log them to a file.

No particular reason except, y'know. UNIX.

utmp: the infernal census

The plan was simple. I'd create a process that attached to a new psuedoterminal (and unless you've just plugged in a teletype or acoustic coupler you're attached to a pseudoterminal) and dump everything that showed up into a log file. The only trick was to convince wall that there was a user logged into that psuedoterminal. And that's where it gets fishy.

Wall doesn't scan all the pseudoterminals on the system looking for users. Instead, it uses a file called /var/run/utmp. Somewhere beneath the green sod lies the rattling skull of the druid who remembers why this file was called utmp, but we will leave them to their fecund peace and move on.

utmp is basically an inventory of who is currently logged into the system, where they're logged in from, and what terminal they're attached to. This information is cleanly formatted as a series of comma-seperated values, one entry per line. Ha, no, just kidding, it's a miserable platform-specific blob of struct utmps that you need to access via a series of decreasingly documented function calls. If you really want to stew in the unholy juices, I recommend starting with a hearty man utmp at your local command line. Highlights include:

Warning: utmp must not be writable by the user class "other", because many system programs (foolishly) depend on its integrity. You risk faked system logfiles and modifications of system files if you leave utmp writable to any user other than the owner and group owner of the file.

and

Linux utmp entries conform neither to v7/BSD nor to System V; they are a mix of the two.

and

This man page is based on the libc5 one, things may work differently now.

Thanks. Good to know.

It turns out that there are standard library functions for reading and updating the utmp file: getutent, getutline, pututline, setutent, and the like. You can read the man pages if you like. If you do, they'll remind you that

New applications should use the POSIX.1-specified "utmpx" versions of these functions; see CONFORMING TO.

which means you actually want to use getutxent, getutxline, pututxline, etc. instead. Of course, these aren't reentrant so you'll want to use the thread-safe variant if you're doing multithreaded code. The man page even helpfully provides a code example of how to add a line to utmp (using the non-x, non-reentrant versions of the functions of course). But wait! Wouldn't you rather just make a single call to login_tty instead? Go ahead and read the man page. Does it do what we need? Maybe!

But this is all useless bullshit, because we never had the ability to add an entry to utmp in the first place. You'll need to have write access to /var/run/utmp first, and for that, you'll need to either be root or a member of the utmp group. And you're neither.

Meet the gatekeepers

So who does update utmp? Let's start with the obvious one: the login program. It gets to write to utmp because it's normally run as root. In fact, if you go ahead and type login at your terminal right now, it will tell you

login: Cannot possibly work without effective root

"Oh, no, I couldn't possibly. What were you thinking?" But of course you're not always using login. Sometimes you'll use ssh to log in remotely. And again, /usr/sbin/sshd runs as root. So it appears that ordinary user processes just can't register themselves in the utmp. That would be good and fair, except for one minor detail: users do it all the time.

Take xterm. When you create a new xterm, a new pseudoterminal is created and a new line is added to utmp. But xterm is an ordinary user process without any setuid or setguid permissions. How does it manage to modify utmp? Well, it uses a library called utempter. It's kind of ingenuous to call utempter a library, though, because it consists of two parts: a traditional-looking shared library, and a standalone utility /usr/lib/utempter/utempter/utempter that, yes, has the setguid permission that allows it to run as a member of the utmp group. Every time you launch xterm, it makes a call into utempter library that then launches the /usr/lib/utempter/utempter utility with a command-line parameter or two (add hostname or del respectively) to add or remove a utmp entry.

(And yes, by the way, if you've read the utmp manpage you'll see that it describes in detail how xterm generates a utmp entry, and yes, by the way, it's entirely wrong.)

So I built a little test program against utempter, and hooray! Success! There was only one minor problem: utempter isn't installed on tilde.net. Why should it be? Tilde doesn't have xterm.

Luckily there are other terminal emulators in the sea. Let's look at gnome-terminal. Hey, it doesn't use utempter! What does it use? It uses libvte, which is a library that includes a binary called /usr/lib/libvte/gnome-pty-helper, which... you see where this is going. Again, this isn't installed on a headless server like tilde, nor should it be.

Is that... good?

From one perspective, this is good news. If you rely on utmp for any layer of your security or reporting, you shouldn't allow user processes to just go modifying it willy-nilly. The bad news is that if you've installed any one of a number of fairly standard packages user processes can go modifying utmp willy-nilly, so you absolutely shouldn't be relying on utmp for any layer of your security or reporting.

The biggest problem here is that it's really difficult to describe what utmp is for, or, hell, even what its name means. Sure, it's used by w and who and a host of other utilities, but the documentation is all over the place and frequently wrong. It's a crazily ambiguous way to indicate who is using your system. What does that even mean? If I have logged out but still have a process running, am I using the system? What if that process basically grants me a shell? Or what if I'm just using an expect script at the other end of my login? As the sages ask, what doth life?

So now what?

Well, we have a few options.

  1. We can ask the system administrator to grant our binary setgid privileges for the utmp group
  2. We can ask the system administrator to install utempter or another "workaround" package
  3. We can remotely run a process that logs in through ssh and monitors the connection to collect walls
  4. We can give up
  5. new We can ask someone else who's already got it running

Options 1 and 2 are essentially grovelling. Option 3 is fairly straightforward, but not really very interesting and definitely leans towards cheating. Option 4 is clearly the most attractive, as it lets me stop typing and get on with my day. Option 5 is ongoing.

back to ~phooky