The Party Line

For reasons which I am not going to delve into here (this is a topic for another post), we are going to get rid of about half of our books.

There are some (low) hundreds of books for the taking, slightly more than half in English, the rest being mostly Russian with a sprinkling of Danish here and there.

Fiction, non-fiction, textbooks, science fiction, you name it.

So, if you are in Copenhagen area and are interested, write me a note and consider coming over to have a look, maybe you’ll find something you’d like to keep. All books are to be had for free, although we would not mind selling them if you will insist.

Often I want to know how long it took for a particular command to finish.

An obvious solution to use the time(1) command does not work without a degree of anticipation on my part that I do not normally posess.

At some point I became sufficiently annoyed to actually add some hooks to my .zshrc. All commands executed in an iteractive shell are timed, but the reporting is done only for those that took longer than 10 seconds to execute.

This ugly code does the job:

note_remind=0
note_ignore="yes"
note_command="?"

note_report()
{
    echo ""
    echo "note_report: $note_command completed in $1 seconds"
}

preexec()
{
    if [ "x$TTY" != "x" ]; then
        note_remind="$SECONDS"
        note_ignore=""
        note_command="$2"
    fi
}

precmd()
{
    local xx
    if [ "x$TTY" != "x" ]; then
        if [ "x$note_ignore" = "x" ]; then
            note_ignore="yes"
            xx=$(($SECONDS-$note_remind))
            if [ $xx -gt 10 ]; then
                if [ $TTYIDLE -gt 10 ]; then
                    note_report $xx
                fi
            fi
        fi
    fi
}

Enjoy.

Me and my wife are going to Paris in April. So, some time ago we have ordered tickets from a popular Danish travel site, rejsefeber.dk.

The tickets are for 2009-04-18.

This morning I have got a shiny (HTML) email from them which said the following (loosely translated from Danish):

Bon Vojage.

Hello Anton.

In a moment you will be traveling to Paris! There are often many things that should be taken care of before the journey, and we would like to help you with practical details. We hope that you will find in this mail something that will make your journey even better. Have a good journey!

After realizing that today is 2009-03-18, I had my moment of panic, frantically searching for the PDF with the electronic ticket, verifying that I have not made a major fuckup and that our tickets are indeed for April the 18th, not March the 18th.

So the fuckup is not mine. Fine. But it would be nice to be absolutely really positively sure, so I called their customer service. The robot helpfully told me that

• this is a paid call (I don’t remember the exact amount per minute, but it was not peanuts);
• I am number 10 in the queue.

And there is no contact E-mail on the website.

So what do you think - is it a simple, albeit embarrasing programming error on behalf of rejsefeber.dk programmers, or a secret plot to get some more money out of customers induced into a state of panic?

At any rate, I do not think I will be using their services again. If I need some excitement in my life, there are better ways to obtain it.

Some years ago I’ve made a little web application which allowed one to browse FreeBSD ports collection by tags, à la delicious.

The tags were not created by users but were instead generated from a couple of fields taken from every port’s Makefile, so it was not exactly a “social” software.

There was some limited amount of discussion on FreeBSD mailing lists, and a publicly accessible readonly SVN repository was created by my friend Erwin, but the overall interest was rather low.

Over time I moved on and basically stopped working on the project, but recently I had an idea - not exactly to re-surrect it, but to make it more easy for people who are interested to contribute.

Enter port-tags at github. Github is a tool to host git repositories of your open-source projects. Anybody can easily clone your repository, fork it completely, or submit their changes back to you. I only started using it today, so I cannot say much about its features and how convenient they are, but from what I’ve heard, it is very very nice.

So, if you are interested, and have got round tuits to spare, please hack on port-tags - maybe some good will eventually come out of it.

Let’s suppose that you have a huge logfile and would like to quickly extract lines from it that relate to a given small time interval. How would you do it?

Since the lines are ordered by time specification, the fastest way (provided you do not keep indexes of any sort) is to do the old good binary search, doing all necessary housekeeping to account for line boundaries and converting the timestamp from whatever format it is in the logfile to epoch seconds for comparison with the target interval boundaries.

Since you are dealing with Perl here, it would be natural to first look on CPAN for a module which somebody else has already written to do just this.

And of course somebody has. Enter File::SortedSeek by Dr. James Freeman. The module interface is a bit weird, so it pays off to read the documentation carefully.

At any rate, here is a complete program that handles the task, assuming that the timestamp (in pretty much any format) is at the beginning of each line of the logfile:

#! /usr/bin/perl
use strict;
use warnings;
use Getopt::Long;
use File::SortedSeek;
use Time::ParseDate;

my ($from, $to);
usage() unless GetOptions("from=s" => \$from, "to=s" => \$to);
usage() unless @ARGV == 1;
$from = parsedate($from) if $from;
$to   = parsedate($to)   if $to;

my $filename = shift;

open L, "< $filename" or die "unable to open $filename: $!\n";
File::SortedSeek::set_silent(1);

my $end = File::SortedSeek::numeric(*L, $to, \&time2sec)   if $to;
my $beg = File::SortedSeek::numeric(*L, $from, \&time2sec) if $from;
$end ||= 0;  $beg ||= 0;
while (<L>) {
    print;
    $beg += length($_);
    last if $end && $beg > $end;
}

sub usage
{
    print STDERR <<EOF;
usage:
\t$0 --from date-time [--to date-time] filename
\t$0 -f date-time [-t date-time] filename
EOF
    exit 1;
}

sub time2sec
{
    my $line  = shift;
    return undef unless defined $line;
    my $r = parsedate($line, FUZZY => 1);
    $r;
}

Nifty, eh?

Model: HP 2510p.

We have:

The docking station covers about 90% of the outlet and some substantial part of the inlet.

Make of this what you wish.

There is a number of very good, but not very well known Perl modules on CPAN.

Sometimes I’ll be writing short posts about such modules which I use and appreciate.

When you are dealing with date and time in Perl, inevitably you will reach a point when you need to do more than is immediately available through Perl builtins and the POSIX module.

Then you try to find a module for what you want on CPAN, and you drown in literally hundreds of modules dealing with dates and times.

Luckily, there is a clear winner in this “modules war” - everybody (or at least everybody sane) recommends to use the DateTime module, and for the things that it cannot do, various other modules from the same namespace.

So life is bright for a perl programmer on the date/time front, until you have a need to parse a date represented in one of a multitude of “human-readable” formats, and you don’t know in advance which one it is going to be.

The DateTime itself cleverly refuses to deal with this task at all, and instead recommends to use one of the DateTime::Format:: modules.

You will be relieved to know that you can easily and quickly create parsers for your own date formats - that is, if you are able to remember that you should use the module aptly named DateTime::Format::Builder::Parser::Regex.

The documentation for DateTime::Format::Bork is also very enlightening.

Aaaaanyway.

I prefer to go against the flow here, and use a module somewhat unfortunately named Time::ParseDate. I mean, it could just as easily be Date::ParseTime or something, right? Worse, for years I had trouble remembering what distribution this modules comes from (it, very obviously for everyone but me, can be found in the Time-modules distribution).

At any rate, if we forget for a second about the funny names, this module is truly a wonder:

$ perl -MTime::ParseDate -le 'print parsedate("Sat Feb 14 00:31:30 2009")'
1234567890
$ perl -MTime::ParseDate -le 'print parsedate("2 days ago")'
1236443283
$ perl -MTime::ParseDate -le 'print parsedate("18:30")'
1236619800

It exports a single function, which takes a single parameter (unless you want to specify some options which are rarely needed in practice), and you get your epoch seconds back in return. Very simple, very elegant, gets the job done. I wish there were more “straight to the point” modules like this one.

There is a number of very good, but not very well known Perl modules on CPAN.

Sometimes I’ll be writing short posts about such modules which I use and appreciate.

There is a common task of executing a script from cron periodically, subject to the following conditions:

• a script can occasionally run a relatively long time (longer than the interval at which it is launched by cron);
• such long runs do not happen often;
• running two or more instances of the script at the same time will lead to all sorts of strange things happening and must be avoided;
• skipping a single run is no big deal.

A usual method to prevent strange things from happening is to use a lockfile, like in this example:

use strict;
use warnings;

use Fcntl qw(:DEFAULT :flock);

sysopen(L, "/var/run/myprocess.lock", O_WRONLY | O_CREAT)
    or die "cannot open lockfile: $!";
flock(L, LOCK_EX | LOCK_NB)
    or die "cannot obtain lock: $!";

# ... do work here ...

unlink "/var/run/myprocess.lock"; # optional
close(L);

(You might want to silently exit when flock fails in order to not spam yourself with useless cron mails).

Nothing fancy, really. But over time it becomes boring to write all this housekeeping code in every little cronjob, especially since in some cases the “do work here” part can be comparable in size to the locking part.

The solution, of course, is to use the CPAN magic and to find a module which sweeps all this complexity under the rug, leaving us with a clean and simple interface, so that we can concentrate on getting the job done.

As usual with CPAN, there is not one, but several modules which were written to perform this task. Most of them are rather powerful, which is unfortunate, since we want simplicity of use above all else. The bells and whistles provided by those modules might be needed in certain situations, but for the purpose described above they just get in the way.

There is, however, a wonderfully simple (and a rather clever) module by Elizabeth Mattijsen, Sys::RunAlone. To get the same functionality as the code above, all you need to do is this:

use strict;
use warnings;
use Sys::RunAlone;

# ... do work here ...

__END__

That’s it. Nothing else to write.

There are only two minor things to remember about this module, if you want to avoid problems.

First, it uses the script’s DATA handle to do the locking (that is, it actually uses the script’s file itself). So if you have several symlinks pointing to the same script, you cannot run them at the same time for it is still one physical file and one DATA handle.

Second, and for the same reason, if you modify the script while it is running and then launch it again, it will fail to detect that another instance is already running, since the DATA handle will be different.

Just keep this in mind when you use it.

Last time I ranted about one thing that is wrong with rrdtool.

It turns out that I, unfortunately, have more gripes to share. Let’s talk today about the legend control.

The rrdtool’s language for controlling how the graph is going to look and feel is pretty rich and complex. Among other things, it has a bunch of graphing commands that are able to manipulate the legend section of the graph. Some of those commands do nothing else, while the other commands only modify the legend as a side effect, and primarily exist for doing something else.

Let’s take as an example the graph below:

rrdtool graph x.png
  --end 1235800000 --start end-150000
  'DEF:data=some.rrd:data:AVERAGE'
  'CDEF:sine=data,POP,1235800000,TIME,-,10000,/,SIN'
  'LINE1:sine#0000ff:a sine of foo\l'
  'CDEF:cosine=data,POP,1235800000,TIME,-,5000,/,COS'
  'LINE1:cosine#00ff00:a cosine of bar'

The trouble is that there is no instruction which can produce a so called “legend box” (a small blue box next to the “a sine of foo” legend text) without doing something else in the process. All of the commands that actually graph something can optionally produce a legend box, but there is no other way to make it.

If we extend our sine example with some nice downtime background (in Soviet Russia sines can have a downtime!), we get a nice picture like this:

rrdtool graph x.png
  --end 1235800000 --start end-150000
  'DEF:data=some.rrd:data:AVERAGE'
  'CDEF:downtime=data,0,0,IF,
    TIME,1235769489,GE,TIME,1235779489,LE,*,
    TIME,1235689489,GE,TIME,1235729489,LE,*,+,+'
  'CDEF:sine=data,POP,1235800000,TIME,-,10000,/,SIN'
  'LINE1:sine#0000ff:a sine of foo\l'
  'CDEF:cosine=data,POP,1235800000,TIME,-,5000,/,COS'
  'LINE1:cosine#00ff00:a cosine of bar\l'
  'TICK:downtime#ffe0e0:1.0:downtime'

But let’s suppose that we’d also like to add background coloring for uptime, and indicate that in a legend. Let’s use another CDEF for uptime, which will be just the opposite of our downtime CDEF:

rrdtool graph x.png
  --end 1235800000 --start end-150000
  'DEF:data=some.rrd:data:AVERAGE'
  'CDEF:downtime=data,0,0,IF,
    TIME,1235769489,GE,TIME,1235779489,LE,*,
    TIME,1235689489,GE,TIME,1235729489,LE,*,+,+'
  'CDEF:uptime=downtime,0,1,IF'
  'CDEF:sine=data,POP,1235800000,TIME,-,10000,/,SIN'
  'LINE1:sine#0000ff:a sine of foo\l'
  'CDEF:cosine=data,POP,1235800000,TIME,-,5000,/,COS'
  'LINE1:cosine#00ff00:a cosine of bar\l'
  'TICK:uptime#e0ffe0:1.0:uptime'
  'TEXTALIGN:left'
  'TICK:downtime#ffe0e0:1.0:downtime'

Oops. Of course. The graphing commands are processed in sequence, so the uptime “background” has obscured our actual sinusoidal data! We will have to put the uptime TICK before anything else to avoid this:

rrdtool graph x.png
  --end 1235800000 --start end-150000
  'DEF:data=some.rrd:data:AVERAGE'
  'CDEF:downtime=data,0,0,IF,
    TIME,1235769489,GE,TIME,1235779489,LE,*,
    TIME,1235689489,GE,TIME,1235729489,LE,*,+,+'
  'CDEF:uptime=downtime,0,1,IF'
  'CDEF:sine=data,POP,1235800000,TIME,-,10000,/,SIN'
  'TICK:uptime#e0ffe0:1.0:uptime'
  'LINE1:sine#0000ff:a sine of foo\l'
  'CDEF:cosine=data,POP,1235800000,TIME,-,5000,/,COS'
  'LINE1:cosine#00ff00:a cosine of bar\l'
  'TICK:downtime#ffe0e0:1.0:downtime'

See the problem? Now the legend boxes are not in the right order.

Naturally, it is pretty trivial to solve - we just remove the legend (it is optional) from the uptime TICK, and add another graphing command in the end with the right legend text. We’ll simply have to make sure that this “graphing command” does not actually graph anything:

rrdtool graph x.png
  --end 1235800000 --start end-150000
  'DEF:data=some.rrd:data:AVERAGE'
  'CDEF:downtime=data,0,0,IF,
    TIME,1235769489,GE,TIME,1235779489,LE,*,
    TIME,1235689489,GE,TIME,1235729489,LE,*,+,+'
  'CDEF:uptime=downtime,0,1,IF'
  'CDEF:sine=data,POP,1235800000,TIME,-,10000,/,SIN'
  'TICK:uptime#e0ffe0:1.0'
  'LINE1:sine#0000ff:a sine of foo\l'
  'CDEF:cosine=data,POP,1235800000,TIME,-,5000,/,COS'
  'LINE1:cosine#00ff00:a cosine of bar\l'
  'TICK:uptime#e0ffe0:0.0:uptime'
  'TEXTALIGN:left'
  'TICK:downtime#ffe0e0:1.0:downtime'

Just like the last time - the problem is easily solvable, but it annoys me, since rrdtool again forces us to work against it instead of working with us. It would be nicer to have a way to put a legend box where we want it in some more direct way (maybe a variant of the COMMENT command)?

Last time I spread some rrdtool love around. Love with no hate won’t do, however. Taking a page from the way the BBC and many other news outlets cover certain conflicts, it seems prudent to present an unbiased view of rrdtool. And what better way to do that if not by spreading some hate this time?

Some time ago I had a task of adding background coloring to the portions of a graph, representing downtime of a device. This information was not available in the RRD archive itself, but could be obtained from elsewhere.

The information was in the form of epoch ranges, like this:

1235639489-1235646689,1235725889-1235729489

After some thought, I decided that making a CDEF with some simple RPN magic, in combination with a TICK graphing command would do the trick. The TICK command draws a vertical line of a specified height when a data point is not a zero or an UNKNOWN value, so I naïvely thought that this simple CDEF will achieve what I want (one long line here is split into several lines for your browsing convenience):

CDEF:downtime=0,TIME,1235639489,GE,
    TIME,1235646689,LE,*,
    TIME,1235725889,GE,
    TIME,1235729489,LE,*,+,+

The boolean values in rddtool’s RPN are represented using the usual C-like convention (0 = false, 1 = true), so for any given time point the result of the CDEF above will be 0 if the point is outside any of the downtime intervals, 1 if it is inside one of the intervals, and more than 1 if it is inside several intervals (which should not happen, but is possible if we specify overlapping downtime epoch ranges).

If you did not understand how this is supposed to work, go read the rrdgraph_rpn manpage.

Anyway, that was easy enough, even if it was not very clean. So where is the promised hate, you’ll ask? Here it is:

rrdtool graph x.png --end 1235800000 --start end-150000
    'DEF:data=some.rrd:someds:AVERAGE'
    'CDEF:downtime=0,TIME,1235639489,GE,TIME,1235646689,LE,*,
     TIME,1235725889,GE,TIME,1235729489,LE,*,+,+'
    'TICK:downtime#ffe0e0:1.0'

ERROR: rpn expressions without DEF or CDEF variables are not supported

You’ve read that right! Rrdtool insists that everything you put into a graph must ultimately come from some data source in some RRD archive. How stupid is that?

And the answer is - very stupid indeed, for at least two reasons.

First, it tries to pretend it knows what I might want to do better than me, and I really hate that in software. Whatever has happened to the idea that a tool must not get in the way of its user?

Secondly, this check is easily circumvented by adding a real DEF somewhere in the CDEF and then (mis)using RPN language to get rid of any value which comes from the DEF. There are many ways to do it, some more creative than others, but even this ham-fisted “solution” will do the job:

CDEF:downtime=data,0,0,IF,TIME,1235639489,GE,
    TIME,1235646689,LE,*,
    TIME,1235725889,GE,
    TIME,1235729489,LE,*,+,+

So we have a pointless restriction, and we are forced to work around it. Some might say that working against the tool to reach the goal is somehow challenging and maybe even fun. If you ask me, this is akin to programming in Brainfuck - a kind of activity better performed as a hobby, if that’s your idea of how to spend your copious free time.

To rephrase the Perl’s motto, rrdtool makes simple things hard.