Ignat Ignatov talked about physical formulas. When I was planning to attend this talk, I thought it is going to be some sort of symbolic formulas computation, possibly with an analysis of dimensions of the physical quantities.
However, despite my (a bit long in the tooth) background in physics, I did not understand a word of it. Apparently, some sort of unification of physical formulas, not entirely unlike the periodic table in chemistry, was presented, with almost no comprehensible details and with scary words like co-homology and algebraic topology. The fact that half of the slides were in Russian, while irrelevant for me personally, probably did not help matters for the majority of the people in the audience. I did not expect any questions at the end of the talk, but there were at least two, so I was probably wrong about general level of understanding in the audience.

Laurent Dami talked about SQL::Abstract::FromQuery. He presented a query form of the Request Tracker and said that it is too complex - a premise many would agree with. The conclusion was that some more natural way to allow the user to specify complex queries is needed. Surprizingly, the answer to that was to use a formal grammar and make the user adhere to it. To me this sounds weird, but if one can find a non-empty set of users that would tolerate this, it may just work.

Denis Banovic talked about Docker, a virtualization container. I did not know much about Docker until this point, so it was useful to have someone to explain it to me.

The next talk was long, 50 minutes (as opposed to a somewhat standard for this conference 20 minutes) Peter "ribasushi" Rabbitson presented a crash-course in SQL syntax and concepts. It looked like a beginner-level introduction to SQL, but it became better and better as it progressed. I even learned a thing or two myself. ribasushi has a way of explaining rather complicated things concisely, understandably, and memorizably at the same time. Excellent talk.

Then there was a customary Subway sandwiches lunch.

Naim Shafiyev talked about network infrastructure automatization. Since this is closely related to what I do at my day job, I paid considerable attention to what he had to say. I did not hear anything new, but hopefuly the rest of the audience found the talk more useful. It did inspire me to submit a lightning talk though.

osfameron talked about immutable data structures in Perl and how to clone them with modifications, while making sure that the code does not look too ugly. Pretty standard stuff for functional languages, but pretty unusual in the land of Perl. The presentation was lively, with a lot of funny pictures and Donald duck examples.

The coffee break was followed by another session of lightning talks, preceeded by a give-away of a number of free books for the first-time YAPC attendees. Among the talks I remembered were SQLite virtual tables support in Perl by Laurent Dami, web-based database table editor by Simun Kodzoman, LeoNerd's presentation about XMPP replacement called Matrix, a Turing-complete (even if obfuscated) templating system by Jean-Baptiste Mazon of Sophia (sp!), and annoucements of Nordic Perl Workshop 2014 (Helsinki, November) and Nordic Perl Workshop 2015 (Oslo, May).

Again, I did not go to the end-of-the-day keynote.

As a side note, the wireless seemed to be substantially more flaky than yesterday, which has affected at least some lightning talk presenters.

Posted Sat Aug 23 18:00:00 2014

When I came to the venue 15 minutes before the official start of the registration, people at the registration desk were busily cutting sheets of paper into attendees' badges. Finding my badge turned out to be a tad not trivial.

This conference is somewhat unusual not only because it is conducted over the weekend instead of in the middle of the week, but also because the keynotes for every day are pushed till the end, even after the daily lightning talks session.

The welcome talk from Marian was about practical things such as rooms locations, dinner, lunches, transportations and so on. Then I went on stage to declare the location of YAPC::Europe 2015 (which is Granada, Spain by the way). After that Jose Luis Martinez from Barcelona.pm did a short presentation of YAPC in Granada, and Diego Kuperman gave a little present from Granada to Sofia.

Mihai Pop of Cluj.pm presented a talk called "Perl Secret". It was basically a 20-minutes version of BooK's lightning talk about Perl secret operators, somewhat duluted by interspersing references to minions. It was entertaining.

The great Mark Overmeer talked about translation with context. He went beyond the usual example of multiple variants of plural values in some languages, and talked about solving localization problems related to gender and so on. The module solving these problems is Log::Report::Translate::Context. As always, great attention to details from Mark.

After lunch (sandwiches from Subway), Alex Balhatchet of Nestoria presented hurdles of geocoding, with solutions. I and my co-workers had encountered similar problems on a far smaller scale, so I could understand the pains, and had a great interest in hearing about the solutions.

Then I attended a very inspiring talk by Max Maischein from Frankfurt about using Perl as a DNLA remote and as a DNLA media server. I immediately felt the urge to play with the code he published and try to adapt it to my own TV at home. There was even a live demo of using DNLA to stream to Max's laptop a live stream of the talk provided by the conference organizers. And it even worked, mostly.

Ervin Ruci talked more about geocoding — this talk was partially touching the same problems Alex Balhatchet was talking about. Unfortunately, it was substantially less detailed, so I was somewhat underwhelmed by it. The presenter mentioned cool things like dealing with fuzzyness of the input data using hidden Markov models, but did not expand on them.

StrayTaoist described how to access raw data from space telescopes using (of course) Perl. Very lively talk. There was a lot of austronomy porn in here.

Luboŝ Kolouch from Czech Republic talked about automotive logistics, and how open source solutions work where proprietory solutions do not. The software needs to be reliable enough to make sure that it takes only 1.5 hours between the part order and its physical delivery to the factory.

After coffee break with more mingling the inimitable R Geoffrey Avery choir-mastered an hour of lightning talks. Most talks were somewhat "serious" today; I hope we see more "fun" ones in the next coming days.

Unfortunately, I missed the first keynote of the conference from Curtis "Ovid" Poe, so cannot really say anything about it.

Finally, we went to Restaurant Lebed for the conference dinner. The location is superb, there is a great view over a lake. The food was great, too. We also got to enjoy some ethnic Bulgarian music and dancing, not too much, and not too little.

Lots of cheers to Marian and the team of volunteers for organizing what so far turns out to be a great conference.

Posted Fri Aug 22 16:51:19 2014

Today I've spent quite some time chasing a bug in a legacy code at work. In retrospect, the problem is trivially simple.

It can be illustrated by the following snippet.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body>
<script src="irrelevant.js" type="text/javascript"/>
<script type="text/javascript">
function meow() { alert("meow"); }
</script>
<a href="http://dailyotter.org/"
 onclick="meow(); return false;">
click for meow</a>
</body>

So, why does it show you otters instead of meowing, and how long did it take you to spot the bug?

Posted Thu May 10 15:30:23 2012 Tags:

Here's a tiny ugly patch to make xpdf remember positions in files and restore them. You open a PDF document, read it a bit, then quit xpdf. Next time you open the same file, the same page will be shown as it was when you quit.

The positions are stored in ~/.xpdf.bookmarks.

If you are using xpdf from FreeBSD ports, just put the patches into /usr/ports/graphics/xpdf/files directory and rebuild.

Patch 1
Patch 2

Posted Wed Mar 7 16:47:16 2012 Tags:

I've been using ikiwiki as my private wiki for several months now, and have been very happy with it.

It's the ultimate geek wiki. You get:

  1. A simple, yet well-known default input format, markdown.
  2. A real version control system of your choosing, as opposed to some ugly bolted on custom thing (I chose git).
  3. An ability to edit the content using your favourite text editor, which is, frankly, a huge improvement over HTML text areas.
  4. It is written in Perl, and the code is reasonably clean and well thought-out. Since Perl is the language I've been using the most, it is a pretty big bonus for me.
  5. The actual wiki content is a collection of generated static HTML pages, so you don't actually have to think about CPU resources spent by the server.

Those are the big points. You get much more than that, of course, but those were the sellers for me personally.

Ikiwiki can also work as a blog, and since today, it powers this blog as well. I'll do a comprehensive writeup on what it took to convert this blog from Movable Type to ikiwiki once I am reasonably sure everything works to my satisfaction.

Posted Tue Dec 21 13:17:09 2010 Tags:

For months, I've been plagued by intermittent mouse freezes on one of my boxes.

It started after a regular Xorg upgrade. According to various mailing lists, that particular upgrade caused similar problems to a lot of people, so I tried different suggested fixes. No luck.

A bit later, Xorg on FreeBSD was modified to fix the reported problems. But the upgrade did not fix my problem.

Eventually I came to a realization that it is likely that the problem is not with the mouse driver or with any other part of Xorg. Rather, it was a problem with synergy client interaction with the new xcb. I even found a problem report with a supposed fix to the problem. By the time I've found it, the fix was committed to the synergy port, and was subsequently rolled back because it lead to other problems. I tried the patch in the PR anyway. Still did not help me.

Not wanting to spend too much time on this, I was coping with the delays and only occasionally, when annoyed more than usual, was trying to find another fix. Unsuccessfully, I must add, until this morning, when I discovered synergy+, a maintenance fork of the original synergy. I was not aware that synergy+ is basically a drop-in replacement to synergy, the binaries having the same names as in the original. Better still, synergy+ client works just fine with the original synergy server. So I've decided to give it a shot, removed the synergy package, and installed the synergy+ port. Voila, the freezes are gone. I am a happy camper now.

Posted Thu Jan 7 16:26:28 2010 Tags:

With the recent (2009-12-23) update to FreeBSD's sysutils/smartmontools port smartctl stopped working if run as non-root. I did not investigate whether it is because of the change in the way smartctl operates, or whether it just stopped to be setuid root.

Normally I don't mind going root to run smartctl by hand, but it presents a bit of a problem for the hddtemp_smartctl Munin plugin.

One possible solution is to add the munin user to the operator group, add the following two lines to /etc/devfs.conf:

perm ata 0660
perm xpt0 0660

And finally, run sh /etc/rc.d/devfs restart.

Being the dummy that I am, I only thought about a simpler solution when composing this post: just add user root into the [hddtemp_smartctl] section of your munin/plugin-conf.d/plugins.conf file. Besides being simpler, this method has an added advantage: an updated version of the sysutils/munin-node port can easily incorporate this change. Dag-Erling: hint, hint. :-)

Posted Tue Jan 5 14:00:26 2010 Tags:

Today at work I needed to locate and extract, automatically, some information from a website.

There was no direct URL to the information I needed, some fields had to be filled and some POST forms had to be submitted.

Normally I would use WWW::Mechanize for such a task, but in this particular instance the situation was made somewhat less managable because the site in question was implemented with ASP.NET.

The problem with this is that every link has an associated JavaScript event handler which does some housekeeping, assigns things to funnily named hidden input fields like __EVENTTARGET and __EVENTARGUMENT and then POSTs a form.

My first thought was to try and find a CPAN module which handles those complications. Not surprizingly, there is one, aptly named HTML::TreeBuilderX::ASP_NET.

According to its documentation, the module works in combination with the standard LWP::UserAgent and HTML::TreeBuilder, and converts ASP.NET JavaScript posting redirects into an HTTP::Request object which can be fed to LWP::UserAgent's request() method. Just what the doctor ordered.

However, it turned out that my joy was a bit premature:

  • it requires Perl 5.10, which we do not yet have on our production systems;
  • documentation is incomplete and inaccurate at times - it insists naming its httpRequest() method as httpResponse();
  • it fails its own tests, not only on two machines I have tried to run them, but also on a lot of other systems according to CPAN Testers.

After a bit of pondering I decided that spending time on trying to fix the HTML::TreeBuilderX::ASP_NET module is a bit counter-productive - I needed the working code soon.

So what to do?

One thing we should keep in mind is that those JavaScript postbacks do not do anything fancy. The hidden fields that are filled in depend on what was clicked on the page, nothing else. After they are filled, a normal POST occurs.

So if we know what to POST, we could just use WWW::Mechanize and get the job done easily and quickly.

So the solution naturally splits into two parts - finding out what fields to set, and automating the process.

The first part is to launch a browser, do clicking and entering by hand, and capture what gets POSTed at each step. This capturing could be done by a variety of methods:

  • tcpdump/wireshark - listen to 'em on the wire!
  • having a proxy which outputs the POSTed parameters;
  • using a browser extension that shows POSTed parameters.

I have chosen the second option, since I had a script similar to what I need already, and since it is easy to filter out any parameters which I did not want to see, like __VIEWSTATE, which can easily be several kilobytes long.

Enter spyproxy.pl:

#! /usr/bin/perl
use strict;
use warnings;
use HTTP::Proxy;
use CGI;

my $proxy = HTTP::Proxy->new(host => "localhost");
$proxy->logmask(32); # 32 - FILTERS
$proxy->push_filter(
        request => Spy::BodyFilter->new(),
);
$proxy->start;

package Spy::BodyFilter;
use base qw(HTTP::Proxy::BodyFilter);

sub will_modify { 0 }

sub filter
{
    my ($me, undef, $req) = @_;
    print $req->method, " ", $req->uri, "\n";
    return unless $req->method eq "POST";
    my $body = $req->content;
    my $q = new CGI($body);
    for my $p ($q->param) {
        next if $p eq "__VIEWSTATE";
        print "$p\n\t", $q->param($p), "\n";
    }
}

Launch it locally in a terminal, set your browser's proxy settings to localhost:8080, and watch the output in the terminal.

The second part of the puzzle is to use the wonderful WWW::Mechanize::Shell. It provides an interactive shell, in which we can issue GET requests, see the content of the responses, view links, forms, and form fields with their values, follow the links, set the value of the fields, click on buttons and submit the forms. Best of all, after getting what we are after we can issue a script command and get a piece of Perl code that will perform all the tasks we've just done.

So the final solution looks like this:

  1. Load the start page in your browser (through the spyproxy).
  2. Load the same page in WWW::Mechanize::Shell.
  3. In the browser, fill in any fields that need filling, and click where you want.
  4. Observe the spyproxy output, note any fields that need setting. In a typical ASP.NET application, you will want to ignore the vast majority of the fields at any given moment. Don't worry, humans are good at this sort of pattern recognition. :-) Pay special attention to __EVENTTARGET and __EVENTARGUMENT fields.
  5. Set the same fields to the same values in the shell (use value fieldname fieldvalue).
  6. If __EVENTTARGET was set, type submit in the shell; otherwise, find the name of the button that was pressed (see step 4), and type click buttonname in the shell;
  7. Examine the content of the response (content in the shell) to make sure that what you've got in the shell makes sense.
  8. If more clicking and entering is to be done, go to step 3.
  9. Type script script-name.pl in the shell.
  10. Go edit script-name.pl - remove any prints you do not need, change constants you entered in the fields with variables where needed.
  11. Your custom scraping script is ready to use.
  12. ...
  13. Profit!

I hope this trick will be of use to somebody. Enjoy!

Posted Wed Aug 26 21:37:37 2009 Tags:
Books giveaway by tobez

For reasons which I am not going to delve into here (this is a topic for another post), we are going to get rid of about half of our books.

There are some (low) hundreds of books for the taking, slightly more than half in English, the rest being mostly Russian with a sprinkling of Danish here and there.

Fiction, non-fiction, textbooks, science fiction, you name it.

So, if you are in Copenhagen area and are interested, write me a note and consider coming over to have a look, maybe you'll find something you'd like to keep. All books are to be had for free, although we would not mind selling them if you will insist.

Posted Tue Mar 24 11:24:07 2009 Tags:

Often I want to know how long it took for a particular command to finish.

An obvious solution to use the time(1) command does not work without a degree of anticipation on my part that I do not normally posess.

At some point I became sufficiently annoyed to actually add some hooks to my .zshrc. All commands executed in an iteractive shell are timed, but the reporting is done only for those that took longer than 10 seconds to execute.

This ugly code does the job:

note_remind=0
note_ignore="yes"
note_command="?"

note_report()
{
    echo ""
    echo "note_report: $note_command completed in $1 seconds"
}

preexec()
{
    if [ "x$TTY" != "x" ]; then
        note_remind="$SECONDS"
        note_ignore=""
        note_command="$2"
    fi
}

precmd()
{
    local xx
    if [ "x$TTY" != "x" ]; then
        if [ "x$note_ignore" = "x" ]; then
            note_ignore="yes"
            xx=$(($SECONDS-$note_remind))
            if [ $xx -gt 10 ]; then
                if [ $TTYIDLE -gt 10 ]; then
                    note_report $xx
                fi
            fi
        fi
    fi
}

Enjoy.

Posted Thu Mar 19 19:55:14 2009 Tags:
Subscribe to The Party Line: RSS Atom