December 2003 Archives

Recently we heard from Kake Pugh about the OpenGuides project, a wiki-based collaborative city guide system; previously, we heard from Brian Ingerson about his Kwiki wiki implementation. Guides, wikis, blogs ... the new fashion in software engineering at the moment is the use of software to help organize, document, and facilitate collaboration -- the social software movement is gaining momentum, and Perl is one of the best languages for it.

In this article we'll look not just at some of the existing social software tools in Perl (focusing, naturally, on my own Bryar blog toolkit), but we'll look at some ways to break, bend, and embed them in other tasks.

Basic Blogging with Blosxom

When I finally decided that the world would benefit from hearing my internal monologue, I naturally looked around for nice, simple blogging programs. Blogs are, essentially, simple things, and so I didn't want all-singing, all-dancing web-based database-backed content management systems. I knew that if I were going to blog, it would have to be easy for me to do so, and there's nothing easier than firing up a text editor in the next available terminal window and doodling my thoughts into it.

Thankfully, Rael Dornfest has the same sort of laziness as I, and created Blosxom, a very simple blogging tool, which simply reads a bunch of files from the file system, finds the most recent and relevant, packages them up together, and sends them at a browser. That was how I saw blogging.

Getting Blosxom up and running is, in keeping with the whole theme of Blosxom, quite simple. You need to download the blosxom zip file, unpack it, drop the blosxom.cgi file in your web server's cgi-bin directory, and then edit the first few lines of it to tell it the name of your blog and where the entries will live, and you're done.

Posting into the blog is just a matter of creating a file called something.txt in the blog directory. It doesn't even matter what the something is. All that matters is the first line of the file, which becomes the title, and the rest, which becomes the story in raw HTML:

First post!

<p> Today I set up a blosxom blog. It was really easy! </p>

You can then, if you like, style your blog with custom header and footer files, CSS, custom HTML for each blog post, and so on.

Enter Bryar

Of course, back when I started using Blosxom, things were not quite so easy, and templating required hacking the Blosxom CGI file itself, and things just didn't seem as neat as I wanted them to be. I liked Blosxom's simplicity, but didn't want to pay the price for it in terms of flexibility.

At the same time, I was learning the wonders of the Template Toolkit, and thought that a well-designed blog should merely collect Post objects and send them to a user-defined template to do what it wants with them.

So I started writing Bryar, which is designed to be a modular, extensible replacement for Blosxom. To make this happen, I needed to provide a default set of components that simply Does The Right Thing. Let's take a look at that default set before we go any further.

When we install Bryar from CPAN, we're encouraged to run a command to set up a new blog:

You probably want to run bryar-newblog in a likely home for
your blog once we've finished installing.
    

The bryar-newblog command simply sets up a decent default set of templates, creates a first blog post, and drops in a bryar.cgi driver, similar to blosxom. At this point, we're ready to go.

Bryar's default collection of modules consists of a Bryar::DataSource::FlatFile, which emulates Blosxom's use of text files in the file system as a database of blog posts; there are Bryar::Frontend::CGI and Bryar::Frontend::mod_perl, which describe how URL parameters are retrieved from the server translated into a request for documents, Bryar::Document which represents each post, and Bryar::Collector which ties it all together. With the benefit of hindsight, that should have been called a Bryar::Controller, since the whole thing can be described as a Model-View-Controller application. But more on that later!

Since all of these things can be overridden, we need something to keep track of which classes we're using, and the Bryar::Config class does that, reading an optional configuration file but otherwise providing mostly sensible defaults.

In the rest of this article we're going to look at some interesting ways to override those classes, and build some more funky tools out of this social software framework.

Blogging the Past

Now it may surprise some people to find that I'm not really a programmer, although it may not come as a surprise to some of my colleagues.

I was trained in modern and classical Japanese language and literature, with a side order of linguistics. Programming was a hobby that turned out to make a bit more cash than conjugating 10th century verbs.

But as I began thinking a bit about blogging, I found an old Japanese diary of a lady-in-waiting to the Emperor, called the "Pillow Book", by Sei Shonagon. It was something I'd read before and found quite interesting, but picking it up again in a blog context put it in a completely new light. It read just like a 10th century LiveJournal.

Here's a sample entry:

Small children and babies should be fat. Provincial governors should be fat too, and other people who have done well for themselves. If they're too thin, it makes you think they're miserable.

As well as thoughts, observations, tales of life at the court, the book contains a large number of random lists, which really made me think of LiveJournal polls and surveys. Here's another example:

Things I think are elegant

• A white coat over a violet waistcoat
• Duck's eggs
• Sherbet ice, liana syrup, in a nice new silver bowl
• Rock crystal
• Wisteria
• Snow on plum blossoms
• A pretty little girl eating strawberries

I decided that it would be quite fun to translate the blog -- uh, book -- in a fairly modern, relaxed style, use fictitious but plausible dates for the entries, and run it as a real blog.

Naturally, I wanted to do this with Bryar, but there was going to be a slight problem; Bryar takes its entry dates from the Unix timestamp of the file, and try as I might, I could not persuade Unix to recognize epoch times a thousand years before the epoch. I had to find another way of getting the times.

But since Bryar is extensible, this is really easy -- I just write a new data source class that gets the entry timestamp from somewhere else. I simply created a subclass of Bryar::DataSource::FlatFile called ...FlatFile::Dated, which takes its date from the first line, and title from the next line.

Telling the Bryar blog to use the new data source was simply a matter of writing a little config file called bryar.conf:

source: Bryar::DataSource::FlatFile::Dated
name: Notes on a Pillow - Sei Shonagon's blog
baseurl: http://blog.simon-cozens.org/shonagon/bryar.cgi

And the rest, of course, is just a simple matter of classical Japanese translation.

Blogging the House

But as well as a linguist, I'm also a geek, and I use Perl to control various things around the house -- the house network, the division of the phone bill, and so on. This is all driven by Template Toolkit and Class::DBI.

It seemed useful, at the time, to add a blog into this great mass of centralized control, so we could pass phone messages around. Of course, in the end, it was easier to just put a big whiteboard in the lounge. XP's "the simplest thing that could possibly work" wins again.

But it was fun, and it allowed me to use Bryar in a new way. This time, we're using the original data source, collector, and other modules, but we're plugging them into an existing templating system. We already have code to deal with the web server, handle the URL processing, and the output; we already have code to transform objects into HTML; all we need is to collect the recent documents and hand them on to the templates.

I did this by adding a Bryar object to our template variables, and calling the collect_current method on the Bryar collector, using Richard Clamp's Template::Plugin::Class module. This gave a bunch of Bryar::Documents, which the template system could manipulate however it desired:

[% INCLUDE header %]
[% PROCESS blogmacros %]

<h2> Recent blog postings</h2>

[%
USE c = Class("Bryar::Collector");
FOR post = c.collect_current(blog);
    display_post(post);
END;
%]

Everything's a Wiki

So much for blogs, for the time being. The other movement in social software at the moment is Ward Cunningham's wiki. A wiki has three important characteristics: a simple mark-up language, the ability for the viewer to modify the page, and the ability to create links from one wiki page to another.

Despite initial predictions that they'd be unusable due to widespread graffiti, vandalism, and other kinds of kicking down the sandcastle, wikis have turned out to work quite well in practice, with sites like Wikipedia collecting a great store of information from contributors both regular and occasional.

CPAN has a fair collection of wiki implementations, including Kwiki, CGI::Wiki, Apache::MiniWiki, and others. However, there may well be times when it makes sense to create a new implementation, particularly when the wiki is to be embedded in another application.

What does it mean to embed a wiki? An example will demonstrate it best. Imagine a web-mail system like IMP or SquirrelMail. Now we want to add the ability to write a short note or annotation on a given message, so that it can be referred back to later. If I get a mail from my friend Al arranging to meet up for a drink at the pub, I'd want to annotate the mail with Al's mobile phone number, directions to the pub, and the fact that I need to take the book that I borrowed from him last month. Then, in an ideal world, the mail system would collect all the notes, work out the date and time of the meeting, and tell my calendaring software about it, but that's another story for a different time.

The fact is that once we have these sticky notes for mail, we might want to use them to refer to other mail, group them together in fluid topics, and so on. For instance, if I mentioned in the annotation that this was an AlMeeting, I would expect that text to be magically marked up as a link to all the other AlMeeting mails. Here we've got the three elements of a wiki -- annotation editing, a simple markup language, and the ability to link.

While the editing is a simple matter of linking your CGI parameters with your database, something that should be second nature to those who want to avoid writing code, the other two components take a little bit of work. Thankfully, chromatic has written a handy module, Text::WikiFormat.

With WikiFormat, you can pass a text string marked up in wiki format to Text::WikiFormat::format, and it will return you HTML:

print Text::WikiFormat::format(<<EOF);
= A Test Post =

I found this '''very''' interesting WikiLink recently.

EOF

__DATA__
<h1>A Test Post</h1>

<p> <br />
I found this <strong>very</strong> interesting 
<a href="WikiLink">WikiLink</a> recently. <br />
  </p>

This gives us the simple markup, but it doesn't do much about the linkability; for an embedded wiki, we're going to need finer-grained control over how the links are constructed. For instance, we might want to pass the link name to a particular URL with a particular CGI parameter. Thankfully, Text::WikiFormat supports two optional parameters in addition to the text to be formatted: the first is a hash reference overriding how the wiki text is marked up to HTML, and the second is a hash reference of options, such as the URL prefix for links.

So, if we have an application that displays a page given a parameter in the URL, we can just say:

 
Text::WikiFormat::format($text, {},  
        { prefix => "http://www.myhost.int/wiki?page=" }); 

And there we have it, an instant wiki.

Of course, in the context of our webmail application, we may have to do something more convoluted. For instance, we might want a link to SimonCozens to resolve to all mail from that person, if it can be understood as a person who sends email, or to a similarly marked-up set of messages, like an AlMeeting, if it can't. For this, we need to introduce a bit of logic and decide how to form the link.

As we mentioned above, Text::WikiFormat allows us to specify replacements for the way markup is translated to HTML. WikiFormat uses a hash-ref internally to turn wiki "tags" into HTML, and we can, in fact, put in a subroutine reference for the link tag and have that emit the appropriate HTML link.

 
sub markup { 
    my $text = shift; 
    Text::WikiFormat::format(shift, { link => \&my_link });
} 

sub my_link {
    my $linkname = shift;
    my $name = $linkname;
    ($name=$linkname) =~ s/(?<=[a-z])(?=[A-Z])/ /g;
    if (have_mail_from($name)) {
       # Valid correspondent, link to page of their mail
       my $id = correspondent_id($name);
       return qq{<A HREF="/mail/search?correspondent=$id">}.$name.
              "</A>";
    }
    # Not a correspondent, just a wiki page
    return qq{<A HREF="/wiki?$linkname">}.$linkname.
           "</A>";
}

What's going on here? First, we receive the link name in StudlyCaps every time format sees a link. We then convert it to a real name, by inserting a space in the gaps between a lowercase character and an uppercase one. Now we have a name like "Simon Cozens" that we can search our mailstore for, and if we find someone, we can return the URL to a page about their mail. Otherwise, we return a wiki page. We've used our webmail annotations to refer to either real people or to other annotations, stealthily embedding a wiki in the application, with minimal effort.

Creating the Bliki

With that example of how to create a custom wiki from an existing application, let's return to Bryar and our blogging software.

The concept of a bliki -- a cross between a wiki and a blog -- may seem like a bizarre one at first, but it's such a natural synergy that it's been independently invented by multiple people, including Martin Fowler and many others.

Depending on how you look at it, a bliki is either an ordinary wiki with a blog-style front-end showing the latest posts, or an ordinary blog with wiki-like markup and interconnections between the posts. The only difference between these views is how easy it is to edit the posts. We'll stick with the "ordinary blog" viewpoint, since editing posts over the Web is a boring and solved problem, and one not really related to the technology we're looking at today.

How are we going to extend Bryar to become a bliki? It's obvious that Text::WikiFormat is going to be involved somewhere, and it would be helpful to use the Bryar::DataSource::FlatFile class too. First, we inherit from the FlatFile class, and override the make_document method, which is responsible for turning a filename into a Bryar::Document:

package Bryar::DataSource::FlatFile::Bliki;
use base 'Bryar::DataSource::FlatFile';
use strict;
use Text::WikiFormat;

sub make_document {
    my ($self, $filename) = @_;
}

Here the format will be slightly different -- the filename (such as "MyWikiPost.txt") will be used to form both the title and the ID of the document, and the content of the document will be transformed via WikiFormatter wholesale.

sub make_document {
    my ($self, $file) = @_;
    return unless $file and open my $in, $file;
    my $when = (stat $file)[9];
    my $who = getpwuid((stat $file)[4]);
    $file =~ s/\.txt$//;

    my $dir = dirname($file);
    $dir =~ s{^\./?}{};
    my $category = $dir || "main";

    local $/;
    my $text = <$in>;
    
    return Bryar::Document->new(
        title    => $file,
        content  => Text::WikiFormat::format($text, {}, {prefix => "id_"}),
        epoch    => $when,
        author   => $who,
        id       => $file,
        category => $category,
    );
}

We change the prefix for links to be a relative link of the form id_PostName, since this is how Bryar deals with references to other blog posts, by default. (Of course, we could change this by modifying the FrontEnd class, but this seems to work fine.)

It took 30 lines of code to convert Bryar into a simple bliki, many of which were stolen from the superclass. Now let's look at something a bit different.

Everything's a Blog

I mentioned when introducing Bryar that it was based on the Model-View-Controller (MVC) pattern, although I didn't realize this while designing it.

The MVC pattern is described in this entry in Ward Cunningham's own wiki, as being made up of three parts:

• A model, therefore is an object representing data or even activity, e.g. a database table or even some plant-floor production-machine process.
• A view is some form of visualization of the state of the model.
• A controller offers facilities to change the state of the model.

All kinds of things follow the MVC design. In Bryar, the model is the data source object; the view is the templating object; the controller is the Collector object. A web browser might be described as a controller that contains all its logic, a view which contains the user interface, and the pane of rendered HTML, and the model of the entire Worldwide Web.

But some web-based applications follow Bryar's MVC approach particularly closely. Imagine, for instance, a company's product catalog on the Web. The catalog data will be stored in some kind of database, and will be presented via a templating system, with a controller that will decide which products to show. Isn't that exactly the same as reading some blog posts from a database (albeit potentially one that looks a lot like a filesystem), rendering them via templating system, with a controller deciding which posts to show?

Indeed, many of the views you will want of the catalog data are exactly what you'd want from a blog: display the most recent products, display all products, search by product name or description, and a detailed display of a single product.

We've seen it take 30 lines to turn a blog into a bliki. To turn a blog into a complete product catalog web site takes a subclass of the Bryar::DataSource::DBI class (to talk to the product database) and the Bryar::Document class (to take more details, a URL to a photo, and so on). The rest is a matter of templating, and that's a designer's job!

Social Software

I consider the emergence of interest in social software to be one of the most fascinating trends in software engineering this year. Two of the most powerful and popular aspects of this, wikis and blogs, are particularly well-suited for extension and embedding, and Perl is a particularly well-suited language for achieving this.

Although part of the point of this article was to demonstrate Bryar, there were several other important points. First, that there are plenty of Perl implementations of both wikis and blogs that you can choose from; second, that Perl makes it really easy to create your own blog or wiki and customize to your own purposes, including embedding them in an existing application.

But finally, the point was to encourage you to think about good design and the power of extensible applications; if you can create a tool that is both powerful and generalizable -- just like Perl itself -- it may end up doing wildly different things to what you initially intended!

Another week, another late summary. Luckily it's been a quiet week so I should get this written faster than usual. As is traditional, we start with perl6-internals

Parrot build system tinkering

Andy Dougherty and other discussed extending the Parrot build system to allow for building parrot itself outside the main source directory (useful when Parrot's mounted read only via NFS for example. It's not (quite) possible to do this just yet because of a couple of generated files in the IMCC tree. Leo suggested only generating those when Configure.pl --maintainer is run, on the grounds that a maintainer shouldn't be building from a read only source. Dan asked for a volunteer to fix things up.

http://groups.google.com/

Object stuff

I (and others I presume) got ready to do the "Happy! Happy! Joy! Joy!" dance of delight; this was the week that Parrot finally got objects. Or, as Dan described them, 'objects-ish'. Parrot now has single inheritance objects. They're not the final word on the matter, but it's definitely better than a poke in the eye with a sharp stick.

Dan went on to discuss what needs to be done with them before we're ready for Parrot 0.1 (Objects are one of the criteria for making a point release).

   if _some_test() goto LABEL

Slim Devices made their name in 2001 with the SLIMP3, a networked MP3 and Internet radio player. The SLIMP3 won a five-out-of-five mouse rating in Macworld magazine, and was featured in GQ magazine and on TechTV's The Screen Savers. This year, they're trying to repeat the success with Squeezebox, an 802.11-enabled version.

The interesting thing about the SLIMP3 and the Squeezebox from our point of view, though, is that the server which drives the player is written in Perl, and developers are allowed -- in fact, encouraged -- to hack on it and make the devices do interesting things. A developer community has sprung up around the SLIMP3, and some interesting third-party hacks have been produced.

We caught up with Dean Blackketter from Slim Devices, who took some time out from moving Squeezeboxes to answer a few of our questions.

perl.com: What made you decide, first of all, to write the server code in Perl?

Dean: Sean [Adams, Slim Devices founder] wrote the very first version of the software in Perl because he was able to get it up and running quickly. When I first encountered Sean and SLIMP3, I was a little afraid of Perl, having been primarily a C and C++ programmer, mostly because when I tried to read it, I got confused by idiomatic usage.

I first decided to rewrite the thing in C, but I couldn't really get started until I could read the existing code. So I bought a copy of Learning Perl and read it in two evenings. Halfway through the first chapter something clicked and I said, "That's so cool!" Then, about every 10 pages after I'd repeat, "That's so cool!" again, but a little louder. By the end, I was sold. At that point I dug in and started making the software better.

What features of Perl were particularly helpful when you were writing the SlimServer?

The greatest feature of Perl, for us, is that so many people know it. Since our software is open source, we benefit from as many users as possible being able to read and improve our code base. Anybody who's thrown together a little Perl CGI or script is a potential contributor to our software.

Second, the ability for us to offer our software on a wide array of platforms (Mac OS X, Windows 98/ME/NT/2000/XP, BSD, Linux, Solaris) and also be deployed on server appliances makes it possible for us to get our product running as broadly as possible.

Third, even though our product's primary feature is playing back audio, the bulk of the software really is processing text. Our built-in HTTP server, command-line interface, device-user interface, and even music meta-information is all text-based, and Perl is nearly the perfect language for this.

What issues did you have to face when considering how to distribute a Perl-based consumer application, and how did you approach them?

Distributing a Perl application on the traditional Unix-like platforms (BSD, Solaris, Linux) is easy. Things got a bit harder on Mac OS X, since command-line solutions aren't acceptable to Mac users. Initially, I wrote a simple graphical application (in AppleScript!) that acted as a launcher for our server software, but this was really not good enough for Mac users. One of our Open Source contributors, Dave Nanian, came to the rescue and built us a System Preferences pane and installer application that work as they should on a Mac.

Windows is the hardest platform for us to support. Luckily we had another Open Source contributor, Sam Saffron, who decided that building a shell application for the server software would be a great way to learn MFC, and he contributed a nice little app for this. Another contributor, Chris Eastwood, put together an installer for us. But it was the fine folks at ActiveState with their Perl Dev Kit that really made it possible for us to release on Windows with their PerlApp (for making Perl-based .exe files) and PerlSvc (for making Perl-based Windows Services).

Even with all this help, Perl on Windows is different enough in subtle ways to make it a real pain to maintain. Maybe if we had more hardcore in-house Windows experience things would be easier, but it's a struggle.

For consumer devices like the SLIMP3, the manufacturer has a choice between open and hackable, and "sealed box." You guys went for the hackable approach, GPLing the code and setting up developers' forums. What was the rationale behind that?

When I got into this, the total of my Open Source experience was running a Linux server in my basement. Sean initially released the SLIMP3 software under the GPL, which made it possible for me to help out. I quickly realized that our customers have a lot of talent and energy that they are willing to contribute to make the product better. We're lucky to have a business model where we can make money on the hardware and give the software away. When our customers contribute, they make their own SLIMP3s (and now Squeezeboxes) more valuable to themselves and make the new units we sell to new customers more valuable as well. Everybody wins.

How did you go about building a developer community behind the SlimServer project?

I like to think that the community built itself. We provided a forum (our discussion lists) and Sean and I participate on a daily basis; acting on the requests, suggestions, bug reports, and patches that people post. We're spending more effort lately setting up our own list server and CVS system, as we had problems with our previous forums (Yahoo Groups and SourceForge, respectively), but these will give us a little more control of our virtual spaces.

How do you decide which developer extensions to keep as third-party projects and which to integrate?

If a contribution will make the product better for a substantial portion of our customers and won't make the product worse for any of them, then, generally speaking, it's included. Submitted features that make the software harder or less intuitive to use, require obscure or platform-specific software or hardware to be installed, generally don't make it in. Contributions that are useful to a large fraction of our customers and don't diminish the product's ability to play back music, make it in.

I act as final arbiter and (hopefully) benevolent dictator, and the community has been really supportive of this.

Finally, we notice that you've decided to donate 10 percent of your profits to the EFF. What's the motivation behind this?

Digital freedoms are important to the folks at Slim Devices, both personally and at a corporate level. We've been frustrated by not being able to offer our customers the ability to play the music that they've bought due to digital rights management systems and onerous licensing fees for various patents. The EFF shares our goal to free our customers' music.

The Squeezebox is available from Slim Devices and their resellers, and the server code can be downloaded from their downloads site.