In a perfect world, all characters would be in Unicode, and all software would know how to deal with them. Although Unicode adoption has grown, especially on the web, there’s still a lot of software out there that uses good old ASCII. As a developer, I still regularly need to convert between ASCII and some numeric format — usually decimal or hexadecimal, but sometimes even octal or binary. I used to refer to this online ASCII table, but waiting for the page to load and then visually finding the desired entry all take time — a few seconds at least.

Then I realized that I could perform most conversions at a shell prompt. For instance, to get the numeric value of the character ‘A’, you can use Ruby:

$ ruby -e "print ?A"

But I wanted something with a little less typing that would be easier to remember. So I created a Ruby script named ‘ascii’, which you can download below or grab from the BitBucket repository.

If you invoke it without arguments, you get the full ASCII table. If you supply arguments, they’re evaluated. If they result in numeric values, then you get the equivalent ASCII values. Otherwise, you get the values of each character in the argument. You can also override the numeric evaluation and specify that you want all arguments treated as string using the -s option. For each value the script outputs, you get the decimal, octal, hexadecimal, and character representations.

ASCII only goes up to 255 in its 8-bit form, but this script actually allows higher values. In that case, the character representation will be modulo 256 — but you can use this ‘feature’ to perform decimal/octal/hex/binary conversions. Since ascii accepts any numeric argument, you can prefix it with 0b for binary, 0o (or just 0) for octal, or 0x for hex. You can even do arithmetic evaluations like ‘?A+32′ (the value of ‘A’ plus 32) or ’65|0×20′ (another way of getting the same value). Note the quotes to prevent the shell from evaluation the ? or the |.

See the included man page for details.

“What’s missing from aqua?”

“Not”

“Naught’s missing? That’s great!”

“No, you need to add NOT.”

“‘Add not’ means leave it alone — it’s perfect.”

“No, it’s not perfect. It needs NOT.”

“It has to be needy to be perfect?”

So, I aded a logical not operator to aqua. If the first character of a token is ‘!’, that’s a not.

I also tightened up the code a bit, at which point I realized that this utility’s parser doesn’t need any data structures other than the incoming stream of characters and the single, resulting lambda that performs the requested operation — composed of lambdas for each test, bound in closures that obviate the need for any describing data structure. It’s the first time I’ve written a parser in such a functional style. Maybe I should rewrite it in Haskell just for kicks.

I like using mutt for my email, and abook for my contacts. They’re simple but powerful programs designed to work together without locking you into one way of doing things. If something doesn’t work the way you like, you can plug in your own replacement for that piece without giving up on the whole product.

Abook provides a mutt-query feature that’s intended as the query_command for mutt — that is, how to find an email address for one of your contacts. The feature works well, but it has some limitations:

• It only searches the name and email fields
• You can’t combine search criteria
• You can only search for literal strings

Since mutt’s query_command provides a well-defined interface, I decided to create a replacement that still uses the abook contacts, but allows you to:

• Search all contact fields
• Search only specific fields
• Combine searches with logical and/or
• Specify precedence of grouping with parentheses
• Use regular expressions

The result, which you can download below or snatch from its BitBucket repository, is aqua — Advanced Query Using Abook. Plug it into .muttrc:

set query_command="aqua %s"

See the included man page for details, and then you can do things like:

Query: notes=geek

So you can tell them all about it!

See Chip sweat

Not knowing where to look for SQLite3 maximums at the time (here they are), I became a little nervous as my Bayesian spam filter database approached 2 gigabytes. When it passed that milestone and continued working, I breathed a sigh of relief. Then one day, I noticed that the spam totals were wrong — my total number of spam messages had been replaced with a nil!

Fortunately, I discovered the problem on the day it first occurred. I restored the database from the previous day’s backup — then started to try to figure out what was wrong. I was just about certain that 2GB was the culprit. Probably a pointer being cast to int somewhere in the implementation of SQLite3 or the gem. Using irb I verified that I could reproduce the problem with a simple insert. So I logged an issue on the sqlite3-ruby Google Group.

Luis Lavena responded almost immediately. In trying to help him reproduce the problem without uploading 2GB of data, I tried out each step myself as well. Suddenly I found that I couldn’t get an insert to work, even on an empty database! That let the 2GB boundary off the hook. What could I have possibly screwed up?

It turns out that all I had done was to upgrade the gem, which came through on a portupgrade. Even though I still can’t find it anywhere in the docs, Luis let me know that the parameters to a query need to be passed as an array now, instead of as individual arguments (UPDATE: it’s been added now). Be warned, and please upgrade your copy of isspam before you get version 1.3.0 or greater of the sqlite3 gem. As usual, the tarball link is below, or you can pull it from Bitbucket.

In case you’re curious, SQLite3 supports a maximum of 1073741823 pages out of the box, with a default page size of 1024. That means a maximum file size of about 1 terabyte. But if you tweak the page size, you can approach 32 terabytes.

See Chip smack his forehead

I had a V-8 moment when I discovered that the sqlite3 rubygem allows you to pass a code block to its database queries. That block gets called once per row as they are being extracted from the database. That made my paged query obsolete, so I ripped it out.

See Chip code another feature just for the fun of it

Ever wonder how an individual word affects spam score? Of course, you could always do echo word | isspam -p, but that only gives you that word’s spam score. It doesn’t tell you how many times that word has been seen in spam or non-spam. So I added a -w option, which takes all the phrases from the input and gives you the detailed statistics on those phrases. You can control the length of phrase to analyze with -m, so if you only want individual words, use -m 1.

As my isspam Bayesian spam filter learns about the spamminess of new words and phrases, its SQLite database grows correspondingly. It’s now up to 1.8 gigabytes, containing more than 6.5 million unique phrases. Because it’s an SQL database, the size doesn’t impact performance much, except when gathering database-wide statistics. The ‘isspam -d’ command, which dumps all of the data to stdout, would run for hours and exhaust available memory (in the previous version). That’s because SQLite/Ruby wants to return the result of a SELECT operation as an array of rows. 6.5 million rows.

That just won’t do, especially as the database continues to grow. So I reworked the ‘dump’ option to read the database in pages of 10,000 rows at a time on one thread, while the main thread formats the output to the screen. I generalized the concept slightly by monkeypatching SQLite3::Database to add a ‘paged_execute’ method to which you can pass the desired page size, and which returns a PagedRows object that has an ‘each’ method and a ‘count’ attribute. Thus, you can use it in place of a Database#execute for most operations that will potentially return an enormous number of rows.

The main thread self-balances its sleep time between checking for input from the database query thread, in order to minimize overall runtime.

I’ve included the code and docs in the tarball below, or you can grab them from the BitBucket repository. But in case you’re not interested in all that spam filter code, here’s the bit that adds paged queries to SQLite3::Database:

# class to create an array-like set of paged database query results
class PagedRows
  # count of items retrieved so far
  attr :count

  # Create a paged query
  #  db = SQLite3::Database
  #  pagesize = int, number of rows per query
  #  query = select statement without limit or offset
  #  param = parameter for query
  def initialize(db, pagesize, query, param)
    @db = db
    @query = query
    @param = param
    @pagesize = pagesize
    @count = 0
  end

  # Iterate over all rows from multiple queries
  def each(&block)
    queue = []			# pages yet to be processed
    done = false
    mutex = Mutex.new

    # worker thread for database queries
    t = Thread.new {
      offset = 0
      begin
	# grab a page
        enq = @db.execute(@query + " LIMIT #{@pagesize} OFFSET #{offset}", @param)
	mutex.synchronize do
	  queue << enq		# add to queue
	end
        offset += @pagesize
        @count += enq.size
      end until enq.size < @pagesize
      enq = nil			# so last page gets released after processing
      done = true
    }

    nsleep = 0.5		# how long to wait for worker thread
    deq = nil			# establish scope of deq
    while (!done)
      sleep nsleep
      while queue.size > 0	# any rows already fetched?
	mutex.synchronize do
       	  deq = queue.shift
        end
	deq.each &block		# process the page of rows
	deq = nil		# release for GC
	Thread.pass		# give the worker thread a chance
	nsleep = [0.001, (nsleep*0.6666666667)].max	# Ok, we waited long enough (will be bumped below)
      end
      nsleep *= 1.5		# wait a little longer next time
    end

    t.join			# this should be a formality
  end
end

class SQLite3::Database
  # Create a paged query
  #  pagesize = number of rows per query
  #  query = database select statement
  #  param = parameter for query
  def paged_execute(pagesize, query, param)
    PagedRows.new self, pagesize, query, param
  end
end

Now that I’ve been using XMonad as my window manager on FreeBSD for a while, I find that when I have to work on Windows one of the more painful aspects thereof is, ironically, window management. I spend way too much time hunting for the right window or arranging windows so I can clearly see all the ones I need at once. Windows Vista and 7 have the ability to view windows side-by-side or stacked, but that’s not adequate to my needs and it virtually requires using the mouse. I desperately want an automatic tiling window manager like XMonad.

So, I created the next best thing: a command-line utility that will arrange all the windows in a tiled fashion, according to a pattern specified per monitor. In the download below you will find all the sources, plus in the Release subdirectory I’ve included an executable built with Visual Studio 2010.

For usage, type ’tile’ at a command prompt without arguments. But, to give you some examples:

tile 1 2

This tiles both the primary and second monitors using a column-based scheme, with no maximum of columns. Each display will be divided into the number of columns needed to hold all of the windows on that monitor.

tile 1:2r 2:3c

This tiles the primary monitor into 2 rows, with each row split as evenly as possible to accomodate all of the windows. Thus, if the primary window contains three windows, the first window will occupy the entire top row, and the second row will be split between the other two windows. On monitor #2, we’ll have three columns — so if we have five windows there, we’ll get one window in the first column and two windows stacked in each of the two remaining columns.

If you specify more rows or columns than there are windows on that monitor, it’s automatically reduced to the number of windows. Tile silently ignores most errors, such as specifying a monitor that doesn’t exist. Tile does not affect windows that are maximized or minimized. In order to be considered within a monitor, at least 10 pixels of the window in both directions must intersect the monitor’s rectangle.

As noted above, all parts of an argument except for the monitor number are optional. If omitted, the max defaults to 99, and row/column defaults to column.

So, now I can choose my optimal layout, put it in a little batch file named ‘t.bat’, and whenever I need my windows to snap into place I can just type ‘t’ at a command prompt.

I’ve been a much happier feed consumer since I started using Newspipe to pump my feeds to mutt. Subscribing to new feeds, however, got a bit more complicated. I’d have to edit my OPML file and make sure I got the new element inserted with all the right syntax. Even before that, I’d have to find the link to the feed I wanted. You’d be suprised how many sites don’t provide a direct link to their feed, so I’d often have to view the page source and search for an RSS autodiscovery link.

Then a few days ago, a reader asked me if I knew of a utility that would list all the autodiscovered feeds on a site. Since I could obviosuly use the same thing myself, I wrote it. It’s called feeds.rb. You just type a command like this at the command prompt:
feeds.rb http://example.com

… and it lists out all the feeds that have autodiscovery links on that page.

The next step, of course, is to automate including one of these links in my OPML file. Yes, I’m lazy — it’s one of the traits of an efficient programmer. I created a second script, called opmlsub.rb:

opmlsub.rb myopmlfile.opml -s http://example.com > myopmlfile.new

This one takes the given URL and adds an outline element for it to the incoming OPML file, spitting the result to stdout. If the URL is a feed, it will just use it directly. If it’s an HTML page, it will look for autodiscovery links and use the first RSS 2.0 link, if available, or the first ATOM link if an RSS 2.0 link can’t be found (I have to prefer RSS 2.0, being a member of The Board).

You can also instruct opmlsub.rb where to place the new link. The option -i TEXT specifies the value of the “text” attribute of an existing “outline” entry inside which the new link will be placed, as the new last child element. If not specified or not found, then the new element will be the last child of the “body” element.

I didn’t include options for editing and deleting elements. That’s easy enough to do with vim on the rare occasions when it’s needed.

Finally, I wanted to be able to do all this while looking at a page in Firefox (or Chromium, when it’s released on FreeBSD). So I created a third script named ‘clipsub’, which takes a URL from the clipboard (using xclip) and adds it to my OPML file without any intervention. I then mapped that to a key shortcut (mod4+shift+S) in my window manager, xmonad. Because opmlsub.rb validates a subset of the feed, I don’t have to worry about accidentally invoking this when a non-URL is on the clipboard. If any error occurs, clipsub pops up the error message in an xmessage window.

So now, when I see a site to which I’d like to subscribe, I just press ctrl+L (highlight the URL in the address bar), ctrl+C (copy it to the clipboard), and mod4+shift+S (subscribe) — and the posts start magically showing up in my feeds folder in mutt.

Mercurial repository on BitBucket

The other day I received a meeting request sent from Microsoft Outlook. This is the first such request that I’ve received since converting to mutt as my Mail User Agent (MUA). Thus, the request came as a file attachment of type “text/calendar”, which I could view as text. If you’ve ever looked at an iCalendar file, you know that they aren’t organized for easy reading. Finally, after I gathered the details, I added the appointment to my when calendar.

“I should automate this,” thought I. So I did. I’ve added some scripts to the remindwhen project to handle iCalendar attachments, including one “glue” script that’s specifically designed for use with mutt and when. See the README for details. As usual, tarball link is on the button below, or you can clone the Mercurial repository from BitBucket.

With the icalmutt.rb script in .mailcap for ‘text/calendar’, I can now view an iCalendar attachment without all the meaningless extra garbage that Outlook throws into it. At the bottom, it asks me whether I want to respond with accept, decline, tentative, or not now. If one of the first three, then the script creates an iCalendar response to the meeting and emails it to the meeting organizer using mutt (so it goes in my sent folder, too). If the response is “accept”, then the script also adds the event to my when calendar, with any specified advance alarm for remindwhen.

At some point, I’d like to add a script for creating a meeting as the organizer, but right now I’m tired of looking at the iCalendar specs and the sparse documentation of the icalendar rubygem (I ended up reading the source instead a lot of the time).

Matt Might published three proof-reading scripts to detect so-called “weasel words” (words you really shouldn’t use in good writing), passive voice, and “lexical illusions” (word duplication across a line boundary). I like these scripts, but I needed to adapt them to my evil purposes.

First and foremost, all of Matt’s scripts require file arguments. For use from within vim, I find piping more useful. So I’ve modified them all to allow either.

Second, I found some of the scripts to be too complex. I think one optional weasel-word file should be enough. I also reduced the size and complexity of the duplicate word detector by rewriting it (in Ruby).

Finally, I built spell-checking into the suite by invoking aspell(1) in pipe mode and filtering the results.

Of course, I advise against relying on these tools to replace the human eye and brain — they aren’t foolproof. But they can fill in some of the gaps that said human apparatus might miss.

I didn’t see any license mentioned on Matt’s site, but I’m releasing my versions under the Open Works License (OWL) as usual.

Overall I’ve been very happy with my getlessmail spam filter. Using a simple Ruby script to describe the rules for weeding out the canned content posing as real meat provides enough flexibility without complexity most of the time. However, spammers are a smart lot (if you overlook the questionable decision to get into the spamming game in the first place). They take great pains to defy simple rules for identifying them as spammers. I needed something additional.

A few years ago, I read Paul Graham’s essay A Plan for Spam and its sequel Better Bayesian Filtering. The simple yet convincing logic of his approach fascinated me, and I itched to try implementing it. Now that I have a reason to do so, I wrote one in Ruby. I implemented it as a class (IsSpam), with a command-line utility wrapper (isspam). The former can be easily built into a getlessmail script without spawning another process (not that there’s anything wrong with that), while the latter can be used from an MUA or cron job to populate the database.

You can grab the tarball at the link below, or clone the Mercurial repository from BitBucket. See the README for an overview. I’ve also included a man page for the command-line utility, and RDoc pages for the Ruby class.

I’ve followed Graham pretty closely in translating the algorithms from Lisp to Ruby. Some exceptions include how I handle non-word characters and how I test for phrases.

For non-word characters, I do two things: I test the words without them, and with them. The exception to this are the punctuation characters [.:;,], which I don’t include in a word if it is followed by whitespace (this pattern can be overridden by the ‘word_split’ attribute). Additionally, if a word ends in [?!] (overridable via the ‘trailing’ attribute), then I also test the same word without that character, recursively. Thus, “buy!!!” tests as “buy!!!”, “buy!!”, “buy!”, and “buy”.

In addition to testing each word individually, I test the combination of adjacent words up to the value of the ‘max_phrase_length’ attribute (3 by default, but it can be overridden). Certain combinations of words should have their own score, but if you test for too long a combination, then you penalize performance for little or no gain. In any case, I don’t test any phrase longer than 256 characters.

So far, the results look promising. My only problem is that I don’t yet have enough data. For the first time in my life, I find myself looking forward to receiving more spam, so I can collect a better sample size.

On my FreeBSD system, I like to keep various statistics displayed in my xmobar at all times. Most of the system statistics that are built into xmobar require the /proc filesystem procfs(5), which isn’t mounted by default in FreeBSD for security reasons. Thus, I have rolled my own pipes to grab this infomation from sysctl(8) and other places. One such statistic that proved difficult was per-CPU usage. The top(1) utility shows this information if you specify the -P option, but you can’t get that output to a pipe in order to filter or reformat it.

After digging around in top’s sources (/usr/src/contrib/top and /usr/src/usr.bin/top), I was able to clone top’s algorithm with an output that mimics iostat(8). Behold:

% pcpustat -c 10
               cpu 0               cpu 1               cpu 2               cpu 3
  us  ni  sy  in  id  us  ni  sy  in  id  us  ni  sy  in  id  us  ni  sy  in  id
  12   1   5   0  83   2   1   3   0  94  79   1   7   1  13   1   0   0   0  99
  46   0   5   0  50   2   0   2   0  97  50   0   2   0  47   1   1   5   0  93
  30   1   5   1  64   3   1   4   0  92  44   0   5   0  51  14   0   5   0  81
   2   2   1   0  95   0   0   6   0  94  92   0   5   0   3   1   1   0   0  98
   6   1   6   0  87   8   2   4   0  87  76   0   8   0  16   1   0   5   0  95
   0   0   6   0  94  33   2   3   0  62  49   0   8   0  44  10   0   2   0  89
   2   1   3   0  94  26   0   5   0  70  69   1   5   0  25   0   0   1   0  99
   2   1   7   0  91  12   1   2   0  85  80   0   5   0  14   2   0   0   0  98
   1   0   6   0  93   7   1   3   0  88  86   1   8   0   5   1   0   1   0  98
  26   1   6   0  68  50   0   4   0  47   8   0   1   1  90  14   0   2   0  84

I have a ‘make buildworld’ in progress, otherwise it would be almost all idle time.

I’ve included options to filter the states and CPUs you want to monitor, as well as to flip the percentages (i.e., not in that state). For example:

pcpustat --not --idle --quiet --wait 2

generates four columns every 2 seconds, representing the overall utilization of each CPU.

Note how I have both long and short forms of each switch. This utility provided a good chance for me to become familiar with the getopt_long(3) function in C. It also serves as a good example of not only how to use it, but also how to provide help for it.

Rather than putting everything into the README, I decided this would be a good opportunity to write my first man(1) page. I’ve included that, and if you want to read it right where it is (after you extract the tarball below or clone the BitBucket Repository), you can use:

man -M project-path/man pcpustat

or copy it to /usr/local/man.

I’m thinking this utility might be a good candidate for my first FreeBSD port. I don’t think it would port well to non-BSD systems, because it makes use of sysctl(3).

Since the MetaWebLog API provides “getPost”, “editPost”, and sometimes “deletePost” methods, I’ve added corresponding operations to rump (my text-editor-oriented blogging tool) via three new command-line switches:

-u ID

Updates the post identified by ID. This overrides the default operation of adding a new post. It has no effect if -p (preview), -f (fetch), or -d (delete) are also specified.

-d ID

Deletes the post identified by ID. This overrides the default operation of adding a new post, or any -u switch. It has no effect if -p (preview) or -f (fetch) are also specified. Since deletePost is not included in the MetaWebLog API specification, you’ll have to see whether your blogging framework provides it. WordPress does.

-f ID

Fetches the post identified by ID. This overrides the default operation of adding a new post, or any -u or -d switch. It has no effect if -p (preview) is specified.

When fetching, replacements are performed on the resulting post, which is then sent to $stdout in rump source format. That allows you to edit it and then resubmit it either as an update to the original or as a new post.

This opens up a number of new possible uses for rump. You could use it to backup your blog to rump files. You could retrieve and edit posts in vim (or your preferred text editor, as long as it can read/write pipes) without ever saving them to a local file. You could construct a fetch/update pipe in a loop to perform mass edits. Your blog is now entirely pipable.

BitBucket repository

In one of my curious moods, I began to wonder how difficult it would be to figure out the location of an email sender based on the IP address shown in the “Received” header fields. It turns out to be more difficult than you may have thought, because:

1. An email often contains multiple “Received” headers, one for each relay point. The innermost (last) is the original sender.
2. However, the original transmission is often within a local network, so the first one or few IPs may be in the reserved local range.
3. No free, global, authoritative database exists that contains the location of all IPs. At least, not that I’ve found. However, there are some free databases you can download that are updated from time to time.
4. The owner of the IP address may not be located at the same place as the connection. In fact, it usually isn’t, but it may be close.

Despite these impediments, I have implemented IP Geolocation for Ruby, and created a method specialized for GetLessMail that uses it.

The two scripts IPGeo.rb and IPGeoMail.rb should be placed somewhere in your Ruby require path. The example database, which I downloaded from http://linuxbox.co.uk/ip-address-whois-database.php, should be placed in /usr/local/share/IPGeo (or you can modify the script to access it wherever you choose). The included dot.getlessmail shows how you could use it to add an “X-IP-Location” header that provides the IP Location data, if found.

As I intimated, you could also use IPGeo.rb outside of the context of email. It would be trivial to write a script that accepts an IP Address and prints out the information. Like so:

require 'IPGeo'
$<.each do |line|
  puts IPGeo.locate IPGeo.get_ip(line)
end

Of course, this information is only as good as your database. The one I've included hasn't been updated since August 2009. You can probably find better databases out there, if you're willing to spend some money on them. I'm not.

You can get the updated tarball using the button below, or scrape it out of the BitBucket.

As I’ve been using my new chat client, Jab, I’ve been tweaking it for usability. You can download the updated tarball below, or grab it from BitBucket.

The first thing that bothered me is that some chat agents send status updates frequently, even if status hasn’t changed. So I changed Jab to only notify of status change if the status is different than what it was the last time we received one for that user ID. Of course, when you ‘tap’ a user, Jab will forget any previously known status so you can see the update.

I wanted to set my status to :away when I close a connection. So I added the concept of events to Jab. You can hook as many procs as you want to any event, and you can define and signal as many events as you like. Jab predefines and signals several. You can see a list of these in the README. For my status purpose, I just hook the :disconnect event in my .jabrc:

on :disconnect do
  status :away, "Swapped out"
end

Another thing I noticed is that I often didn’t notice when new jabs came in, because I had another workspace active. Since I use xmobar to display the number of new email messages, I thought I should include the number of new jabs as well. I decided to define ‘new’ jabs as any jabs that came in since I entered the most recent command in Jab. Thus, it was fairly easy to define hooks for :input and :jab, and then zero or increment the count at each hook, respectively. I’ve included that code in a file named ‘notify-new’, which you can simply ‘source’ in your .jabrc to get the same action. This code writes the number to a file (~/.jab/new). Then, I can create an xmobar monitor in my .xmobarrc as follows:


Run Com "cat" ["~/.jab/new"] "jabs" 150


Simple, huh?

Let’s say you wanted to play a sound whenever a jab came in. You could do that as follows:

on :jab do |body|
  system 'mplayer -quiet ~/.jab/new-jab.wav >/dev/null 2>&1'
  body
end

or use whatever media player you prefer.

I don’t chat a lot. It’s too disruptive to use for most communications. But I do have a few friends and colleagues with whom I enjoy the occasional real-time text conversation. In order to make myself available for those people to ping, I have to keep a chat client open. Up until a few days ago, that client was Pidgin.

Pidgin is a nice piece of software — reliable, and pretty well designed from a user interface perspective. But there are a couple of things I don’t like about it. Pidgin has a large memory footprint — it’s like having another browser open. Granted, on my 4GB system, I don’t really miss 100+MB, but it’s the whole idea that you could blow 100MB on something a simple as chat that makes me uncomfortable.

A bigger concern for me is screen real estate. I like to keep chat and email open on the same workspace, which I divided with xmonad into 80% mutt, 20% pidgin. Pidgin, though, is not very happy with 20%. Perhaps if I could figure out a way to use a smaller font it would be OK, but as it stands you can’t see much of a conversation in that space, and not all the buttons are visible. Plus, each conversation wants to pop up a new window, which crunches all the others down. I really don’t want to dedicate more space to chat.

Like most X11 programs, Pidgin favors the mouse. I think there probably are ways to use the keyboard to achieve most tasks, but Pidgin was definitely designed for use with the mouse. The X11 requirement also means that you can’t use it from a virtual console. Sure there’s finch, but I didn’t find that very impressive.

What I needed was a lean, compact, text-based, customizable chat client that runs in a terminal window. So I wrote one.

I call it jab, because it uses XMPP (Jabber), and it’s written in Ruby (the contrarian in me decided to drop the ‘r’ in Jabber rather than adding one). All chat actions are Ruby statements that you enter as commands within a single terminal window. Responses are displayed in the same window, but they begin with a ‘>’ to distinguish them. You can also apply color filters to help visually separate conversations with different people, or different kinds of notifications.

Because jab runs as a read-eval loop in Ruby, it’s essentially a specialized irb. In fact, I’ve separated the Jab class into its own file, so if you prefer you could even run your chat session from irb instead of jab. Because you have the full power of Ruby in your hands, you can customize jab in an infinite number of ways. Naturally, it supports rc files to set up your preferences for each session.

I’ve tested jab with Jabber and Gmail accounts on both ends, running on FreeBSD 8 and Windows Vista (though Vista reports errors from Ruby’s Readline module if you don’t have terminfo and stty — I use MKS’s version). I haven’t tried any other hosts or client platforms, so if you do please let me know your results.

For full documentation, see the README. For concise documentation, start jab and type ‘help’.

You can download the tarball below, or clone the sources from BitBucket.

Right after posting my previous entry, I realized that rump needs a preview capability. Embedded code is hard to get right the first time, and so are any other embedded HTML tags. Plus, you want to see how images line up and check the links. I’d like to be able to make any corrections to those in vim before submitting a draft to WordPress.

So, I’ve added the -p switch to preview the post. You have to supply an RHTML file for formatting, and then you can generate HTML (or anything else, for that matter) from your post. I’ve also provided some scripts to make it easy to see how to send that output to Firefox, w3m, or whatever browser you choose. See the README file for details.

If you want your RHTML template to produce a page that looks just like your blog, you can do what I did:

1. Fetch a single, existing post from your blog. For example:

fetch -o preview.rhtml "http://chipstips.com/?p=511"

2. Edit the file to:
a. add the necessary require (as per the README): <% require "rump_preview" %>
b. change all occurrences of the title to <%=title%>
c. replace the content of the post with <%=content%>
d. replace the categories with <%=categories.length > 0 ? categories.join(",") : "Uncategorized" %>
e. replace the tags with <%=tags.size > 0 ? tags : "None"%>

3. Voila! As long as your stylesheets and scripts are linked by absolute URL, your previews should look just like your blog posts.

You can download the updated tarball below, or get it from the BitBucket repository.

Ever since moving to vim as my primary editor, I’ve wished that I could also use it as my blogging platform, without having to copy/paste and reformat everything. Well, now I can — in fact, I’m posting this directly from vim.

I created a Ruby script named rump — it’s an acronym for Ruby Upload Metaweblog Post, but it also reflects the fact that blogging is a form of showing your ass. You can download this script below, or access it from its Bitbucket repository.

See the included README file for details. Essentially, you write your blog post as Ruby code — but rump provides some methods to keep that code to a minimum. For instance, in this post, I have a command for title, categories, tags, and content — the latter using a Here document as its parameter. Let me illustrate:

title "Rump - Ruby Upload Metaweblog Post"

tags "ruby, blogging, metaweblogapi, vim"
category "Ruby"
category "Web"

content <<EOF

(content goes here)

EOF 

The rump script takes this file, or any number of files, as arguments. So, I use a configuration file as the first argument to set up static things like what server to access, followed by the file that contains the post. You can also override settings in the command line with -e “commands”. The special file name “-” represents STDIN.

To make this even easier, I set up a separate directory on my local system for each blog, then put all the configuration commands into a file named .rump. Then, I mapped F12 in vim to post using that file, by adding the following to my .vimrc:

:map <F12> :w !rump .rump -^M


That “^M” is an actual return character.

So, now that I’m done with this post, I’ll press F12 and it will magically appear on Chip’s Tips.

I’ve added some new features to my add-ons for the when calendar program.  You can download the tarball below, or pull it from BitBucket.

Audible reminders

I found that I wasn’t always noticing my reminders in xmobar, so I decided to add sound effects.  I changed remindwhen.rb to look for a file named ~/.when/reminder.wav.  If it exists, that file is played (using mplayer) whenever a reminder is output.

Since my xmobar checks reminders every minute, and I sometimes want advance warning on a reminder, I found that the beeping once a minute can get pretty aggravating.  So I wanted a way to mute the alarms temporarily.  Thus, I created a shell script to turn the alarm on or off (by renaming reminder.wav) and tied it to a key shortcut in xmonad.  I then added a visual indicator to xmobar when alarms are muted.  How that works is all described in the README.

I also found that for some reminders I don’t want an audible alarm ever.  So I added a ‘quiet’ option within the text of the reminder, and I changed remindwhen.rb to strip that out (as well as any advance) before outputting the event.

Calendar format

I was so proud of my enhancements to when that I showed my wife my new scheduling tools.  “Yuck,” she said, “how can I see it as a calendar?”

Hmm, that’s true.  When looking for open days, you need to see the whole grid with weekdays – not just the already scheduled events.

So I wrote a couple of more scripts to pipe when’s output to pcal to create an HTML calendar that includes the event text on the right days.  Then I created another script that, with one command, invokes when piping the result through the first script and into w3m for viewing on a terminal.

By-product

As part of the implementation of the alarm support, I wrote a very handy little script to do an in-line, ternary conditional echo.  It’s called, oddly enough, “echoif”:

echoif “likeit” “applause” “`criticism`”

For details on how to use these new scripts, see the README.

Richard Barndt sent me a first pass implementation of a split method for Regex – that is, the ability to split a string at the location of an expression.  He was also nice enough to include several test cases using assertions.  I’ve adapted his syntax to my own preferences and included Regex.split in version 2.1.4 of Synthesis, which you can download below.

Whereas Richard’s version returns an ArrayList of strings, I elected to return an ls of Vars.  An ls can be used as an Arraylist, but it adds many useful methods.  Var is the normal way to store strings in an ls, for various reasons.  I also elected not to include the static Regex.split method – it seems needlessly repetitive, since you have to supply an expression anyway.  I did, however, add a split method to Var for convenience.

This implementation of split uses the GlobalSearch option (g) to determine whether to split at every occurrence of the expression or only the first one.  Captured groups within the expression are returned in the array as well at their respective positions – otherwise the match to the expression is not returned.  There’s also an option to include empty strings that result from the split, which is true by default.

Thanks, Richard, for the idea, the code, and the test cases!

In my quest to find more lightweight, componentized, customizable tools for my work, I decided to use when for my calendar.  When uses a simple text file for calendar events and sports a command-line interface, yet its handling of dates and recurrence is quite sophisticated.

When doesn’t have any mechanism for reminders, which is something my quinquagenarian brain finds necessary.  But that turns out to be good Unix design, because you can easily layer your own reminder mechanism over when, as I have in the scripts you can download as a tarball below, or pull the files from the BitBucket repository.

When’s event format is simple:  each line begins with a specification of the date(s) on which the event occurs, followed by a comma, followed by a free-text description.  There is no requirement or convention for specifying a time for the event.   It therefore just ends up as part of the event’s text.

Some users invoke when as part of their shell profile, so they’re reminded of upcoming events whenever they log on.  But I stay logged on all day, so I need more frequent opportunities for reminders.  Thus, I created remindwhen.rb, which I run once a minute from xmobar so I get a bright yellow reminder at the top of my screen as events become due:

In keeping with the Unix philosophy, though, you can send the output of the script to whatever notification mechanism you prefer.  The script merely parses the when output looking for events that are due within a range of the current time and sends them to stdout.  For an event to be considered, it has to contain a time in the format HH:MM[pm][+n], where HH is the hour, MM is the minute, “pm” is optional, and n optionally specifies an advance warning, in minutes.

To avoid cluttering the regular calendar with daily reminders, these scripts assume that you have at least two calendar files: the default (~/.when/calendar), and one just for reminders (~/.when/reminders).  The remindwhen.rb script searches the latter first, then the former.  So if you just want a reminder, add it to reminders.  If you also want to see it on your daily calendar, add it to calendar instead.

I’ve also created some supporting scripts to make reminders even easier to manage:

remind time description

adds a reminder to the reminders calendar for the specified time on today’s date.

reminders

shows today’s reminders form the reminders calendar

reminders e

edits the reminders calendar.