Planet Larry

Introducing Gaka

The CSS for my blog is now being generated via gaka, a CSS-generating library I wrote this afternoon. It's extremely simple, but it got the job done for me. I turned around 600 lines of CSS into around 250 lines of Clojure without much effort. It looks like this:

user> (require '(gaka [core :as gaka]))
nil
user> (def rules [:div#foo
                  :margin "0px"
                  [:span.bar
                   :color "black"
                   :font-weight "bold"
                   [:a:hover
                    :text-decoration "none"]]])
#'user/rules
user> (println (gaka/css rules))
div#foo {
  margin: 0px;}

  div#foo span.bar {
    color: black;
    font-weight: bold;}

    div#foo span.bar a:hover {
      text-decoration: none;}

Gaka is partly inspired by Sass, which I found very pleasant to work with recently. And it's partly inspired by Hiccup, which is a delicious way to generate HTML in Clojure.

There's more info and more examples on github.

spamassassin / procmail with vpopmail accounts

motivation

i’ve had lots of trouble with mail the last two years with lots of spam still passing spamassassin. but that wasn’t so bad since we already had greylisting [1] running. that however still meant 8 spam mails per day with subjects as:

• Hot news for js! 70% off all June!
• Totaler Ausverkauf von Top Zeitmesser
• Yes, js, today -80% to all prices. Office accessible free
• This crysis never ends
• VIAGRA ® Official Site -95%

however, that changed after i started to collect the spam to train the bayesian filter [2] used by spamassassin. this posting is about how i’ve done it!

in contrast: after using bayes i receive one spam mail every 4th day.

i still have to monitor the spam, which is filed in a folder called ‘spam’ in one of my ‘imap’ folders but since it can be identified as spam by 100% most of the time it’s simply a copy’n'past operation of all files in ‘spam’ to ‘spam_train’. the ‘spam-train’ folder is used in order to train spamassassin. of course there are false-positives and false-negatives as well. but handling those is very easy.

is my setup special in any way?

no i don’t think so at all. however: most sites do encourage admins not to use bayes filters per vpopmail account. and because of that and other factors there is not much documentation of how to do so. since i had some issues with how to call spamassassin from procmail it did not work for a very long time. recently i bought my nokia n900 mobile phone and using mail on this device is very nice. receiving a spam mail per hour means a notification on the n900 which IS annoying if the notification mostly contains spam slogans.

my .procmailrc

# qmail Lazydog procmailrc file

SHELL=”/bin/bash”

VHOME=”/var/vpopmail/domains/lastlog.de/js”

VERBOSE=”no”

LOGFILE=”/var/vpopmail/domains/lastlog.de/js/procmail.log”

:0fw

| spamassassin –siteconfigpath=/var/vpopmail/domains/lastlog.de/js/.spamassassin/ -p /var/vpopmail/domains/lastlog.de/js/.spamassassin/local.cf

:0:

* ^X-Spam-Flag: YES

/var/vpopmail/domains/lastlog.de/js/.maildir/.spam/new

:0:

* ^(From|Cc|To).*news@aktuell.conrad.de

/dev/null

# (other rules below)

my local.cf file, the only config file i changed:

# cat local.cf

required_score          3.2

rewrite_header subject _SCORE(0)_ |

use_bayes 1

use_dcc 1

use_pyzor 1

use_razor2 1

bayes_auto_learn 1

bayes_path /var/vpopmail/domains/lastlog.de/js/.spamassassin/db/bayes

bayes_file_mode 0777

report_safe 1

#add_header all Flag _YESNOCAPS_

ok_languages            en de

ok_locales              en de

# Google SafeBrowsing Plugin

loadplugin Mail::SpamAssassin::Plugin::GoogleSafeBrowsing

#body GOOGLE_SAFEBROWSING eval:check_google_safebrowsing_blocklists()

google_safebrowsing_apikey ABQIAA#VJ#J#VAQFbnTQ4uqKBRgArBR3gWufhSOf_c-vzV4UEN0steDDKD

google_safebrowsing_dir /var/cache/spamassassin

# scores for each url hit in message body

google_safebrowsing_blocklist goog-black-hash 0.3

google_safebrowsing_blocklist goog-malware-hash 0.5

how to experiment with spam

no blog or documentation talks about this but i think it’s very important. say you have a spam mail and you want to test your spamassassin configuration do this:

ls -la /var/vpopmail/domains/lastlog.de/js/.maildir/cur

-rw-rwx—+ 1 vpopmail vpopmail     2630 26. Dez 2009  msg.zQJ68:2,RS*

-rw-rwx—+ 1 vpopmail vpopmail    21905 16. Dez 2009  msg._zr78:2,S*

-rw-rwx—+ 1 vpopmail vpopmail     7433  9. Mär 2009  msg.zssm8:2,S*

-rw-rwx—+ 1 vpopmail vpopmail     2332  6. Mai 17:27 msg.ZuFm8:2,S*

-rw-rwx—+ 1 vpopmail vpopmail     3418 21. Nov 2008  msg.ZvFh8:2,S*

-rw-rwx—+ 1 vpopmail vpopmail     3237 23. Feb 13:07 msg.zZcg9:2,S*

let’s pick one of these messages: ‘msg.zf8y8:2,S‘

now, let’s see what spamassassin thinks about this message with:

cat msg.zf8y8:2,S | spamassassin -t -D -p /var/vpopmail/domains/lastlog.de/js/.spamassassin/local.cf >mail 2>debug

-t -D are only important for debugging

also have a look at:

tail -n 200 /var/vpopmail/domains/lastlog.de/js/procmail.log

creating the configuration for vpopmail usage of spamassassin

the message is processed by spamassassin and written to a file called ‘mail’ and the debug output going to stderr is written to a file called ‘debug’. this helped me a lot to verify that spamassassin was reading the right config files.

cat msg.zf8y8:2,S | spamassassin -t -D –siteconfigpath=/var/vpopmail/domains/lastlog.de/js/.spamassassin/ -p /var/vpopmail/domains/lastlog.de/js/.spamassassin/local.cf >mail 2>debug

watch for things like this:

[7582] dbg: conf: finish parsing

[7582] dbg: plugin: Mail::SpamAssassin::Plugin::GoogleSafeBrowsing=HASH(0x14e3a10) implements ‘finish_parsing_end’, priority 0

[7582] dbg: bayes: tie-ing to DB file R/O /var/vpopmail/domains/lastlog.de/js/.spamassassin/db/bayes_toks

[7582] dbg: bayes: tie-ing to DB file R/O /var/vpopmail/domains/lastlog.de/js/.spamassassin/db/bayes_seen

[7582] dbg: bayes: found bayes db version 3

[7582] dbg: bayes: DB journal sync: last sync: 0

[7582] dbg: config: score set 3 chosen.

[7582] dbg: message: main message type: multipart/mixed

check: no loaded plugin implements ‘check_main’: cannot scan! at /usr/lib64/perl5/vendor_perl/5.8.8/Mail/SpamAssassin/PerMsgStatus.pm line 164.

initial bayes, sa-learn

since i’ve been learning the db wrong for some time i decided to relearn everything with:

cd /var/vpopmail/domains/lastlog.de/js/.spamassassin/db/

ls -la

-rw-rw—-+ 1 joachim users    7824 21. Jun 15:17 bayes_journal

-rw-rwx—+ 1 joachim users  643072 21. Jun 15:09 bayes_seen*

-rw-rwx—+ 1 joachim users 5177344 21. Jun 15:09 bayes_toks*

just remove all 3 files (or create backups if in doubt) and then use sa-learn:

cat spamcommand

o “=== <SPAM Train> ====”

echo “/var/vpopmail/domains/lastlog.de/js/.maildir/.spam_train/cur”

nice sa-learn -D 5 -u vpopmail –dbpath /var/vpopmail/domains/lastlog.de/js/.spamassassin/db –siteconfigpath /var/vpopmail/domains/lastlog.de/js/.spamassassin/ -p /var/vpopmail/domains/lastlog.de/js/.spamassassin/user_prefs -C /var/vpopmail/domains/lastlog.de/js/.spamassassin –spam –dir /var/vpopmail/domains/lastlog.de/js/.maildir/.spam_train/cur

echo “=== </SPAM Train> ===”

echo “”

echo “=== <HAM Train> ===”

for i in cur .js@dune2.de/cur .notice/cur; do

echo “learning from: /var/vpopmail/domains/lastlog.de/js/.maildir/${i}”

nice sa-learn -D 5 -u vpopmail –dbpath /var/vpopmail/domains/lastlog.de/js/.spamassassin/db –siteconfigpath /var/vpopmail/domains/lastlog.de/js/.spamassassin/ -p /var/vpopmail/domains/lastlog.de/js/.spamassassin/user_prefs -C /var/vpopmail/domains/lastlog.de/js/.spamassassin –ham –dir ”/var/vpopmail/domains/lastlog.de/js/.maildir/${i}”

echo “=== </HAM Train> ===”

i execute this script every night using a cronjob, and so far it is working great! handling false positives and false negatives is easy as well since you only have to copy the wrong mails into the right folder and the next cronjob probably learns it right. i’m not sure on this but so far it is working.

./spamcommand

==== <SPAM Train> ===

/var/vpopmail/domains/lastlog.de/js/.maildir/.spam_train/cur

[7913] info: archive-iterator: skipping large message

…

Learned tokens from 2515 message(s) (2531 message(s) examined)

=== </SPAM Train> ===

=== <HAM Train> ===

[22465] info: archive-iterator: skipping large message

…

Learned tokens from 1100 message(s) (1171 message(s) examined)

[19782] info: archive-iterator: skipping large message

….

Learned tokens from 191 message(s) (199 message(s) examined)

=== </HAM Train> ===

./spamcommand  291,01s user 12,00s system 12% cpu 40:05,37 total

rights management

final words

links

[1] http://de.wikipedia.org/wiki/Greylisting

[2] http://wiki.apache.org/spamassassin/BayesFaq

[3] http://wiki.apache.org/spamassassin/BayesInSpamAssassin

[4] http://standbytux.blogspot.com/2005/07/using-spamassassin-and-procmail-to.html

[5] http://faisal.com/docs/salearn

New blog engine up and running

Well, my new blog is up and running. Sorry for the temporary lack of cows in my layout. I'm dogfood-testing the blog engine in a fairly vanilla state until I work out some of the bugs. This layout is based upon barecity, a minimalist Wordpress theme that I adapted easily enough to my blog.

As a bonus, I applied a dirty hack to my RSS feed that I think should help avoid screwing up people's RSS readers with duplicate entries.

I'll write again soon with some info about the blog engine and some things I learned writing it.

(As mentioned previously, here's the code.)

Managing ZIP-based file formats in git

git rocks, everybody knows that, but this morning I ranted about the problems with putting binary files into it due to that removing the option to actually see the differences. Splitbrain came to the comments and pointed out that Microsoft’s new file formats (like pptx, docx and such) are just zipped packages of XML files so at least some diffing should be possible. Remembering what I wrote about versioning PDF files in git I thought it should be a quick hack and in fact it really is:

Open your ~/.gitconfig file (create if not existing already) and add the following stanza:

[diff "zip"]
textconv = unzip -c -a


What it does is using “unzip -c -a FILENAME” to convert your zipfile into ASCII text (unzip -c unzips to STDOUT). Next thing is to create/modify the file REPOSITORY/.gitattributes and add the following

*.pptx diff=zip

which tells git to use the zip-diffing description from the config for files mathcing the given mask (in this case everything ending with .pptx). Now git diff automatically unzips the files and diffs the ASCII output which is a little better than just “binary files differ”. On the other hand to to the convoluted mess that the corresponding XML of pptx files is, it doesn’t help a lot but for ZIP-files including text (like for example source code archives) this is actually quite handy.

speed up firefox (reloaded)

I’ve posted before about this little hack on speeding up firefox. The key is that you actually move the entire .mozilla folder from disk to memory.

First you have to mount /tmp to memory (some linux distributions may do this by default) adding this line to /etc/fstab and rebooting:

none /tmp tmpfs size=512M,nr_inodes=200k,mode=01777 0 0


Then it’s safe to do:

cp -R /home/user/.mozilla /home/user/.mozilla_save
mv /home/user/.mozilla /tmp/mozilla
ln -s /tmp/mozilla /home/user/.mozilla


I updated the script so I can use it as a system init script:

#!/bin/bash

start() {
    mkdir -p /tmp/mozilla
    rsync -avi --delete /home/user/.mozilla_save/ /home/user/.mozilla/
}

stop() {
    size=`du -xs /home/user/.mozilla/ | awk '{print $1}'`
    digits=`expr length $size`
    if [ $digits -gt 4 ]; then
        rsync -avi --delete /home/user/.mozilla/ /home/user/.mozilla_save/
    else
        echo 'no!'
        exit 0
    fi
}

case "$1" in
start)
     start
     ;;
stop)
     stop
     ;;
*)
     echo $"Usage: $0 {start|stop}"
     exit 1;;
esac

So I placed this script at /etc/init.d/, made it executable and created a link inside /etc/rc.5/

ln -s /etc/init.d/ffsync.sh /etc/rc5.d/S99ffsync


rc5 because the default runlevel on Fedora is 5. You can see yours from /etc/inittab.

So every time my system boots the above script runs with the start option (executing the start function) and every time it halts/reboots it runs with the stop option (stop function).

In order to be sure that no data loss will occur in the unlikely event of sudden shutdown, I have a cronjob that saves the mozilla folder every 15 minutes.

*/15 * * * * /etc/init.d/ffsync.sh stop


Trust me. With the above hack you’ll see a significant difference on firefox’s speed. Especially if you’re you using firefox’s awesome address bar to search through your browser’s history.

Breaking links is easy to do

I apologize in advance to everyone who subscribes to my blog's RSS feed, but this week your RSS reader is probably going to suddenly find 25 "new" posts from me.

My blog currently uses /blog/title as the URL scheme, with similar URLs for categories and tags etc. Soon, I'm probably going to change it to /blog/123/title, as part of the impending release of version 0.2 of my blog engine. (The code-in-progress is in a branch on github, for the daring and foolish among you.)

This way, I can change the title of a post without breaking everything. I have heretofore lacked this ability. It's easy to code, you just tell Compojure to ignore everything after the number in a route. Something like this:

(defroutes foo
  (GET ["/blog/:id:etc" :id #"\d+" :etc #"(/[^/]*)?"] [id]
    (pages/post-page id)))

It's only a few lines of code to change, but the ramifications are widespread. It'll instantly break every link to my blog, for example. At least it's pretty easy to set up a bunch of redirects in Compojure to avoid that. I think this'll work:

(require (blog [db :as db]
               [link :as link])
         (oyako [core :as oyako])
         (ring.util [response :as response]))

(defn redirect-post [name]
  (when-let [post (oyako/fetch-one db/posts :url name)]
    (response/redirect (link/url post))))

(defroutes redirect-routes
  (GET ["/blog/:name" :name #"[^/]+$"] [name]
    (redirect-post name)))

(Oyako here is the experimental ORM-like library I'm using to interface with PostgreSQL nowadays, having ditched Tokyo Cabinet.)

Changing my URL scheme is also going to mess up RSS though, because I (foolishly) used post URLs as the GUIDs in my RSS feed up to this point. This problem I don't know how to avoid. I might reduce the number of posts included in my feed temporarily, to limit the damage.

Excellent Post About Free Software Purists

Edit: Almost forgot to mention: I heard about this blog post from flameeyes.

There was a really, really good blog post that actually clarified to me what was for so long a nuisance in discussions with Free Software purists (and I couldn’t reason out why at the time). Read this quote:

A long time ago, Larry Wall gave a “State of the Onion” speech. Larry is the guy behind Perl, the programming language, and every year he gave a talk about what he believes Perl is all about and where it’ll be going over the following twelve months. Larry’s not really a programmer; he’s a linguist who does a lot of programming. So his view on what a computer language should be is uniquely interesting, and should be read. Anyway, he spoke of freedom, and characterised the ends of the spectrum on the subject as “Bill” and “Richard”, those two being Bill Gates (at the time head of Microsoft) and Richard Stallman (head of the Free Software Foundation): the idea here was that those two represented the ultimate expressions of their philosophies. Richard Stallman was at the far extreme of belief in free software; Bill Gates was at the other far end, decrying the very concept of freedom. It was a throwaway joke, at the time; a caricature for the purpose of a laugh in a presentation.

I encourage everyone to read the original post, since it really brings some light out on the unreasonableness and extreme nature of the Free Software purist position.

Reparenting QGraphicsItem during Mouse Drag

QGraphicsItem::setFlag(ItemIsMovable) is a wonderful feature. With one line of code, an item becomes dragable with the mouse for repositioning. Unfortunately, things get a bit tricky when reparenting during drags.

Let’s say you have two QGraphicsItem called plate1 and plate2, in a subclass called PlateGraphicsItem. Inside plate1 and plate2, you have a few child items, food1, food2, food3, food4, etc, in a subclass called FoodGraphicsItem. Foods are inside of plates. What if we want to move one piece of food from one plate to another, smoothly dragging it there? Intuitively, we should be able to do this (after remembering to run setFlag(ItemSendsGeometryChanges)):

QVariant FoodGraphicsItem::itemChange(GraphicsItemChange change, const QVariant &value)
{
	if (change == ItemPositionChange && scene()) {
		QRectF newRect = mapRectToScene(boundingRect());
		newRect.moveTo(parentItem()->mapToScene(value.toPointF()));
		foreach (QGraphicsItem *item, scene()->items()) {
			if (item != parentItem() &&
			    qgraphicsitem_cast< PlateGraphicsItem* >(item) &&
			    item->mapRectToScene(item->boundingRect()).contains(newRect)) {
				setParentItem(item);
				break;
			}
		}
	}
	return QGraphicsItem::itemChange(change, value);
}

As the user drags the food, we see if it’s boundingbox fits inside the bounding box of a different plate. If it does, we switch parents. But this doesn’t happen. As soon as the item is reparented, it leaps across the screen to the same relative position it was in while inside of the old plate. If it was on the bottom-left corner of the old plate, it moves to the bottom-left corner of the new plate. Not what we want. So it seems like we should be able to modify the inside of that loop like this:

				QPointF oldPos = scenePos();
				setParentItem(item);
				return QVariant(item->mapFromScene(oldPos));

But alas, this does not work either. As soon as you move the mouse again (while still dragging), the item leaps to where it was before the above fix. And remember: this leap puts the item far away from underneath the mouse cursor. So it works initially, but the next time Qt’s MouseMove event handler is called, we’re SOL.

After digging through the Qt source for quite a bit of time, it turns out that Qt keeps track of where the mouse was pressed originally, in the scene coordinates, and uses this to compute the new location relative to the old parent coordinates. Not good. Mixing coordinate systems like this is not okay once the item is reparented. It turns out, we have to simulate a mouse release event to clear Qt’s internal state. But this doesn’t completely do it, because the mouse event still has the scene coordinate of where the button was pressed, which means we also need to intercept and tamper this information before Qt sees it. Essentially what this amounts to is reversing Qt’s coordinate equation, solving for zero, and adding back the location that we want. And this has to happen every time the user moves the mouse, but it should only happen once the item has been reparented. And if the item is reparented back to the original plate after a series of drags, we still have to do this transformation, since we have already cleared Qt’s internal state, which means we need to store the click location once we reparent.

So all and all, this horrible hack amounts to adding this (and making the above modification):

void FoodGraphicsItem::mouseMoveEvent(QGraphicsSceneMouseEvent *event)
{
	//BIG PHAT UGLY HACK!
	if (event->buttonDownScenePos(Qt::LeftButton) == m_manualMouseReleaseAt ||
	    !parentItem()->contains(parentItem()->mapFromScene(event->buttonDownScenePos(Qt::LeftButton)))) {
		m_manualMouseReleaseAt = event->buttonDownScenePos(Qt::LeftButton);
		QGraphicsSceneMouseEvent mouseRelease(QEvent::GraphicsSceneMouseRelease);
		QGraphicsItem::mouseReleaseEvent(&mouseRelease);
		event->setButtonDownScenePos(Qt::LeftButton, parentItem()->mapToScene(pos() + transform().map(event->buttonDownPos(Qt::LeftButton))));
	}
	QGraphicsItem::mouseMoveEvent(event);
}

This works, but it’s awful. Really awful. And it could break in future Qt versions. But for now, it works.

So I’m wondering — is this behavior by design, or have I uncovered an odd Qt bug?

Update: It looks like others have encountered the same problem. Well, here’s a work around. But is there a more correct solution?

Vim regex - remove kind-of-matching lines

I have a file where every line starts with a number (followed by whitespace and a bunch of other stuff). Every number appears on either one or two lines, and if two, the second line always has a b after the number.

I need to delete every line for which there's a corresponding b line. But if there's no corresponding b line I want to leave the original line there.

Before:

123 foo bar
456 blarg
789 quux
123b foo baz
789b quux blurble

After:

123b foo baz
456 blarg
789b quux blurble

Except in my real file, I have a thousand lines and it'd take a year to do by hand. Vim to the rescue:

:sort
:%s/^\v((\d+).*\n)(\2b.*)/\3/

And that is why Vim is awesome. Can you think of a shorter way to do this, in Vim or Emacs?

Ads on license plates?

What if when your car stops at a red light, your license plate displays ad banners? What could possibly go wrong?

Quoth the person(?) who wrote this bill:

"We're just trying to find creative ways of generating additional revenues," he said. "It's an exciting marriage of technology with need, and an opportunity to keep California in the forefront."

The forefront of annoying the hell out of people. Certainly what I need is more distractions on the road. I mean, what if there's a new brand of toothpaste and I didn't find out yet? Someone somewhere needs to earn a dime for telling me about it by any means necessary.

I'm just waiting for the first company to propose paying new parents a few hundred dollars to tattoo ads on their babies.

Biting the Hand That Feeds

Granted, I’m not a Groklaw junkie. Lawsuits are the epitome of bureaucratic boring and are not a creational activity. So, keep in mind that I only read the occasional major headlines from The SCO Group’s escapade in futility. It does come as a relief that the show is finally over.

The situation in many ways paralleled the AT&T UNIX lawsuit of the ’90s. It is every bit as ironic considering, for instance, that BSD folks were largely responsible for the success of UNIX. In SCO’s case, their OS (SVr4 UNIX) is based largely on the work of Research UNIX, BSD, and even includes a wide selection of GNU tools in userland.

Except this time, no offending code was ever demonstrated, just a straw-man argument and utter defeat.

Still, I feel an overwhelming sense of sadness over the whole affair. Santa Cruz Operation and System V UNIX were respectable in their time. My guess is that SCO along with UnixWare (the natural evolution of System V) will fade into oblivion along side the countless other dead UNIX implementations. Linux zealots often take jabs at other implementations, but I think UnixWare could have held a viable, if niche, place in the enterprise had been under proper stewardship.

As time goes on, the UNIX diaspora seems to be waning. We are left with, essentially:

• Linux, with a mostly GNU userland as the heavyweight contender
• The BSDs, perhaps equal or greater in architectural quality but relatively unknown giants. We can however lump Mac OS X in here which is the most widely used.
• Solaris, which might be considered an open source System V fork. An interesting OS that has a a great lineage and potential, but lacks in trust and certainty for contributors at the moment.
• AIX, a System V and BSD hybrid with plenty of IBM thrown in for good measure. Perhaps the gold bar and only remaining competitive mid-iron standard due to IBM’s silicon prowess.
• HP-UX, an older System V linage perhaps on a slow deathbed due to reliance on the vapid Itanium

History has been unforgiving to those companies that try to unfairly weasel programmers and users in this market. The lesson is to work with and encourage your development community and not bite the hand that feeds. My eyes are on Oracle for the time being. Sun had a hard enough time nurturing the Solaris community despite being a favorable company, and Oracle can just as easily kill this operating system through boneheaded maneuvers if it is not careful.

Related posts:

1. Linux and the PC Mentaility Here’s a minor thought dump of the moment: no hard...
2. Mobile phones spur cross-platform applications. Open Source is mainstream. I had an interesting day today.  At school, we had...
3. My thoughts on software and complexity My thoughts on the growth of the Linux kernel and...

Paludis 0.48.0 Released

Paludis 0.48.0 has been released:

• --reset is now the default for git syncers. If Paludis is used to sync a git repository with modifications, those modifications will be lost. Use --no-reset to stay with the old (git pull) behaviour, but note that it is probably better to change your workflow such that Paludis never directly works with any repository it does not manage.
• environment.conf is now general.conf. Your existing environment.conf should be renamed.
• When switching UIDs, we now also pick up supplemental groups.
• World is now updated for package moves.

Printer spam: what could possibly go wrong?

As further evidence that there are no depths to which companies won't stoop when it comes to advertising, HP has come up with a great idea: Get people to hook their printers up to the internet and then spew advertisements out of their printers.

Well, it's a win-win situation for the companies doing the advertising: Not only will people see your ads, they'll pay for the ink and paper to print them. Maybe not such a great situation for the end-user though.

And then there are the privacy implications of targeting ads based on geolocating the IP address of the printer. Which I find a bit disturbing, but I guess advertisers already do that with online ads. But wait, there's more:

Ads can also be targeted based on a user's behavior as well as the content, said Vyomesh Joshi, head of the HP's Imaging and Printing Group.

Looking at what I'm printing so you can try to sell me things? Just a bit creepy.

Most troubling to me is the intrusiveness of the whole thing. They're taking control of a physical object in my house and using it against me. May as well kidnap my cat and train him to spell out "BUY PEPSI" in his cat litter.

Quote some slimeball at HP:

"What we discovered is that people were not bothered by it [an advertisement]," Nigro said. "Part of it I think our belief is you're used to it. You're used to seeing things with ads."

Translation: "We know this is a really horrible idea, but if people are complacent enough to sit there and take it without complaint, what's stopping us?"

He's right though, people are used to it.

I guess TV, radio, internet, phones, product placement in movies and games, print media, billboards and the postal service just aren't enough. Clearly what the world really needs is another ad-delivery mechanism.

Restoring ssh connections on resume

I use pm-utils for hibernation support.
It has a hooks system which can execute stuff upon hibernate/suspend/thaw/resume/..., but they run as root.
If you want to run stuff as a regular user you could do something like

su $user -c <command>

The solution is very simple. Use this:

sudo pm-hibernate && do-my-stuff.sh

Adobe support, the FSF and double standards

Flameeyes has a great article over on his blog about the recent shift in the perception of Adobe in the FLOSS world.
So I am not the only one who sees some weird stuff going on

Effective data visualisation for the WIPUP dashboard.

Data visualisation is a big, big topic. It’s often underestimated by the majority of society. But what is data visualisation? It’s not so much how effectively you present data – no, it’s how effectively you deliver information. Today since work has resumed on WIPUP I decided to take a look at how to implement effective data visualisation in the WIPUP dashboard. Before we continue, let’s take a look at how the dashboard looks like now:

Obviously unfinished, the dashboard looks like somebody vomited. Disregarding the obvious visual glitches such as the 0% segments in the pie chart and questionable use of colours, the most obvious thing that immediately catches our eye is … nothing. That’s right. It’s a mosaic of data, no clear correlations or focus is delivered to the user. Well, what would you expect if 6 charts were thrown at you? It’s not hard to see that there is a problem here to be fixed.

The first step is to consider, based on the results of dogfooding, which graphs aren’t realy useful to the user at all. The two obvious ones are the comment activity and the popularity based on subscribers. If, hypothetically, there was a popular WIPUP user, the usefulness of these two will increase, but overall it’s quite agreed that the majority of users will not find it useful, and even if showing a variety of data, we have to look at the purpose behind comments – to initiate discussion. Such a qualitative concept is not well presented with quantiative means and thus should not be attempted. Looking at the subscribers chart, it may be useful, but doesn’t warrant its own graph.

The second step is to consider exactly what medium we are using to present the data – in this case we are using line charts and pie charts. Line charts are appropriate here as it is used with a timescale, but as for the pie charts – well, experts unanimously agree that there is absolutely no usecase for piecharts. Pie charts should not have been invented. They are useless at showing data. It is hard to determine which segment you should focus on, and the colours distract rather than emphasise the data. Even worse, I have commited the ultimate evil of creating a 3D pie chart. Here’s something you should try out – cover all the percentages of the “activity per project” pie chart, and look at the two large segments (purple and green). Looks about the same proportion, eh? Nope. There’s a stunning 10% difference between them. Try the same with the orange and yellow together – big relative difference again. This trick even works with groups – the orange and yellow segments together look around the same as the pink and lime green, no? Yep, again, there’s a big difference. Conclusion: piecharts suck. The only benefit is that they look pretty.

The third step is to search for correlations. The most obvious one is how many updates I have made and how many views I’ve received (duh). The important thing here is that from the user perspective, they want that correlation. They want to see that the more they update, the more views they get. Thus we should try and emphasize this correlation. How better than to merge the two graphs? This way any mismatch of this ideal correlation (ie, an update they thought would be interesting, but from the statistics, show otherwise) would be obvious.

After a quick brainstorm, I copied the raw data onto OpenOffice Calc (equivalent of Microsoft Excel), and tried this alternative:

As you can see, I’ve compressed the data of 5 graphs into 2 graphs (say, I didn’t know Calc could do 2 y-axes). On the left, we see a stacked bar chart showing activity, kudos and subscribers (trackers are added as a subscriber to all projects, which is why it is all equal). Immediately the user can create a focus and see correlations. For example, this allows us to sort the data in terms of most to least. In this case, I’ve spent the most of my updates on uncategorised updates. However, we can now easily determine the correlation that even though the activity spent on uncategorised work is the most, it doesn’t create as big an impact as say, the Evan project, of which it is clear that the Kudos+Subscribers:Activity ratio is much greater. Infinitely more useful, don’t you think?

On the right, we can see we’ve showed the correlation by merging the two line graphs. Now we can clearly see that there is a slight correlation misalignment in the update I did on the 07/06 – perhaps the update wasn’t very interesting? Or perhaps it’s simply because I wasn’t as actively giving out updates as before – all very interesting conclusions.

Well folks, I hope I shared some useful stuff today. Next step, implementing the changes and revamping the dashboard.

Related posts:

1. The Eadrax Dashboard
2. After the WIPUP release, the stats are in.
3. The WIPUP 21.02.10 stats are out.

Introducing falscan

falscan is a very simple port scanning utility that I have written based entirely off of a program called pscan I found which was originally written in Ruby. As it’s name implies its written entirely in the Falcon programming language.

There are some caveats at this point though. Mainly it doesn’t really work. The basics are there but there are a couple of rough spots I need worked out before I’d really call it working. Hopefully someone will submit a patch. I have, after all, done all of the hard work.

You can find everything you need here:
http://github.com/steveno/falscan

Enjoy the Penguins!

Taming Firefox memory usage

Firefox seems to eat memory like rats in a wheelie bin.

Successful programs often become so complicated that they are like operating systems in themselves, so complicated that they struggle with system type issues such as having perfect garbage collection, Firefox is one of these!

The irony is that Firefox's predecessor Mozilla, become too complicated, bloated and resource hugging. So Firefox was re-factored out of the code base as a small and fast browser. Now it is repeating the same cycle. This is not an entirely fair comparison, as Firefox has to deal with a Web filled with high resolution images, video and so on that Mozilla never did in its day.

Anyway, when Firefox goes into a crawl I have just ignored it, just restarting Firefox every so often when it hogs RAM. I also occasionally clear out the cache, history and all the other crap in the hope that it will help too.

I noticed today that I had only five tabs left open yet 86% of the laptop's memory is being used by Firefox, when I looked a bit further into it, a good chunk of that was on a process called npviewer.bin - i.e. Flash. 46% of my laptop's memory (2 GB) was being spent on Flash!

Finally something had to be done.

I decided to install an extension called Flashblock. This prevents the downloading of flash content until I agree to it, any Flash content is replaced by a play button.

Extensions can cause memory leaks, so I decided to remove all the extensions I don't use any more, and to disable ones that I use very rarely. We will see if my Firefox is now more nimble.

Memory Settings

One can set configuration settings to limit the amount of memory that Firefox uses to cache pages and store session history. As it happens, I decided not to do this yet, in order to see what effect the above steps I have taken make.

You can look at what is currently in Firefox's memory cache by putting this into the location bar:

about:cache?device=memory

So on the available settings. For each you need to type about:config into the address sbar, in order to add the settings.

browser.cache.memory.capacity

This limits the amount of memory that Firefox will use to store images. Set to the cache size you want in KB. So to set the maximum to 24MB give the number 24576.

browser.sessionhistory.max_total_viewers

This limits the amount of memory that Firefox will use to store old pages. This speeds up going backward and forward through pages.

config.trim_on_minimize

This one only works on Windows. If set to true, when Firefox is minimised, the cache is swapped from RAM to disk.

Browsers and speed

There is an adage I have often heard in mailing lists which goes something like this: 'Software can be fast, cross-platform or beautiful. Choose any two'.

The hope was always that the GNOME project would come out with its own light browser that would be beautiful and efficient, beating the lumbering yet cross-platform Firefox. After many promising developments in Epiphany and Midori, this has yet to happen.

So what browser do you use? Has your Firefox had slow moments too? Any good tips to share with us?

Emacs creating zombie buffers

Note to self:

Using bleeding-edge swank-clojure and Slime, Emacs was creating buffers that didn't point to any file, every time C-c C-k wanted to display warnings. This was really quite annoying as it resulted in a mess of empty buffers being created. This fixed it:

(setq slime-highlight-compiler-notes nil)

Managing Remote Desktop Sessions

open vs. closed

Calling time on mobile internet nonsense?

Data was first sent down a phone line during the second world war. In the 1950s and 60s, the military and universities already had modems connecting computers together. Bulletin board systems were (in theory at least) available to everyone with business or home computers by the early eighties. However, they were slow and (at least in the UK) stupidly expensive to use.

By the mid-nineties 56k modems were increasingly common. 56k was a theoretical maximum, often one was lucky to get half of that. At least for text, the web was becoming somewhat useful, annoying still, but useful.

However, the web only really becomes ubiquitous with broadband. The web works best when you don't have to watch the clock and you don't have to worry about the quantity of bits coming through the pipe.

Would social networking sites work if we had to count bits? Shall I be your friend on Facebook, well if you make 500 updates a year and it costs me 1p for each, is hearing about your dog's toothache worth £5 per year? Is signing up to this blog's RSS feed worth 50p per year? I would leave a comment on this article but I have been online for 58 minutes already and I am not sure I will make it. Video and audio would certainly not work in a bit counting world.

One of the problems of the wired Internet was that for a long time ISPs sold services they could not deliver. An ISP might sell you 1M/s broadband, however if more than a small proportion of the customers actually used 1M/s, then the network would grind to a halt.

A kettle uses up to 2500 Watts. Tonight England is playing in the Football World Cup, at half time, millions of Britons will put the kettle on during the same 15 minutes. Does the electricity supply stop? No it doesn't. If I put my kettle on, does it stop other houses in the street from using electricity? Of course not. Electricity suppliers do not suspend accounts for breaching 'fair use'. What a stupid phrase, we are going to sell you X, but it is only 'fair' if you use X/Y.

Why this scandalous nonsense was never outlawed I don't know. However, my point is that by 2010, at least where I am, now we can just use the wired internet with confidence and not care about counting minutes or bits.

Whenever an organisation loses track of its core purpose then it has entered a downward spiral. When the kernel is removed, all you are left with is a shell. When the fire goes out, you are left with ash. (insert your own metaphor here.)

The role of an Internet Service Provider is to provide a pipe; customers send requests down it to remote computers and receive responses back along the pipe. An ISP should provide great customer service (i.e. responding to blockages in the pipe), but otherwise they should just get out of the way.

Whenever an ISP tries to do something different, for example provide their own content/advertising/tracking systems/censorship/etc then they just run up costs and put off customers.

So now we get to 3G. This is not the first time phone companies have tried to deliver mobile broadband. In the 1990s they offered WAP, where they promised the web but delivered a poor approximation of Teletext. Instead of giving the people what they want, they tried to give people what the phone companies decided they wanted. It failed miserably. For more on this read Why WAP was crap.

When you turn on the shower, a man from the water company does not get into the shower with you. When you flush the loo, a man from the water company does not wipe your bum for you.

With 3G, phone companies are becoming ISPs. Being an ISP is being a utility, as I mentioned above, you provide the pipe and let people get on with their lives. It is not sexy, it does not lead to excess profits, but the future of 3G is to be another pipe for bits. It will only be ubiquitous when we can use our 3G cards and devices without caring about how many gigabytes we download.

If a company is promoting a service as unlimited, then it should be unlimited. Unlimited followed by a disclaimer is not unlimited. Companies need to stop lying and stop selling bandwidth that they have not got. One day we will have a true unlimited service, until then don't call it that.

Mobile Devices need to be better at measuring usage, and the user should be able to set the device to stop working once it reaches the ISP's arbitrary cap. Regulations also need to be tightened up so that overuse is charged at a sensible rate. Paying £5 for the first GB and then £15, £30 or more for subsequent GBs makes no sense. Tricking people into hefty charges is not a fair way to treat customers.

The mobile phone company 02 sold iPhones and service plans on the basis that subscribers would receive 'unlimited downloads', people invested in the expensive device on that basis. Now 02 found that its network cannot cope and has switched to reverse gear. (for more see the BBC report).

It is about time that we as customers put an end to all this nonsense. The way to do that is by providing better information. It is about time that we had a customer's charter detailing what is fair and what is not fair, and an independent website auditing service plans against the charter. Then as customers our responsibility is to vote with our feet, punishing companies that do not comply.

What do you think? Are Mobile Internet plans fair? Is there something regulators should be doing? What would be a fair mobile Internet plan?

Posts drafting away …

A few days ago I went meta again and wrote about writing, I mean if you’ve read more than 3 posts here you know my spiel, I’m basically mutating the same 3 or 4 ideas around new contexts. In that post I talked about how some posts are hard to write, let me be all pretentious and quote myself:

On the other hand there are posts that usually have meandered around in my head for weeks or months, ideas that have kept my mind occupied for a long time and that I try to meld into a post at some point in time. And usually those posts are really hard to write, I trash many of them, rewrite them, trash the rewrite.

This lead to Andy C adding to the comments:

Interesting. Your blogging workflow is very different from mine. I get an idea, I type the words, I fix typos and then I just hit ‘Publish’.

I never have any Drafts sitting there. In fact, the very thought just irritates me.

I replied quickly and thought it was done but on my way home tonight I came back to this, to the process of writing and stuff.

Many people do not blog because they say: “But I don’t have any ideas!” I rarely hear that people don’t have time or skills, it’s usually about “not having anything to write about” or “not having ideas”. Which is something I just don’t believe but I know how that perception might emerge because I know it from myself.

I have ideas for short and long articles all day (which sucks when you are concentrating on writing something and another thing wants your mind’s attention) but obviously I don’t write them whenever they come to my head. So I put them on a mental “yeah write about that” list. That list has a problem, it disappears all the time.

An idea might bounce around in my head for 10 hours throughout the day but when I have time to write recreationally it’s empty, all the cool and awesome posts are gone. A few days later they might pop back up, they might not. I lose many many ideas that way every month and that sucks. Not cause all of them are so bloody brilliant but because if I had more than just one idea to write about I could chose the best one and wouldn’t have to run with the one I got at a particular moment.

For a while I tried writing “Drafts”. I would add some sort of heading and a few very rough bullet points and come back later going through the drafts to decide which one to work on for a post. Basically none of them ended up being written, when the amount of drafts reached a certain point I would just go draft-berzerk and kill them all. I am not a draft person, it’s just not a way that triggers my brain.

I also tried paper for a while but my pants rarely have enough room as it is now and that makes it hard to carry even more stuff so that never worked out. Another thing that kills that kind of process is my lack of a proper handwriting, I usually cannot read my handwritten notes after three days.

I have not given up looking for solutions to this because losing ideas hurts. As I said, not cause they are all awesome, but cause I hate losing something without having had the chance to work on it. But right now I probably have to accept that I lose ideas every day. Writing, as any creative task, is very personal I guess. Things that work for someone else might absolutely kill your workflow and inspiration, on the other hand a pointer from someone else might make you realize how much you’ve been fighting your own brain.

Understanding how one’s own brain ticks, what kind of things capture its attention and what slows it down is probably one of the most important tasks for anyone who wants (or has to be) creative. It’s about finding your way of doing things because most ways won’t work. All those shiny “10 rules to follow for blogging” howtos, all those gurus that tell you about structure or simplicity or whatever: Read it, but read it like someone describing their own favorite dish. You might like it, you might like parts of it, but chances are that you will hate it.

kde activities on multiscreens

It’s not you, it’s me

Dear FSF.

I’ve been using FLOSS (free/libre/open source software) on my computers pretty much exclusively for about 5 or 6 years now and I don’t see me going back to the limited lands of Windows or OSX anytime soon, but I have started to feel some sort of disconnect to one of the leading institutions of FLOSS lately: The Free Software Foundation (FSF).

The FSF has achieved a lot, has funded and supported a lot of the GNU projects and software developers, has defended the GPL (my preferred license for software) in court and in general have had a huge part in bringing free software to where it is today. I am not ungrateful, it’s just … it’s not you, it’s me.

When I look at some of the recent FSF campaigns I feel ashamed for being seen as associated with it: Windows7Sins is just the prime example for the kind of attitude I can no longer stand (DefectiveByDesign isn’t much better). What I can not longer stand is the vocabulary of hate and judgement, the negativity of it all.

Look at the campaign name: Windows7Sins. Sins are a religious concept, one that doesn’t apply to software but one that shows what kind of broken self perception the FSF has: That of a missionary, of someone who can, from the moral high ground, cast judgement down onto everybody else with a holier-than-thou attitude.

Those campaigns are full of negativity, of nagging and most importantly talking about issues that do not even reach their target audience. If you want to make people use your software or support your cause, it’s not enough to just run around screaming how “evil” the others are, that makes you look like a raving lunatic. And that is not my way.

While I do have ideals and while I do consider Freedom Zero to be important, I don’t preach because in the end that is my decision. It’s not my place to tell people what they are not allowed to do because of my rules. What I can do is to lead by example. To be positive.

I am a very positive person, even if I might might look ranty or grumpy from time to time, and these campaigns that focus on whatever other people might have done wrong just bore me. No. That’s not enough. They suck the life out of me.

If you want people to adapt your ideals or products you gotta show them why they are better than what they have been using: Tell them about the brilliant things they get when they use your stuff, tell them about new possibilities. The FSF should focus on outlining what positive things a new users gets from FLOSS: Tell people about VLC that allows them to play basically every type of media without hassle. Don’t talk about why IE is evil but show them what an awesome browser Firefox is. Just drop the hate.

In my life my message has never been one of hate. And I feel that lately our goals and approaches no longer line up the right way. You probably didn’t change, you were always like that, maybe I changed or got tired of that attitude, I don’t know. It’s not you, it’s me, but I think we can’t really let you talk for me anymore.

Thanks for all you have done and good luck with your future projects and campaigns,

tante