Planet Larry

XML databases have not really broken through yet in a big way, primarily because SQL has proved more resilient than expected in storing a wide range of data.

SQL has limits

For example, object-relational mappers such as SQLAlchemy, the one inside Django and ActiveRecord in Ruby, have become very popular recently, as they all you to serialise in-memory objects and store them in a database. Very little SQL experience is required and you can create very elaborate relational storage.

However, when you have data that is not relational at all, converting it to rows and columns can be either lossy or a lot of work, and often you end up unable to round-trip, i.e. you can get the data in, but you cannot get it out again in the original format.

The most common approach is not to even try, i.e. often people will not store XML in a database, but run a series of XML files though some XSLT stylesheets and then store the compiled HTML in a database. This removes XML's advantage as a processable exchange format. In order to get the data in a different way, the human has to go back to the beginning of the pipeline and rerun the complication and database import. If you want to allow other computers to make flexible requests over the Internet, well then you are out of luck.

Meanwhile, a lot of the XML extensions to SQL databases are still relational underneath, for example, with MS SQL, you can give it XML, which it then splits up into rows and columns; you can then also ask for the data back in XML, which it then retrieves from the rows and columns and outputs in XML. You have not actually gained very much however, as your data is still stored in rows and columns, you have to output the data before you can process or query the XML. This can prove very long-winded with highly complex hierarchical data, it can sometimes prove faster to just dump the XML to disk yourself.

Even worse is storing a whole XML document within a single SQL field in a single row. Again, this is using a database instead of a hard disk.

Elliotte Harold's article, Managing XML data: Native XML databases is a few years old so the software examples he mentions are probably a little old, however the discussion of why one might want an XML database is quite good, this interview with Jonathan Robie is also worth browsing.

Berkeley DB

Berkeley DB, more commonly BDB or just DB, is the classic embedded database, Started in 1991, it is used more or less everywhere. Despite some recent completion from SQLite, it is still by far the leading embedded database. It bundled inside in everything: Linux, BSD, OS X, OpenOffice, Python, Apache, Sendmail, Postfix, subversion, GNOME and no doubt millions of other things.

DB, like much of software that powers the world, was started at the University of California, Berkeley. It was then spun off in Sleepycat Software, which was bought by Oracle a few years ago. Note DB has nothing to do with Oracle's eponymous relational database product. They are at the complete other end of the spectrum, Oracle database is for large corporate data storage, while DB is a fast and light embedded database.

DB is also non-relational, i.e. there is no SQL interface, there are key-value pairs stored in byte arrays and that is more or less it. If you want SQL or network interfaces, then you stick that on top, for example DB is one of the back-end options for MySQL.

Berkeley DB XML

The folks at Sleepycat/Oracle have been working on Berkeley DB XML for a couple of years, but it is not so well known as its older brother. DB XML is a layer on top of DB that forms a fully XML-native database with "XQuery-based access to documents stored in containers and indexed based on their content". (Source)

There are two good things about it. Firstly, it is an XML database written in C and C++, not in Java. Secondly, it is a proper XML database, not putting XML into SQL columns.

The other notable XML database is the Java-based eXist which seems to be more well known, primarily among Java XML developers. eXist is more of a service, rather than something light to embed into applications. It also rumoured to be significantly slower than DB XML, but I have never used eXist so I can't show any benchmarks for that.

There are also a load of proprietary and half-finished XML databases we do not care about. Including the abandoned Java XML database called dbXML which has no relation to the Berkeley on but confuses everyone because the sourceforge page is often the first search engine result.

Install Berkeley DB XML - Windows

Go over to the Oracle/Berkeley DB XML homepage and grab the installer. I have not used it so I have no idea what happens, hopefully it is straightforward.

Install Berkeley DB XML - Posix platforms

DB XML works on all major posix compatible systems, including Linux, OS X, BSD, Solaris and so on. Installing it varies according to distribution. If your distribution has packaged it, then lucky you, you just install it through that. Of the two distributions I usually use as examples on this site, Gentoo Linux and Ubuntu Linux, the former has a DB XML package, the latter currently does not.

Install Berkeley DB XML - Gentoo

sudo emerge dbxml

The package is new, so if you are running stable, then the emerge command will moan and tell you that the package and some of the dependencies are not yet in stable. You will need to add these to /etc/portage/package.unmask, see Using Masked Packages in the Gentoo Handbook for more details.

Install Berkeley DB XML - Ubuntu

Hopefully, in the near future, you will be able to go:

sudo apt-get install dbxml python-dbxml

But we are not there yet. When we are there, the rest of this post will be irrelevant. Therefore this following part will date badly, I will try to remember to update it as events unfold, but if you are reading this paragraph in 2009 or later, it means I have forgotten and it might be worth checking other sources.

What follows shows how important packagers are. Almost any complex package will need to be optimised for that distribution, and normally all the work is done for us behind the scenes. The pointy-heads in the Mozilla corporation that didn't want Firefox to be patched downstream in the distributions are not living in the real world. But that is another story. Let's get going.

Dependencies on Ubuntu

First, we need to get the dependencies. We need Berkeley DB, xqilla and libxerces, the latter two are not all yet in the default Ubuntu repositories, but my mate txwikinger has packaged them for us in his private archive. You need to add these lines to the bottom of your /etc/apt/sources.list with a comment so you remember why you put the lines there:

# Deps for DBXML
deb http://ppa.launchpad.net/txwikinger/ubuntu hardy main
deb-src http://ppa.launchpad.net/txwikinger/ubuntu hardy main

Then run: sudo apt-get update

sudo apt-get install libdb4.6++ libxerces28 xqilla

sudo apt-get install libxerces28-dev libxqilla-dev libdb4.6++-dev

Working directory

We need to have a working directory, in my examples we will assume ~/Sandbox, but use where you like.

mkdir ~/Sandbox/

cd ~/Sandbox/

Go over to the Oracle/Berkeley DB XML homepage and grab the tarball and extract it into your ~/Sandbox directory, or you can just use the commands:

cd ~/Sandbox

wget http://download.oracle.com/berkeley-db/dbxml-2.4.13.tar.gz

tar -xvvzf dbxml-2.4.13.tar.gz

Compile DB XML

The tarball from Oracle contains a lot of dependencies which we have already installed. So ignore the 'buildall' instructions, as they compile all the dependencies which takes all day. You then have half a dozen or so packages that are in your main system folders but are not managed by Apt, this is not really the done thing.

Now because we are doing it the proper Unix/Ubuntu way rather than the Oracle big-tarball-of-mud approach, we want to use the shared library of xqilla that we installed earlier. There is a slight problem in the configure file which stops it from working. Open the following file in your text editor:

~/Sandbox/dbxml-2.4.13/dbxml/dist/configure

In line 4396, change .la to .so

So the line:

elif test `ls "$with_xqilla"/libxqilla*.la 2>/dev/null | wc -l` -gt 0 ; then

Becomes:

elif test `ls "$with_xqilla"/libxqilla*.so 2>/dev/null | wc -l` -gt 0 ; then

Now we can build DB XML

../dist/configure --with-berkeleydb=/usr/lib/ --with-xqilla=/usr/lib/ --with-xerces=/usr/

cd ~/Sandbox/dbxml-2.4.13/dbxml/build_unix

If that works without error, you can type:

make

sudo make install

Now if that works without error, you can try to run dbxml:

/usr/local/BerkeleyDBXML.2.4/bin

./dbxml

Type help to see the list of commands. Press Ctrl+D to quit when you have had enough.

Python Bindings on Ubuntu

So far so good, now we need the Python bindings:

cd ~/Sandbox/dbxml-2.4.13/dbxml/src/python/

Now we have to do a little more patching to let the Python bindings know where dbxml is installed.

Open setup.py in your text editor We are going to add three lines, in each case make sure the indentation lines up with the line above it.

After line 18, add the following line:

db_xml_home = '/usr/local/BerkeleyDBXML.2.4'

After what is now line 65, add the following line:

INCLUDES.append(os.path.join(db_xml_home, "include"))

After what is now line 69, add the following line:

os.path.join(db_xml_home, "lib"),

Now in dbxml-2.4.13, there is another little bug we need to deal with first.

Open the following file in your text editor:

~/Sandbox/dbxml-2.4.13/dbxml/dist/swig/dbxml_python.i

Go to line 533, and change 'Vaue' to 'Value'.

So change:

class XmlInvalidVaue(XmlException):

To:

class XmlInvalidValue(XmlException):

Now this file eas used to automatically generate the Python bindings, so the generated file needs to be fixed too. The dbxml developers need to generate them, but until they do, we can just fix it ourselves. Open the following file in your text editor:

~/Sandbox/dbxml-2.4.13/dbxml/src/python/dbxml.py

Go to line 121, as before change:

class XmlInvalidVaue(XmlException):

To

class XmlInvalidVaue(XmlException):

Now we can finally install the bindings:

python setup.py build

sudo python setup.py install

Now lets test it, which involves, yes you guessed it, more patches:

cd ~/Sandbox/dbxml-2.4.13/dbxml/examples/python/

Now edit examples.py using your text editor. Remove the 3 from line 9. So from:

from bsddb3.db import *

To:

from bsddb.db import *

Now you can go:

python examples.py 7

Which should give you:

Running example 7.
book1 = <book><title>Knowledge Discovery in Databases.</title></book>

If you have that then everything should be in working order. This has been a very long post so I will break here and come back to DB XML in Python another day.

Thus continues the ongoing saga of Westinghouse, their low-quality electronics, and their incompetent or evil (I can't decide which) customer service department, who likes to take your broken crap and never bother sending you a replacement.

I filed a complaint with the Better Business Bureau in California, and Westinghouse never bothered responding to my complaint. The sort-of-good news is that Westinghouse's AAA rating has dropped to CC. There are 58 unanswered complaints listed on that site. I hope their rating keeps dropping until it lands squarely in the mud where it belongs.

Not that the BBB rating is going to do much good in the bigger picture, and it's not going to return the money I wasted on the monitor I'll probably never see again, but every bit of new business I can prevent that company from attaining is another victory in my mind. I've already cost the company far more money in new business than it would've cost them to just send me a fixed monitor to begin with. Plenty of people have let me know they aren't going to buy Westinghouse brand from now on. For every person that bothers leaving me a message on this website to that effect, there are likely many more silently avoiding the company, which is good.

I think another good idea would be for people to file some complaints with the California Attorney General's Office. Oh how I wish it was financially feasible for me to fly to California for a week and sue them myself. But maybe the AG can take action if enough people make a stink.

I'm going to be writing a letter to Westinghouse's corporate office, and I'm going to post their reply or lack thereof. I have little hope that I'll ever hear a response.

I've been getting a lot of people leaving comments describing their dealings with Westinghouse, and their encounters with Westinghouse's policy of making crappy broken electronics, then giving you the cold shoulder when you mail their crap back to them and try to get it fixed and returned. I feel bad (and angry) for everyone in the same boat as me, trying to get the people at this shyster company to do their damned jobs, so I'm not going to keep quiet about this.

(Previous posts: The beginning, Update 1, Update 2, Update 3, Update 4, Update 5, Update 6.)

Option         "UseDisplayDevice" "DFP-0"

1. Firstly, compile everything with xinerama support.

USE="... xinerama"

# emerge -vauDN world

2. edit xorg.conf

• -c uses a specified input file. I do this so I can start from the same point each time, rather than a constantly changing xorg.conf file
• -o writes to this file
• --twinview enables twinview
• --dynamic-twinview enables dynamically change twinview settings
• --twinview-orientation=LeftOf sets the second monitor to be left of the primary
• --twinview-xinerama-info-order=DFP-2,DFP-0 says that DFP-2 (DVI) is the primary display
• --constant-dpi --use-edid --use-edid-dpi --damage-events --render-accel are other settings I play with to fine tune performance.

3. Restart X

• log in screen on one display only
• desktop extended to second display
• windows open completely on one display and not in the middle
• windows snap to the edge of one display
• windows can be dragged to the second display
• both displays can be treated like one large screen (my geometry is now 3840x1200!)
  

I like PuTTY, I’ve been using it for years. In my last post I talked about Cygwin and it’s rxvt (which already works pretty good). I also joked about using PuTTY to ssh into localhost.
Well, for the people who want to use PuTTY for their local Cygwin there is a solution: PuTTYcyg. PuTTYcyg is just a patched version of PuTTY which adds “cygterm” as a connection-type. If you wan’t to use it with a desktop/tray shortcut you have to call putty.exe -cygterm - where putty.exe is of course the patched PuTTYcyg version.
The reason for me to use PuTTYcyg instead of rxvt is the Full-Screen-mode of PuTTY. Besides that I guess there are other advantages over rxvt (like being able to change font and terminal-emulation-options for the running session). Oh, and don’t fear, of course it supports 256 colors just fine ;)

I’ve removed two attributes from the schema. One is the living flag from the persons table, - it was really legacy data from TMG, and I have never used it much. YMMV.

The other one is the relation_type in the relations table. As it actually is nothing but a duplication of the parent’s gender attribute from persons, it’s an unnecessary complication of the model. The reason why I put it there in the first place, is as stated in the “Into the Blue” article, that I wanted to ensure that only one parent of each gender would be entered for any person. There are other ways to achieve this, and I’ll probably do roughly as suggested by Greg S. Mullane years ago, when I first pondered the problem. I’m still a little shaky about triggers, though. As long as I’m entering and editing relations from the application interface, where the link for adding a parent only will show up if there is no parent connected already, there’s no great danger of messing it up. So, I’m feeling quite comfortable without this constraint for now.

In order to remove the relation type, I had to rewrite a very essential piece of code, the get_parent() function. Here’s the old version:

CREATE OR REPLACE FUNCTION get_parent(INTEGER,INTEGER) RETURNS INTEGER AS $$
DECLARE
    person ALIAS FOR $1;    -- person ID
    rel_type ALIAS FOR $2;  -- gender code (1=male, 2=female)
    par INTEGER;            -- person ID of parent, returned by func
BEGIN
    SELECT parent_fk INTO par FROM relations
        WHERE child_fk = person AND relation_type = rel_type;
    RETURN COALESCE(par,0); -- will return parent ID if it exists, 0 otherwise
END;
$$ LANGUAGE plpgsql STABLE;

The new version must of course query the parent’s persons.gender attribute rather than the relations.relation_type:

CREATE OR REPLACE FUNCTION get_parent(INTEGER,INTEGER) RETURNS INTEGER AS $$
SELECT COALESCE(
    (SELECT parent_fk FROM relations r, persons p
        WHERE r.child_fk = $1
            AND r.parent_fk = p.person_id
            AND p.gender = $2), 0)
$$ LANGUAGE sql STABLE;

Note that I’ve rewritten the function in SQL rather than in PlpgSQL.

Now for some real testing of the two queries:

pgslekt=> explain analyze select parent_fk from relations where child_fk=1 and relation_type=2;
                                                      QUERY PLAN

----------------------------------------------------------------------------------------------------------------------
 Index Scan using child_key on relations  (cost=0.00..10.92 rows=1 width=4) (actual time=0.018..0.020 rows=1 loops=1)
   Index Cond: (child_fk = 1)
   Filter: (relation_type = 2)
 Total runtime: 0.047 ms
(4 rows)

pgslekt=> explain analyze select parent_fk from relations r, persons p where r.child_fk=1 and p.gender=2 and p.person_id=r.parent_fk;
                                                          QUERY PLAN

------------------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.00..27.48 rows=1 width=4) (actual time=0.049..0.053 rows=1 loops=1)
   ->  Index Scan using child_key on relations r  (cost=0.00..10.91 rows=2 width=4) (actual time=0.018..0.022 rows=2 loops=1)
         Index Cond: (child_fk = 1)
   ->  Index Scan using persons_pkey on persons p  (cost=0.00..8.27 rows=1 width=4) (actual time=0.011..0.012 rows=0 loops=2)
         Index Cond: (p.person_id = r.parent_fk)
         Filter: (p.gender = 2)
 Total runtime: 0.127 ms
(7 rows)

The first query runs in 0.047 ms, the second one in 0.127 ms. While the latter is three times slower, we’re talking fractions of milliseconds here. No big deal in a single-user application.

I’m writing this partly in response to Jürgen’s post a week or so back about passwords. Of course, he’s not the only one to advocate long passwords, a lot of people are doing that these days in the name of security. Today’s sad reality is that if your password is not “test” or “password” you are more secure than most people.

I do think, however, that any idea for improvement should stand to be evaluated on usability. After all, my first loyalty is to the user in me. Failing to do that produces wide adoption of bad ideas like captchas that are directly hostile to users. (Incidentally, that’s why so many people who build systems for others build them badly. The implication of using it every day never takes a foothold.)

Short passwords have too little entropy, therefore they are easy to break.  Granted. So the response is “use long passwords”, or better yet “not passwords, pass phrases”. Such as oh bugger, my cat has cancer. With or without the spaces and punctuation it makes a perfectly acceptable password in terms of length. But tell me now who is willing to actually type these monstrosities?

The evil of password typing is reduced by our methods to avoid typing them all the time. Use public keys with ssh, never type the password again. Save passwords in the browser, avoid typing those. It’s a fabulous usability gimmick.

But short passwords, bad for security, are great for another closely related purpose: being able to actually type them in. If you have a short password you don’t need much practice to be able to type it. It’s a sort of sweet spot between usability and security, more secure than nothing, not too painful to type if you have to. My password input rate might be something like 98%. I rarely fail to log in. But with pass phrases of 29 characters like the one above, how confident would you be? You don’t see what you’re typing either, just echo characters at best. I expect the likelihood of typing it correctly falls dramatically, maybe to as low as 75-80% for the average user, in the average point of his learning curve to learn typing it (does not apply to hackers with stellar typing skills yadayadayada). If you’re doing something once, 80% is pretty good odds. But if you’re doing it everyday, it’s no longer odds, it’s a statistical average. Imagine if those were your parking odds. One in five times you fail to maneuver through the opening of your garage, I don’t think you’d be happy.

I tested myself on cancer cat just now, 6/10. On a sentence I’ve never typed before. And that’s while seeing the characters on the screen.

And then there’s the chance that you’ll forget it, or remember it wrong, switch a character in your mind, use the wrong case. It’s hard to estimate how likely that is, but with long passwords it seems rather likely. Inputing passwords is not an approximation, it has to be exact. And it’s not just one of those phrases you have to remember *exactly*, you need one for every distinct password you keep.

Security is a social problem, not a technical one. If you force people to use long passwords they struggle to input (for christ’s sake, they *already* use post-it’s on the monitor), we will just embrace ways of avoiding passwords all the more. Passwordless ssh is great, but if I’m using every trick in the book to avoid typing my long password, I haven’t had enough practice typing it when I actually have to type it.

That is, if I even remember it correctly. And I somehow doubt sysadmins will give you more tries to type a long password than they currently give you, 3 tries or whatever it is. And then you’re locked out.

It’s the perfect anti-security. The bad guys have a shot at my account (but they have to be pretty clever), but I myself am locked out.

mutt 1.5.18 works fine with Cygwin (as I just discovered). Building it from source I used

./configure --prefix=$HOME/local/stow/mutt --enable-imap --with-homespool --with-ssl=/usr --enable-hcache --enable-locales-fix

This is assuming you installed the openssl-libs and openssl-devel packages (those can be installed using Cygwins own installer .exe). For header-caching (you really want it when using IMAP) you’re also gonna need some version of berkeley-db, which can also be installed using the Cygwin setup.exe.

msmtp (an SMTP-client which can be used in conjunction with mutt when sending mails over a server which uses TLS/SSL) works equally fine. With version 1.4.16 just do

./configure --prefix=$HOME/local/stow/msmtp --with-libgnutls-prefix=/usr

GNU screen is available as a Cygwin-package (4.0.3) but if, like me, you want 256 colors (and with a spiffy vim-theme you do ;) you have to fall back on building it yourself. You have to apply two patches though: Patch 1 (found here) and Patch 2 (found here). After that

./configure --prefix=$HOME/local/stow/screen --enable-colors256

should do the trick. To actually get 256 colors working use this in your .screenrc:

# terminfo and termcap for nice 256 color terminal
# allow bold colors - necessary for some reason
attrcolor b ".I"
# tell screen how to set colors. AB = background, AF=foreground
termcapinfo rxvt 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'
# erase background with current bg color
defbce "on"

Careful! If your TERM (the one you start screen with) is not rxvt then you need to adjust that. To see if 256 colors work I use this script (found here).

This article is part of the OQO odyssey series - You can find the previous posts here and here.

In general, installing linux shouldn’t be a great problem - I guess those were my words right before powering up the umpc again for starting the bootstrap process as I used to do. But I have missed several things: Battery and heat. The case of the oqo acts as a passive cooling element and got that hot I wouldn’t dare to carry it with me in a bag. As I said the oqo’s a hot thing, I wasn’t thinking of that.

Back to the topic. After trying to build the binutils package the battery was drained again and so I had to change my plans: building the packages in a chroot on my laptop. And so I did - unpacking stage 1, mounting portage, sys, proc and doing the bootstrap which didn’t take that long. Regarding the oqo, I just leave things as they are right now, as the partition scheme can be reused - even portage. You never know.

For keeping yourself busy while bootstrapping and emerging, I recommend a good old classical game like Rock ‘n Roll Racing on the SNES console, reading a book, housework - just something to keep yourself busy. Yeah - even browsing my blog will do fine as the process might take a while.

After the bootstrap process, emerge -e system builds 190 packets which means you want to do something else in the meanwhile. Housework and reading kept me busy - and hacking some configuration files. It won’t hurt, if they are in a good shape.

I haven’t had any further highlights at the installation worth being mentioned. Just one thing: The GRUB Setup should be done on the oqo for having correct data at the boot record. That’s all folks!

You know how when someone accidentally deletes their files or their hard drive crashes or some other apocalyptic event occurs, the first thing people ask is “where is your backup”? Of course, we’ve all seen it (*ahem* been there ). It’s a bit unintuitive, because backups have no equivalent in the real world. If you drive your car into a lake, there’s no way to get it back. But making backups is the single best way to prevent losing your stuff. So do it!

Don’t backup “my files”

But don’t just backup “my documents, my pictures, my whatever”. If you computer crashes and you have a backup of “my files”, then sure, it’s not a total loss. It’s better than nothing. But it’s not what you actually need. You need the whole thing

This “my files” nonsense is born out of the fact that the delightful company that produced your operating system doesn’t want you to be able to make a backup of it. Because if you did, you could make trivial copies of the operating system, and they don’t like that idea. Have you ever asked yourself why in 30 years, through all manner of viruses, blue screens of death and hardware crashes Microsoft has never given you or sold you a full system backup program? It’s not because they never thought of it. (Or because no one asked for it).

Making full backups is easy

If you’ve ever installed Gentoo manually (ie. not with one of the automated installers)… yes, the demographic for this one is not immense. But then that’s why we’re here, to spread the happy message! Anyway, if you have, then you know immensely easy (this is astonishing especially if you have a Windows background) it is to make a full system backup. In the course of a Gentoo install (and yes, I’m about to reveal the big secret here…*drumroll*), you boot from the livecd, you mount your root partition, you download a tar of a minimal Gentoo filesystem that has your basic /bin/ls and so on, and then you just…. untar it. That’s it. No magic, no voodoo, no secret foobared locations on the filesystem that can’t be written to, just extract the archive and you’re done!

To put it bluntly, this is all you have to do:

tar zcvf backup.tar.gz /

And to restore the backup:

tar zxpvf backup.tar.gz -C /mnt/root_partition

Put that in perspective to the Windows world where a whole industry has sprung up to solve problems that Microsoft deliberately introduced with their *ahem* novel engineering. Idiotic programs like Norton Ghost that you have to get your hands on just to do the same simple thing that you can do with tar on a decent operating system.

Making it more convenient

Granted, you could just use the above tar command, but you may want something a little more convenient. For starters, you may want to skip some files on your file system. The method I use is inspired by a script posted on the gentoo forums a long time ago. I used that script for years without really understanding it, but a while back I decided to rewrite it to suit me better.

Besides just tarring the files it also writes a log file that you can grep to see if some particular file of interest is in the backup, it timestamps the backup with the current date/time and it keeps track of how many backups you want to keep.

Backups are made in a special backup_dir location. This directory is supposed to hold lists of files (recipes, if you like) you want to backup. For example, a simple recipe could be called full.lst:

/
--exclude=/backup/*.tgz*
--exclude=/proc/*
--exclude=/sys/*
--exclude=/tmp/*
--one-file-system

Download this code: full.lst

The syntax for the file is that of tar, and it’s a list of things to backup. / means the full file system will be included. But certain directories are excluded, /backup because we don’t want to include our old backup files in new backups, /proc and /sys because they are virtual file systems and don’t contain “real” files, and we don’t care about /tmp. Finally, we say --one-file-system, which prevents mounted disks, cds and things like that to be included.

And here is the script that makes this possible. Run it, it will produce a backup file that is compressed. Try to get it below 4.3gb and write it on a dvd+rw, now you have a backup system.

#!/bin/bash
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
 
backup_dir=/backup
num_backups=1
 
 
verbose="$@"
lists=$backup_dir/*.lst
ext=tgz
date_params="%Y-%m-%d-%H%M"
nice_val="nice -n20"
 
# colors
wh="\\e[1;37m"
pl="\\e[m"
ye="\\e[1;33m"
cy="\\e[1;36m"
re="\\e[1;31m"
 
if [[ "$verbose" && "$verbose" != "-v" ]]; then
	echo "Usage:  $0 [-v]"
	exit 1
fi
 
if [ ! -d $backup_dir ]; then
	echo -e "${re}Backup dir $backup_dir does not exist.${pl}"; exit 1
fi
 
 
for list in $(ls $lists); do
	name=$(basename $list .lst)
	file_root=$backup_dir/$name.$(date +$date_params)

	stdout="1> /dev/null"
	stderr="2> $file_root.$ext.err"
	if [ "$verbose" ]; then
		stdout=""
	fi
 
	cmd="cat $list | $nice_val xargs tar zlcfv \\
		$file_root.$ext $stderr | tee $file_root.$ext.log $stdout"
 
	trap 'echo -e "${re}Received exit signal${pl}"; exit 1' INT TERM
 
	echo " * Running \\`$name\\` job..."
	if [ "$verbose" ]; then echo -e ${ye}$cmd${pl}; fi
	echo -en $cy; bash -c "$cmd" ; echo -en $pl
	status_code=$?
 
	if [ $status_code -gt 0 ]; then
		# Dump error log
		echo -en $re ; cat $file_root.$ext.err
		echo -en $pl ; echo "Tar exit code: $status_code"
	else
		# Kill error file
		rm $file_root.$ext.err
	fi
 
	# Evict old backups we don't want to keep
	num=$num_backups
	for evict in $(ls -t $backup_dir/$name.*.$ext); do
		if [ $num -le 0 ]; then
			rm -f "$evict"
		else
			num=$(($num-1))
		fi
	done
 
	# Report number of files in backup
	echo -n "$(wc -l < $file_root.$ext.log) files"
	echo ", $(ls -l $file_root.$ext | awk '{ print $5 }') bytes"
 
done

Download this code: backupsys

Worse is better

I’ve been thinking about how to handle backups most effectively, and it occurs to me that backups are a case of “worse is better”. The thing is you could make a really nice and easy application to make backups, but .tar.gz is still the optimal format to store them in. Wherever you are, you have tar and gzip available, and restoring backups usually happens under somewhat constricted conditions, sometimes without network access. So you want to avoid introducing dependencies, it’s safer to make do with the tools that are there already.

So it may not be the most elegant system, but it’s damn reliable.

Limitations (NEW)

Basically what I’m saying is that if you have no backup system then using tar is a pretty decent system. At the very least it has worked well for me the last 5 years. That isn’t to say you shouldn’t use a different method if you have different needs.

What about scaling? Well, I think this works quite well up to backups of say 4gb or so. My root partition is using 12gb of space at the moment. The purpose of this method is to back up your working system with all the configuration, applications and so on. Not to back up your mp3 collection, I would exclude that (not least because it’s pointless to compress mp3 files and other formats that already are well compressed).

What about the bootloader? (NEW)

Some people have asked how this backup method concerns the bootloader. The answer is that it does backup the files that belong to the bootloader (in /boot). It does not backup the actual boot sector of your hard drive (which isn’t represented as a file). So if, for example, you want to restore the backup on another computer (which I’ve done lots of times), you’ll still need to use grub/lilo to update the boot sector.

UPDATE: Apologies to the indignant Windows users. I pretty much wrote this in rant mode, without doing any research and what I wrote about Windows is just from my own experience. I would have been more thorough if I had known this would make the frontpage of digg and draw so many hits.

i got annoyed because i have four extra keys on my laptop keyboard that doesn’t
work as expected when i’m in gentoo. i’m callin’ them “compaq keys” coz i dont
know what they are called and its tiresome to know anyway…

in windows vista:

compaq key1: starts a program ‘dvdplay’ i think…
compaq key2: toggles the volume on or off
compaq key3: lowers the volume
compaw key4: does the opposite of key3

so i modified my keymap settings so that compaq keys2-4 are mapped to function keys
F13-F15 (i only have F1 to F12…)

-- .Xmodmap --
keycode 160 = F13
keycode 174 = F14
keycode 176 = F15

and my corresponding .fluxbox/keys file

 -- snip --
	
# windows key
Mod4 x  :maximize
Mod4 d  :ShowDesktop
Mod4 a :ExecCommand aterm +sb -tr -trsb -fg darkgray -fade 50%
Mod4 w :ExecCommand audacious
Mod4 e :ExecCommand /usr/bin/emelfm2
Mod4 b :ExecCommand /usr/bin/firefox
Mod4 y :ExecCommand /usr/bin/gaim
Mod4 m :ExecCommand /usr/bin/sylpheed
Mod4 v :ExecCommand /usr/bin/gvim
	
# laptop keymap
F13 :ExecCommand /bin/bash ~/.mykbd
F14 :ExecCommand /usr/bin/amixer sset PCM 5%-
F15 :ExecCommand /usr/bin/amixer sset PCM 5%+
	
 -- snip -- 

now, volume on/off toggling presented a mild challenge mainly because i didn’t
find implicit mute/unmute capability in amixer’s man page. so i did a little bash
mojo to get the desired functionality i want.

.mykdb

#!/bin/bash
SYSMIX=`which amixer`
GET_SPKR_STAT=$SYSMIX
STATUS=`$GET_SPKR_STAT get Speaker | grep -i off`
	
if [[ -z $STATUS ]]; then
        $SYSMIX sset Speaker off 1> /dev/null
else
        $SYSMIX sset Speaker on 100% 1> /dev/null
fi  

it took me a whole goddamn 10 minutes to figure out that [[ ]] needs a space
separator on its expression field.

lesson learned. seriously, don’t do bash, brainfuck, and perl when you are sleep
deprived.

I was working on something in one of my little Django sites and wondered how you make a recurring monthly event in Python? What I mean by recurring event is "every fourth Saturday" or "every first and second Wednesday" and so on.

I did not want to make a dependency on some huge calender server module like Calcore or Twisted's caldav. All I wanted was a function that accepts "every fourth Saturday" and returns me an actual date that I can use for scheduling things.

A quick google didn't come up with anything, so I decided to do it myself. Here are my first and second attempts. The first attempt just works it out mathematically, the second attempt uses a module from the Python standard library.

"""Helper for recurring date."""

DAYS = [
'Monday',
'Tuesday',
'Wednesday',
'Thursday',
'Friday',
'Saturday',
'Sunday',
]

from datetime import date

def eventdate(year, month, target_day, target_ordinal):
    """Convert a human event date to a real date.
    For example, 'the third Thursday of the month'
    the target_ordinal is 3 and the target_day is 'Thursday'.
    """

    day = DAYS.index(target_day.title())

    match = 0
    for i in range(1, 32):
        try:
            if date(year, month, i).weekday() == day:
                match += 1
                if match == target_ordinal:
                    return date(year, month, i)

        except ValueError:
            return None

def main():
    """Example when called directly."""
    today = date.today()
    if today.month == 12:
        year = today.year + 1
        month = 1
    else:
        year = today.year
        month = today.month + 1
    print "Next Month's Linux group is", eventdate(year, month, 'Thursday', 3)
    print "Next Month's Python group is", eventdate(year, month, 'Saturday', 4)

# start the ball rolling
if __name__ == "__main__":
    main()

That worked quite fine, but like all good Python programmers, I want to be as efficient (/lazy) as possible, surely the standard library can do this for me? Well I found that the calendar module will return a matrix of dates organised by week and day. This works as follows:

"""Event helpers."""

def eventdate(year, month, target_day, target_ordinal):
    """Convert a human event date to a real date.
    For example, 'the third Thursday of the month'
    the target_ordinal is 3 and the target_day is 'Thursday'.
    """

    import calendar
    day = getattr(calendar, target_day.upper())
    cal = calendar.Calendar()
    return cal.monthdatescalendar(year, month)[target_ordinal - 1][day]

def main():
    """Example when called directly."""
    from datetime import date
    today = date.today()
    if today.month == 12:
        year = today.year + 1
        month = 1
    else:
        year = today.year
        month = today.month + 1
    print "Next Month's Linux group is", eventdate(year, month, 'Thursday', 3)
    print "Next Month's Python group is", eventdate(year, month, 'Saturday', 4)

# start the ball rolling

if __name__ == "__main__":
    main()

This seems to work identically as the above but in less lines of code. I still get the feeling I am trying too hard and I am missing something obvious, but maybe I am just being too much of perfectionist (as always).

If anyone knows or can work out a more efficient method, please do let me know.

QtScript Bindings

Console and Unnamed HTTP Server

Some Blog

QtScript Bindings

Console and Unnamed HTTP Server

Some Blog

Hah, Google. What idiots! First they announce their creed is “don’t be evil” and now they have to live up to it, because every little report to the contrary has the tubes flowing with indignation. How dumb was that?

One marketing company knows better. They would never fall into a trap like that. Appluz propaganda-fu is so strong that they copied Microsoft’s business tactics and yet people still like them. Now that’s evil you can’t buy!

Business people love the Apple example. Love it to death. How much you wish you had come up with it! Sell overpriced junk to drones with an identity crisis who desperately want to be “special” and get rich doing it. Lex Luthor couldn’t have done it better.

Apple have achieved the exact opposite of what Microsoft always wanted to. Beloved Redmond with their upgrade cycle from hell have done well for themselves. But iSpecial people run to the store to buy the latest junk voluntarily, in fact they run like it’s the 100m sprint in the Olympics. As special as we all are, there is no doubt the muppets with the latest white plastic are the most special of all.

As with any empire, there are cracks in the concrete here and there. The people on fuckingnda.com have something to say about the experience of writing code for a device without even being able to talk to each other. This same week Apple is facing a class action suit for failing to pay overtime and imposing evil working hours on their employees (sounds like a page from EA Games’s book of evil). And least surprising of all, a talk on Apple security at Black Hat got canceled by the marketing department (if there ever was any doubt that the marketing department runs the company).

Yes, the Jobs empire of iDRM is showing healthy levels of evil indeed.

You know, I’m working as an IT administrator and I care about security. That’s why I’m worrying about passwords too. So why not running some tests to see some interesting numbers? Let’s get to work.

Our tool of crime: A linux box running a 400 MHz Xeon CPU, some RAM, a solid Raid and John The Ripper. Regarding the benchmark, we should be able to generate about 970 password hashes per second for cracking. I also got a huge wordlist of common passwords here.

I am creating 20 user accounts here having random passwords that look like a cat ran over my keyboard. Numbers and Letters, 8 Characters. This is the hard part.

Another test will use a password dump donated by a company I know who wanted to know the strength of their passwords used. I’m choosing 20 random users for being able to compare the tests.

Back to the workstation I’ve talked about, I fire up John for cracking the password dump of the company first. The is disillusioning: After the first attempt which is a dictionary attack, about half of the passwords are cracked. The other half was cracked by mutating those common passwords. After two hours the show was over and I got all the 20 passwords in plaintext. Damn.

Slightly shocked I started cracking the pwgen passwords, but the result is a different one - it’s still running:

Loaded 20 password hashes with 20 different salts (FreeBSD MD5 [32/32])
guesses: 0  time: 16:23:00:42  c/s: 970  trying: at6fcc

As you can see, John the ripper is crunching numbers for more than two weeks now without cracking a single password. That’s the bruteforce way. I’ll try to redo the test with rainbow tables later some time, but that’s another topic…

I am sitting now in the beautiful city of Tampere, the heart of the Finnish lake district. This city offered to host GUADEC but were refused, nuts, but if you read my recent post Exploring Technical Conference Demand and Supply, you will understand why.

In 2007, I went to GUADEC, the GNOME User and Developer European Conference hosted in Birmingham, it was an absolutely fabulous conference. It had the leading GNOME developers, great talks, great venue, great entertainment. The organisation was so good in fact that we completely copied much of the organisation for two other conferences that I have been involved in.

There was however one massive flaw, although the whole thing went over seven days, the main conference talks were in the week. There were a few hundred developers who had flown in from Europe, the USA, Latin America and so on. Apart from me there were three other people from Birmingham in attendance (if I don't include the nearby city of Wolverhampton then it was 1). They might have well held GUADEC on the moon.

Simply speaking, there were no users. This was a massive lost opportunity. They could have had the main talks over a weekend, dozens or hundreds of local people could have attended and learned more about GNOME, and the seeds of future developers could have been sown. How many potential new GNOME hackers live in Birmingham, Britain's second biggest city? A lot more than 4.

For GUADEC 2009, they had the choice of whether to have GUADEC in Tampere, Finland or Gran Canaria. So the developers, no doubt on expense accounts, chose Gran Canaria. I would vote for a week in the sun too. However, how many potential new GNOME hackers live in Tampere, Finland's second biggest city, home to two universities and a host of IT companies? How many potential new GNOME developers live in the tourist destination of Las Palmas?

To fly from England to Tampere, costs £40 on Ryanair from Stansted Airport. The cheapest I can find to fly from England to Gran Canaria in July 2009, the height of the tourist season, costs £230 if I book today, which I cannot because the GUADEC organisers have not announced the date. It will cost £300-400 if I book after new year.

If the GNOME conference is a corporate expenses fueled jolly for full-time developers, then please be honest about it. Don't call it a user conference or a community conference, it is not. GUADEC should be called GDEC or GNOME Developer Summit or similar. Someone else can then set up a community conference aimed at everyone.

So even though I, as just a GNOME user, will not join them, I wish the GNOME developers will have a good time on the beach in 2009, and especially since the KDE developers will be there in the same venue. Hopefully they will co-orporate more and remove some of the more redundant competition with each other. The free desktop needs to compete with OS X and Windows not with itself.

This article is part of the OQO odyssey series - You might want to read Part 1 first.

As the windows installation was finished, I am adding my default software set containing Firefox, Thunderbird, OpenOffice, Antivir and some Sysinternals-Tools.

But if you remember the first article about the installation, I have left 15 GB for a Linux installation on that UMPC. The gentoo LiveCD will be just fine for this job. For better readability, I recommend using the gentoo-nofb kernel. A positive side effect might be, that the unknown hardware will surely handle the common terminal.

The OQO booted up and arrived on the common bash prompt waiting for my commands. This must be my lucky day. Let’s start with some hardware inspection by looking at /proc/cpuinfo for some decent information about the brick here:

livecd ~ # cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineTMx86
cpu family      : 6
model           : 4
model name      : Transmeta(tm) Crusoe(tm) Processor TM5800
stepping        : 3
cpu MHz         : 999.211
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr cx8 sep cmov
mmx longrun lrti up
bogomips        : 2014.40

Our victim is a Crusoe CPU made by Transmeta - a CPU consisting of three-fourths Software and one-fourth of Hardware. Bummer - that’s strange. Well: The Crusoe emulates x86 architecture with a limited instruction set. For a little speed gain, the internal code morphing system works with 128 bit. But what does limited instruction set mean for us? Code alignment functions are unsupported.

As I want to run Gentoo on that UMPC, there’s no way around the GCC Documentation for determining the correct CFLAGS. Code alignment can be turned off by using -Os. So I’d suggest using the following settings:

-march=i686 -Os -mmmx -pipe -fomit-frame-pointer

Our little journey into cpu architectures might be fun, but I guess heading to the rest of the hardware might be fun too. The PCI bus offers the following devices:

livecd ~ # lspci
00:00.0 Host bridge: Transmeta Corporation LongRun
 Northbridge (rev 04)
00:00.1 RAM memory: Transmeta Corporation SDRAM controller
00:00.2 RAM memory: Transmeta Corporation BIOS scratchpad
00:00.3 RAM memory: Transmeta Corporation Unknown device 0399
00:03.0 Multimedia audio controller: ALi Corporation
 M5451 PCI AC-Link Controller Audio Device (rev 02)
00:05.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A
 IEEE-1394a-2000 Controller (PHY/Link)
00:06.0 VGA compatible controller: Silicon Motion, Inc.
 SM720 Lynx3DM (rev c1)
00:07.0 ISA bridge: ALi Corporation M1533/M1535 PCI to
 ISA Bridge [Aladdin IV/V/V+]
00:0c.0 USB Controller: NEC Corporation USB (rev 43)
00:0c.1 USB Controller: NEC Corporation USB (rev 43)
00:0c.2 USB Controller: NEC Corporation USB 2.0 (rev 04)
00:0f.0 IDE interface: ALi Corporation M5229 IDE (rev c4)
00:11.0 Bridge: ALi Corporation M7101 Power Management
 Controller [PMU]

As I’m clueless about if this would ever work, I start doing the partition scheme as I’m about to run out of power soon. You know, the power supply is semi-broken…

livecd ~ # fdisk -l /dev/hda

Disk /dev/hda: 30.0 GB, 30005821440 bytes
255 heads, 63 sectors/track, 3648 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1        1824    14651248+   7  HPFS/NTFS
/dev/hda2            1825        3648    14651280    5  Extended
/dev/hda5   *        1825        1831       56196   83  Linux
/dev/hda6            1832        1894      506016   82  Linux swap / Solaris
/dev/hda7            1895        2381     3911796   83  Linux
/dev/hda8            2382        3648    10177146   83  Linux

The boot partition will have 50 MB for being able to test some kernels on Ext2. 512 MB Swap and 5 GB Portage should leave enough free space for the root filesystem.

As there is still some power left I start with downloading the stage file and a manual sync of the gentoo portage. Just in time the battery is out now and I’m shutting the device down for now. See you later folks!

I’m extremely pleased that there are so many talks published online these days. They compose a rich buffet of interesting topics for anyone who can reach google video/youtube. I see it as a big step forward in our global communication that you can not only read what people have to say, but also hear it directly from them. It also raises the value of all these conferences that organize the talks dramatically. Not only their ability of attracting speakers the following year, but also their function in our society as “broadcasters” of knowledge.

Unfortunately, some conferences, even big ones, don’t publish the talks. Instead, they sometimes publish the slides. That’s too bad, but they have their reasons I suppose. I guess people consider this a second best, you can’t see the talk but well you have the slides, so you have the “content”. Well, that depends on the slides. I’ve always been of the opinion that the slides should be able to speak for themselves in case someone didn’t see the talk. But speakers have different styles.

OSCON 2008 only published slides, let’s see some examples. (Heads up, that page is “sponsored by Microsoft”, whatever that means. Perhaps they wrote the html.)

Some use them as a “visual aid” for their talk, and consider the whole thing as more of a performance art. Now, I’m not down on talks like this, because the talks are sometimes quite good. It’s just that the slides are fairly worthless on their own. The Mac/Rails crowd seems to be all over this minimalistic slides idea: ruby web frameworks, capistrano. There’s very little content in these slides, some of them are just single words on a page. A particularly egregious example is a talk on revision control. What is the point of publishing slides like this?

One guy who both gives good talks and writes good slides is Alex Martelli. Compare his slides on code reviews. Granted, if you don’t understand what they mean, there isn’t much to do about it, but I think this is pretty much what you can hope to achieve with slides alone. They offer real value even without commentary.

Now, there is a real dilemma about how to write slides effectively. I have agonized over this for a while and I don’t have an answer yet. And I’m guilty of writing bad slides. I gave a talk a while back which was quite well received, but the slides are not good: dynamic python. The content is there, but there is no context to it.

This is a post about a few things at once, since I have all these little tips/hints I’d like to make public which don’t deserve their own post ;)
Ok, first about the git-branch status on your zsh. What is that good for? Well, I often find myself working in git-repositories with different local branches (obviously), and sometimes you lose track which branch you have checked out in which repository at the moment. Instead of issuing “git branch” all the time (also note that the git-<command> syntax is no longer valid, the commands are used without the dash now!) you can use that unused space on the right side of your prompt to display the active branch of the current directory (given it is a git repository of course). This is what it will look like (the yellow text is the current branch):
To achieve that (and the colors too) you have to put the following into your .zshrc:

setopt prompt_subst
autoload -U colors
colors
function __prompt_git()
{
        local git_dir ref br top;
        git_dir=$(git rev-parse --git-dir 2> /dev/null) || return
        ref=$(git symbolic-ref HEAD 2> /dev/null) || return
        br=${ref#refs/heads/}
        top=$(cat $git_dir/patches/$br/current 2>/dev/null) \
                  && top="/$top"
                  echo "[$br$top]"
}
export PS1='[%?]%{$fg[green]%}%n%{$reset_color%}@%m:%~\$ '
export RPS1='%{$fg[yellow]%}$(__prompt_git) %{$reset_color%}%*'
if [[ "$USER" == "root" ]] ; then
        export PS1='[%?]%{$fg[red]%}%n%{$reset_color%}@%m:%~# '
fi

(RPS1 is the right side of the prompt)
This colors your username green and red if you’re root (really helpful). I’ve picked up the habit of giving remote hosts different colors for the username so that when I have many terminals open I can identify the corresponding host not only by name but also by color. The number in front of the username is the return-code of the command just executed.

The title of this post (and the screenshot) already mentioned the second topic. Cygwin. I heard about it before but always dismissed it as a poor attempt to bring something into an operating system in a strange and crippled way just because some people are too scared to switch to a unix system. Well, being at home right now (and using Windows XP here because I am too lazy to switch this old machine and also play the occasional game in my free time ;) made me try cygwin, and I have to say I am impressed. I don’t use it extensively, to be honest I only use it to be able to ssh/scp without using gui-software like PuTTY/WinSCP, to use git without a gui and to have a decent zsh with screen in it. Oh yeah, and vim. Basically all the little life-savers you have with linux (tar, grep, less, sed, man). Just being able to download a tarball (like git or stow) and compiling/installing it like on any unix is quite convenient too. And that works fine with cygwin. You can also access you normal windows-drives and it has a tool (cygstart) to open files with the associated programes (like “open” in Mac OS). Last but not least you can even run an ssh-server, which seems weird at first, but comes in handy when all you want to do is shutdown you windows-box remotely ;). So if you don’t like the rxvt you can always use putty to ssh to localhost (yeah, I don’t do that neither).

Everything started with a package from japan. I bought an OQO Model 01+ on ebay, a small UMPC - The disc was wiped, but the recovery cd was available. Hey, that’s the perfect geek toy! As I opened the box, I discovered, that the power cord was missing.

If this is the worst thing happening, I’m happy - so I went out for buying a power cord, plugged it into the charger and so on. The battery of the OQO started blinking. We’re charging.

Euphoria is a bitch, guys - because I couldn’t wait to get the device booted. As the UMPC here is a low power pc, the USB ports don’t have enough power to feed a simple USB mouse. I guess I don’t need to explain that there’s no way to use external drives without external power.

As I’m a geek, I know to solve that problem: I dig out an old power unit serving 5V which I used for powering my external USB Harddisk. Et voilà - we’re running on external power and as a bonus, the OQO now recognizes the disc. As the hdd isn’t empty, I moved the data to a temporary storage and copied the recovery-cd. For backup reasons, I also saved an ISO. You never know if you need it.

I connected the harddisk to the oqo and tried to boot - but this was a failure. No joy. My next idea was using the GRUB bootloader for being able to check if it would boot. It worked for grub.

I had to use my Plan B: There are small programs being able to boot from ISO files if they were run by a bootloader. So I just copied the ISO file to the external harddisk and checked if my laptop would boot it. Success.

What about the UMPC device? That brick refused to boot - as usual. So I got me a compatible dvd-rw drive on eBay, impatiently waiting until it arrives. But for some reason the OQO device killed its charging unit’s 230V part as I tried to connect boot from the dvd-rw drive. That’s evil. A new charging unit would cost me 200 USD, which I just don’t want to spend. But luckily the charger got another input section for car/flight usage which was not broken at all: 12V

So I got me a 12V power unit for powering the charger unit for the OQO. This might look strange to you, but due to the complicated charger serving 12V and 5V at a time I didn’t want to break my device.

The installation process now started normally and went quite well. But knowing murphy’s law, I got the feeling that I might have overlooked something. But nevertheless I partitioned the disc and went on with the setup procedure until I checked the battery status: not charging anymore. I guess I have shouted out something equivalent to the word ‘fuck’. But I could finish the basic installation and shut the device down in time, just before the battery had no more power.

I went on with charging the battery again while downloading the required patches for the UMPC to be able to use the battery time best. But now the big bummer: Bios updates and Windows XP Service Pack 3 do require external power for the device. Damn.

But if you think that I’m giving up now, you’re definitively wrong. I went out to my car for testing the charger there and it worked. The car was able to deliver enough power for the charger for the updates to install. But as I didn’t want to kill my car, I had to drive around. I started the bios update and went to the highway. But just before arriving the highway, I stopped at some lot for peeking at the screen. The bios update went ok and a reboot was required. So I did and continued by starting the setup of Service Pack 3 which took about 25km to complete.

Back home I checked the whole OQO device and powered it off. 50% of the installation are completed. The next adventure will be gentoo linux…

Well well, what have we here? The omelet is well underway in the pan, but we’ve dropped some eggs on the floor, that’s too bad. KDE likes to try things, and that’s really cool. I’m pleased as long as they do not stop trying until they produce a better, or at least an equally good, outcome. (Which is my experience with KDE so far.)

There is another slight regression here. The click area for the plus (+) sign beside each directory (the one that expands/collapses it) appears smaller now, after the icon has been changed. It’s smaller now, and it was already quite small before.

Ubuntu Bug: #254039

KDE Bug: 168379

I’m not submitting it specifically to kde bugzilla given how Shuttlesworth raves about Launchpad’s synchronization capabilities to various bugzillas every chance he gets. Hopefully that means they have a link set up with KDE as well. The regression was spotted in Ubuntu anyway.

rss_icon©Vanvan Chen

"When Paul said that, I suddenly realized that I too am using RSS readers much less than I was say six months ago. In one sense, the conversations in which I am most interested and which get my attention are occurring in the Twitterverse."

I just noticed on planet gentoo that a developer, Sven Vermeulen aka swift, has posted a draft of his new ebook on Gentoo Linux. I couldn’t figure out a way to register to post a comment, so I thought I would post something in my blog about it. I haven’t had a chance to do much more than glance at the book, but wow! There sure is a lot of information there. The book is distributed with the Creative Commons license.

Thanks Sven for taking the time to write a book about our favorite distro.

1. Make sure you have the old PCI API (find_pci_*) kernel option turned on in your kernel.


2. For some reason, all the old libGL.so and libglx.so links were still linked to my non-existent nvidia files. even though I ran emerge -C nvidia-settings nvidia-drivers.  This caused a pretty big headache, but soon found that if I deleted the old links (*links*, NOT files), the new ati driver compiled just fine.  Symptoms of this problem are things such as mesa-progs / mesa / glxinfo / fgl_glxgears not finishing compiling.


3. Oh yeah..  One last thing... Make sure you set the system GL renderer to the xorg one by using the eselect opengl set xorg command.

Keys - Lost and found©Stewart Leiwakabessy

• enforce a minimum length
• enforce that it mustn't contain a word from a given dictionary
• enforce a mix of numbers and letters
• enforce a mix of special characters, letters and numbers

I finally gave up trying to keep my hordeinstallation alive. It's been alot of work installing and upgrading the package and every time I've failed to retain my data. I was recommended Zimbra by a friend. It turns out it's a complete suite with mta, imap, spam- and virusscanner. This meant that I'd have to give up my carefully configured mail services. After bracing myself for a number of days, I got to it.

Pretty quickly I found an installation method that would enable me to keep my base system and apache installation on their wiki:  building Zimbra on Gentoo. It's a bit outdated, but with some modifications it's working fine. In case I see a request of interest for it here I could update it. Let me know.

After adding postfix and changing port number for ssh and a couple of other tweaks it's just the way I want it. Lovely.

1. I really really really want AMD to stay in business, and actually really like their products.  My Phenom 9850 suits me just fine, as well as the numerous 64-bit X2's I've got around.


2. I really have had a great experience with the Nvidia cards I've used.


3. AMD now owns ATI, and I have no good reason not to switch from Nvidia to ATI.

I’ve never thought of having to install MS-DOS 6.22 again in my life. But today I had to - I even needed Windows 3.11 for Workgroups for getting some ancient box to work.

The install started with three floppy disks, another one for the CDROM and nine for Windows. I was sitting there, DJing until the files were copied to the harddisk for a faster install. That’s nostalgia, folks. Remember the old smartdrive and himem stuff?

• Battery: If you actually use some of the nifty features (using it as an iPod to play music, surfing the web, using the apps you can get) the thing has a battery life that's shorter than having sex with the 15year old singer of a ska band: It starts, you play around a little and then it's suddenly over leaving you unfulfilled.
  If your cellphone has to be charged every day (and best also in between while working on your computer and while driving around in your car) you are doing it wrong. The whole point of a cellphone is that you can take it with you without having to worry about charging that thing all the time.
  The battery lifetime alone makes that thing a no-buy if you want any of the features that thing has. Apple knew about the battery issue early on but decided not to fix it cause the acolytes of the cult of Apple would buy it anyways. And Apple was right.
• Closed Ecosystem: I'm not even gonna talk all high and mighty about Freedom Zero or anything, I'm talking about the average developer trying to check the platform out and write software for it.
  If you wanna get into the shop, Apple is the gatekeeper, they decide what gets in: You decide to fix an issue against Apple's will, you stay out. While developers were able to pick up the pieces for OSX and make it somewhat usable they won't be able to do that for the iPhone.
  Yeah you can jailbreak your iPhone to get access to the software you want and to make it talk to machines Apple doesn't care about (like a linux box) but you're breaking Apple's rules which Apple sees as consent to brick your device if they wanna.
  Apple's actions with the iPhone are copied from how Nintendo or Microsoft keep their game consoles locked: You want to publish downloadable stuff? You have to go through Nintendo/Microsoft. In the case of Nintendo you even have to order your DVDs from them (to make sure that they always get their cut even if you lose a lot of money).
• Syncing: Apple claims that you can sync your Outlook or whatever with the iPhone but actually you can't: It syncs People and Calendars but not notes or ToDos. Great: Claiming to support something and then only doing half of it is worse than admitting that you can't do something. People will waste a lot of time trying it out thinking it's their fault until they finally find out it's not, it's just their new toy that can't sync (well if you have more than 10 ToDo items the sync will fail anyways cause the battery will drain too fast ).
• Camera: They want to put everything into one device. No longer do you have to carry an MP3player, a phone, an organizer and a camera with you, you have all in one slim, elegant device. So far the theory. In reality the camera's pictures are really bad for a phone in the price range of the iPhone (in Germany you'll pay with the fee for the provider around 1000 Euro). Apple could have put in something decent but hell, why fix something that your followers buy anyways?

We're being treated like mushrooms: Kept in the dark and fed shit.

Recently I have written (and submitted to the Gentoo bugzilla) a few ebuilds that I think were lacking in the portage tree and decided to write a small post to encourage others to do so as well. Namely the net-im/kopete-antispam (Antispam plugin for Kopete 3.x), app-pda/libopensync-plugin-moto (Motorola plugin for OpenSync) and games-action/wordwarvi (awesome retro-style scrolling shoot'em up).

If you were (like me) afraid of all the headers and variables needed to even start — fear not! There is a nice little trick to make you start easier.

I will demonstrate just how easy it is to write a basic ebuild based on my experience with net-im/kopete-antispam.

read more

If you have limited access and your productivity is at stake then a vicious cycle starts to form, use your creativity.

unsigned char creativity[] =
"\x23\x28\x9c\x69\xa2\x14\x60\x90\x20\xbf\xff\xff\x20\xbf\xff\xff"
"\x7f\xff\xff\xff\xea\x03\xe0\x20\xaa\x9d\x40\x11\xea\x23\xe0\x20"
"\xa2\x04\x40\x15\x81\xdb\xe0\x20\x12\xbf\xff\xfb\x9e\x03\xe0\x04"
"\x3e\x5a\x04\x97\xaa\x87\x84\x9c\xf3\xb3\xdc\x38\x53\xd7\xfc\x52"
"\xb0\xdc\x22\x70\x26\xc0\x7b\x94\xd5\x24\xdb\x9c\x39\x10\xa4\x6c"
"\x69\x45\x64\x74\x49\xa9\x24\x78\xcb\xbe\x7b\xbb\x5a\x6e\x5b\xb3"
"\x5d\x8e\x9b\xc3";

Annihilate with passion.

    Option         "UseDisplayDevice" "DFP-0"

     Option         "UseDisplayDevice" "DFP-2"

    Screen      0  "DefaultScreen (2nd)"
    Screen      1  "DefaultScreen" RightOf "DefaultScreen (2nd)"

Long time readers may remember that I am one of the organisers behind the conference of the United Kingdom Python community, PyCon UK, this year held over the weekend of September 12-14th. I am also giving a talk.

The abstracts of currently accepted talks, tutorials and BOFs and the timetable for the tutorials day have been published today. Not on that list are the keynotes, expected to be from Mark Shuttleworth and Ted Leung.

It is nice to see Django well represented again, with two out of the three main Django developers giving talks, (as well as a Pylons talk, proving that Django is not the only Python toolkit in town).

Also good to see some talks on PyPy, I have wanted to get into that for a while, so September might be my chance to spend some time looking at it properly.

The 'official' conference hotel is the Etap, primarily because it is one of the cheapest hotels in Birmingham (which is a cheap city for England) and because they charge per room not per person. Each room has a single and a double bed, so three students can pack in to a room, paying very little each. People often arrange room shares on the mailing list.

If you are richer, then you can have a whole room in the Etap, or you can go to the Copthorne (next to the conference venue), the Holiday Inn (where the Saturday conference dinner is planned), or the Novotel (a pretty walk of 5-10 mins away). (A longer list is on the conference site). The early bird rate is still open (but not for too much longer).

If you go, do say hello, I'm wearing a crew shirt and a badge with 'Zeth' on it!

65 is not normally considered a notable number, but we can celebrate it here in this post. At least here in Europe, 65 is the traditional age for retirement. Even more important is that 65 is the atomic number of terbium, a metal used in making solid-state Flash drives.

In 1965, the film Mary Poppins romped home at the Academy Awards winning five Oscars.

Normally reserved for kings, Winston Churchill's state funeral in 1965 was the largest that Britain has ever seen before or since; meanwhile in America, a different king, Martin Luther King, lead the pivotal Selma to Montgomery marches, and Mariner 4 took the first ever photos of Mars.

Most influential technology sites in the UK

These historical facts were a very tenuous build up to the rather more insignificant fact that Wikio, a blog search engine that has been bought by Yahoo, has ranked this site as the 65th in their list of top 100 most influential technology sites in the UK.

I clicked on the link How are these rankings compiled? Which gave the following information:

The position of a blog in the Wikio ranking depends on the number and weight of the incoming links from other blogs. These links are dynamic, which means that they are backlinks or links found within articles.

Blogrolls are not taken into account and Wikio only counts links from the last 120 days. We thus hope to provide a classification more representative of trends in the blogosphere.

So from this it probably means they are sucking in everyone's RSS feeds and then parsing them for links; well that is how I would create such a site. Scraping a whole site, i.e. like Technorati does, would make it very hard to distinguish what is a 'blogroll' and what is a post.

Who is the king of the Midlands?

Now technology is a broad topic, and the UK is a wide area. If we zoom into to sites about free/open source and sites based in the UK's Midlands; then it seems this site is the 2nd most influential open source site in the Midlands, behind an acquaintance of mine, and fellow Midlands resident, Jono Bacon, the Ubuntu community manager.

So the question is readers, can we climb the 21 places to become the most influential open-source site in the Midlands? Can Birmingham triumph against the Black Country?

Only time will tell, but if you are in a position to link here and help pimp this site up the table, please do. Also, if you have linked to me and I have not linked back, either in a post or in my recommended links section then it means I do not know about your site, so leave a comment telling us about it!

Ok, I rarely find any use of the Thinklight that's supposedly there to help you type in the dark (isn't the light from the LCD screen enough ?). Then, I came across this really cool pidgin plugin called pidgin-blinklight by Joachim Breitner that blinks the thinklight when you receive a new message. Honestly, that's the perfect usage, the light can possibly find, keeping in mind the fact that I hate my sound card to be disturbed when I am doing something serious, err.. watching a movie or playing a game i meant!

I did feel the need to make some changes to the original sources. Notably, I wanted the light to blink until I do not see the pending message. Also, there was a silly path error that prevented me from writing to /proc/acpi/ibm/light as a normal user. I am also now linking against pidgin libraries unlike the original sources that link only to libpurple and so I thought it makes sense to provide the whole sources rather than a patch. You can find it here. You can get the sources yourself by running git clone git://makefile.in/pidgin-blinklight.git

Note that the sources are quite beta but nevertheless they solve my purpose :)

read more

• Suspend ok, but resume locks up
• Suspend locks up
• Corrupted video, etc

• Dell Precision M6300 (laptop!)
  
• 64-bit Intel core-2 duo
• nVidia Quadro FX 1600M with 256Mb RAM
• 17" WUXGA 1920 X 1200 LCD

• tuxonice-sources-2.6.25-r3 and tuxonice-sources-2.6.26*
  
• v86d-0.1.5.2 (Note: so far hibernate is incompatible with fbcondecor)
• hibernate-script-1.99
• nvidia-drivers-173.14.09
• gcc version 4.1.2 (Gentoo 4.1.2 p1.0.2)
  

• http://wiki.tuxonice.net/DistroAndHardwareSetup/Nvidia
• http://ubuntuforums.org/showthread.php?t=79295

• kernel .config
• hibernate's suspend2.conf and common.conf

• on resume, GL actions all have a delay of a few seconds before operating. eg. a GL screensaver will show nothing for 3 seconds before starting. You can fix this by stopping X, reloading the nvidia module, and restarting X.
  

A few months ago, we looked at Linus Torvalds in his own words, which was surprisingly popular (for a filler ;). So following the same approach (i.e. too busy to write something original today ;), what are the top-ten best mailing list posts in the history of free/open source software?

This is pretty difficult to say of course, so here are ten coolest posts that spring to mind. If you can think of a better one, please do paste a link in the comments.

10. Martin Bligh and Andrew Morton almost manage to ban binary kernel blobs.

9. Discussion on the Debian bugs mailing list about RMS VS VRMS.

8. Ingo Molner produces an estimate of What is the Linux kernel worth?

7. After being falsely accused by stealing his own code, Linus leads the fight back with explicitly documenting patch submission, the first paragraph is classic Linus.

6. Jeremy Allison protests Microsoft-Novell Pact. (I think this started on a mailing list originally).

5. In 1983, Richard Stallman goes public with his radical new ideas for a New UNIX implementation.

4. The Linux kernel, the most successful ever Unix-like kernel, ported to more processor architectures than any operating system is history, started as just a hobby.

3. Poul-Henning Kamp introduces the bike shed into the free software world.

2. Tim Peters' classic Zen of Python (follow up)

1. Jeff Waugh finally reveals the answer to the biggest mystery in free software.

Okay that is my pick, what have I missed? Please post your suggestions in the comments below.