Posts for Friday, December 31, 2010

avatar

Urchin 0.1.0 released

A few months ago I wanted to teach myself some new shell tricks. One thing led to another and I ended up writing my own shell in Ruby. Urchin exists mostly just for the sake of it, but I’m looking forward to experimenting and implementing some things that aren’t found in other shells.

Urchin does not aim to be POSIX compliant. Finding POSIX shell scripting pretty nasty was one reason I started this project. However, I generally really like the existing standard shells, so many things work in a way you would expect: pipelines, redirections, job control, globbing, environment variables, aliases and tilde expansion are all there.

Urchin recently passed my criteria for a first release – “run as my login shell for a week without crashing or annoying me”. For sure, there are some problems (tab-completion, quoting and escaping issues spring to mind), but I had to draw a line somewhere and just release something.

Trying it out

This still feels very much like a beta release. The chances are you don’t use a shell exactly like me and this isn’t yet a viable Bash/Zsh replacement.

I’ve only tested this release on Linux. I’d love to hear how it runs on other platforms. Future versions will be tested on more platforms.

Warning: I’d advise testing it out for a while before you get too trigger happy and chsh!

If I haven’t managed to put you off yet, follow the instructions on the Urchin homepage to try it out.

2010 in passing

So 2010's been a year.

Nearer it's start some friends and I competed for the second time in MIT's BattleCode, this time getting second non-MIT spot, or 18th overall. These competitions have been good for us: they are fun, it's a good group project, and we work on our group project skills like planning, coordinating, and so forth. We spend so much time on our own, or in school even, working solo it's good to work these skills as they will be needed later. Also it's fun to learn about and catch up on low level AI stuff, like swarming and flocking movement/coordination techniques etc.

I also entered a school project into BCNet's Broadband Innovation Challenge and got awarded second place. My project was "Cortex" a P2P processing app that runs with no software install entirely in your web browser. It was comprised of a small Java Applet webserver used as a backbone for communication and then a JavaScript front end, with all the control logic of the P2P network also written in JavaScript. I pretty much wrote a P2P app in JavaScript just using Java only to get around the AJAX/Server of Origin security policy issue. It was an interesting and challenging project and I'm pleased with how it did in the competition.

Over the summer I was in China which was amazing.

Then in the fall while finishing off my degree in CS once and for all I also competed in the Google sponsored University of Waterloo AI Contest. This, while being a simpler solo competition, was notable for me as it was my third major project undertaken in Lisp. I thoroughly enjoyed the challenge and again learned lots more about Lisp and again improved my Lisp style. Lisp and the emacs environment just take longer to learn and wrap my head around. And since I don't get to work in them constantly, between work and school, it takes time. I'm by no means a master, but after convincing a friend to take a stab at the same competition in Lisp for his first try with Lisp, I at least see how far I've come. I'm getting more used to thinking functionally, especially with respect to using Lisp mapping functions instead of loops to modify, filter, or build on data. I placed disappointingly poorly due to lack of time, but I'm satisfied with what I learned (and also proud by association that the winner was a Lisp entry!). It was a good experience. I look forward to being able to undertake some more Lisp projects in the new year.

I also boned up on my Python this fall for a small work project, a multi threaded web crawler for a client. Played successfully with Python's threading, so that was fun.

And that brings us to now. I'm in Colombia for the holidays, and in my vacation spare time I've finally gotten around to looking at the codebase to my school project "Cortex". As school projects are, it worked, and well, but the codebase was a bit of a mess due to strong time constraints. Now that I have some time I'm doing some massive cleanups and adding a few features I'd wanted to but didn't have time to. Hopefully early in the new year it'll be in shape that I can release it. That would be nice.

So 2010 was a great year. I got to write a lot of cool code in several different language. I got to travel more than I ever have before, and I read a lot more than 2009 (traveling facilitates a lot of reading :)). It's been a good year.

For 2011 though, now that I'm done with school, I'd like to start by releasing more code, starting with Cortex; getting more paying work; and looking at maybe starting a startup. I'd like to spend more time working on both AI (if you hadn't noticed, obviously a hobby of mine) and in Lisp, starting with getting back into my signed copy of Peter Norvig's "Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp" (yes getting it signed was awesome and a ridiculously geeky moment) and moving on from there. I'd like to at least keep up with the reading. I have high hopes for it to be an interesting year.

So here's to 2010, you've been great, lets see if I can't build on that for a more amazing 2011.

Posts for Thursday, December 30, 2010

avatar

Why I have backups

You often read stories about people who have data loss and did not keep any (recent) backups, and are now fully equipped with a state-of-the-art backup mechanism. So no – no such failure story here but an example why backups are important.

Yesterday I had a vicious RAID/LVM failure. Due to my expeditions in the world of SELinux, for some odd reason when I booted with SELinux enforcing on, my RAID-1 where an LVM volume group (with /usr, /var, /opt and /home) was hosted (coincidentally the only RAID-1 with 1.2 metadata version, the others are at 0.90) suddenly decided to split itself into two (!) degraded RAID-1 systems: /dev/md126 and /dev/md127. During detection, LVM found two devices (the two RAID metadevices) but with the same meta data on it (same physical volume signature), so randomly picked one as its physical volume.


Found duplicate PV Lgrl5nNfenRUg9bIwM20q1hfMrWylyyL: using /dev/md126 not /dev/md127

Result: after a few reboots (no, I didn’t notice it at first – why would I, everything seemed to work well so I didn’t look at the dmesg output) I started noticing that changes I made were suddenly gone (for instance, ebuild updates that I made) which almost immediately triggers for me a “remount read-only, check logs and take emergency backup“-adrenaline surge. And then I noticed that there were I/O errors in my logs, together with the previously mentioned error message. So I quickly made an emergency backup of my most critical file system locations (/home as well as /etc and some files in var) and then tried to fix the problem (without having to reinstall everything).

The first thing I did – and that might have been the trigger for real pandemonium – was to try and found out which RAID (md126 or md127) is being used. The vgdisplay and other commands showed me that only md127 was used at that time. Also, /proc/mdstats showed that md126 was in a auto read-only state, meaning it wasn’t touched since my system booted. So I decided to drop md126 and add its underlying partitions to the md127 RAID device. Once added, I would expect that the degraded array would start syncing, but no: the moment the partition was added, the RAID was shown to be fully operational.

So I rebooted my system, only to find out it couldn’t mount md127. File system checks, duplicate inodes, deleted blocks, the whole shebang. Even running multiple fsck -y commands didn’t help. The volume group was totally corrupted and my system almost totally gone. At that time, it was around 1am and knowing I wouldn’t be able to sleep before my system is back operational – and knowing that I cannot sleep long as my daughter will wake up at her usual hour – I decided to remove the array, recreate it and pull back my original backup (not the one I just took as it might already have corrupted files). As I take daily backups (they are made at 6 o’clock or during first boot, whatever comes first) I quickly had most of my /home recovered (the backup doesn’t take caches, git/svn/cvs snapshots etc. into account). A quick delta check between the newly restored /home and the backup I took yielded a few files which I have changed since, so those were recovered as well. But it also showed lost changes, lost files and just corrupted files so I’m glad I have my original backups.

I don’t take backups of my /usr as it is only a system-rebuild away. As /etc wasn’t corrupted, I recovered my /var, threw in a Gentoo stage snapshot (but not the full tarball as that would overwrite clean files) and ran a emerge -pe world –keep-going.

When I woke up, my system was almost fully recovered with only a few failed installs – which were identified and fixed in the next hour.

Knowing that my backup system is rudimentary (an rsync command which uses hardlinks for incremental updates towards a second system plus a secure file upload to a remote system for really important files) I was quite happy to have only lost a few changes which I neglected to commit/push. So, what did I learn?

  • Keep taking backups (and perhaps start using binpkg for fast recovery),
  • Use 0.90 raid metadata version,
  • Commit often, and
  • Install a log checking tool that warns me the moment something weird might be occurring

Posts for Wednesday, December 29, 2010

mac os x, virtualbox

motivation

running mac os x from virtualbox helps to experiment with software deployment for that platform. i’m using gentoo linux as host system with 2.6.34-gentoo-r6 kernel on a 64 bit core 2 duo machine with virtualbox 3.2.10 OSE. there are various blog postings about how to get it running so i won’t repeat this here.

the most important thing is to use the snow leopard retail dvd. as the dvd coming with the mac (a mac book pro in my case) won’t work. you can get the retail dvd on ebay or various other sources. note: the retail dvd is not labeled ‘snow leopard retail dvd’ but something like ‘snow leopard update dvd’ in most auctions (don’t blame me if that is wrong).

how i use it

having snow leopard not only native, but also in a virtualbox image, makes testing of software easier.

hints

disk image size

i use about 40 gb here, and about 25 gb are actually in use with:

  1. latest patch (1 gb)
  2. xcode (6-8 gb)
  3. trolltechs qt sdk (2 gb)
  4. mac os x (8 gb)

i wasted a lot of disk space with a default installation. but i don’t want to spend too much time on that right now so i will leave it that way. i removed the printer drivers later on, saved me 600mb.

when installing software, map the image (dmg) using samba (running on the host machine)

import dmg images, which are used to install software from, using samba from the host. this way you do not have to copy the dmg onto the virtual machine using ssh or by mapping an external harddrive (note: the OSE version i am using, does not support USB host mapping anyway) into the virtual machine.

this also reduces the host image size by a few gigabyte of disk space. initially i was doing it wrong: i copied the file, installed from it and finally removed it. however, removing it did not have much of an effect, as virtualbox does not know about hfs+ partition scheme, therefore the file is only deleted inside the virtualmachine, still the host image uses the diskspace.

example:

my current disk image: mac os x [snow leopard].vdi is at 25Gb. but inside the virtual machine is is only 18gb.

smb.conf

[global]
  workgroup = MYGROUP
  server string = Samba Server
  security = share
   dns proxy = no
[rw-files]
   comment = Temporary rw files
   path = /home/user/smbshare/rw
   read only = no
   writable = yes
   public = yes
[ro-files]
   comment = Temporary ro files
   path = /home/user/smbshare/ro
   read only = no
   writable = no
   public = yes

don’t forget to:

  1. create the paths listed above
  2. secure the network as a rw path might be accessible from your lan

consider using a user/password for that.

downloading xcode

use wget to download it. it is about 3,6gb and it failed several times here. also resuming did not work well. after several attempts downloading it (must be about 12gb i’ve already downloaded) i had a, yet again, broken download at 99%. i thought: “not again”! and as there was no resuming (using a webbrowser) i downloaded a torrent.

next i used the torrent and started the download, which created a file with the same filename. i closed the torrent client (rtorrent) and moved the previously metioned 99% download file, to the place where the torrent expected it’s download. then i restarted the torrent download.

thanks i did not have an offset error (this can happen on wrong resumes, anyway i did not use resume).

finally the torrent checked all chunks and finnaly there was only 1mb left to download. so i repaired my download, using bittorrent.  all i had to do is wait 5minutes.

next thing was file integrity. it seems that apple does encode some individual information into the xcode download. if rumors are true, it is no wonder that users sue apple, see [1]. anyway, this means we can’t create a md5sum. it would be interesting to know what is going on.

what we do know already is that most of the file, apparently 99% of the download is generic. if not, then i would not be able to repair the xcode download with the torrent content of a different user. see discussion at [2] for more input on this.

create snapshots

virtualbox supports snapshots, use them! they are a great way to update various system components as xcode without having to uninstall any components. this results in a cleaner system, saves time & resources as deleted files are only deleted virtually inside the vm.

image growth

every virtually deleted file is not really deleted as virtualbox works on a block device layer. that means: if hfs+ decides to use deleted inode for a newly created file it will be good as vm disk and real disk are in sync. but i guess more often than not this is not the case and the real disk image will grow over time.

links

[1] http://www.heise.de/newsticker/meldung/Apple-wegen-angeblicher-Weitergabe-von-Nutzerdaten-verklagt-Update-1159403.html

[2] http://discussions.apple.com/thread.jspa?messageID=12770938


avatar

HTC Tattoo running Android 2.2.1

Yesterday I had some time to kill so I decided to look around for an android upgrade for my HTC Tattoo. HTC only provides (and supports) android 1.6 for the Tattoo, which feels a bit outdated. I like to run a little more up to date OS ;)

So after some help from google I found a guid on the unlockr. Following that guide and then using the SPQRom 0.7 is basically all there is to it.

The downside of this upgrade is that the FM and camera are no longer working. The FM is a sad thing (but hey I still have my rockbox-enabled-ipod) but the camera is something I can live with, since well it made crappy pictures anyway!

But other than some things not working I am very pleased with android 2.2.1 it runs smooth and after installing all the apps (and some new onces) my phone feels up to date again!

Posts for Monday, December 27, 2010

Merry Christmas! To: My Phone, From: The Internet

So early this year I bought my first Android phone, an LG Eve, from Rogers Mobile. It turns out it came with Android 1.5 which was a year old at the time, the current version was 2.1 and 2.2 was released that spring. Rogers has been promising an update to 1.6 for the phone pretty much since I bought it and still no release. It was pushed back twice and has since gone silent. This could be the end and a sad story at that, with just over 2 more years left on my contract.

However, this is Android! And so heroically the CyanogenMod people have been following the Android code base and releaseing an free up to date version that works on some more common phones. The good folks of Open Etna have taken that work and customized it and made sure it works on the LG Eve specifically. So following the instructions on their and other sites, I have been able to upgrade my phone to Android 2.2, and it's awesome. There is a lot more software available, voice commands, live wallpapers, and a JIT compiler so software should run faster, and more that I'm just discovering (ships with a good terminal app). And just generally newer default apps. So thank you very much to those communities of volunteers for doing vastly more than what my Phone company was incapable of.

And now briefly what I did:

I followed the Open Etna Installation Guide. It's pretty straight forward and simple. However there were a few other resources. I can't stress enough how important good backups are, so make sure your contacts are synced to your google account and make a list of your favorite apps because you are about to wipe everything clean. Better instructions on how to run the backup procedure are at www.zacpod.com/p/71. Bellow the videos on the Open Etna guide is a important reminded to install the Google apps. Considering how important this is I'm surprised they placed it a bit out of the way, and it's important because it allows you to then resync all your contacts and access all the other Google goodies that help make these phones great (and who wants to reenter all their contacts when all you have to do is hit a button to restore them).

So that's about it. Now my phone is to date and has a whole lot more life to it. Thanks Android community!

avatar

Internet TV... without the internet TV bit.

So I bought a 32" LED backlit (not LED pixel) LCD Samsung a couple of months ago.  It's hard to compare "Internet" TVs because each brand puts their own spin on exactly what "internet" means.

The Samsung series 6 has an Ethernet port, and by default Windows (Vista and higher) detect it and ask if you want to "trust" it or not.  The TV detects DLNA servers you might have easily on your network too.

But what about this "internet" claim?  When I first bought it, there were some simple internet "apps" that you can choose from and install.  Mostly boring stuff from a TV perspective like Google Maps, Picasa, Facebook, Youtube, etc.  This is all good but not exactly what I'd call internet "TV".



Today I noticed there were some internet app updates, including Bigpond TV and Twitter.  After downloading and running the small updates I ran the Bigpond TV app.  A nice video tells me I have to run a software update.  About half an hour later the software on the TV is updated from 001014 to 003003.  The first thing I noticed is the TV guide updates the screen when loading new EPG data, instead of waiting till the end.

Finally at almost half past midnight I'm looking forward to seeing what Internet TV channels are available, and I'm greeted with "Sorry, we are currently unable to retrieve Time Information.  Please call us on ... for help."  woot, is that it?  No, it appears I can still load the channel list (albeit with no previews).

That is until it locks up because I scroll too fast.  Yup, the whole TV has locked up.  My faith is partly restored as the watchdog timer kicks in and reboots the TV.  However, all Internet TV settings are reset, I have to agree to the copyright and disclaimer again, and now all my apps need updating again.

And now for the commentary: If you want Internet TV to be preferred over a computer, it should be better (easier, faster) than a computer to watch up to date, relevant, live content.

Posts for Friday, December 24, 2010

package management done differently: nix

motivation

package management is great when it just works. but often ‘edge cases’ reveal shortcomings in the respective package manager. a very innovative approach to package deployment is implemented in the NIX package manager.

if you are only interested in the technical side of my recent findings about NIX, skip chapter::’my linux journey’ and read on at chapter::’NIX OS’, below.

my linux journey, so far…

originally i started my linux journey with redhat, impressed by the debian package manager apt, i soon changed over to debian linux.

i’ve used debian when kde 3 was finally released and altough i was quite happy with kde 2 soon i realized that kde 2 had to give way for the kde 3 integration. if i recall correctly it was quite hard to use kde 2 in parallel. at that time kde 3 (the first days/weeks after the release in debian) was unusable and crashed all over the place. that keept this way until the relese of kde ’3.5′.

so the main problem with kde 3 was that it contained arts (an audio daemon like ESD) which was tightly coupled to ‘kde 3.x’. a normal desktop user would never had issues with that but as i always was interested in jackd, the ‘audio connection kit’ for low latency audio stuff. the main problem was that the dependencies for arts somehow made it very complicated when wanting also jackd installed on the same system. as there were no useflags, as in gentoo linux, i didn’t have much joy, as i had to decide between kde or jackd.

in the end i experimented with gentoo linux and that’s what i’m stuck with until today. but back then one could install kde 3 and jackd at the same time without much burden. but soon after i moved from debian to gentoo there was a kde 4 release. this resulted in horrible kde3 vs kde4 problems. finally i had to remove my beloved, back then stable 3.5 kde installation and was forced to use the kde 4 alpha release quality software.

in gentoo there had been support for a ‘prefixinstallation of ‘kde 3.x and 4.x‘ which also made several parallel kde 4 installations possible. of course one could also install kde3 and kde4 at the same time. but support for the ‘kde prefix’ installation is probably gone by now and with it the kde3 support.

so NIX OS does solve two problems at the same time:

  • to some degree it has use flags, as gentoo does
  • all installed software is installed as done in gentoo when doing a kde prefix installation

anyway, enough kde history, let’s see what nix actually is…

NIX OS

i’ve been experimenting with NIX OS lately. NIX is a package manager, or in nix terms a ‘component deployment tool’. Nix has some very unique and innovative features i’ve not yet seen yet.

here is a incomplete list of features i like:

  • one can install several different version of a single program at the same time
  • still only one ‘selected’ version is executed, when typing “firefox” for instance
  • installation of components is atomic, no system breakage after hitting ‘reset’ too early
  • if a upgrade to a more recent version fails, one can roll-back nearly instantly
  • there is no difference between source deployment and binary deployment anymore
  • all packages are now ‘read only’, so one could say that a package could be copied between systems even after installation
  • nix can be used ad-hoc in linux distributions and even on windows and mac os x
  • nix can be used to produce cross platform builds or to automate processes with dependencies in general
  • services (or in nix terms compositions) can be cloned from one computer to another very easily
  • NIX replaces Make/Autotools/CMake and similar tools

most points listed here, can also be found at [1].

in other package managers as (portage, apt, …) there is only a global namespace which is dictated by the distribution. this name dictates package names and dependencies. i would like to call this a ‘build time’ namespace. nix moves this ‘build time’ namespace issue into a runtime namespace issue as one can have two different versions of the same program installed but only one can be marked as active (read: used in a profile).

how is a ‘Nix component’ composed?

to learn how Nix works i decided to make the ‘evopedia’ [2] project available on Nix OS.

1. creating a component

looking at the nix manual [3] and the very informative phd-thesis [4] from Eelco Dolstra i’ve decided to start in:

cd /etc/nixos/nixpkgs/pkgs/applications/science/misc/

mkdir evopedia

cd evopedia

vi default.nix

2. default.nix (for evopedia)

{stdenv, fetchurl, bzip2, qt, libX11}:

stdenv.mkDerivation rec {
  name = "evopedia-0.4.2";

  src = fetchurl {
    url = "http://evopedia.info/src/${name}.tar.gz";
    sha256 = "79e372d78e34e252e363a64c55cf6952d4930b226c01110452770faf189ff88b";
  };

  configurePhase = ''
    qmake PREFIX=$out
  '';

  buildInputs = [ bzip2 qt libX11 ];

  meta = {
    description = "Offline Wikipedia Viewer";
    homepage = http://www.evopedia.info;
    license = "GPLv3+";
    maintainers = with stdenv.lib.maintainers; [viric];
    platforms = with stdenv.lib.platforms; linux;
  };
}

3. injecting the new component into the system

/etc/nixos/nixpkgs/pkgs/top-level/all-packages.nix should contain this:

  evopedia = callPackage ../applications/science/misc/evopedia {
    qt = qt4;
  };

compared to c, this is like a #include “foo.h” statement, inserting the evopedia expression somewhere.

4. installation

nix-env -i evopedia

summary

evopedia can be found in nixpkgs already, so you don’t have to do this in order to use evopedia.

think about this as an example how to deploy your software on Nix os.

thanks

i’m very thankful for the support from developers at irc.freenode.net#nixos

links

[1] http://nixos.org/

[2] http://evopedia.info

[3] http://hydra.nixos.org/build/565033/download/1/manual/

[4] http://nixos.org/docs/papers.html


Posts for Thursday, December 23, 2010

evopedia is running on arch linux

evopedia [1] is now working on arch linux! i’ve uploaded the PKGBUILD to AUR [2], which means that it is now very easy for arch linux users to use evopedia.

with some luck a developer will pick up that package and put it into the main distribution. in case you want to use the package directly (without waiting for official support,  read on)

how to install

download the PKGBUILD source from [2]. then read [3] and in short:

pacman -S base-devel # as root

makepkg PKGBUILD # as normal user

pacman -U evopedia-0.4.2-1-x86_64.pkg.tar.xz # as root

if you get any trouble, read the documentation ;-)

the source

# Maintainer: Joachim Schiele
pkgname=evopedia
pkgver=0.4.2
pkgrel=1
pkgdesc='Offline Wikipedia Viewer'
arch=('i686' 'x86_64')
url='http://evopedia.info/'
license=('GPL3')
depends=('bzip2' 'qt')
makedepends=()
md5sums=('2f9672ebd550f01a4b71bf67a4140911')
#source="http://lastlog.de/misc/${pkgname}-${pkgver}.tar.gz"
source="http://evopedia.info/src/${pkgname}-${pkgver}.tar.gz"

build() {
  cd "$srcdir/$pkgname-$pkgver"
  /usr/bin/qmake evopedia.pro PREFIX="$pkgdir"/usr
  make
}

package() {
  cd "$srcdir/$pkgname-$pkgver"
  make install
}

# vim:set ts=2 sw=2 et:

links

[1] http://evopedia.info

[2] http://aur.archlinux.org/packages.php?ID=44675

[3] https://wiki.archlinux.org/index.php/Creating_Packages


avatar

A peek into the future.

It’s nearing 2011, which means we’re smack in the middle of Christmas, many things are finishing and many things are about to begin, and tradition states that now is the time to stop, reflect, realign, and shape up. I haven’t been blogging regularly as of late (given my 2 week absence) and my usual efforts to try and post every alternate day has evaporated. Mostly it’s because WIPUP has absorbed a lot of my usual verbose documentaries on the current progress of my projects (as was its purpose), but also because I haven’t exactly been doing loads lately. In fact, I’m now proudly spending some quality wasted time, and I feel as though I’ve deserved it.

Let’s take a look at what’s been going on so far. The ThoughtScore project has been relaunched and has some excellent momentum going for it. Pictures are churning out slowly as texturing is a tedious process, but on the other hand at least the storyline is getting a lot of love. WIPUP recently had a big release, as well as an appearance in Google Code-In, which may or may not result in awesome KDE integration. I’ve got myself a VPS, and learned the ins and outs of setting up a DNS server and a mail server, thus migrating thinkMoult and my main email. Various other doodads also popped up including private git hosting for my projects, and a public ADOM game server, which is quite active and seeing regular improvements and updates (and bugfixes!) There has been quite a bit of private enjoyment such as reading, learning C++, photography and music composition, and in time they will mature to see their place on WIPUP. I’ve also been publicly insulting the KDE website and discussing/proposing solutions with the kde-www team, and this is currently very much in progress as well. I also recently received a charming package from KDE eV from the KPresenter design competition, including a wicked t-shirt, a postcard and a sticker. Of course, I’ve also been doing freelancing work with the folks over at OmniStudios and the workload should increase as I start university.

Speaking of university, let’s see what’s coming up on the horizon. I’ll be off to Canada early Christmas morning (Christmas on a plane!), and will likely be having a quick 1-2 day kde-www sprint there in the midst of mingling with relatives. The days are very, very packed, and so I have a short time to reunite with Malaysia and within the same week, off to begin my new life in Australia. In other words, a lot more of "real life" is coming up.

My current plan is somewhat along the lines of "don’t panic". I might tweak it a little later :)

Related posts:

  1. Holiday plans.
  2. Well, let’s learn to cook.
  3. KDE.org relaunch with a brand new design!

Posts for Wednesday, December 22, 2010

Issues and challenges for the post-Cablegate world

Already the birds are chirping (they're not tweeting anymore) about WikiLeaks and its Cablegate affair, how it came to be, how it got blocked, how it got filtered, DDoS'ed, removed from Amazon's cloud service, PayPal, MasterCard, Visa, how its founder became wanted by the Interpol and arrested for sex crimes, how a 16-year old boy was arrested for DDoS'ing etc. etc., so I won't bother you with this. Also I am not really interested much in what the cables say and whether they should have leaked or not.

No, this post is about what I think we have to watch out after Cablegate.

In all honesty, I think that traditional media and the general public will get bored with WikiLeaks and forget about it in about a months' time. That is unless WikiLeaks has some pretty well-planned and well-timed tricks up his sleeve (which still may be the case).

But only then things will start to become interesting! And not so much in diplomacy — sure, diplomacy will again become more secure, pick its words wiser and in general be more discreet again, but that is nothing new. I think the consequences will be a lot severer and with a wider impact …and it could be even more dangerous, that it will (try to) be done silently. Not just for the openness' sake, but because what is at stake.

So, the cat is out of the bag. While everyone is running after it (whether they claim to see a kitty or a tiger), let us stop, sit down and watch what the bag and the people chasing the cat are doing …
<!--break-->

Legislation — privacy, anti-terrorist, national security, copyright, ACTA, SHIELD, Digital Agenda

Probably the first and if carefully watched, most visible issue will be the legislation.

After the national security fiasco that resulted in Cablegate most governments around the globe (with the US at its front) will be hard-pressed to "do something about it". Because neither police, nor court, nor governement can (or rather should) operate against the law and current law does not direcly criminalise WikiLeaks, they will have to change the law.

Apart from privacy considerations, such laws could have a huge impact on copyright and freedom of speech. Just like how anti-terrorist and child pornography laws can strengthen surveillance and internet filtering, so can anti-WikiLeaks legislation. Actually in this case the laws could be direcly aimed at diminishing freedom of speech and privacy. And once the pandora's box is open, it would be very likely that in time both government and commercial interests would broaden the scope of filtering and spying. The common denominator here is censorship.

Initially I thought it would take cca. two months before governments would silently start pushing for laws that would try to "prevent another WikiLeaks" — this is also the reason why I did not finish this article before. But it apparently in the US senator Liebermann has already proposed a law directly aimed at WikiLeaks. The so called SHIELD Act is directly aimed at amending the Espionage Act in a way that would make WikiLeaks fall under espionage and therefore a criminal activity.

All this only means that it is already high time we watch for the laws that are being planned — be it about anti-terrorist, anti-piracy, national security, ICT or even copyright. All could be (mis)used for trying to "prevent another WikiLeaks" and used to.

And let us not forget that ACTA is still hanging above our heads. This is not the time to get sidetracked. In fact, we should fight harder to stop ACTA! Chances are that governments might try to pass it as soon as it can in the wrong fate it would prevent future such affairs.

We need to remember that this problem was not triggered by Cablegate, it merely put it in the limelight. The issue of law not adapting to new technology — and obviously the internet demands an even greater paradigm shift then the printing press, radio and television put together — has been present ever since internet came to be and is not limited just to privacy. In fact I would dare to argue that anything less then rethinking copyright, other "intellectual property rights", privacy, "ownership" of data and bits and many other laws together as a whole, would be idiocy. I would go as far as to claim that to some degree we would have to rethink our social, political and economic systems as well.

This may sound a bit extreme, but the harsh reality is that in current system where more and more laws are made and amended it is increasingly harder to see the bigger picture. In relation with that also changes in law are mostly only made as reactions to specific issues. And when addressing such a broad paradigm shift with such a narrow-sighted reactionist legislation the problems can only escalate.

Bottom line: we need to keep an eye on ACTA, SHIELD, privacy, public access, national security, anti-terrorist, copyright and other IPR laws. The ideal, though sadly not likely, solution would be to sit back and rethink the whole system.

Technology — new internet, filtering, spying, cloud computing etc.

When state of the art technology and law are not going hand-in-hand, there are always two solutions — either adapt the law or break the technology.

Since the start of the internet it was said that the internet cannot be controlled. While ultimate control may indeed be impossible in the internet as we know it, there are those who wish it would be so and will try hard to make it happen. Therefore we have to watch out how the future of the internet technology will evolve. Even if currently it is not possible, future technology could at the very least immensly cripple net neutrality and freedom of speech.

From time to time we hear reports about pushes to make the future Internet (whether Internet2 or other project) technically easier to control. Whether it is an issue only for the tin-foil-hatted or a real potential major risk is debatable, but if we learnt something in the past few weeks it is that anything is possible. How far such networks are already developed and how likely they are to go mainstream is again debatable, but this is an option we have to keep in mind.

For example only a few days ago the UN CSTD made a very contraversial decision to create a Working Group on Improvements to the IGF with a membership made up only of governments.

Of course as EU Commissioner Cecilia Malmström has noted, governments will have to protect themselves and more importantly vital infrastructure against cyberattacks. But again the crucial bit is to figure out what to protect, how and in what the implications will be on the society in general.

For even the thickest heads, Cablegate finally made it very clear, it is that when depending on cloud computing is risky, you are at the mercy of ToS of the service provider.

Even before WikiLeaks, there were criticisms that the current DNS system is too centralised and that a new P2P DNS system would have to be made. Of course a top-down DNS system and domains brings predictibility and compliance, but questions arise if this is still the best solution. If DNS was more dispersed we could (at least for a while) give power to admins who could use it arbitrarily as was the case with spam black and gray listing in the 1990's. On the other hand though we see that ICANN has been criticised over and over again by different stakeholders that it is overpowered, abusive of it and not neutral in its decisions. In my humble opinon, DNS would be harder to solve in a P2P environment, but a solution would be possible. I not a network expert, so I will leave this question to them. What I can add here is that I think that even if ICANN's current job was done by an apolitical professional international NGO (e.g. made from already existing national academic computer networks like SURFnet and ARNES), it would be a lot better.

Then there is also the question of blocking websites. Technically speaking there are many methods (by IP, on DNS servers, etc.) but whichever large-scale method or combination thereof will always bring false negatives and false positives. Imagine it like spam filtering — although GMail Yahoo Mail have a ton of users cooperatively teaching the filtering AI, it is still subpar compared to a simple custom Bayesean filter you can set up in your e-mail client. So in both cases the only way to get it done right is to have the end user set up and teach its own filters. But this does not suit the idea of top-down censorship (whether by state or business). Apart from the false positives and false negatives problem, an immense (and increasingly so) issue is that if country was actually to make a halfquarter-way effective blocking/filtering system, it would open the Pandora's box. The moment such an (insanely costly) system would be up and legal, the commercial content businesses would start lobying for making use of it to globally block everyone and everything that would allegedly infringe their IPRs. And once we are there, all goes as long as money flows.

When using technology (or code if you will) instead of legislation, we also have to keep in mind that code is not law and therefore should not replace it. If that happens we not only lose the benefits of a legal prosecution — believe it or not, they are huge, we just take them for granted — but also the ebility to appeal, benefit of doubt, of saying your side of the story. Using technology instead of law is worse then the inqusition!

Society — proprietarisation, consumerism

From the social perspective, we see an increasingly bigger move towards consumerism and proprietarisation of intangible goods. Even though proprietarisation of intangible goods (via copyright and other "IPRs" and much was said about that already) is a grave issue, an even bigger issue in the future could be consumerism.

You can consume a sandwich or a glass of wine (although I would rather enjoy them), but you cannot consume music, art, thoughts, feelings or a bitstream. The internet can be used, but cannot be consumed. If that was the case, with more people using the internet, the scarcer it would be, but exactly the oposite is true. The same holds for many other things — tangible and even more so intangible.

Consuming something means that after doing so it does not exist anymore, therefore anything that can be consumed has to be by definition possible to have and destroy. Anything that can be consumed ergo has to be property.

Because of this "buy once — use/destroy once, buy another" nature, consumerism is an ideal state of mind for the content industry.

But us take a thought experiment to see what can happen if we continue to let it go wild.

Imagine that in the near future a piece of digital data would be consumable (via complex DRM, ToS, laws and/or whatever) and therefore property. Not just internet, but also any bitstream or string of bits as an "materialised" form of information is intangible, not a scarce good and is not destroyed upon use. What is then the difference between that and an also intangible idea? The internet in nature is so intangible that if that were to happen, one could well imagine that mere thoughts, ideas, dreams —even life — could be deemed as consumable. And if these could be property of someone, there is just a small step to transfer the property to your thoughts, dreams, feelings …life to someone via their ToS.

This may sound crazy now, but then again, copyright, patents, other "IPRs" and related rights are very close to this.

So let us stop acting as consumers of the internet and its services and start acting as what we are: customers and users! At the very least we should read the ToS and PP and do not use the service if we do not agree with them.

It is not about Jullian Assange or WikiLeaks

These problems existed well before WikiLeaks, but Cablegate brought it more to public attention. And here I mean both leaking of confidential information as well as censorship.

There are already first direct spin-offs of WikiLeaks emerging: OpenLeaks, BrusselsLeaks and IndoLeaks. But more importantly, as Bruce Schneier analyses nicely, the issue is not WikiLeaks (or any *Leaks project), but in national security on one side and on the other the push for more open government. Which website or medium or technology is irrelevant.

It is not (just) about internet

Without a doubt the internet has immensly changed our lives. It has penetrated our lives so much, that we take it as an everyday thing — no-one really calls it the "cyberspace" anymore (at least no-one who uses it often). Can you imagine a life without online shopping, e-mail, IM, VoIP, social network sites, reading news and articles online, modern banking (cash machines, credit cards, quick wire transfers, online banking), knowing when the next train or bus will arrive etc.?

And this is exactly the gist of the problem — the battle for net neutrality is not a battle just for freedom on the internet, but a battle for freedom of speech in general.

To support this claim, I would like to point out that the internet is (and technically always was) a two-way communication. Many even argue that the internet is by design a P2P system and here I would agree. But even if we put P2P aside, it is clear that therefore the internet cannot be called a medium, which is a one-way means of communication only.

My generation

The question of the future of the internet has a lot to do with the generation gap.

Basically there are three generations involved.

The eldest generation I would call here the pre-internet generation. It consists of people who were born well before the internet era they either do not use it or think of it as a medium. There are of course exceptions here in the form of internet pioneers, early adopters and those who adapted to it, but on average they do not think of the internet as their turf.

The youngest generation I would like to call the (for the lack of a better term) cyber-born. These are kids who were already growing up with and on the internet, they feel it is their world and cannot possibly imagine a life without it. Internet an integral part of their lives, so much that they (almost) cannot protest outside it.

And then there is generation of cyber-grown. This is the generation of those who spent a considerable part of their youth without the internet and a considerable part online, remembers dial-up, Usenet etc.my generation. What is special about this generation is that they remember the life before the internet, yet have started using it so early on in their lives that it is second nature to them — in a way we are a generation of cyber-amphibians.

What is happening lately more and more often is that we see a clash of the generations. Let us analyse the current WikiLeaks-related events a bit from this perspective. After the pre-internet generation started to block WikiLeaks, cut its revenue stream etc., the cyber-born protested by grouping online, "like-ing" on social networking sites, flaming, DDoS'ing — that is their version of protest: respectively of public gatherings, singing/screaming together, wearing banners and at the very end throwing bricks and burning cars. Now the pre-intenet generation will respond (or already has) by calling them (cyber-)terrorists, criminals etc. As you can well imagine, the misunderstandings between these two generations due to and regarding the internet are enormous and can very well escalate.

In many ways you could compare the fight for net neutrality, freedom of the user (be it by movements for free software, open source, free culture etc.) to the hippie and punk movements of the previous century. The major difference though is that those were fighting for new freedoms, while the current battle is about already existing ones.

But here is where my generation comes in — as the only generation which understands both worlds, we are the ones who can try to mediate between these two, we are the ones who can slowly push for better laws, better understanding and in general stop the war on internet before it happens! A lot is at stake and if we do not want our children to live in an even more Orwellian society then already we do, we have to do something about it.

…And remember to "like" is not enough!

Issues and challenges for the post-Cablegate world

Already the birds are chirping (they're not tweeting anymore) about WikiLeaks and its Cablegate affair, how it came to be, how it got blocked, how it got filtered, DDoS'ed, removed from Amazon's cloud service, PayPal, MasterCard, Visa, how its founder became wanted by the Interpol and arrested for sex crimes, how a 16-year old boy was arrested for DDoS'ing etc. etc., so I won't bother you with this. Also I am not really interested much in what the cables say and whether they should have leaked or not.

No, this post is about what I think we have to watch out after Cablegate.

In all honesty, I think that traditional media and the general public will get bored with WikiLeaks and forget about it in about a months' time. That is unless WikiLeaks has some pretty well-planned and well-timed tricks up his sleeve (which still may be the case).

But only then things will start to become interesting! And not so much in diplomacy — sure, diplomacy will again become more secure, pick its words wiser and in general be more discreet again, but that is nothing new. I think the consequences will be a lot severer and with a wider impact …and it could be even more dangerous, that it will (try to) be done silently. Not just for the openness' sake, but because what is at stake.

So, the cat is out of the bag. While everyone is running after it (whether they claim to see a kitty or a tiger), let us stop, sit down and watch what the bag and the people chasing the cat are doing …
<!--break-->

Legislation — privacy, anti-terrorist, national security, copyright, ACTA, SHIELD, Digital Agenda

Probably the first and if carefully watched, most visible issue will be the legislation.

After the national security fiasco that resulted in Cablegate most governments around the globe (with the US at its front) will be hard-pressed to "do something about it". Because neither police, nor court, nor governement can (or rather should) operate against the law and current law does not direcly criminalise WikiLeaks, they will have to change the law.

Apart from privacy considerations, such laws could have a huge impact on copyright and freedom of speech. Just like how anti-terrorist and child pornography laws can strengthen surveillance and internet filtering, so can anti-WikiLeaks legislation. Actually in this case the laws could be direcly aimed at diminishing freedom of speech and privacy. And once the pandora's box is open, it would be very likely that in time both government and commercial interests would broaden the scope of filtering and spying. The common denominator here is censorship.

Initially I thought it would take cca. two months before governments would silently start pushing for laws that would try to "prevent another WikiLeaks" — this is also the reason why I did not finish this article before. But it apparently in the US senator Liebermann has already proposed a law directly aimed at WikiLeaks. The so called SHIELD Act is directly aimed at amending the Espionage Act in a way that would make WikiLeaks fall under espionage and therefore a criminal activity.

All this only means that it is already high time we watch for the laws that are being planned — be it about anti-terrorist, anti-piracy, national security, ICT or even copyright. All could be (mis)used for trying to "prevent another WikiLeaks" and used to.

And let us not forget that ACTA is still hanging above our heads. This is not the time to get sidetracked. In fact, we should fight harder to stop ACTA! Chances are that governments might try to pass it as soon as it can in the wrong fate it would prevent future such affairs.

We need to remember that this problem was not triggered by Cablegate, it merely put it in the limelight. The issue of law not adapting to new technology — and obviously the internet demands an even greater paradigm shift then the printing press, radio and television put together — has been present ever since internet came to be and is not limited just to privacy. In fact I would dare to argue that anything less then rethinking copyright, other "intellectual property rights", privacy, "ownership" of data and bits and many other laws together as a whole, would be idiocy. I would go as far as to claim that to some degree we would have to rethink our social, political and economic systems as well.

This may sound a bit extreme, but the harsh reality is that in current system where more and more laws are made and amended it is increasingly harder to see the bigger picture. In relation with that also changes in law are mostly only made as reactions to specific issues. And when addressing such a broad paradigm shift with such a narrow-sighted reactionist legislation the problems can only escalate.

Bottom line: we need to keep an eye on ACTA, SHIELD, privacy, public access, national security, anti-terrorist, copyright and other IPR laws. The ideal, though sadly not likely, solution would be to sit back and rethink the whole system.

Technology — new internet, filtering, spying, cloud computing etc.

When state of the art technology and law are not going hand-in-hand, there are always two solutions — either adapt the law or break the technology.

Since the start of the internet it was said that the internet cannot be controlled. While ultimate control may indeed be impossible in the internet as we know it, there are those who wish it would be so and will try hard to make it happen. Therefore we have to watch out how the future of the internet technology will evolve. Even if currently it is not possible, future technology could at the very least immensly cripple net neutrality and freedom of speech.

From time to time we hear reports about pushes to make the future Internet (whether Internet2 or other project) technically easier to control. Whether it is an issue only for the tin-foil-hatted or a real potential major risk is debatable, but if we learnt something in the past few weeks it is that anything is possible. How far such networks are already developed and how likely they are to go mainstream is again debatable, but this is an option we have to keep in mind.

For example only a few days ago the UN CSTD made a very contraversial decision to create a Working Group on Improvements to the IGF with a membership made up only of governments.

Of course as EU Commissioner Cecilia Malmström has noted, governments will have to protect themselves and more importantly vital infrastructure against cyberattacks. But again the crucial bit is to figure out what to protect, how and in what the implications will be on the society in general.

For even the thickest heads, Cablegate finally made it very clear, it is that when depending on cloud computing is risky, you are at the mercy of ToS of the service provider.

Even before WikiLeaks, there were criticisms that the current DNS system is too centralised and that a new P2P DNS system would have to be made. Of course a top-down DNS system and domains brings predictibility and compliance, but questions arise if this is still the best solution. If DNS was more dispersed we could (at least for a while) give power to admins who could use it arbitrarily as was the case with spam black and gray listing in the 1990's. On the other hand though we see that ICANN has been criticised over and over again by different stakeholders that it is overpowered, abusive of it and not neutral in its decisions. In my humble opinon, DNS would be harder to solve in a P2P environment, but a solution would be possible. I not a network expert, so I will leave this question to them. What I can add here is that I think that even if ICANN's current job was done by an apolitical professional international NGO (e.g. made from already existing national academic computer networks like SURFnet and ARNES), it would be a lot better.

Then there is also the question of blocking websites. Technically speaking there are many methods (by IP, on DNS servers, etc.) but whichever large-scale method or combination thereof will always bring false negatives and false positives. Imagine it like spam filtering — although GMail Yahoo Mail have a ton of users cooperatively teaching the filtering AI, it is still subpar compared to a simple custom Bayesean filter you can set up in your e-mail client. So in both cases the only way to get it done right is to have the end user set up and teach its own filters. But this does not suit the idea of top-down censorship (whether by state or business). Apart from the false positives and false negatives problem, an immense (and increasingly so) issue is that if country was actually to make a halfquarter-way effective blocking/filtering system, it would open the Pandora's box. The moment such an (insanely costly) system would be up and legal, the commercial content businesses would start lobying for making use of it to globally block everyone and everything that would allegedly infringe their IPRs. And once we are there, all goes as long as money flows.

When using technology (or code if you will) instead of legislation, we also have to keep in mind that code is not law and therefore should not replace it. If that happens we not only lose the benefits of a legal prosecution — believe it or not, they are huge, we just take them for granted — but also the ebility to appeal, benefit of doubt, of saying your side of the story. Using technology instead of law is worse then the inqusition!

Society — proprietarisation, consumerism

From the social perspective, we see an increasingly bigger move towards consumerism and proprietarisation of intangible goods. Even though proprietarisation of intangible goods (via copyright and other "IPRs" and much was said about that already) is a grave issue, an even bigger issue in the future could be consumerism.

You can consume a sandwich or a glass of wine (although I would rather enjoy them), but you cannot consume music, art, thoughts, feelings or a bitstream. The internet can be used, but cannot be consumed. If that was the case, with more people using the internet, the scarcer it would be, but exactly the oposite is true. The same holds for many other things — tangible and even more so intangible.

Consuming something means that after doing so it does not exist anymore, therefore anything that can be consumed has to be by definition possible to have and destroy. Anything that can be consumed ergo has to be property.

Because of this "buy once — use/destroy once, buy another" nature, consumerism is an ideal state of mind for the content industry.

But us take a thought experiment to see what can happen if we continue to let it go wild.

Imagine that in the near future a piece of digital data would be consumable (via complex DRM, ToS, laws and/or whatever) and therefore property. Not just internet, but also any bitstream or string of bits as an "materialised" form of information is intangible, not a scarce good and is not destroyed upon use. What is then the difference between that and an also intangible idea? The internet in nature is so intangible that if that were to happen, one could well imagine that mere thoughts, ideas, dreams —even life — could be deemed as consumable. And if these could be property of someone, there is just a small step to transfer the property to your thoughts, dreams, feelings …life to someone via their ToS.

This may sound crazy now, but then again, copyright, patents, other "IPRs" and related rights are very close to this.

So let us stop acting as consumers of the internet and its services and start acting as what we are: customers and users! At the very least we should read the ToS and PP and do not use the service if we do not agree with them.

It is not about Jullian Assange or WikiLeaks

These problems existed well before WikiLeaks, but Cablegate brought it more to public attention. And here I mean both leaking of confidential information as well as censorship.

There are already first direct spin-offs of WikiLeaks emerging: OpenLeaks, BrusselsLeaks and IndoLeaks. But more importantly, as Bruce Schneier analyses nicely, the issue is not WikiLeaks (or any *Leaks project), but in national security on one side and on the other the push for more open government. Which website or medium or technology is irrelevant.

It is not (just) about internet

Without a doubt the internet has immensly changed our lives. It has penetrated our lives so much, that we take it as an everyday thing — no-one really calls it the "cyberspace" anymore (at least no-one who uses it often). Can you imagine a life without online shopping, e-mail, IM, VoIP, social network sites, reading news and articles online, modern banking (cash machines, credit cards, quick wire transfers, online banking), knowing when the next train or bus will arrive etc.?

And this is exactly the gist of the problem — the battle for net neutrality is not a battle just for freedom on the internet, but a battle for freedom of speech in general.

To support this claim, I would like to point out that the internet is (and technically always was) a two-way communication. Many even argue that the internet is by design a P2P system and here I would agree. But even if we put P2P aside, it is clear that therefore the internet cannot be called a medium, which is a one-way means of communication only.

My generation

The question of the future of the internet has a lot to do with the generation gap.

Basically there are three generations involved.

The eldest generation I would call here the pre-internet generation. It consists of people who were born well before the internet era they either do not use it or think of it as a medium. There are of course exceptions here in the form of internet pioneers, early adopters and those who adapted to it, but on average they do not think of the internet as their turf.

The youngest generation I would like to call the (for the lack of a better term) cyber-born. These are kids who were already growing up with and on the internet, they feel it is their world and cannot possibly imagine a life without it. Internet an integral part of their lives, so much that they (almost) cannot protest outside it.

And then there is generation of cyber-grown. This is the generation of those who spent a considerable part of their youth without the internet and a considerable part online, remembers dial-up, Usenet etc.my generation. What is special about this generation is that they remember the life before the internet, yet have started using it so early on in their lives that it is second nature to them — in a way we are a generation of cyber-amphibians.

What is happening lately more and more often is that we see a clash of the generations. Let us analyse the current WikiLeaks-related events a bit from this perspective. After the pre-internet generation started to block WikiLeaks, cut its revenue stream etc., the cyber-born protested by grouping online, "like-ing" on social networking sites, flaming, DDoS'ing — that is their version of protest: respectively of public gatherings, singing/screaming together, wearing banners and at the very end throwing bricks and burning cars. Now the pre-intenet generation will respond (or already has) by calling them (cyber-)terrorists, criminals etc. As you can well imagine, the misunderstandings between these two generations due to and regarding the internet are enormous and can very well escalate.

In many ways you could compare the fight for net neutrality, freedom of the user (be it by movements for free software, open source, free culture etc.) to the hippie and punk movements of the previous century. The major difference though is that those were fighting for new freedoms, while the current battle is about already existing ones.

But here is where my generation comes in — as the only generation which understands both worlds, we are the ones who can try to mediate between these two, we are the ones who can slowly push for better laws, better understanding and in general stop the war on internet before it happens! A lot is at stake and if we do not want our children to live in an even more Orwellian society then already we do, we have to do something about it.

…And remember to "like" is not enough!

Posts for Tuesday, December 21, 2010

virtualbox extends my host system with several computers on the same network

to experiment with deployment i’ve installed several linux distributions using virtualbox (based on iso images). i had some problems with the networking but today i got rid of the complicated setup, used previously.

what i want

|-------------------------------------------|
|       fedora      debian       windows    |
|  10.0.0.100    10.0.0.101   10.0.0.102    |
|          |          |          |          |
|          --------|||||||-------|          |
|                     | switch              |
|---------------------|---------------------|
                      |
                      | vboxnet0 10.0.0.1/24

virtualbox adds the vboxnet0 (host only network adapter) using the preferences dialog in the main gui. additionally i added masquerading, using a external script, in order to have all hosts inside the ‘host only networking’ accessing the internet. the switch is created by virtualbox automatically when selecting ‘host only networking’ using vboxnet0 per appliance.

i was using ‘=app-emulation/virtualbox-ose-3.2.10′.

how to do it

the setup is quite easy, i just wanted to write it down anyway in case it is of help for others.

masquerading

in order to get each virtualbox appliance connected with the internet, masquerading is used. this is basically a one way internet connection but in my scenario there are no services provided to the outside world.

iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

echo 1 > /proc/sys/net/ipv4/ip_forward

dhcp server

as the virtualbox-built-in dhcp server did not work at all, i decided to use =net-misc/dhcp-3.1.2-p1 instead. so here is the configuration:

dhcp.conf
# option definitions common to all supported networks...
option domain-name "example.org";
option domain-name-servers 10.0.0.1;

default-lease-time 600;
max-lease-time 7200;
ddns-update-style ad-hoc;

log-facility local7;

subnet 10.0.0.0 netmask 255.255.255.0 {
  option domain-name-servers 8.8.8.8;
  range 10.0.0.10 10.0.0.200;
  option routers 10.0.0.1;
}

subnet 192.168.56.0 netmask 255.255.255.0 {
  option domain-name-servers 8.8.8.8;
  range 192.168.56.101 192.168.56.202;
  option routers 192.168.56.1;
}

i run this server from a script which is also started at boot time:

dhcp eth0 vboxnet0

this is helpful as i can configure static IPs for known mac addresses. but also automatic assignment should be quite stable as all parameters can be tuned as used to.

the clients

as mentioned already, each client has to be assinged to the ‘host only network’ adapter: vboxnet0. most linux distributions will try to get the host address via dhcp automatically. also each host can communicated with each other, so it is easy to do maintainance stuff.

update: 2010-12-21 modified article as it’s not clear why some things are done the way they are


evopedia is running on gentoo linux

gentoo linux logo (copied from commons.wikipedia.org)

gentoo

evopedia [1], the offline wikipedia reader, has been ported to gentoo linux. the ebuild can be found at [2].

i would like to say thanks to all devs @ irc.freenode.net#gentoo-sunrise! especially to those who reviewd my ebuild.

hope this ebuild will be moved into the official distribution anytime soon.

source

/usr/local/portage/sci-misc/evopedia/evopedia-0.4.2.ebuild

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=3

inherit qt4-r2

DESCRIPTION="Offline Wikipedia Viewer"
HOMEPAGE="http://evopedia.info/"
SRC_URI="http://evopedia.info/src/${P}.tar.gz"

LICENSE="GPL-3"
SLOT="0"
KEYWORDS="~amd64"
IUSE=""

RDEPEND="
        >=x11-libs/qt-gui-4.4:4
        >=app-arch/bzip2-1.0.6
"
DEPEND="${RDEPEND}"

src_configure() {
        eqmake4 evopedia.pro PREFIX="${EPREFIX}"/usr
}

links

[1] http://evopedia.info

[2] http://bugs.gentoo.org/show_bug.cgi?id=349210

update: 2010-12-24: added the source (contents of evopedia-0.4.2.ebuild)


Posts for Sunday, December 19, 2010

avatar

Migrating blogs from Drupal to Pyblosxom

pyblosxom is a pretty cool blogging platform written in python.
Like many of the modern minimal blog engines it works with plaintext files only (no database), has a relatively small codebase, supports many plugins (like markdown support), is written in a proper scripting language, has a simple and clean file structure, is seo-friendly, and so on.
The one feature that sets it apart from other minimal blog engines is that it supports comments, and doesn't just rely on an external service like disqus, but stores comments as plaintext files as well.
Some features seem a bit overengineered (like, multiple possible locations to store themes (known as "flavours") and templates; I'm a fan of convention over configuration and keeping things simple), but discussing this with the maintainer revealed this is because pyblosxom is meant as a reimplementation of the original perl-based bloxsom project. Over time features could be simplified and/or redesigned.
So I plan to migrate this blog from drupal to pyblosxom.
To do this, I'm building the tool drupal-to-pyblosxom.
The goal is to convert posts, associated metadata (publish time, tags) and comments from the drupal database to pyblosxom files. Source code display should be converted too (merely a matter of converting between different plugin conventions), and images shown should be downloaded. Currently I'm about halfway, if there's anyone out there with a similar use case, help is welcome ;)

Posts for Saturday, December 18, 2010

avatar

An useful qmail hack

My mailserver runs qmail + vpopmail + dovecot. Most of the outgoing traffic is automatically generated by the websites (joomla, moodle etc.).

Looking at the logs I found out that a lot of traffic was generated by the moodle administrator and targeted to a small group of users. With the beginning of the lessons an increase of traffic was expected, but not this much.  So this was worth an investigation.

Unfortunately qmail-smtp just logs sender and reciver of the mail, but not the content itself, and there is no way (that I know), to have the body of the message logged.

Here I found a nice idea: make a script to check whether the sender is the user that I am monitoring, and, to forward to me a copy of all its mail.

First thing I did was to create a user that will receive a copy of the mails, then I aliased the user to my main account.

vadduser bigbrother@example.com

valias -i andrea@example.com bigbrother@example.com

Qmail uses qmail-remote to send mail, so here is where I’m putting my script:

cp /var/qmail/bin/qmail-remote /var/qmail/bin/qmail-remote.orig

vim /var/qmail/bin/qmail-remote.hacked

And here’s the content of the script:

#!/bin/sh
sender="$2"
firstrecip="$3"
if [ "$sender" = "moodleadmin@example.com" ] ; then
tmpfile=`mktemp -t spying.XXXXXXXXX`
cat - >"$tmpfile"
env NEWSENDER="$sender" \
DTLINE="Delivered-To: <$firstrecip>"$'\n' \
/var/qmail/bin/forward bigbrother@example.com <"$tmpfile"
/var/qmail/bin/qmail-remote.orig "$@" <"$tmpfile"
rm "$tmpfile"
else
exec /var/qmail/bin/qmail-remote.orig "$@"

fi

I will now make my script executable, give it the right permissions, and copy it to replace qmail-remote. I won’t delete the qmail-remote.hacked file, since it will be useful to have it there, just in case.

chown root:qmail /var/qmail/bin/qmail-remote.hacked

chmod a+rx /var/qmail/bin/qmail-remote.hacked

cp /var/qmail/bin/qmail-remote.hacked /var/qmail/bin/qmail-remot

Now all the outgoing mail from moodleadmin@example.com will be forwarded to bigbrother@example.com, that is an alias to my main account.

Of course this will generate a lot of noise in my inbox, but fortunately I have sieve up and running, so I just added this lines to my sieve config to have all the messages sorted in a dedicated folder:

require "fileinto";

if header :contains "Delivered-To" bigbrother@example.com"{
fileinto "
BigBrother";
stop;
}

Oh, if you are curious, the traffic was generated by a couple of users that decided to have an email digest for every single post in the forum. That sounds weird in 2010, considering that moodle allows nice ways to be notified only for relevant news, anyway, it’s their choice.

\n’ \

/var/qmail/bin/forward bigbrother@example.com <”$tmpfile”
/var/qmail/bin/qmail-remote.orig “$@” <”$tmpfile”
rm “$tmpfile”
else
exec /var/qmail/bin/qmail-remote.orig “$@”
fi

#!/bin/shsender=”$2″firstrecip=”$3″if [ "$sender" = "moodleadmin@example.com" ] ; thentmpfile=`mktemp -t spying.XXXXXXXXX`cat – >”$tmpfile”env NEWSENDER=”$sender” \DTLINE=”Delivered-To: <$firstrecip>”

I will now make my script executable, give it the right permissions, and copy it to replace qmail-remote. I won’t delete the qmail-remote.hacked file, since it will be useful to have it there, just in case.

chown root:qmail /var/qmail/bin/qmail-remote.hacked

chmod a+rx /var/qmail/bin/qmail-remote.hacked

cp /var/qmail/bin/qmail-remote.hacked /var/qmail/bin/qmail-remot

Now all the outgoing mail from moodleadmin@example.com will be forwarded to bigbrother@example.com, that is an alias to my main account.

Of course this will generate a lot of noise in my inbox, but fortunately I have sieve up and running, so I just added this lines to my sieve config to have all the messages sorted in a dedicated folder:

require "fileinto";

if header :contains "Delivered-To" bigbrother@example.com"{
fileinto "
BigBrother";
stop;
}

Oh, if you are curious, the traffic was generated by a couple of users that decided to have an email digest for every single post in the forum. That sounds weird in 2010, considering that moodle allows nice ways to be notified only for relevant news, anyway, it’s their choice.

\n’ \/var/qmail/bin/forward bigbrother@example.com <”$tmpfile”/var/qmail/bin/qmail-remote.orig “$@” <”$tmpfile”rm “$tmpfile”elseexec /var/qmail/bin/qmail-remote.orig “$@”
fi

I will now make my script executable, give it the right permissions, and copy it to replace qmail-remote. I won’t delete the qmail-remote.hacked file, since it will be useful to have it there, just in case.

chown root:qmail /var/qmail/bin/qmail-remote.hacked

chmod a+rx /var/qmail/bin/qmail-remote.hacked

cp /var/qmail/bin/qmail-remote.hacked /var/qmail/bin/qmail-remot

Now all the outgoing mail from moodleadmin@example.com will be forwarded to bigbrother@example.com, that is an alias to my main account.

Of course this will generate a lot of noise in my inbox, but fortunately I have sieve up and running, so I just added this lines to my sieve config to have all the messages sorted in a dedicated folder:

require "fileinto";

if header :contains "Delivered-To" bigbrother@example.com"{
fileinto "
BigBrother";
stop;
}

Oh, if you are curious, the traffic was generated by a couple of users that decided to have an email digest for every single post in the forum. That sounds weird in 2010, considering that moodle allows nice ways to be notified only for relevant news, anyway, it’s their choice.

Posts for Friday, December 17, 2010

Yahoo is death

So now Yahoo is shutting down Delicious. I really can't think of a single service that has been made better under Yahoo, and plenty that have died horrible slow deaths after taking off so quickly and getting bought for decent prices. Yahoo is where you go to die, even if you are healthy.

Anyways, a quick and simple backup command to get an XML file of your bookmarks from Delicious care of Ixiaus 9 on Hacker News

curl https://{your username}:{your password}@api.del.icio.us/v1/posts/all > bookmarks.xml

Ha, coupled with massive lay offs just before xmas I think Yahoo's holiday spirit can be summed up in "Bah Humbug!"

Posts for Thursday, December 16, 2010

Paludis 0.56.1 Released

Paludis 0.56.1 has been released:

  • We now show the number of skipped and failed packages in “x of y” output.
  • We now run pkg_pretend even if certain confirmations are required.
  • Various minor bug fixes and documentation tweaks.

Filed under: paludis releases Tagged: paludis

Posts for Wednesday, December 15, 2010

What I learned in the last few days?

You can write about cyberwar, business ethics, weird German censorship laws, cultural techniques or privacy, nothing will get people checking out your  blog like dropping the “U”-bomb. One post with Ubuntu in the title and the hits on this page multiply.

It is kind of a trend: The posts I write here that I personally consider the most interesting or challenging usually neither get comments nor any significant amount of readers. All the posts I wrote that got any traction were basically 5 minutes of unfiltered braindump.

Either my taste significantly differs from that of my readers or … I don’t know…

flattr badge large What I learned in the last few days?

7 problems GSoC admins encounter

At the always-excellent Mentor Summit for the Google Summer of Code, I ran a session titled “Best Practices for GSoC Admins.” Many of these practices appear specific to a program like GSoC at first glance, but they easily transfer to recruiting new contributors outside GSoC; just imagine the same ideas with less process behind them. Here I want to share the main points of our session and expand upon some of them in the hopes that it can help future GSoC admins and other people interested in recruiting new developers.

We focused primarily on the biggest problems we face as admins. Intriguingly, although perhaps 30 people attended, nearly all the problems were universal (fortunately, some groups had already solved them!). The only exception to their universal nature was that smaller organizations seemed not to require the same level of progress tracking, because missing/poor progress quickly became obvious to everyone in a small group.

Here are the top problems, with suggested solutions:

  1. Tracking progress. Require weekly updates from both students and mentors. This means admins don’t need to personally track every student or ensure the mentor is around. Blogs or wiki pages (“journals”) work for many projects, although some have issues with blogs. A key point is to offload work to mentors so they tell you whether students are on track. Keep a spreadsheet (possibly public for tracking and shame?) to stay on top of things, because it’s easy to lose track after a few weeks.
  2. Knowing student skills. Model the type of things they would do during GSoC on a smaller scale. Require a patch during the application period to prove they can build and modify your software. Additionally, require that students interact with your community so you can consider how (and whether) they will fit in.
  3. Avoiding failure. Check in with students at “quarter points” — halfway between the start and the midterm, and halfway between the midterm and the final. This leaves time to fix any show-stopping problems before they guarantee failure. During the application period, get a calendar of when both students and mentors will be gone so you can take this into account. Investigate problems early to avoid failure instead of waiting until it’s inevitable. In the case of conflicts between students and mentors, admins can act as neutral mediator — make sure everyone knows this when the summer starts so they don’t feel helpless. Some students communicate poorly (grad-school model of independent work), so try to catch this early and push them. Are there non-binary solutions, ways to do something besides just pass or fail? Can we withhold T-shirts, pay less money based on final “grade”, increase number of payments, pay late, etc.
  4. Disappearing/lazy mentors. One major problem here is figuring out what motivates mentors: what are the incentives and punishments? The most common response to unacceptable mentoring was blocking that person from any future mentoring. Is that enough? Nobody knows; it seems to be mostly an after-the-fact solution that may not fix things during the summer.
  5. Inexperienced mentors. Pair new mentors with experienced mentors and/or backup mentors. Admins should offer to be “mentor-mentors,” teaching the beginners how it’s done.
  6. Increasing the number of proposals. Two student audiences exist: those familiar with your project and those who discover it through GSoC. For the first, target non-accepted students from previous years (Reject gently!). Publicize GSoC internally on your mailing lists, websites, etc. For the second, publicize your project in blogs, to college profs, etc. Have a good ideas list (where good means fun and exciting, so students apply to your project). Increase the time between org acceptance and student deadline so students have time to discover exciting organizations and ideas. Have more flexible ideas that give students some ownership (they must expand upon them!).
  7. Improving the quality of proposals. A high-quality application template is key. Problem: at least one organization saw a correlation between adding a template and getting fewer proposals. Could applying be made a two-step process, so that the template is displayed after a student commits to applying to a specific organization? Require a timeline in the proposal to ensure they understand project details at a level sufficient to code them, but allow it to flex once coding starts. Ask specific questions to gauge both understanding and enthusiasm. Do live interviews by IRC or phone, possibly with live coding.

If you have any suggestions for these problems, or more problems you’ve encountered, please let me know in the comments!


Tagged: gentoo, gsoc

Posts for Monday, December 13, 2010

Ubuntu-isms suck

Do you wanna know why Ubuntu-isms suck? Because they make software non-portable. I have just spend a few hours to try to get Synapse, an alternative to the stagnant and buggy Gnome-Do, into my Gentoo Overlay.

I knew that I had to package Zeitgeist and a few other packages but I was ready for it, I thought: “How bad could it be?”. It could be bad.

With many devs only developing on Canonical’s GNOME fork every other distribution that might be running on a more vanilla GNOME is fucked. You want a launcher to start programs? Be ready to pull in Canonical’s weird indicator-library that is a systray, but just not a systray. Then there’s this helper library, and that helper toolkit, all hosted on Launchpad, no information on how to build it apart from (get it from our PPA or the current Ubuntu version) and all heavily depending on each other.

Now this is not only a Canonical problem but a bigger one: If you as a developer only target one specific Distribution and their specific GNOME fork, you gotta understand that you basically locked the rest of the GNOME people out, if they are not willing to pull in half the foreign distribution and their stuff.

I understand that often you might not even think about it, you just use the libraries you find in front of you, but you have to understand that you are making it hard for yourself as well as others. If a library can only be installed via binary packages for one distribution it could as well be proprietary because nobody else will find out or will be willing to figure out how to build you package and its dependencies.

When it comes to dependencies the train of thought shouldn’t be “why not?” but “Why?” for every single package. Every dependency you add makes your program harder to install for someone: The library you use to save two lines might be uninstallable somewhere, the library you use for that one fancy effect might contradict another distribution’s license policy.

If you add dependencies that are only custom tailored to one distribution (like the whole Ubuntu-Gnome-Additions stack) then supply a configure switch or runtime checks along the lines of “–disable-ubuntu” so those among us that run something different can access software. It’s not like I’m not willing to do the work, I’m just not really motivated to try to figure out 5 random packages without documentation just to satisfy your build-dependencies even though I will never use the runtime features.

Target user needs and not some specific distribution. Thanks!

flattr badge large Ubuntu isms suck

Interesting comic regarding Wikileaks

Not KDE/Kubuntu related, but hits a note on freedom...

Posts for Sunday, December 12, 2010

Community :: Ethics :: Should Julian Assange be extradited to Sweden?

I own the state.

So to start with, let me be open about my own beliefs so you know where I am coming from.

As citizens and taxpayers we own the government, and as owners we have a right to know everything about our property. I believe in freedom of government information, and that the freedom of information act is a nice start but only a small step to transparent accountable government, more must be done. We must know that the government is serving us, the citizens, we must be shown that the government has not been captured by minority elites or corporate interests.

I believe the words of Jesus in John 8:32, namely that "the truth will set you free". Governments should not have secrets, or at least should have as few as possible. The date and location of the Normandy landings in 1944 is the kind of thing I think of as appropriate for a state secret, everything below that level should be public. Information governments hold about Oil or Pharmaceutical companies up to no good should and must be in the public domain. Tittle-tattle about what the head of the Bank of England might think about the shadow chancellor after a few drinks does not qualify for legal protection as a 'state secret'.

A lot of what is in the Wikileaks cables so far is the result of US diplomats writing down various unfounded rumours and slander about foreign leaders, people should not be being paid to write this nonsense down in the first place. Meanwhile real facts based on evidence should be put into the public domain. If the US government had taken this approach then there would be nothing to leak. The problem is with the people who wrote the cables in the first place, not the people who published them when they leak.

Still the biggest stick

The United States armed forces is a trillion dollar investment with more hi-tech weaponry than the rest of the world put together and well over 2 million highly-trained forces in active duty or in reserve. The "People's Liberation Army" of China has a slightly larger nominal headcount but is decades behind in technology and training. No one can argue that US hegemony of the world is affected by Wikileaks. America's hegemony is not based on secrets or 'soft power', it is based on overwhelming capability. The more it is known about, the more it deters enemy nations from attacking America.

The right-wing hacks and government insiders moaning about Wikileaks are whistling into the wind. Major established newspapers such as the Guardian and the New York Times already have all the cables and are co-publishing them with Wikileaks. If they somehow magically made Wikileaks disappear it would not change anything since the newspapers will still press ahead as planned. At time of writing, Wikileaks has a network of 1697 mirrors; it is statistically likely that a minority will be broken at any one time, but even so that is enough to make the website content more or less impossible to take offline.

Why is Assange being extradited to Sweden?

Julian Assange is the public face of Wikileaks, he is also now detained at Her Majesty's pleasure in Wandsworth prison, remanded in custody and awaiting a potential extradition to Sweden. A guardian article explains his conditions.

During August 2010, Julian Assange was in Sweden on Wikileaks business when he had intimate relations with two women. What actually happened we have no idea, at the moment all we have to go on is third party he said/she said-type rumours; and these are bizarre. Assange is wanted for questioning, no charges have actually been filed yet, and so all the media links that follow have to be taken with a truckload of salt.

A lot of the media so far that have looked at the scant information we do know have already tended to side critically with Assange. Richard Pendlebury from the Daily Mail was sent to Sweden to have a go at putting the chronology together. As do Israel Shamir and Paul Bennett in their article. Mark Hosenball in an article I read in the Toronto Star has a different take on it.

Naomi Wolf argues that Assange is a jerk, but that does not make him a rapist. I can think of several instances in my life where I have been a complete jerk (please don't write in listing them here!), therefore the Wolf article does sadly ring quite true, but that probably says more about me and the people that Wolf has dated than Assange (Wolf herself has been the alleged victim of another high profile alleged sexual harassment but that is another story altogether). Assange's lawyer has been publicly expressing the "Hell hath no fury like a woman scorned" argument, that the women were expecting a real relationship with Assange and when they found out that he was sleeping with both ended up at the Police station and part of a rape charge.

The general theme of the articles is that the women later met together, and at that point generated or clarified various concerns about the encounters with Assange, and then went together to the police who constructed these concerns into a case. The descriptions are complicated by the complexities and differences of Swedish law and by the fact that Assange had slept with two politically active women with twitter accounts, Youtube videos, links to political parties and backgrounds in sexual politics. More on this in David Edwards' article. Assange broke one of life's golden rules - don't date wannabes - this includes political wannabes as well as actresses and performers, especially when they are not very attractive!

Katrin Axelsson from Women Against Rape wonders "at the unusual zeal with which Julian Assange is being pursued for rape allegations" when often clearer cut and violent cases languish without giving their victims justice. In Shamir and Bennett's article (mentioned already above), one of their arguments is that the CIA "threatened to discontinue intelligence sharing with SEPO, the Swedish Secret Service" unless the government worked against Assange and the whole thing might be a "honey trap".

US via Sweden?

Jemima Khan argues for the extradition theory, the idea that a Swedish prosecutor is attempting to extradite Assange as part of an eventual extradition out of the EU altogether to the US:

"I believe that this is about censorship and intimidation. The timing of these rehashed allegations is highly suspicious, coinciding with the recent WikiLeaks revelations and reinvigorated by a rightwing Swedish politician. There are credible rumours that this is a holding charge while an indictment is being sought in secret for his arrest and extradition to the US. An accusation of rape is the ultimate gag. Until proved otherwise, Assange has done nothing illegal, yet he is behind bars."


One argument against the extradition theory is to ask the rhetorical question of why should it be easier to extradite Assange from Sweden rather from than the UK?

Well due to the Gary McKinnon case, the British public mood for sending suspects across the Atlantic is low, (mis)using the extradition treaty again for a case that is not a bomb wielding terrorist may lead to the treaty being repealed. The US would certainly want to avoid that.

Sweden is not the liberal paradise as it is sometimes portrayed, there is more to Sweden than Abba's Dancing Queen, there is an authoritarian streak also. Sweden had a forced sterilization program from the 1930s until 1976, forced crime sterilization is a crime against humanity. Also, the United Nations ruled that Sweden violated the global torture ban in 2006 by knowingly co-operating with US process that led to asylum seekers being transferred from Stockholm to Egypt and then being tortured there. Sweden was involved in other illegal rendition flights going from and through Sweden. Sweden was also part of the war in Afghanistan. Sweden also does not have a Jury system, trials are performed for judges alone.

The possibility of Sweden being more likely to extradite Assange may not be the motive, the motive may be to keep Assange within countries likely to extradite to the US. If Assange went to Venezuela, Ecuador, Russia or any other place outside America's Empire then he would be out of reach. If he is being held on remand in the UK, or being tried in Sweden, then he is kept in place.

So according to the extradition theory, part of it just opportunity. Assange admits he had, what he saw as consensual, sex in Sweden; and Sweden has a prosecutor willing to push the case, albeit that the Stockholm prosecutor decided there was no evidence, so a second prosecutor from Gothenburg was brought into play. This gives a reason to keep Assange in place.

It will all no doubt come out in the wash, one way or another. Either there is evidence in Sweden of sexual offences or there is not. A holding strategy could not work for very long. We have a right to free movement across the EU, especially for work purposes, and Assange could have been questioned at Scotland Yard or at the Swedish embassy in London; so if it turns out that the Swedish prosecutor cannot get a conviction then serious questions will be need to be asked about European warrants and using them for fishing exhibitions.

The Wikileaks cables shows governments up to all sorts of stranger-than-fiction hijinks, just look at all the shenanigans that happened with Abdel Baset Ali al-Megrahi. Even so the extradition theory is based on weaving together circumstantial evidence and seems a bit far fetched, (but that does not necessarily make it untrue). It is a 'who ordered the death of JFK type question'. If you think it was the military-industrial complex or whatever then the extradition theory could be credible. If you think JFK was killed by a nut then life is just sometimes random - America wants to punish Assange and he just happens to be an alleged rapist.

Dr Kirk James Murphy's article laying out the possible conspiracy is let down for me by the phrase "just happens" the implication that it is to coincidental to be true, and he says sarcastically "Small world, isn’t it?". Well actually yes, it is a small world, especially when it comes to capital cities, universities, political parties and activist movements and so on, you always seem to see the same old faces again and again.

Throwing out the baby with the bathwater

Extreme cases like Wikileaks are a poor basis for reform of freedom of speech, as the adage goes - hard cases make bad law. A worrying sign is that in response to Wikileaks, freedom of speech may be further restricted. Such laws will have no effect on situations like Wikileaks but will no doubt have negative unintended consequences further down the line.

Well that was my attempt to make sense of it. Please leave a reply and let me know what you think. Should Julian Assange be extradited to Sweden?

Posts for Thursday, December 9, 2010

No Nonsense Logging in C (and C++)

A lot of times people do zany things and try and reinvent wheels when it comes to programming. Sometimes this is good: when learning, when trying to improve state of the art, or when trying to simplify when only Two-Ton solutions are available.

For a current daemon project I need good, fast, thread-safe logging. syslog fits the bill to a tee and using anything else would be downright foolish — akin to implementing my own relational database. There’s one caveat. For development and debugging, I’d like to not fork/daemonize and instead output messages to stdout. Some implementations of syslog() define LOG_PERROR, but this is not in POSIX.1-2008 and it also logs to both stderr and wherever the syslog sink is set. That may not be desired.

So, the goals here are: continue to use syslog() for the normal case as it is awesome, but allow console output in a portable way. Non-goals were using something asinine like a reimplementation of Log4Bloat or other large attempt at thread-safe logging from scratch.

Using function pointers, we can get a close approximation of an Interface or Virtual Function of Object Oriented languages:

void (*LOG)(int, const char *, ...);
int (*LOG_setmask)(int);

These are the same parameters that POSIX syslog() and setlogmask() take. Now, at runtime, if we desire to use the the “real” syslog:

LOG = &syslog;
LOG_setmask = &setlogmask;

If we wish to instead log to console, a little more work is in order. Essentially, we need to define a console logging function “inheriting” the syslog() “method signature” (or arguments for non-OO types).

/* In a header somewhere */
void log_console(int priority, const char *format, ...);
int log_console_setlogmask(int mask);

And finally, a basic console output format:

/* Private storage for the current mask */
static int log_consolemask;

int log_console_setlogmask(int mask)
{
  int oldmask = log_consolemask;
  if(mask == 0)
    return oldmask; /* POSIX definition for 0 mask */
  log_consolemask = mask;
  return oldmask;
}

void log_console(int priority, const char *format, ...)
{
  va_list arglist;
  const char *loglevel;
  va_start(arglist, format);

  /* Return on MASKed log priorities */
  if (LOG_MASK(priority) & log_consolemask)
    return;

  switch(priority)
  {
  case LOG_ALERT:
    loglevel = "ALERT: ";
    break;
  case LOG_CRIT:
    loglevel = "CRIT: ";
    break;
  case LOG_DEBUG:
    loglevel = "DEBUG: ";
    break;
  case LOG_EMERG:
    loglevel = "EMERG: ";
    break;
  case LOG_ERR:
    loglevel = "ERR: ";
    break;
  case LOG_INFO:
    loglevel = "INFO: ";
    break;
  case LOG_NOTICE:
    loglevel = "NOTICE: ";
    break;
  case LOG_WARNING:
    loglevel = "WARNING: ";
    break;
  default:
    loglevel = "UNKNOWN: ";
    break;
  }

  printf("%s", loglevel);
  vprintf(format, arglist);
  printf("\n");
  va_end(arglist);
}

Now, if console output is what you desire at runtime you could use something like this:

LOG = &log_console;
LOG_setmask = &log_console_setlogmask;
LOG_setmask(LOG_MASK(LOG_DEBUG));

LOG(LOG_INFO, "Program Started!");

In about 60 lines of code we got the desired functionality by slightly extending rather than reinventing things or pulling in a large external dependency. If C++ is your cup of tea, it is left as a trivial reimplementation where you can store the console logmask as a private class variable.

Some notes:

  1. You should still call openlog() at the beginning of your program in case syslog() is selected at runtime. Likewise, you should still call closelog() at exit.
  2. It’s left as a trivial exercise to the reader to define another function to do logging to both stdout and, using vsyslog(), the syslog. This implements LOG_PERROR in a portable way.
  3. I chose stdout because it is line buffered by default. If you use stderr, you should combine the loglevel, format, and newline with sprintf before calling vprintf on the variable arglist to prevent jumbled messages.
  4. Of course, make sure you are cognizant that the format string is passed in and do not allow any user-supplied format strings as usual.
Share this article: Reddit HackerNews Slashdot Facebook StumbleUpon Google Bookmarks FSDaily Twitter Identi.ca Digg del.icio.us Print email PDF

Related posts:

  1. Why VIM is not my favorite editor UPDATE: clang_complete is what the people want and what the...
  2. To users that miss xorg.conf and complain about it I get requests from users and see questions all the...
  3. Bulletproof your server to survive Digg/Slashdot implementing scale up for web 2.0 sites with current practices...

Planet Larry is not officially affiliated with Gentoo Linux. Original artwork and logos copyright Gentoo Foundation. Yadda, yadda, yadda.