Planet Larry

Posts for Wednesday, April 28, 2010

Windows7 Crash of the Day

Crash

I should start a running series of these.

Posts for Tuesday, April 27, 2010

In my opinion, everybody who codes around databases should read this gem of an article from «The Database Programmer»: Minimize Code, Maximize Data. One of the corollaries is that you should never hard-code a bunch of rules which could be stored in the database itself.

Myself, I had this Big Ugly Function called _my_expand() which translated between internal «shortlinks» stored in a compact form, and expanded them on the fly to fullblown hyperlinks. For instance, a link stored as “[sk=25658|67|side 66a]” is expanded to <a href=”//www.arkivverket.no/URN:sk_read/25658/67″>side 66a</a> at runtime. The primary reason is not to save space; my database is about 50 MB on disk which is just a drop in the ocean on a 400 GB drive. Rather, I find the shortlinks far easier to manage and work with, as I have a very large number of references to relatively few base URLs.

The _my_expand() function started with a single link expansion, the online scanned church books of Norway:

CREATE OR REPLACE FUNCTION _my_expand(TEXT) RETURNS TEXT AS $$
-- private func, expand compact links
DECLARE
    str TEXT = $1;
BEGIN
    -- Scanned church books [kb=book reference|image reference|link text]
    str := REGEXP_REPLACE(str,
            E'\\[kb=(.+?)\\|(.+?)\\|(.+?)\\]',
            E'<a href="//www.arkivverket.no/URN:kb_read?idx_kildeid=\\1&uid=ny&idx_side=\\2">\\3</a>', 'g');
    -- More definitions here
    RETURN str;
END
$$ LANGUAGE plpgsql STABLE;

It was easy to extend this function to include ever more shortlink expansions. But there comes a point when you have to question the sanity of such a scheme. The last time I looked at this code, I thought: «Hey! It would be great to stash the search and replace strings into a table, and then loop through that instead of this big ugly function». It turned out to be very simple, although as usual I stumbled a couple of times.

CREATE TABLE short_links (
    link_type CHAR(2) PRIMARY KEY,
    short_link TEXT,
    long_link TEXT,
    description TEXT
);

Let’s try with a link to a scanned probate protocol. The raw text stored in the database of the record with source_id #23091 is:

[sk=25658|67|side 66a]. Vabakken under Klyve vestre i Solum 07.07.1784: [p=6947|Isach Jonsen].

The [p=xxx|yyy] construct is a compact internal link to a person. That is dealt with in a previous post.

pgslekt=> INSERT INTO short_links (link_type, short_link, long_link, description) VALUES
pgslekt-> ('sk', E'\\[sk=(.+?)\\|(.+?)\\|(.+?)\\]',
pgslekt(> E'<a href="//www.arkivverket.no/URN:sk_read/\\1/\\2" title="Lenke til bilde av skifteprotokollside">\\3</a>',
pgslekt(> 'Scanned probate registers [sk=protocol|image reference|link text]');
INSERT 0 1
pgslekt=> select regexp_replace((select source_text from sources where source_id=23091),
pgslekt(> (select short_link from short_links where link_type = 'sk'),
pgslekt(> (select long_link from short_links where link_type = 'sk'), 'g');
                                          regexp_replace
-----------------------------------------------------------------------------------------------
 <a href="//www.arkivverket.no/URN:sk_read/25658/67" title="Lenke til bilde av
skifteprotokollside">side 66a</a>. Vabakken under Klyve vestre i Solum
07.07.1784: [p=6947|Isach Jonsen].
(1 row)

Beautiful. Now I’m going to replace the Big Ugly Function with a loop reading values from a table:

CREATE OR REPLACE FUNCTION _my_expand(TEXT) RETURNS TEXT AS $$
-- private func, expand external compact links
DECLARE
    str TEXT = $1;
    links RECORD;

BEGIN
    FOR links IN SELECT short_link, long_link FROM short_links LOOP
        str := REGEXP_REPLACE(str, links.short_link, links.long_link, 'g');
    END LOOP;
    RETURN str;
END
$$ LANGUAGE plpgsql IMMUTABLE;

And It Just Works. Marvellous. In the future, I might even write a PHP interface to manage shortlinks

Mark Somerville

United Kingdom

Unix signal programming in Ruby

I recently thought I would be able to use Unix signals to solve a problem in a Ruby program I’m writing. It turned out not to be workable, but was a fun journey into Unix signal handling and how they work (or don’t) with MRI.

I’m not a signals expert – corrections and opinions are very welcome! Also, I started to feel a bit out of my depth when patching the longjmp() calls. I’d love to find out more about this stuff.

_{Photo by David Blaikie}

Brief signals primer

Signals are used to alert processes or threads about a particular event. Synchronous signals are usually the result of errors in executing some instruction (such as an illegal address reference) and are delivered to the thread that caused the error. Asynchronous signals are external to the execution context and are probably the ones you’re more familiar with – they can be sent between processes using things like kill or delivered when needed by the kernel.

When a signal is generated it is immediately put into the “pending” state. If the process has a thread that has not blocked signals of that type, it is delivered straight away. If that type of signal is blocked by all threads in the process, it remains pending until they are unblocked in one of the threads, at which point it is delivered immediately. Delivered signals can be ignored (often the default response) or processed by a signal handler. In Ruby, we define a SIGUSR1 handler like this:

Signal.trap("USR1") do
  puts "USR1 caught"
end

Why might we want to block signals?

Blocking signals is often used when we have a section of code that must not be interrupted. To enable this, each thread maintains a signal mask. This is the list of signal types that the thread is blocking, which we can examine and change using pthread_sigmask() (sigprocmask() in single-threaded programs). A new thread inherits the signal mask from the parent. However, each thread does not have its own set of signal handlers – these are shared throughout the process. Asynchronous signals that are delivered to the process can be processed by any thread that has not blocked those signals.

Signals in MRI

Unfortunately, MRI isn’t really very friendly to Unix programmers wanting to play with the signal mask, as we’ll see.

MRI defines the Signal module, that only contains two methods: Signal.trap and Signal.list, which provides the mapping of signal names to numbers for your platform. Since none of the other libc signal handling functions are defined, I created a library to provide them (and some other system calls sometime). syscalls is built using the lovely FFI library. This mirrors the libc functions closely, with a couple of Ruby-style shortcuts added.

Ruby 1.8

As Joe Damato found, MRI 1.8 with pthreads enabled is rather rt_sigprocmask() happy. It seemed obvious that all that mucking about with the signal mask would cause strange behaviour when blocking signal, but let’s see how:

require "syscalls/signal"

mask = Syscalls::Sigset_t.new.to_ptr
Syscalls.sigemptyset(mask)
Syscalls.sigaddset(mask, "USR1")

puts "Block and roll!"
Syscalls.sigprocmask(Syscalls::SIG_SETMASK, mask, nil)

puts "Looks fine so far - let's raise an exception..."

begin
  raise
rescue
end

puts "Aw-naw!"

What’s going on?

Using strace with ruby 1.8.6 (2009-08-04 patchlevel 383) [x86_64-linux] gives us:

write(1, "Block and roll!\n", 16Block and roll!
)       = 16
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0
write(1, "Looks fine so far - let's raise "..., 48Looks fine so far - let's raise an exception...
) = 48
rt_sigprocmask(SIG_BLOCK, NULL, [USR1], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0
write(1, "Aw-naw!\n", 8Aw-naw!
)                = 8

Line 3 is from Ruby calling getcontext – it is pretty harmless since it is passing SIG_BLOCK along with an empty set of signals, which adds nothing to the existing signal mask. Line 4 is our own call to sigprocmask – note we’re using SIG_SETMASK, which replaces the existing mask. So far, so good. However, on lines 7-9 Ruby stores the old mask (our SIGUSR1), replaces it with an empty mask and then immediately replaces that with our SIGUSR1 mask again.

But, the mask is only empty for a fraction of a second – I think I’ll be alright!

Think again! It’s tempting to think that this wouldn’t be a problem in most real-world situations, but you may recall that when a signal cannot be delivered because it’s blocked it is put into a pending state. When that signal type is unblocked, the signal is immediately delivered. This means there can actually be plenty of time to queue up a signal to cause a problem here.

The REE stuff below all applies to the other MRI 1.8 flavours I tested too – that’s 1.8.{6,7} on 64-bit Linux.

REE 1.8.7-2010.01

REE has Joe’s --disable-ucontext patch applied, which meant a lot fewer sigprocmask()s to wade through! In fact, it nearly worked – just our old SIG_SETMASK friend set during the exception handling:

write(1, "Block and roll!\n", 16Block and roll!
)       = 16
rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0
write(1, "Looks fine so far - let's raise "..., 48Looks fine so far - let's raise an exception...
) = 48
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(1, "Aw-naw!\n", 8Aw-naw!
)                = 8

Time to dig around and see why Ruby is doing that.

Calling raise resulted in a call to rb_longjmp(), which appears to be a reimplementation of siglongjmp() (or _longjmp() – I don’t know which). This in turn calls rb_trap_restore_mask(), which sets the signal mask back to the mask that was stored when Ruby starts up or the last call to Signal.trap was made.

Patching Ruby 1.8 – including 1.8.6, 1.8.7 and REE

This might be dangerous or not even sensible. Let me know if you find out!

As far as I can tell, simply removing the call to rb_trap_restore_mask() shouldn’t break anything since the places that the trap_last_mask variable is set are very limited. It may not the best place for the fix (if rb_longjmp() is actually siglongjmp(), this might break the reimplementation), but it does at least appear to work.

Here’s the truly tiny patch.

Ruby 1.9

Ruby 1.9.1-p376 is a slightly more tricky case. As you’ll be aware, MRI 1.9 maps each Ruby thread to a native C thread and uses the GIL to ensure only one runs at any one time. The interpreter uses a thread to trigger an interrupt in order to schedule threads. This thread is created on initialisation of the interpreter and means that even very simple programs have two native threads running.

As we can see in the abbreviated strace below, the signal mask is empty when the timer thread is created and this will be inherited. This means that if we block a signal type in our main Ruby thread, they will still be able to be delivered and handled by the timer thread.

Below that we can see rb_trap_restore_mask() emptying the mask when it is called from rb_longjmp(). The sigaltstack() and following sigaction()call on lines 4-5 tell Ruby to handle segfaults on a different stack.

rt_sigaction(SIGHUP, {0x48b9f0, [], SA_RESTORER|SA_SIGINFO, 0x3deca0f0f0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGUSR1, {0x48b9f0, [], SA_RESTORER|SA_SIGINFO, 0x3deca0f0f0}, {SIG_DFL, [], 0}, 8) = 0
...
sigaltstack({ss_sp=0x1b0baf0, ss_flags=0, ss_size=16384}, {ss_sp=0, ss_flags=SS_DISABLE, ss_size=0}) = 0
rt_sigaction(SIGSEGV, {0x48bce0, [], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x3deca0f0f0}, {SIG_DFL, [], 0}, 8)
...
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
...
clone(Process 23020 attached
child_stack=0x7f3b2c188ff0, flags=CLONE_VM|CLONE_FS| CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM| CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f3b2c1899e0, tls=0x7f3b2c189710, child_tidptr=0x7f3b2c1899e0) = 23020
...
[pid 23019] write(1, "Block and roll!", 15Block and roll!) = 15
[pid 23019] write(1, "\n", 1
)           = 1
[pid 23019] rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0
[pid 23019] write(1, "Looks fine so far - let's raise "..., 47Looks fine so far - let's raise an exception...) = 47
[pid 23019] write(1, "\n", 1
)           = 1
[pid 23019] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 23019] write(1, "Aw-naw!", 7Aw-naw!)      = 7
[pid 23019] write(1, "\n", 1
)           = 1

Patching Ruby 1.9.1

First off, we need to mask all signals as soon as the timer thread is created. This makes sure that all signals can only be delivered to the main Ruby thread (until we create some more). I suppose this will actually slow down signal delivery by some small amount.

Ruby itself blocks some signals for a time to allow sections of code to run without interrupts, using rb_disable_interrupt() and rb_enable_interrupt(). These functions mask and unmask all signals. We need to make Ruby save the existing signal mask while it blocks all signals, then restore the old mask, rather than unblocking everything.

Here’s the patch. Again, I don’t think this has any negative side-effects, but it also might not be a good idea. Let me know if you find out!

Evan interpretive score released.

Remember Evan? No, not the guy – the piece. The piece named after the guy. That piano piece. Yeah, the one I composed almost 8 months ago. The one I never announced I had finished composing.

The reason is partly because I didn’t finish composing it. I slapped on a makeshift ending to wrap it up, became utterly disgusted at the formulaic tonic and then slapped on another note right after because that’s what felt right. It was exactly what the piece was about. But was the piece itself finished? In a sense, yes – the main ideas were there – but they were undeveloped little mud puddles that tried to jump from one puddle to the next. All in all – a hard piece to perform.

It’s not hard just because it has a few technical areas and a heavy chordal passage – it’s hard because it rushes the performer into a frenzy. The piece is divided into two clear sections both of what would first seem like distinctly different characters, but further study of the piece’s theme would reveal that they are not – in fact, they are one single, continuous development. In a way, it’s not the change in emotion that makes this piece interesting (though most would easily say that it is), it’s more of the restraint in emotion the performer has to show.

This restraint must be shown to let the various smaller melodies creep through the heavy banging in the middle. Simple dexterity is not enough. The performer has to understand what exactly the hurry is all about – is it an anger? Is it a frustration? Is it simple mischeviousness? One climax after another – or should they be climaxes?

Many of the melodies are admittedly not for the piano. The main theme itself in one such interpretation is much more fit for violin – long, slurred bows with the piano peeking through with light droplets to add contrast. It will end hanging – with a slight yearning, a slight sadness – or another much grander interpretation: a confident, full sound with a nimble backdrop to emphasise its majesty.

The only clearly interpreted section is the last – one of stillness. A slight murmur – a few sharp breaks of light – a heavy, determined, and resolute ending.

So a while back I had to record Evan to send off with my portfolio – obviously having not played it for 8 months and the piano untouched for a while due to other responsibilities, I will readily admit that my playing – and resulting recording, was an abomination. It was crap. An insult. There was a brief section played just right – but the rest was just note after note – a disgusting sequence of sound that didn’t deserve the 5.5 minutes of MP3 I sent off. As a result I’m not going to show the recording here until it’s redone properly.

However the score itself is another matter. As you might’ve garnered from the description above – I didn’t explain the emotion of the piece – not my emotion with it, at least – but I did give suggestions. The score is unmarked and completely interpretive. I invite anybody who wants to to play it – and yes, it is unfinished. It will probably stay that way for a while. I would appreciate any recordings of people playing it!

Click here to download it (second attachment) in PDF format.

Posts for Sunday, April 25, 2010

Roeland Douma

The Netherlands

qbook

This morning I was up rather early (well not that early but the others were still asleep ). So I decided to do a little coding and came up with a very simple Qt desktop client for facebook. Since it would of course be nice to have client sitting in your task bar showing new notifications, later this can be extended but for now that will do.

All this was created in a about an hour but would not have been possible without qfacebookconnect.

It is far from finished and no notifications are really visible yet but a nice start is made (not bad for 45 minutes coding). See the git repository of qbook if you want to take a peak, all paths are currently hard coded so you will need to do a little hacking, this is mainly because qfacebookconnect is not yet in portage so I can properly link to it.

Maybe next time when I am up early I will first fix an ebuild for that one

Posts for Thursday, April 22, 2010

Jürgen Geuter

Germany

An experiment about rules

So after some recent events I thought it would be fun to combine three things I really enjoy in one blog post:

Doing an experiment
Drawing awesome pictures
Pissing some people off

Now as you will see, while I do have some skills in two of those areas, one of them is not exactly something I am very talented with but it's my blog so nobody can kick me out for sucking (yet another reason everbody should have a blog

).

Let's just start with one of my brilliant, awesome, artistic and heartwarming pictures:

Pretty, isn't it? And (as any piece of art) it is open for interpretation: Maybe the person depicted is a clown working every day at the Happy-Fun-Land amusement park, maybe he's a Doctor curing cancer, maybe it's not a dude but a woman, maybe a model on the catwalk or Computer-science-Barbie come to life. It's up to you my dear reader to think of something, I just ask you to think of something cool. You got a minute, I'll wait.

Thought of something? Great! Then let us continue with the experiment, which will blow your mind. I can piss people off by just adding a word to the image.

BAM BAM BAAAA. Now the picture depicts the Prophet Mohammed which is forbidden by the rules of the Islam and opens me up for being burned in hell or at the stake or somewhere. Am I a rude fuckhead for posting this?

Let's look at yet another picture:

This is a picture of the judeo-christian god. The 10 commandments state explicitly that

"You shall not make for yourself an idol, whether in the form of anything that is in heaven above, or that is on the earth beneath, or that is in the water under the earth."

which is a very similar rule though it does not bring the hate and flames that depicting Mohammed does.

Now this is not a pisstake on the Islam or a sign of how much better Judaism or Christianity are. Let's look at a few aspects of what happened.

What we see is that there are more than just one religion that forbid the creation of idols and that is pretty much exactly what a rule like "don't depict the prophet Mohammed" is about: Don't make you deity or holy people comic book characters, keep them abstract to avoid having too "worldly" gods like for example those the greeks had, gods that stole, lied and fucked around. From that background the rule itself isn't bad (always given the fact that you subscribe to some sort of religion).

But some people seem to consider any picture displaying the Prophet Mohammed heresy and whatnot, consider any picture like that an insult to their religious belief. Those people don't seem to understand the intention of their own rules: Your religious rules don't apply to people outside of your own religion, they are meant for you to support your belief and make sure that you don't stumble off the way. Complaining that some atheistic guy in Germany drew your Prophet is like a Jew complaining that some Christian is eating non-kosher meat: Retarded.

That does not mean that there might not be pictures of your prophet that can be considered insulting or propagandistic but most of them aren't. And if people who are not believing into your religious concepts do make some fun about it (as in satire) that is also fair game, just as you can think that we of the "wrong belief" will all go to hell or whatever punishment you can come up with. We can make fun, you can feel better than us. And everybody wins.

Rules are important and are what makes many of the constructs of our society working but rules have a wording and an intention and especially what you and your club come up with new rules make sure to think of them as something that has an intention and not something that has to be followed by the letter.

The Zen of Python (a set of rules or principles for Python) has two notes about exactly this problem:

Special cases aren't special enough to break the rules.
Although practicality beats purity.

Yes you have rules and you should try to apply them in a generic way but at some point you need to use your brain and understand what the rules was meant to achieve in the first place. I tired of having to explain these simple things. If your god is so smart why didn't he make everyone smart enough for these things?

Well I hope I didn't insult too much here. Let the flames come in the comment section

Posts for Wednesday, April 21, 2010

Clete Blackwell

Apple

Two years ago, I purchased my first Apple product. After the iPhone had dropped in price from $599 to $299, I was able to afford one. I purchased the white iPhone 3G 16GB model. I was amazed at the speed and ease of use. The iPhone was and is definitely a revolutionary device. Before the iPhone, most smartphones were very primitive. The iPhone was the first phone to implement multitouch and the first phone to implement a touch screen in an easy-to-use and user-friendly way.

I am very impressed with the iPhone. I upgraded to the white iPhone 3GS 32GB when it was released last summer. The speed of the 3GS was amazing and the battery life was decent.

In November, my Dell laptop died and I had many bouts with Dell over receiving a suitable laptop for my needs that was comparable to the one that I originally had. After Dell was unable to offer all that I needed, I decided to buy a MacBook Pro and sell the one that Dell gave me. I had had it with Dell and Windows in general. I wanted to move to a sleek, simple interface that offered the robustness of UNIX. Mac OSX is perfect for that.

Three days ago, my MacBook Pro arrived. It is absolutely amazing. It is fast and simple to use. Everything about it is polished beyond belief. I have no major or minor complaints so far. I have been adjusting some to the new UI and the new way of finding things.

Installing applications is just as easy as downloading a single file and dragging it into the applications folder. *.dmg files are really just CD/DVD images, so burning a program to a CD/DVD is extremely simple. The OS takes care of all of it for you. Additionally, the $70 Magic Mouse is definitely worth the purchase. I like it because I can use it left-handed easily. I also like the multitouch functionality of it. I’m blown away by Apple’s OS.

It is no marvel that Apple yesterday reported their best non-holiday quarter ever.

The only thing that I must shun Apple on is their unwillingness to open source their products. That is it.

Here is a screenshot of my workspace. I currently have my MacBook Pro connected to my TV VIA Mini DisplayPort -> HDMI.

Posts for Tuesday, April 20, 2010

Brian Carper

USA

DD-WRT: 1, Microsoft: 0

An xbox360 wireless card is $100, and it's theoretically the only wireless card that works with an xbox, so if you want wireless internet, you have to buy that card. I guess the idea is to sell the console cheap (for sufficiently large values of "cheap") and then gouge customers on proprietary cables and addons afterward. Microsoft isn't the only company that does this, by far. (Not nearly as bad as $20 for 8MB worth of PS2 memory card. Ughhhh.)

Fortunately, if you have a spare Linksys router lying around (as I do) you can throw DD-WRT on it, put it into Client Mode, connect your xbox to the router via a bit of ethernet cable, and there you go. I can also plug my aging desktop machine (sans wireless card) into the same router, and two other devices if I can find any.

Installing DD-WRT was surprisingly straightforward if you take the time to read through the wiki instructions first very carefully. A bit of healthy paranoia of turning your hardware into a brick goes a long way.

15 minutes, $100+ savings. Thanks again, Linux and open-source community.

Jürgen Geuter

Germany

No it's not 'logical'

Today we'll be looking at one of my pet peeves and from this you will be able to see a rant coming so let's not beat around the bush anymore and dive into the usual rambling

When people try to explain something to you the probability of somebody saying a phrase like "But it's so logical!" approaches 1. They might be telling you the way to a point of interest or their way to paint a bikeshed, it doesn't matter. Everything is always logical.

The adjective "logical" means that something behaves according to the laws of logic which luckily are not all that many: You take true and false, add variables and a bunch of operations like "and" and companions and you have what we call logic.

There are a few different approaches to what logic actually is: Some believe that Logic is the one structure that can never change, that is true for every conceivable world, others believe that we see logical structures behind things because our brains are hardwired for logic. Either way: Logic is something you have a hard time arguing with because a logical term has the useful property to be either true or false.

But why is it that when after a 3 hour lecture on how to properly organize a directory layout somebody talks about how his/her opinion is oh so logical we hear nothing about "true" or "false" and also no logical operations? Because we've been bullshitted. Opinions hardly ever have anything to do with logic, what people mean when they say "It's logical" is "I personally find that rather plausible." But since they claim to base their opinion on logic you cannot argue with it anymore.

It's actually a neat trick to win an argument. The best thing about it is that you don't have to understand any logic, you can just magically make your opinion "logic". How useful and how much better than spending the time to understand what logic means and how it's different from opinion and plausibility.

Just repeat after me: Logic does probably not mean what you think it means. Write it a hundred times.

Posts for Saturday, April 17, 2010

Matija Šuklje

Slovenia

DFD 2010 in Slovenia — report

The first Document Freedom Day in Slovenia has passed and it went pretty good.

I know that for the biggest impact I should have reported about it the very same day or at least the next one, but a) I was too tired b) I had to much other important tasks to do and c) I wanted to gather everything so I can submit a nicely rounded off report. Warning: longer post ahead.

The impatient of the more audiovisual persuasion can check out the photos of the DFD cake transfer at the Supreme Court here, the evening event's pictures on Kiberpipa's Flickr account and the lectures and the Q&A session on their video archive. Others read on.

DFD Cake

Quarter to eleven a group of us (Katja Guček, Rok Papež, Igor Kolar and me) met in front of the Court palace in Ljubljana. Despite heavy rain and some shenanigans including our cake, the court's security x-ray scanner and several people giggling, the cake reached its destination safe and sound. There we met the representatives of the Supreme Court — Alenka Jelenc Puklavec, (temporary) president of the Supreme Court; Janko Marinko, secretary of the Supreme Court; Gregor Strojin, head of the PR office; Bojan Muršec, director of the Centre for Informatics and head of the open source and open standards projects.

What followed was a full hour of chatting while eating (our) cake and supping (their) coffee not only about the importance of open standards, but — what we honestly did not expect — a deeper insight into how important these are to the courts, their painful experience with closed formats that stopped being supported, how hard they fought to be able to use open formats and open source solutions. What was really touching was how much dedication these people (as well as the Supreme Court's former president Franc Testen and others) have put into this. It showed that the biggest hurdle for the migration to open standards along with some open source solutions was neither finiancial nor technical nor practical, but solely pressure from the politics and the business side.

The courts were also the first in the country to produce a study on how much a migration to open source alternative (mainly OO.o at the time) would cost. It showed — oposed to the later governmental study — that the open source solution would in the longer run cost less, be a lot more reliable and tweakable. In the means of openness as well as in using standards, code and access to and from the people, it surprised us that our courts are at the very top of the EU — on par with the Finnish!

This all shows that our courts trully "get it" — we can only wish the government, parliament and others follow suit!

This is one cake and coffee I think neither party will easily forget. We were reassured of our choice and they were happy that a civil initiative is starting to form to be vocal about the importance of document freedom and free software.

Of course pictures were made and are available here.

The evening event

The event was quite a success as well.

We met in Cyberpipe/Kiberpipa at six in the evening and there was about 25-30 people present all the time. Because the people at Kiberpipa were so kind to record the whole event, I can provide you with the video and audio. The whole event was quite informal and people were mingling and talking while munching snacks between presentations.

After the greeting, I made a short introduction into open standards and open formats [video&audio]. I took the oportunity to promote the FSFE and our Fellowship group a bit as well.

After that Gregor Strojin explained a bit about the Supreme Court's need of open standards and Bojan Muršec held a presentation of the courts' centre of informatics and on how the courts migrated to ODF and OO.o during the years 2003-2006 [video&audio].

The last presentation was held by our Fellowship's own Andrej Čremožnik, who presented several open formats for multimedia and compared them to their proprietary counterparts both technically and otherwise [video&audio].

We concluded the formal part with a round table, giving the microphones also to the public. This resulted in a nice Q&A session [video&audio].

Of course, there are pictures from the evening event available as well.

Media coverage

We could have made even better use of the oportunity, but all in all it was more then good for our first event. A list of internet articles (we found about 20) who mentioned the event is available on the FSFE Wiki. Apart from that the DFD was mentioned in the IT section of Delo (one of the two biggest newspapers in Slovenia) and mentioned on at least two radios. I gave an oral statement for STA, a short live interview for Radio Slovenia International and in the evening Radio Val 202 (both very popular national stations) came to check out the event and interview all of the lecturers. A list of links can be accessed on the Fellowship Wiki.

Althought it was quite a handful, it went good. I hope the rest of the DFD teams had and equally successful day.

P.S. As a cherry on top, we were later told that the rest of the cake went to the IT department — to the people who did the hands-on technical work of migration and everything — where everyone got a piece and they made a small party out of the occasion as well. So the cake was shared equally between the policy makers and the techincal staff. Despite the rain, it could not be a better day for document freedom in Slovenia.

Posts for Friday, April 16, 2010

Brian Carper

USA

Printing a nicely formatted plaintext table of data in Clojure

I use this fairly often while data-munging, when I want to quickly view a list of hash-maps of data in a simple table format. Usually this is coming out of database. Maybe someone will find it useful.

(defn table
  "Given a seq of hash-maps, prints a plaintext table of the values of the hash-maps.
  If passed a list of keys, displays only those keys.  Otherwise displays all the
  keys in the first hash-map in the seq."
  ([xs]
     (table xs (keys (first xs))))
  ([xs ks]
     (when (seq xs)
       (let [f (fn [old-widths x]
                 (reduce (fn [new-widths k]
                           (let [length (inc (count (str (k x))))]
                             (if (> length (k new-widths 0))
                               (assoc new-widths k length)
                               new-widths)))
                         old-widths ks))
             widths (reduce f {} (conj xs (zipmap ks ks)))
             total-width (reduce + (vals widths))
             format-string (str "~{"
                                (reduce #(str %1 "~" (%2 widths) "A") "" ks)
                                "~}~%")]
         (cl-format true format-string (map str ks))
         (cl-format true "~{~A~}~%" (repeat total-width \-))
         (doseq [x xs]
           (cl-format true format-string (map x ks)))))))

Then you can do this:

user> (def data [{:name "Brian" :job "Code monkey" :age 29}
                 {:name "Bob" :job "Janitor" :age 97}
                 {:name "Johnny McLongname" :job "None" :age 3}])
#'user/data
user> (table data)
:name             :job        :age 
-----------------------------------
Brian             Code monkey 29   
Bob               Janitor     97   
Johnny McLongname None        3    
nil
user> (table data [:age :name])
:age :name             
-----------------------
29   Brian             
97   Bob               
3    Johnny McLongname 
nil

Martin Matusiak

Netherlands

lessons from “Coders at work”

I already mentioned Coders at work in an earlier entry. The point of this one is not to write a review, but to make a note for myself of what I’ve gotten out of the book. I think I could do better to read more books with a pen and a pad so I have a better chance of exploiting the content.

So these are notes to myself. I wouldn’t take it upon myself to summarize a more general listing of notes that would somehow apply to the average person, because I think we’re all in very different places in the universe that is called “learning to program (well)”, and every person has to figure out for himself what he most needs to learn relative to where he now is.

Advice: Read code

Read other people’s code, “open black boxes“. This is something I never really do, I should start. Just take some codebase and check it out, get used to the practice. Reading code is not the easiest thing to get into, so here are some tips:

First, get it to build.
Sometimes everything you have to do to build it already teaches you a number of things about the codebase. And once you have it built, you can start making changes to it and try out little things dynamically.
Read while building.
Making builds for any codebase can be hairy and painful, so parallelize this activity with code reading. Great way to use the time you’d otherwise waste in between debugging the build.

Advice: Write unit tests for new library

You’ve found a library for something that you’ve never used before: how do you figure out how to use it? Write unit tests. Some libraries have bad unit tests (or no tests) to begin with, so it could be a way to improve it. In any case you can test your basic hypotheses of how the library works.

Ideas to investigate

OO and classes vs prototypes (JavaScript).
“There is a lack of reuse in OO because there is too much state inside”. Libraries must expose too much of their innards through APIs, functional programming model should be better at this.

Pointers

Articles:

Richard P. Gabriel – Worse Is Better

Books:

Douglas Crockford – JavaScript: The Good Parts
In the absence of the book, Crockford’s lecture series on JavaScript is probably a good start.
William Strunk, Jr. and E.B. White – The Elements of Style
For writing better English.
Steve McConnell – Code Complete
On software engineering process and best practices.
Gerald Weinberg – The Psychology of Computer Programming

Talks:

Joshua Bloch – How to Design a Good API and Why it Matters

Mark Somerville

United Kingdom

Notes on setting up Gentoo

I’m installing Gentoo on my laptop – here are some notes (for myself, really).

Gentoo installs the flash plugin to a path with “netscape” in it. There is a problem with this http://groups.google.com/group/opera.linux/browse_thread/thread/9e9a3c21cd413c2f/3949ce2e8f96f84a so we need to symlink the plugin to the opera plugin path.

Mark Somerville

United Kingdom

Geotagging my Flickr photos

Tonight I finally got round to adding my photos to the Flickr mapping feature. I guess I got inspired by taking and uploading my first photo using Zonetags with my phone (Nokia 6680) and my Holux bluetooth GPS receiver.

Zonetags

Zonetags is a trivially easy-to-use beta-ish application for Series 60 Nokia smartphones that acts as a wrapper for the standard camera software. It can record the cell id (I think it uses the last known value if you’re not in reception) and attach it to each photo you take. You can also connect an external GPS device to add much more accurate longitude and latitude information to the photo. Once you have taken a photo you’re asked if you want to upload it to Flickr. If you do, Zonetags can optionally add action tags to do things like rotate your photos as well as suggestting extra tags based on your location and the tags that other people have given to nearby photos.

For example, when I uploaded a photo coming down from Snowdown at the weekend, “Caernarfon” and “United Kingdom” tags were added along with the geotagging and celltagging ones. Over time, this could get really smart.

Unfortunately, my camera can’t use my bluetooth GPS. I think as a workaround way of geotagging those photos, I might also take one with my phone and upload it with private permissions via Zonetags (to get the geotags into Flickr) and just replace the photo with the better camera shot when I get back to a computer. We’ll need to wait and see if that is too much of a pain in the arse to be practical when out in the hills though.

Flickr map

I’ve wanted something like this for a while now – being able to browse photos by location is nifty stuff. It’s super easy to get photos on the map (if you’re using a supported browser, of course…), but I find it is limited by the poor quality of the maps, at least for most of Scotland. Of course, Flickr can’t help this and it will improve given time, but it would all be so much cooler if the maps I use most were worth looking at! There are 3rd party apps that are more accurate or you can hand geotag the photos with the lat/longs yourself and import them instead, but I haven’t bothered looking at any of those options yet since my priority was to just get them all geotagged.

In some crazy alternate reality where I actually have some spare time, I will use the Flickr API in my desktop mapping software to give the same functionality in a faster and much more detailed environment. That’s after re-writing it so that it’s not just a prototype, adding in GPS tracklogging and a million other neat things… sigh.

The good

Flickr map is very cool, I like it
Zonetags is very cool, I like it

The bad

Yahoo! Maps isn’t as good as Google Maps
Yahoo! Maps is vastly inferior to Google Maps
Scotland has pish quality maps – ces’t la vie
Zonetags doesn’t allow you to change your mind and upload a photo you previously took with it – pretty poor
My phone/bluetooth/software combination only allows one phone -> bluetooth connection at a time

The ugly

The admin map interface doesn’t work in Opera – boo

From that list it looks like I’m not too positive about this stuff, but that couldn’t be further from the truth – it’s excellent and is going to keep getting better.

Update (26/10/2006)

I forgot to mention that I was a little annoyed that Flickr didn’t add geotag tags to my photos when I added them to the map. Last night I found this excellent bookmarklet which embeds Google Maps into Flickr, giving you much more accuracy. It totally rocks. Or, it will totally rock if it starts to work in Opera. For now it just rocks. It adds the geotags as well as giving your photos a link to Google Maps (you can see the singletrack!). There is a little discrepancy between Yahoo! and Google Maps (about 100-200m), but it is definitely preferable for me to use Google Maps for the forseeable future. Now I need to go back through my photos again…

Mark Somerville

United Kingdom

Migrated blog software (yet again)

I’m checking out Mephisto just now, RSS/Atom readers might have just got some double posts – sorry! I’ve written some more mod_rewrite rules so most of the important old URLs should work fine and, to be honest, I don’t care about any of the more obscure ones.

Recently, I’ve hardly had any time to write here and I hate switching blog software – so why have I done it yet again? Mostly to abandon the downright abysmal Typo and Mephisto offered a simple migration process away from that. So far so good, but we’re a whole two hours in…

We’re also (thankfully!) at least considering using Rails for some projects at work and every little helps when getting back in the swing of things.

Posts for Thursday, April 15, 2010

George Kargiotakis

Greece

Greek spammers email addresses blacklist

GrRBL
In the beginning of the year I announced my RBL for Greek spam emails. The blacklist is growing larger by the day, thanks to some really kind people forwarding me their Greek spam emails, and has reached more than 120 IP addresses of verified Greek spammers.This alone though is not enough.

Why
Some spammers use their aDSL lines which have dynamic IPs to send their massive email “newsletters”. These people are split into 2 sub-categories. The ones that use their own PC as an SMTP server and the ones who use their ISP’s mail server as SMTP. I’ve tried to complain to some of their ISPs…some replied back saying that they were willing to look into the issue (but did nothing at all in the end) and others did not even reply to me. For both sub-categories, GrRBL is ineffective since I can’t add dynamic IPs in the blacklist nor can I add the IPs of the email servers of those major Greek ISPs.

Another category of spammers is the one that uses their gmail/yahoo accounts to send their emails. GrRBL is ineffective for this category as well since I can’t add gmail/yahoo to the blacklist…

What
So there was no alternative but to gather all those email addresses of these 2 categories above and add them to a new blacklist, one that will contain email addresses. I use this blacklist with my spamassassin configuration to eliminate Greek spam that GrRBL can’t. Each time I receive (or someone forwards me) a new Greek spam, I add the “From:” email address to this new blacklist. This new blacklist grows far more aggressively than GrRBL since it’s a lot easier to gather the data and already has more than 140 addresses.

Distribution
There are two available formats of the blacklist, one ready for use by spamassassin and another one with clear formatting ready to be used even by SMTPs to drop these spam emails without even touching your inbox.
The blacklist is currently only distributed to a group of well trusted people and it is available only through rsync with a username/password.

Jobby is a generic forking process manager written in Ruby, built with robustness as a primary goal from the start. It was initially developed for offloading long running background tasks from the webserver in Rails applications, but it turned out to be useful in its own right, so we extracted it to work more generally. It uses Unix sockets and a call to fork(), so won’t work on Windows or JRuby, but has been tested on several flavours of Linux and OS X. It is also copy-on-write friendly, should the Ruby interpretter that’s used support it.

If you prefer to read code than blog posts, server.rb does most of the heavy lifting.

After a long struggle of using the old BackgrounDRb, last year we finally threw in the towel. We checked out the alternatives, but nothing really fitted the bill. Since background job processing is quite central to some of our applications, we built our own. From the start we had some requirements:

Run jobs in parallel up to a maximum number or processes, then start queuing
Robust – it just had to be solid
Easy to manage – we hated those BackDRb scripts
Reliable logging
Able to get messages from the background processes

Jobby itself does not provide a mechanism for getting messages from the child processes. I’ll detail a jobby_rails plugin that handles that some other time.

Installing Jobby

gem install jobby will do the trick if for some reason you don’t hate gems. You can also check out the source from github.

Gentoo users can install Jobby from the Gentoo Ruby overlay. Note: I stopped using Gentoo sometime around March 2010. It might still be there.

How does it work?

Jobby is a command line program. It consists of a daemon that listens on a Unix socket and will fork when it receives a connection. The forked child either runs some Ruby code or executes some shell code, depending on how the daemon was started. When you call Jobby, you pass a single parameter to STDIN – this string is passed to the forked child. It could be an ID of some sort, a whole email, a marshalled object or whatever.

There is only one command used, jobby. When this is run it first checks for the existence of the daemon process listening on the socket. If it’s not there, it starts one (and then executes the child command). This is great because it makes daemon startup totally automatic and if the daemon should go down (this hasn’t happened to us yet in the months that we’ve been using it), the next request will start up a new one.

Stopping Jobby

There are two ways to stop the daemon, both using Unix signals. Issuing a USR1 initiates a “very friendly shutdown”, which will stop the daemon accepting any new connections but will allow it to finish the jobs in the queue before terminating. Issuing a TERM signal will stop the daemon forking, then issue a TERM signal to the child processes and terminate the daemon.

Other nifty stuff

When using Jobby to run Ruby code, you might want to have the children load a few libraries before they do any work. Repeating this every time is a pain, so you can instead pass a --prerun parameter to prerun some Ruby code in the daemon before any forking takes place. This will load the libraries (at least in this example) so that they are immediately available in the children when the fork occurs. If you’re using a copy-on-write friendly interpretter, like Ruby Enterprise Edition, you’ll get the memory saving benefits of that too.

You can run multiple Jobby daemons – just call each one by passing a different socket parameter. This is how you can run different code for different job types that you have.

To make your debugging life easier, the forked children will show up in process listings as jobby: /path/to/socket. The actual daemon process will show up as jobbyd: /path/to/socket.

Using Jobby with Rails

Jobby was designed for use in Rails applications, but was extracted since it is generally useful. This post only describes Jobby itself, which you can happilly use from a Rails app using a system call. Sometime I’ll get round to releasing and detailing a Rails plugin for Jobby that adds a communication layer to Jobby and makes Rails intergration nicer. We’re using it just now, but it’s not really releasable as it is. I decided that since the main Jobby code has been so stable for us, it made sense to release it now.

Mark Somerville

United Kingdom

Ensuring epatch works correctly on embedded Gentoo

On my BeagleBoard, the ebuild messages were telling me that the patches had been applied successfully, but inspection of the code showed they hadn’t. It was a nasty problem to fix – it seems like a epatch wasn’t liking the cross-compiled bash.

After compiling bash natively and getting a another shell, the epatching was successful.

Mark Somerville

United Kingdom

Nibbles 0.1 released

I’m pleased to say that I’ve just released version 0.1 of Nibbles, an alternative RSS and Atom feed reader for Maemo. A big thanks to the people who were kind enough to help out with the pre-release testing, you’ve been (and hopefully continue to be!) a great help.

Nibbles screenshot

Nibbles is quite stable, can parse a wide variety of feeds and is pretty bandwidth efficient. Nibbles is finger-friendly and does aim to be a good mobile feed reader, but things like downloading of full offline articles didn’t make it into this initial release (I have some working code for that, but it’s not working enough!). New users might want to know that the zoom in/out hardware buttons hide and show the feed and article lists.

Of course, since this is a first release, things aren’t perfect. The biggest problem for me that I think people should be aware of is that Nibbles uses too much battery power. This is detailed in bug #2342 and is really a problem with ruby-gtk2 rather than Nibbles. As I said, with some luck, the next ruby-gtk2 release will fix this.

You can install Nibbles from the project page.

Mark Somerville

United Kingdom

Nibbles, a feed reader for Maemo

I don’t really get on with the supplied RSS reader on my N800. It doesn’t work like I expect and it lacks mobile features like offline reading that I’d like. So, I’m writing a feed reader for Maemo called Nibbles. It’s still very early stages, but there’s enough for a development release now, I think. Below is a screenshot of the current preview version, 0.0.4:

Nibbles screenshot

It seems to cope nicely with a wide variety of RSS and Atom feeds, despite my proof-of-concept code somehow making it into the app ;).

Currently it’s quite bare of features and can be considered the simplest app that could work (for example, the feeds update at a fixed 30 mins). It will likely stay that way until I get a few bugs worked out and 0.1, the first ’stable’ version, released. After that, I’ll work on some more interesting new features. I’m still very open to suggestions though and all feedback is welcome, so please don’t be shy!

If you’d like to play along and help the development of a better feed reader for your internet tablet, use the .install file. Please report issues to the Maemo Garage tracker.

You can make Nibbles the default feed reader by editing /usr/share/applications/defaults.list. You need to set hildon-news_reader=nibbles.install. This will allow you to tap on the feed icon in a browser window and have the feed added to Nibbles.

I’ve only specified that the app will work in OS2008, but I’m pretty sure it’ll work in OS2007 too. Since I only have an OS2008 machine, I can’t test it easily. If anyone is still using OS2007 and would like to help, please do get in touch.

Mark Somerville

United Kingdom

Creating RubyGems .debs for Maemo

I’m writing an application (more on that sometime soon) using the very nice Ruby bindings for Maemo. I’ve got some dependencies that are packaged as RubyGems that I wanted to be managed through the Application Manager, so I created some .debs for them. ~~Here is their temporary home~~ The rake, hpricot, feed-normalizer, simple-rss, atom and htmlentities gems are available in my repository. I didn’t create the rubygems and librubygems packages themselves – I was sent them, but couldn’t find them hosted elsewhere.

It took me a little while to suss out how to create the .debs so I’ve included the steps here in case anyone finds themselves in the same boat. This works for native Ruby gems and compiled C extensions.

Install the gem inside Scratchbox. Irritatingly, “gem install example” doesn’t always want to do very much, so you may need to dowload the gem file and “gem install example-0.1.gem”.
Make a new directory with the rubygems-<package>-<version> syntax (the rubygems- prefix seemed to make sense to me): mkdir rubygems-example-0.1</lil>
cd rubygems-example-0.1/ ; mkdir gems specifications bin
cp -r /targets/CHINOOK_ARMEL/var/lib/gems/1.8/gems/example-0.1/ gems
cp /targets/CHINOOK_ARMEL/var/lib/gems/1.8/specifications/example-0.1.gemspec specifications/
cp /targets/CHINOOK_ARMEL/var/lib/gems/1.8/bin/example bin/ (if there are any bins)

Create a Makefile with this content (you will need to replace the spaces at the start of the lines with tabs):

all:
clean:
install:
        mkdir -p $(DESTDIR)/var/lib/gems/1.8/gems/
        mkdir -p $(DESTDIR)/var/lib/gems/1.8/bin/     <------ REMOVE THIS LINE IF THERE ARE NO BINS
        mkdir -p $(DESTDIR)/var/lib/gems/1.8/specifications/
        cp -r gems/* $(DESTDIR)/var/lib/gems/1.8/gems/
        cp bin/* $(DESTDIR)/var/lib/gems/1.8/bin/     <------ REMOVE THIS LINE IF THERE ARE NO BINS
        cp specifications/* $(DESTDIR)/var/lib/gems/1.8/specifications/

dh_make -e you@example.com then choose the type of package.
Edit debian/control to suit. Make sure the section is something like user/Utilities, not just Utilities if you want the package to show in the Application Manager.
dpkg-buildpackage -rfakeroot -b
The parent directory should now contain rubygems-example_0.1-1_armel.deb

As a final note, /var/lib/gems/1.8/bin/ may not be in $PATH, so you'll need to deal with that if you have files in bin/.

Mark Somerville

United Kingdom

Mercurial 0.9.5 for Maemo

I’m using Mercurial to track changes in my latest project, a GTK mapping application. I couldn’t find a build for my N800, so I made one myself. ~~I’ll host a .deb until I get around to putting it on the garage or creating a repository.~~

Edit: I’ve started a garage project.

To install python2.5-runtime I needed to add repository.maemo.org to the application catalogue to satisfy some dependencies. Tested with OS2008 beta on an N800 only.

Planet Larry is not officially affiliated with Gentoo Linux. Original artwork and logos copyright Gentoo Foundation. Yadda, yadda, yadda.

Planet Larry

Brief signals primer

Signals in MRI

Ruby 1.8

REE 1.8.7-2010.01

Patching Ruby 1.8 – including 1.8.6, 1.8.7 and REE

Ruby 1.9

Patching Ruby 1.9.1

Further reading

DFD Cake

The evening event

Media coverage

Advice: Read code

Advice: Write unit tests for new library

Ideas to investigate

Pointers

Zonetags

Flickr map

The good

The bad

The ugly

Update (26/10/2006)

Kernel

Wireless card and LEDs

Installing Jobby

How does it work?

Stopping Jobby

Other nifty stuff

Using Jobby with Rails