http://world.std.com/~franl/worm.html

A Slashdot article reminded me of one of my favorite technical articles on the Internet entitled “A Tour of the Worm", an in-depth historical and technical look at the Morris worm. The Morris worm, mistakenly unleashed in 1988, was one of the first significant worms to strike the Internet, and it caused enough damage that it arguably has done the most relative damage of any worm since then.

Check it out if you haven’t read it, or even if you have; it’s a fascinating look at the early days of the Internet. http://world.std.com/~franl/worm.html

Here’s the thing. I hate stating the obvious. It really annoys me. On the other hand, obvious things are sometimes things that most need to be repeated. So I wrestle with myself and I finally decide that I should, because there is a shockingly large number of people out there who don’t realize how obvious this is. See if you can learn something from this mock dialog.

Vendor: Good morning, is this Harry, the CTO*, I’m speaking to?
Client: Yes, how may I help you?
Vendor: Hey Harry, this is Steve from Microsoft. I would like to talk to you about Windows Vista.
Client: What’s that?
Vendor: Why, it’s the brand new version of our Windows operating system.
Client: Oh, that.
Vendor: I was wondering if I could interest you in our product.
Client: You know what, I don’t think so, we are a very security sensitive company, and..
Vendor: But that’s precisely the reason I’m calling, I would like to tell you how you can enhance your security with Windows Vista. You see, we’ve built the operating system with security in mind and it’s the state of the art in operating systems.
Client: Hey, that sounds pretty exciting. So how does this work now, you ship us the source code and…
Vendor: No no, we don’t distribute the source code.
Client: You don’t?!?
Vendor: No, you see it’s a trade secret. (my precious etc)
Client: You’re kidding, right?
Vendor: No, really.
Client: So how do we know that it’s actually secure if we can’t see for ourselves? How do we know there isn’t anything malicious in it?
Vendor: Well you’ll just have to trust us.
*Harry hangs up*
Vendor: Hello? Harry?
*CTO - the highest placed person who makes technical decisions in a company.

How did it go? Did you get it? It was kind of a long thing, huh? Ok, stop racking your brains, I’ll give you the answer: no source code, no security.

Here’s how that works. It’s simple economics, so try to keep up. If they give you the source code, then they put their cards on the table. You can see what the code does, and if it’s doing something stupid (security hole) or nasty (like sending your data to back to the vendor), then you’ll be able to check for this. Now you may say “I don’t know how to check”, and that’s okay. But just by giving you the source code the vendor knows that you can see everything the code is doing. And if you find something nasty in there, they know you’ll never trust them again. So it doesn’t really matter if *you* don’t know how to check, because there are others who do, and sooner or later someone will find the nasty code if it’s in there. Thus, if the vendor gives you the source code, then he’ll be a lot more careful about what’s in there, because he’s risking losing your trust and your business forever. That will keep him honest.

Is there then anything surprising about finding out that Microsoft is putting in backdoors in Windows? No, because how would you know it’s there? You don’t have the source code! In case you were wondering, the words “security” and “backdoor” are mutually exclusive.

So what have we learned today? Is there somehow we could summarize all this in just one sentence? There is: If you want security, ask for the source code. If you can’t get the source code, you know that the vendor isn’t taking security seriously.

  * ERROR: net-fs/nfs-utils-1.1.2-r1 failed.
 * Call stack:
 *       misc-functions.sh, line 652:  Called install_qa_check
 *       misc-functions.sh, line 360:  Called die
 * The specific snippet of code:
 *   		[[ ${abort} == "yes" ]] && hasq stricter ${FEATURES} && die "poor code kills airplanes"
 *  The die message:
 *   poor code kills airplanes

I had to undelete someone's files from a FAT partition today. My first thought was to use good ol' Windows to do so, given that Windows is the unholy ground which spawned FAT to begin with. I remember there used to be an UNDELETE command of some sort in some old version of DOS. But this doesn't seem to exist in XP any longer.

There are however lots and lots of third-party "shareware" programs which can do this kind of thing, as Google reveals. There is in fact an overwhelming number of such shareware programs. Most of these programs are total crap and cost around $30. One program required me to burn a CD and reboot my computer from the CD before I could run it. Many of the programs "intelligently" scan a partition looking for chunks of things that look like JPEGS or WMVs. I tried a few "demos" before I gave up, not having an hour to waste finding the one program that would work. Thus bringing the current score to Windows: 948, Brian: 0.

Instead I brought the drive home and plugged it into Gentoo and used this post as a guide. I dd'ed the partition to a file, fscked around with it a bit, mounted it via loopback, and had my files back. Took 10 minutes, and worked as expected. And it didn't cost me $30.

The moral of this story: I need to burn a Knoppix disk to take to work with me.

My only quibble is that I can never ever remember what Gentoo package contains fsck.vfat. Note to self, it's dosfstools. I can never think of the search terms even to locate that package. I had to google it.

My ninth call to Westinghouse today, about my Westinghouse L2410NM 24" LCD monitor which I RMA'ed back in March, revealed that they did in fact shipmy monitor, supposedly to my house, on April 4th or so. A UPS tracking number confirms it. There are are a few things wrong with this.

1. In spite of the fact that I asked for a phone call to be updated on the status of my monitor whenever it was shipped, I received no such phone call.
2. During the four phone calls (or was it five?) I made to Westinghouse in April, AFTER my monitor was supposedly shipped to my house, no one at the company had any record that it shipped. I was told that by multiple representatives over the past four weeks that my monitor was "in processing".
3. I asked for my monitor to be shipped to workplace, not my house. My nice, safe, cozy workplace with human beings who can sign for large expensive packages. Not my empty house in a neighborhood full of drug addicts, in the property theft capital of the west. In addition to telling the phone representative this, I actually taped a 8.5 x 11 inch sheet of paper directly to the monitor itself (as well as the outside of the box) specifying SHIP TO: and my work address. Even such drastic measures were not enough to catch the attention of whatever magical monitor-repair fairies work at Westinghouse, apparently. Perhaps I should've carved that information directly into the monitor screen.
4. I could possibly overlook the above, except that, as you may have surmised, at the present time, I do not, in fact, have my monitor.

After calling up UPS to ask why their driver left a $450 computer monitor, in a shiny bright blue and white box with pictures of a computer monitor all over it, sitting on my front porch while I was at work without getting my signature, I placed call number ten (yes, I've finally hit double digits!) to Westinghouse, and managed to escalate my issue to the Westinghouse corporate office. Supposedly in 7-10 business days they will send me a brand new monitor.

Oh how I wish I had any confidence that I'm ever going to see that monitor.

In the meantime, this guy was on sale at the local store, so I bought one. Time will tell whether LG brand is any better than Westinghouse. This time, I also bought the extended warranty, having learned my lesson that it can, indeed, be worth an extra $60 to save myself some pain and aggravation later. I'm also going to think twice about buying things like this over the internet in the future. There is something to be said about being able to drive 10 minutes down the road to have your property serviced or replaced by real-life human beings, rather than paying to have things shipped around the world for a month.

I’ve made a difficult decision that I hope doesn’t make anyone feel too bad: effective immediately I’ve removed any feeds from Planet Larry / Larry the Universe that were from developers who have retired from the Gentoo project.

I feel like ex-developers carry a lot of weight with their posts and opinions, and I created the planet feeds mainly for users, by users (speaking of which, don’t be shy, and sign up).

That’s all for now.

"Intellectual Property is the oil of the 21st century"

Computers are a love/hate thing for me. I love all things digital, but I desperately need to get away from it sometimes too. So I had a nice vacation away from my computer last week. I couldn't keep myself from reading some mailing lists and hitting Slashdot once a day, but I didn't write a single line of code and didn't give my websites or work projects or anything much thought.

But now my vacation is over, and it's so easy to fall back into old habits, endlessly looking at webcomics and reading articles about Common Lisp unit testing suites and cringing at the latest drama amongst Gentoo devs and minding my message board like a crusty old beat cop making his rounds. It's the life I've chosen, and I do like it, but I do like getting away sometimes too.

I fulfilled one of my dreams last week when I finally caved and ordered a solid glass mousepad. They're pretty cheap on newegg.com, depending on the color you want. I happened to want green, and it happened to be the cheapest, so all is well. It looks very nice, and it's big and hopefully the surface won't degrade over time; I tend to eat through mousepads via a slow yet inexorable process of erosion.

Unfortunately my laser mouse doesn't work on it. However, I have learned that if I upgrade my mouse's firmware, it will magically be able to work on a solid glass mousepad. Who would've thought my mouse had updateable firmware, let alone that updating the firmware would allow it to work on new surfaces? Not I.

The bad thing is that I need freaking Windows XP to upgrade the firmware on my mouse. I don't have any computer that has XP on it and I'm afraid to try anything in a virtual machine that involves something as dangerous as fiddling with the innards of connected peripherals. So I tried to install XP on my laptop, desperate times calling for desperate measures. But of course the install failed because my XP install CD is so old (pre-SP1, received free from my college 7 years ago) that it didn't recognize most of my hardware. In fact, the XP install CD blue-screened, which set a new record for how low Windows could sink in my opinion.

So I tried slipstreaming SP2 into my install CD. But it failed because, get this, the filenames of some drivers on the CD, namely usbehci.sys, ended up in lower case rather than uppercase and the CD's install program couldn't locate them. I kid you not. Since when is anything in Windows case-sensitive? Is it running Linux? I had to burn another CD after renaming all the files into uppercase. Then the CD worked, but it couldn't find my hard drive, probably due to missing SATA drivers. At that point I gave up, and plan to take my mouse to work tomorrow to upgrade the firmware on a work machine that has XP on it.

And so the score up to this point in my life is Windows: 947, Brian: 0. Windows remains undefeated.

Thanks go out to Logitech for not letting me use Vista (or, say, LINUX) to upgrade my mouse's firmware, and of course to Microsoft, for yet another gloriously broken and frustrating computing experience.

Google’s SketchUp is an incredibly easy-to-use architectural and 3D-modeling tool. The Computer Science Club here at Mansfield has been talking about modeling the university for a 3D perspective in Google Earth, similar to what Google Earth provides for New York City and other metropolitan areas. Recently, I have been experimenting with SketchUp. A free version is available from the website and a professional version can be purchased for about $500. People enrolled in Universities can obtain the professional licenses for $50 a year. The $50 counts towards purchasing a full license, so after 10 years, it’s yours forever. Or, you can pay for the educational license for 3 years and pay the rest of the money up front.

At a first glance, SketchUp seems to be too simple to be worth anything. Upon further investigation, the simplicity seems to come from Google’s innovational perspective. Google has outdone themselves with SketchUp. It is amazingly easy to pick up and create simple objects. More complicated objects can be created with some practice. I have spent about an hour and a half working with the program. First, I watched the beginning tutorials. Then, I went straight into making objects and refining them. It’s amazingly simple. Just make a shape, pull it up to give it depth, draw other objects on it, and manipulate them. It’s amazingly simple.

In about twenty minutes, I was able to make the desk that I use here at school. Keep in mind that it isn’t 100% perfect, nor is it 100% to scale. I have never seen a program this simple. With an hour and a half of experience, I was easily able to make this (click to enlarge):

Here is the front of it:

From an angle:

From the side (notice the arches in the drawers):

And here it is next to a person:

I consider OLPC to be one of the most exciting initiatives of the last few years. When the idea was first circulated it was such an exciting call to arms to do something about the lack of education in poor regions of the world. And the project has produced what appears to be a pretty incredible product, the research of which is now recycled back into the general hardware industry, so it has brought advances that wouldn’t otherwise have happened (now).

I recall pondering the real purpose of the project, asking what is going to be achieved with these laptops. The OLPC project had a very good answer to this. They said the laptops will promote learning in areas where school books are a luxury. Furthermore, the laptop itself is completely tweakable, you press a special key and the source code of the current program pops up. This will promote learning through tweaking and experimentation, so that eventually an industry can be built on these foundations, in regions where little industry exists today and where perhaps the potential for one (in terms of natural resources) is bleak. A beautiful dream, one that could change the world in big ways.

Now Negroponte has changed his tune. Visionary that he is, he failed to convince the clients of the value of free software. So now he’s humming “forget open source, it’s all about the kids!” while preparing to run Windows on the laptop. There is a new smoke screen being constructed:

Negroponte says that the organization is working to ensure that Sugar can run smoothly on Windows.

Riiiight, running Sugar on Windows. Tell me, what exactly is the value of running Windows with an all free software stack? It’s completely useless, that’s what. The whole value of Windows is as a platform, not merely as an operating system. People buy Windows to run Windows applications, not for Windows itself. Or are we actually buying that Egyptian officials are eager to purchase Windows licenses in order to run the free software suite?

Congratulations, Negroponte, you’ve just become a licensed Windows vendor. The kids will no doubt have fun clicking on the Start menu and playing Solitaire. There is a great deal to learn from that, just nothing about the operating system or the applications, you know, actual learning.

OLPC in its original form was about empowering the users, with Windows that capability is entirely destroyed. The fact you cannot mix learning with trade secrets should be blindly obvious to anyone. Open souce is important, but it’s especially important when you want people to learn something.

Furthermore, learning doesn’t happen in isolation. It’s accelerated when it happens in a community of ideas and impulses that flow freely. Resigning OLPC president gets it when he says:

“What comes part and parcel with open source is a culture, and it’s the culture that I’m interested in,” he says. “It’s a culture of expression and critique, sharing, collaboration, appropriation.” And this culture can and should spill into classrooms, he says.

Mal eben zum festhalten:

Einfache Schleife
for i in `ls` ; do echo $i ; done

In Dateien suchen und ersetzen
cat filename | sed ’s/suchen/ersetzen/g’ > filename-new

Dateien umbenennen und hinten etwas wegnehmen
mv $i ${i%-new}

For a very long while now, several years actually, I've been a bit annoyed by the behaviour of terminals under X when you doubleclick links. What the UI considers a word is selected. Selection 'starts' at the point that is doubleclicked and 'spreads' in each direction, stopping at a char it considers to be a word delimiter. A space is probably always considered a delimiter. Sometimes a '?' too, and often ',' as well. This has been very annoying for me as I spend quite alot of time in irc (irssi in a screen, its lovely in combination with bitlbee; icq, irc etcetc in the same screen.) and every now and then someone pastes an url which I want to doubleclick and then paste into my browser. Now the selection stops prematurely as it's not uncommon for a hyperlink to contain one of the characters that is in the word delimiter list. I've thought of this as a limitation to the system I use and, although annoyed, never gave it much thought. Maybe I've been to susceptible to propagandaists telling me gnu/linux is user-unfriendly.

A couple of weeks ago I started to think a little about it and the solution is rather simple. I use Eterm more or less exclusively.

In ~/.Eterm/user.cfg I've put this:

<eterm-0.9>
begin misc
cut_chars "\t\\\`\\\"\'() *,;<>[]{|}"
end misc

This might not be perfect but it's certainly served it's purpose this far as I have been able to doubleclick on the links and then middleclick in the address field (or open a new tab) in the browser. My OS is even better =)

Graduate School Information Request

---

Your request for information has been processed.

Thank you for your interest in the University of Massachusetts Amherst.

Applications directed to addresses within the United States are mailed twice weekly with first class postage; allow 10 days for delivery. Applications directed to non-U.S. addresses are mailed weekly and sent air mail; allow at least 3 weeks for receipt. There is no charge for any of these materials.

The following materials will be sent to you here:

Department of Computer Science
KICT, IIUM
Gombak, 53100 MYS

• CMPSC program materials

Sometimes you may want to check that an email address is not syntactically invalid, i.e. it looks like a recognisable email address. I use this approach in my zetact contact form processor.

Of course, it does not mean the address actually leads anywhere, but at least you know are dealing with an email address that could exist.

This is the code I have been using, albeit I have changed it from a class method to a simple function to make this post simpler.

"""Email check using regex."""

def invalidreg(emailkey):
    """Email validation, checks for syntactically invalid email             
    courtesy of Mark Nenadov.                                               
    See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65215"""
    import re
    emailregex = "^.+\\@(\\[?)[a-zA-Z0-9\\-\\.]+\\.([a-zA-Z]{2,3}|[0-9]{1,3\
})(\\]?)$"
    if len(emailkey) > 7:
        if re.match(emailregex, emailkey) != None:
            return False
        return True
    else:
        return True

I decided it would be more Pythonic to try to do this using the built-in string methods, rather than importing the re module and using a monster regular expression. Here was my first attempt.

"""Email checks using string methods - simple version."""
def invalidemail(emailaddress):
    """Checks for a syntactically invalid email address."""
    try:
        emailitems = emailaddress.rsplit('@', 1)
        emailitems.extend(emailitems[1].rsplit('.', 1))
    except IndexError:
        return True

    if [x for x in emailitems if not x.replace(".","").isalnum()] \
            and emailaddress >= 7:
        return True
    else:
        return False

After a bit of testing and playing with this, a friend pointed me towards the relevant RFC on restrictions of email addresses. While the standard allows the use of many different special characters, in practice email addresses have to be much stricter if you actually want people in the real world to be able to send email to you.

For example, if we allow the email address []@commandline.org.uk, will whatever receives the output of this function be able to use it? As pointed out by Jan Goyvaerts, most software won't actually be able to handle obscure special characters.

We also don't want to water down the syntax check and allow junk for the sake of theoretical but non-existent addresses.

My compromise is to allow these special symbols -_.%+. in the local-part of the email address, and -_. in the domain name. I also do sanity checking on the top-level domain, it needs to be either a generic name or two characters long (country codes are all two letters).

So below is my current version, I added lots of comments and white space to make it easy to read.

"""Ditch nonsense email addresses."""

GENERIC_DOMAINS = "aero", "asia", "biz", "cat", "com", "coop", \
    "edu", "gov", "info", "int", "jobs", "mil", "mobi", "museum", \
    "name", "net", "org", "pro", "tel", "travel"

def invalid(emailaddress, domains = GENERIC_DOMAINS):
    """Checks for a syntactically invalid email address."""

    # Email address must be 7 characters in total. 
    if len(emailaddress) < 7:
        return True # Address too short.

    # Split up email address into parts.
    try:
        localpart, domainname = emailaddress.rsplit('@', 1)
        host, toplevel = domainname.rsplit('.', 1)
    except ValueError:
        return True # Address does not have enough parts. 
    
    # Check for Country code or Generic Domain.
    if len(toplevel) != 2 and toplevel not in domains:
        return True # Not a domain name.
    
    for i in '-_.%+.':
        localpart = localpart.replace(i, "")
    for i in '-_.':
        host = host.replace(i, "")
    
    if localpart.isalnum() and host.isalnum():
        return False # Email address is fine.
    else:
        return True # Email address has funny characters.
    
# Start the ball rolling.
if __name__ == "__main__":
    print invalid("warrior@example.com")

Discuss this post - Leave a comment

[update] Question solved, see bottom of post.

Since Python 2.5 the language got a new built-in method ‘all’ (and it’s nephew ‘any’). I wanted to play around with this a little, combined with generators, so I created a little testcase to test performance.

Here’s the test-case: take a list L of X random numbers in a given range [A, B], and check whether

• all elements in L are >= A
• all elements in L are >= (A + Z) where Z is a number in [0, (B - A)]

The first test should always result True, the second test could result to False.

Here’s the output of a test-run:

In [1]: import random, sys

In [2]: a = [random.randint(100, sys.maxint) for i in xrange(2000000)]

In [3]: len(a)
Out[3]: 2000000

In [4]: #Check whether all elements are >= 100 

In [5]: %timeit all(i >= 100 for i in a)
10 loops, best of 3: 515 ms per loop

In [6]: %timeit any(i < 100 for i in a)
10 loops, best of 3: 454 ms per loop

In [7]: def f(l):
   ...:     for i in l:
   ...:         if i < 100:
   ...:             return False
   ...:     return True
   ...: 

In [8]: %timeit f(a)
10 loops, best of 3: 292 ms per loop

In [9]: #Same thing for 100000, since now the list shouldn't be completely iterated

In [10]: %timeit all(i >= 100000 for i in a)
100 loops, best of 3: 4.73 ms per loop

In [11]: %timeit any(i < 100000 for i in a)
100 loops, best of 3: 4.29 ms per loop

In [12]: def g(l):
   ....:     for i in l:
   ....:         if i < 100000:
   ....:             return False
   ....:     return True
   ....: 

In [13]: %timeit g(a)
100 loops, best of 3: 2.82 ms per loop

In [14]: #For reference

In [15]: %timeit False in (i >= 100 for i in a)
10 loops, best of 3: 531 ms per loop

In [16]: %timeit False in (i >= 100000 for i in a)
100 loops, best of 3: 5.03 ms per loop

It’s as if ‘all’, ‘any’ or ‘in’ don’t break/return when a first occurence of False (or True, obviously) is found. Is this the desired behaviour, and if it is, why? The calculation time difference between using all/any/in or a custom-made function (which is, unlike all etc, not written in C) which breaks whenever it can, is pretty astonishing.

[update] Question solved. It’s pretty normal the function-based approach performs better, since it combines what ‘all’ and the generator provided to ‘all’ do, taking away the generator function-call overhead. Damn

If you’ve been dealing with files for a while you will have noticed that there is a slight semantic gap between how humans see files and how computers do. If you’ve ever seen a file list like this you know what I mean:

Lecture10.pdf
Lecture11.pdf
Lecture12.pdf
Lecture1.pdf
Lecture2.pdf
…

Numbering these files was done in good faith, and a user understands what it means, but the computer doesn’t get it. Sorting in dictionary order produces the wrong order as far as the user is concerned. The reason is that the digits in these filenames are not treated and compared as integers, merely as strings. (Actually, . comes before 0 in ASCII, what’s going on here?)

While we’re not expecting our computers to wisen up about this anytime soon, there is the obvious fix:

Lecture01.pdf
Lecture02.pdf
…
Lecture10.pdf
Lecture11.pdf
Lecture12.pdf

You’ve probably done this by hand once or twice, while cursing.

On the upshot, this is very easy to fix with a few lines of code:

#!/usr/bin/env python
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
#
# revision 1 - support multiple digit runs in filenames
 
import os, string, glob, re, sys
 
def renseq():
    if (len(sys.argv) != 2):
        print "Usage:\\t" + sys.argv[0] + " <num_digits>"
    else:
        ren_seq_files(sys.argv[1])
 
 
def ren_seq_files(num_digits):
    files = glob.glob("*")
    for filename in files:
        m = re.search("(.*)(\\..*)", filename)
        ext = ""
        if m: (filename, ext) = m.groups()
 
        digit_runs = re.finditer("([0-9]+)", filename)
        spans = [m.span() for m in digit_runs if digit_runs]
        if spans:
            spans.reverse()
            arr = list(filename)
            for (s, e) in spans:
                arr[s:e] = string.zfill(str( int(filename[s:e]) ), int(num_digits))
            os.rename(filename+ext, "".join(arr)+ext)

 
 
if __name__ == "__main__":
    renseq()

Download this code: renseq.py

This works on all the files in the current directory. Pass an integer to renseq.py and it will change all the numbers in a filename (if there are any) to the same numbers, padded with zeros if they have fewer digits than the amount you want. So on the example

renseq.py 2

will turn the first list into the second list.

If say, there are filenames with numbers of three digits and you pass 2 to renseq.py, the numbers will be preserved (so it’s not a destructive rename), you’ll just revert to your incorrect ordering as it was in the beginning.

renseq.py will rewrite all the numbers in a filename, but not the extension. So mp3 won’t become mp03.

I had to fix a problem at my dad's company...
"The network was broken."

It was a NetBEUI network connecting some windows stations - it has been running for years - and now suddenly the nodes couldn't find eachother.
One of the boxes (windows 2000 iirc) had 2 network cards, one for the network, the other not used for anything (not even connected). Disabling the latter - not even touching the former - fixed half of the network.

read more

Someone in #lh today told me about Sender Policy Framework, which sounds like a badly-needed enhancement to the Internet’s email protocols. Basically, the idea is to provide a DNS record that informs MTAs “don’t trust emails claiming to be from this domain unless they’re coming from one of my actual servers".

In DNS, this looks like (in my case):

emptymatter.org. IN TXT "v=spf1 a mx ~all"

Some MTAs support SPF but need to be configured, I believe Gentoo’s postfix is one of them. If I’m going to expect other mail servers to support it, I probably should myself. I’ll have to tackle that another day…

Since Heise announced that Microsoft will release the Windows XP Service Pack 3 on the 29th, I didn’t sleep too well, as I really want to grab it as soon as possible. Sure, I use Linux, but this doesn’t mean that I am not dangling with windows boxes from time to time and I am fed up applying about 100 patches before I can even think of security.

But I was laughing too soon - as just one day after the 29th (today), I have seen a posting on the Washington Post Blog that Microsoft has delayed the start of the service pack again. In a written statement they say:

“In order to make sure customers have the best possible experience we have decided to delay releasing Windows XP SP3 to Windows Update and Microsoft Download Center.”

In other words, there seems to be no release date yet. Well - in the meantime I’m rolling my own update pack using the Heise Offline Update. Thanks a ton, guys.

Nothing exciting here. Got my tax refunds. Might build a home-made NAS with a couple terabytes of disk and put it in the basement.

On the DS, I’ve been playing Rondo of Swords and The World Ends With You. Rondo is a pleasant find, a difficult but still reasonable strategy RPG that makes one think and plan ahead, unlike games such as Revenant Wings which are much more “bring a healer and just mob everyone at the thing they’re strong against!” Also, I have a crush on Atlus by this point. There’s no denying it now. I draw their name with little hearts all around when I’m in meetings.

The World Ends With You is refreshingly original, one of those games that, even with it being Square Enix, is a bit surprising that it made it to the States. Very Japanese, and the game makes few concessions to the English audience. Sure, long gone are the times of gratitutous name changes, but even the j-pop/j-rock soundtrack remains intact, and that is, to my slightly jaded mind, a bit commendable. Now, if only the main character didn’t suffer from two vile Square Enix staples: unimaginable thinness and nearly sickening teenage angst. Neku is supposed to get better with the latter; I hope it is soon.

My games to beat are now Etrian Odyssey and Rondo of Swords, one I must beat before Etrian Odyssey II (guess which one) is released here, and the other before the Final Fantasy IV remake reaches the States. I’m excited. If I have time before those, Final Fantasy III and The World Ends With You are my RPGs to beat. FF3 is a cakewalk thus far, but its ease and its crude mechanics compared to Final Fantasy V make it hard to stay with for long.

I didn’t really intend this to become all about video games. I’ve been working on a Gentoo Wiki page for the HP 2133 which has kind of slowed down as most of the parts I’m interested in are supported as best they can be without new versions of drivers, I think. There’s some other hardware that I need to try out (the webcam, for example), but I don’t really care that much, so it’s low priority. Notebooky stuff works.

I have a Waterfield Designs bag coming soon, which I’m excited about. Don’t think it will be suitable for gaming books, but I still have that backpack which is going on 5+ years. The little trooper.

I’ve been meaning to survey the gaming group and associated friends to see what they’re using for IM these days. I think the answer for some is “nothing", with a couple saying “AIM on occasion” or “I idle on Google Talk", so I’ve not really been motivated to test those waters. I want to get a private Jabber conference room running for the group, since the IRC thing kind of sputtered off and died (I still idle there!), but I know it means getting people to switch to Jabber (or at least Google Talk) and then getting them to use a non-Google Talk client (Pidgin, I bet, but maybe Trillian would work). Sigh. If anyone has interest in switching to one network (I highly suggest a Jabber-like ["XMPP” for the techies]), or trying out conferences, or whatever, email/IM me and we’ll play around.

This really is getting rambly, and people might expect me to write long posts all the time. So I’m wrapping this up by saying that spring is finally here, and that’s why it snowed yesterday.

Use Readline shortcuts

At the bash prompt, you can use the default readline keybindings, these are similar to Emacs ones. Many of these are also available within other programs that use readline, such as the Python interpreter.

Here are some useful ones:

Ctrl-A Beginning of Line
Ctrl-E End of Line

Ctrl-U Kill (cut) everything left of cursor
Ctrl-K Kill (cut) everything right of cursor
Ctrl-W Kill (cut) the single word before the cursor
Ctrl-Y Yank (paste) the text back

Ctrl-L Clear Screen
Ctrl-D Exit
Ctrl-R Reverse interactive-search, (attempt to complete what is currently being typed using the history file)

SSH without Passwords

If you login to a remote machine often and you get bored of typing the password, then you can use public key cryptography instead.

The way it works is that the remote machine has a copy of your local machine's public key, it can then use that to check that your local machine is really your machine, and so let you in.

To start with, on the local machine, see if you already have a key pair:

ls ~/.ssh/id_?sa.pub

If not, then make one:

ssh-keygen -t dsa

Now you need to copy your public key to the remote host. On the local machine run:

scp ~/.ssh/id_?sa.pub remotehost:

Now we login to the remote server:

ssh remotehost

Append the public key to your authorized keys file

cat id_?sa.pub >> ~/.ssh/authorized_keys

Now you can login without passwords. Make sure the security of your machines is well thought out. Use disk encyption if possible.

Create a script directory in home directory

I often talk about random Python or bash scripts. The easy way to use them on Linux is to make a dedicated script directory for these.

mkdir ~/bin

Add it to your shell's path. Edit ~/.bashrc and add:

export PATH=$HOME/bin:$PATH

Now all the scripts that you add to ~/bin are always available. This makes things a lot more flexible and fun as you can try out various scripts by dropping them in ~/bin and then deleting them when you are bored of them.

Discuss this post - Leave a comment

1. Logged into a lab computer via XDMCP: I could've used VNC, but that would've required someone to be logged-in on the lab comp.


2. Using my laptop's PulseAudio as the lab computer's default PulseAudio sink: This makes the lab computer's PulseAudio send all sound to my laptop's PulseAudio by default.


3. Connected to my laptop's DAAP share from the lab computer's Rhythmbox: The music on my laptop becomes accessible from the lab computer's Rhythmbox.

This has probably happened to you. You come to a web page that has links to a bunch of pictures, or videos, or documents that you want to download. Not one or two, but all. How do you go about it? Personally, I use wget for anything that will take a while to download. It’s wonderful, accepts http, https, ftp etc, has options to resume and retry, it never fails. I could just use Firefox, and if it’s small files then I do just that, and click all the links in one fell swoop, then let them all download on their own. But if it’s larger files then it’s not practical. You don’t want to download 20 videos of 200mb each in parallel, that’s no good. If Firefox crashes within the next few hours (which it probably will) then you’ll likely end up with not even one file successfully downloaded. And Firefox doesn’t have a resume function (there is a button but it doesn’t do anything ).

So there is a fallback option: copy all the links from Firefox and queue them up for wget: right click in document, Copy Link Location, right click in terminal window. This is painful and I last about 4-5 links before I get sick of it, download the web page and start parsing it instead. That always works, but I have to rig up a new chain of grep, sed, tr and xargs wget (or a for loop) for every page, I can never reuse that and so the effort doesn’t go a long way.

There is another option. I could use a Firefox extension for this, there are some of them for this purpose. But that too is fraught with pain. Some of them don’t work, some only work for some types of files, some still require some amount of manual effort to pick the right urls and so on, some of them don’t support resuming a download after Firefox crashes. Not to mention that every new extension slows down Firefox and adds another upgrade cycle you have to worry about. Want to run Firefox 3? Oh sorry, your download extension isn’t compatible. wget, in contrast, never stops working. Most limiting of all, these extensions aren’t Unix-y. They assume they know what you want, and they take you from start to end. There’s no way you can plug in grep somewhere in the chain to filter out things you don’t want, for example.

So the problem is eventually reduced to: how can I still use wget? Well, browsers being as lenient as they are, it’s difficult to guarantee that you can parse every page, but you can at least try. spiderfetch, whose name describes its function: spider a page for links and then fetch them, attacks the common scenario. You find a page that links to a bunch of media files. So you feed the url to spiderfetch. It will download the page and find all the links (as best it can). It will then download the files one by one. Internally, it uses wget, so you still get the desired functionality and the familiar output.

If the urls on the page require additional post-processing, say they are .asx files you have to download one by one, grab the mms:// url inside, and mplayer -dumpstream, you at least get the first half of the chain. (Unlikely scenario? If you wanted to download these freely available lectures on compilers from the University of Washington, you have little choice. You could even chain spiderfetch to do both: first spider the index page, download all the .asx files, then spider each .asx file for the mms:// url, print it to the screen and let mplayer take it from there. No more grep or sed. )

Features

• Spiders the page for anything that looks like a url.
• Ability to filter urls for a regular expression (keep in mind this is still Ruby’s regex, so .* to match any character, not * as in file globbing, (true|false) for choice and so on.)
• Downloads all the urls serially, or just outputs to screen (with --dump) if you want to filter/sort/etc.
• Can use an existing index file (with --useindex), but then if there are relative links among the urls, they will need post-processing, because the path of the index page on the server is not known after it has been stored locally.
• Uses wget internally and relays its output as well. Supports http, https and ftp urls.
• Semantics consistent with for url in urls; do wget $url… does not re-download completed files, resumes downloads, retries interrupted transfers.

Limitations

• Not guaranteed to find every last url, although the matching is pretty lenient. If you can’t match a certain url you’re still stuck with grep and sed.
• If you have to authenticate yourself somehow in the browser to be able to download your media files, spiderfetch won’t be able to download them (as with wget in general). However, all is not lost. If the urls are ftp or the web server uses simple authentication, you can still post-process them to: ftp://username:password@the.rest.of.the.url, same for http.

Download spiderfetch:

• spiderfetch.rb

Recipes

To make the use a bit clearer, let’s see some concrete examples.

Recipe: Download the 2008 lectures from Fosdem:

spiderfetch.rb http://www.fosdem.org/2008/media/video 2008.*ogg

Here we use the pattern 2008.*ogg. If you first run spiderfetch with --dump, you’ll see that all the urls for the lectures in 2008 contain the string 2008. Further, all the video files have the extension ogg. And whatever characters come in between those two things, we don’t care.

Recipe: Download .asx => mms videos

Like it or not, sometimes you have to deal with ugly proprietary protocols. Video files exposed as .asx files are typically pointers to urls of the mms:// protocol. Microsoft calls them metafiles. This snippet illustrates how you can download them. First you spider for all the .asx urls, using the pattern \.asx$, which means “match on strings containing .asx as the last characters of the string”. Then we spider each of those urls for actual urls to video files, which begin with mms. And for each one we use mplayer -dumpstream to actually download the video.

#!/bin/bash
 
mypath=$(cd $(dirname $0); pwd)
webpage="$1"
 
for url in $($mypath/spiderfetch.rb $webpage "\\.asx$" --dump); do
	video=$($mypath/spiderfetch.rb $url "^mms" --dump)
	mplayer -dumpstream $video -dumpfile $(basename $video)
done
 

Download this code: asx_spiderfetch.sh

Living Code

David Parker famously said that texts are living, once they leave the pen of the author then they have a life of their own, you never know where the text will end up or how it will be modified. For Python code that is even more true.

The beauty of Python is that you can write code fast, share code and modify code. For this to work, your code needs to be readable. Writing code is easy, reading other people's code is much harder, or even reading your own code after a few months or years has past.

Therefore the aim is to make code as readable as possible, even if it causes a little more work when you write it. The way to make your Python code most readable is to keep to the Style Guide for Python Code, also known as PEP8.

Pylint for the Win

It is far easier to keep your code valid to PEP8 as you go along, than to try to move a large codebase to PEP8 at the end. I recommend the use of a tool called pylint.

Pylint is available from all Linux distributions' package managers (e.g. apt-get install pylint or emerge pylint). Here are some instructions for Windows.

If you have ever made a webpage you probably know about HTML-tidy or the online W3C Validator tool. These tell you everything wrong with your HTML.

Pylint is similar, it goes through and tells you both syntax errors and also how your code differs from the PEP8 standard.

There are some corner cases in which you will need to give pylint the finger, but doing it consciously for good reason is better than because you are sloppy.

PEP8 is better than your crappy style

People often don't use PEP8. This is for a variety of (bad) reasons.

Firstly, sometimes people are tourists from another programming language, they do not know any better so they write their Python code like it was Java or C code.

Secondly, Sometimes people think their (cl)own style is better than PEP8 in some technical way. Well that does not matter. I might have a better way to design a plug socket, but if I implemented my better plug socket, I would not be able to buy any electrical devices.

There can only be one standard, and PEP8 is that standard. If you want to change that standard then bribe, sleep with or kill Guido Van Rossum.

Not following the standard makes your code less readable to others, this prevents the quick reuse that Python is designed for (see above).

If you are a free-software/open-source project, then you particularly should be ashamed if you write hard to read code, because allowing other people to read, understand and modify your code is the whole point.

Lastly, some people don't use PEP8 because the document is too circular and verbose for them to remember. I feel your pain, below are the main points in 12 easy rules.

The 12 commandments

Guido, who brought you out of the land of Visual Basic, out of the land of slavery, spake all these words to thee:

• Module names should be in all lowercase - hello.py.
• Class names should be in CamelCase.
• Methods and functions should be in lower_with_underscores
• Implementation-specific 'private' methods _single_underscore_prefix
• Especially private non-subclassable methods __double_underscore_prefix
• Top level constants (i.e. those that are not in a function or class) should be in BLOCKCAPITALS. Overuse of these constants may make your code less reusable.
• If a variable inside a function or method is so temporary and disposable that you cannot give it a name, then use i for the first one, j for the second and k for third.
• Indentation is four spaces per level. No tabs. If you break this rule then you must be stoned in the village square.
• Lines are never more that 80 characters wide. Tip, break lines with a backward slash \. You do not need to do this if there are parentheses, brackets or braces. Don't add extra parentheses just to break lines, use \ instead.
• Spaces after commas, (green, eggs, and, ham)
• Spaces around operators i = i + 1
• Write docstrings for all public modules, functions, classes, and methods. Python is an international community, so use English for docstrings, object names and comments. If you want to provide local translations then use a proper localisation library.

Discuss this post - Leave a comment

The USA’s 2008 election is the first governmental election of any kind that I will have an opportunity to vote in. This is a very important election for everyone. We face major decisions about the war in Iraq and on many major issues. A lot is at stake in this year’s general election. I have followed this election more than any other. I have been involved in a lot of conversations with people who come from very different cultures and backgrounds. Many of these people support my viewpoints, many of them disagree, and many of them could not care less about politics.

I have been appalled at how little it takes for a certain candidate to obtain an undecided and unopinionated person’s vote. Those of us who are strong conservatives or liberals will not be swayed to vote the opposite way for anything in the world (although this is debatable). However, undecided people are swayed way too easily. I have heard at least ten people tell me that they will vote for Obama because he is “young” and relates to “the younger generation” more than any of the “old” people such as Clinton and McCain. I have had one person tell me that, although they are a liberal, they would not vote for McCain if they were conservative because he is “old” and that he will probably “die in office.”

People disregard the candidate’s actual views and opinions and focus in on the unimportant things. They decide to vote for candidates because they “seem like nice people.” I am not sure if it is the college students’ way or if the entire population is this way, but it’s atrocious. If I wanted to, I could run for President, take all of the “correct” views on all of the issues; that is, the ones that would get me elected. I could completely disagree with all of these views, but I could stand up there, smile, act friendly, cry when I hear a sad story, etc. I could be a complete phony and get some of these idiots to follow me just because I am a down-to-earth and likable character.

What has our country come to? Elect someone because they are honest and they agree with your issues. Elect them because they will lead our country in the right (or should I say correct?) direction. But do NOT elect them because they are nice people and they seem friendly. This is ridiculous. Does anyone want someone in office who is nice but leads our country to demise? I don’t think so.

Amendment: It seems that my post has sparked a large debate over at the Gentoo Forums. Check it out here. It looks like most people agree with my basic argument that people are ignorant. However, many people seem to have some insane (read: extreme socialistic / communistic) ideas on the Gentoo Forums.

• Vim: On-the-fly Code Checker for Vim
  

  
  I don't need to say anything about this -- it's simply wonderful.
  

  
  


• Pidgin: Master password support for pidgin
  

  
  Finally, I won't have to type in three passwords when I start Pidgin. gnome-keyring integration ftw!
  

  
  


• GNOME: Reducing memory fragmentation in GNOME
  

  
  Reduced memory is always more than welcome, and it's not an easy task at all :)
  

  
  


• GNOME: SuperRandom: a predictive listening plugin for Rhythmbox
  

  
  This should be interesting. I currently have to manually form lists of songs I like to listen to. Time will tell how well this project fruits.
  

  
  


• GNOME: Make Evolution use Seahorse widgets for encryption and key selection
  

  
  No more UI blocking, and no more f-ugly password prompts! Fun, fun, fun =)
  

  
  


• GNOME: Integrated Media Management for GNOME
  

  
  This has always been a major missing feature in GNOME. This gets implemented, and I owe the student+mentor(s) a beer each *g*
  

  
  


• GStreamer: Media service to transcode video and audio in different formats
  

  
  If this integrates well with the above GNOME project, things will take a turn for the better w.r.t media in GNOME.
  

  
  


• Moinmoin: Extending moin's groups to LDAP and other external sources.
  

  
  We use MoinMoin for our internal wiki, and authentication is done against LDAP. So needless to say, this project will make management of roles using LDAP groups quite straightforward.
  

  
  


• BZFlag: BZFWeb, a web-based BZFS admin interface.
  

  
  Just today we were talking about implementing a basic web interface for BZFlag in Django over the summer, and this thing comes up :)
  Although our idea was to let people select server options using the webinterface, and then connect to the server, this project would be a good thing to keep an eye on.
  

  
  


• FFmpeg: Generic frame-level multithreading support
  

  
  This will rock for everyone with more than one core. This doesn't include me, but includes the Navya Server ;)
  

  
  

Did you hear the exciting news? JRuby 1.1 is out! For real, you can call your grandma with the great news. Wow, that was quick.

Okay, so the big new thing in JRuby is a bytecode compiler. As you may know, up to 1.0 it was just a Ruby interpreter in Java. Now you can actually compile Ruby modules to Java classes and no one will know the difference, very devious. Sounds like Robin Hood in a way, doesn’t it?

The JRuby guys are claiming that this makes JRuby on par with “regular Ruby” on performance, if not better. Hmm. Just to be on the safe side, what size shoes do you wear? Oh ouch, those are going to be tricky to fit in your mouth. And Freud will say you’re stuck in the oral stage. Too much? Okay.

So here is my completely unvetted, dirty, real world test. No laboratory conditions here, you’re in the ghetto. First we need something *to* test. I don’t have a great deal of Ruby code at my disposal, but this should do the trick. How does scanning the raw filesystem for urls sound? The old harvest script actually does a half decent job of turning up a bunch of findings.

Now introducing the contenders. First up, his name is JRuby, you know him from occasional mentions on obscure blogs and the programming reddit past the top 500 entries. He promises to free all Java slaves by giving away free Rubies to everyone!

Aaand the incumbent, the famous… Ruby! You know him, your parents know him, every family would adopt him as their own child if they could. He’s the destroyer of kingdoms and the creator of empires, he’s bigger than Moses himself!

Our two drivers will be racing across a hostile territory. Your track is a 25gb ext3 live file system. During this time, I can promise you that only Firefox is likely to be writing new urls to disk, but I could be lying eheheh. Due to the unpredictable nature of this rally track, regulations allow only one racer at a time, but you will be clocked.

First up is the new kid on the block Jay….Ruby. The Ruby code will not be compiled before execution, we’ll let the just-in-time compiler do its thing.

$ time ( sudo cat /dev/sda5 | bin/jruby harvest.rb –url > /tmp/fsurls.jruby )
real 39m26.547s
user 37m19.072s
sys 1m28.406s

Not too shabby for a first run, but since this a brand new venue, we have no frame of reference yet. Let’s see how Ruby will do here.

$ time ( sudo cat /dev/sda5 | harvest.rb –url > /tmp/fsurls.ruby )
real 78m42.186s
user 62m12.537s
sys 2m18.721s

Well, look at that! The new kid is pretty slick, isn’t he? Sure is giving the old man a run for his money. Let’s see how they answered the questions.

$ lh
-rw-r–r– 1 alex alex 86M 2008-04-21 18:29 fsurls.jruby
-rw-r–r– 1 alex alex 8.6G 2008-04-21 20:58 fsurls.ruby

Yowza! No less than a hundred times more matches with Ruby. What is going on here? Did Jay just race to the finish line, dropping the vast majority of his parcels? Or did father Ruby see double and triple and quadruple, ending up with lots and lots of duplicates? Well, we don’t really *know* how many urls exist in those 25gb of data, but it seems a little bit suspect that there would be in excess of 8gb of them.

One way or the other, it’s pretty clear that the regular expression semantics are not entirely identical. In fact, you might be sweating a little right now if your code uses them heavily.

UPDATE: Squashing duplicates in both files actually produces two files of very similar size (13mb), in which the disparity of unique entries is only a very reasonable 4% (considering the file system was being written to in the process). The question still remains how did Ruby produce 8gb of output.

I couldnt resist posting this, everytime i think xkcd couldnt get any better they go post something like this!

Of course, now xkcd posted this, how long till someone goes off and actually makes this

(incase the image doesnt show, heres the original xkcd post)

That’s a question that’s been bugging me for months now. It’s so vexing to try to find something out and not getting it. All the more so when you look it up in a couple of different places and the answers don’t seem to have much to do with each other. Obviously, once you have the big picture, all those answers intersect in a meaningful place, but while you’re still hunting for it, that’s not helpful at all.

I put this question to a wizard and the answer was (not an exact quote):

A function whose free variables have been bound.

Don’t you love to get a definition in terms of other terms you’re not particularly comfortable with? Just like a math textbook. This answer confused me, because I couldn’t think of a case that I had seen where that wasn’t the case, so I thought I must be missing something. The Python answer is very simple:

A nested function.

It’s sad, but one good answer is enough. When you can’t get that, sometimes you end up stacking up several unclear answers and hoping you can piece it all together. And that can very well fail.

I read a definition today that finally made it clear to me. It’s not the simplest and far from the most intuitive description. In fact, it too reads like a math textbook. But it’s simply what I needed to hear in words that would speak to me.

A lexical closure, often referred to just as a closure, is a function that can refer to and alter the values of bindings established by binding forms that textually include the function definition.

I read it about 3 times, forwards and backwards, carefully making sure that as I was lining up all the pieces in my mind, they were all in agreement with each other. And once I verified that, and double checked it, I felt so relieved. Finally!

I can’t follow the Common Lisp example that follows on that page, but scroll down and you find a piece of code that is much simpler.

(define (foo x)
	(define (bar y)
		(+ x y))
	bar)
 
(foo 1) 5 => 6
(foo 2) 5 => 7

Download this code: closure.lisp

What’s going on here? First there is a function being defined. Its name is foo and it takes a parameter x. Now, once we enter the body of this function foo, straight away we have another function definition - a nested function. This inner function is called bar and takes a parameter y. Then comes the body of the function bar, which says “add variables x and y“. And then? Follow the indentation (or the parentheses). We have now exited the function definition of bar and we’re back in the body of foo, which says “the value bar“, so that’s the return value of foo: the function bar.

In this example, bar is the closure. Just for a second, look back at how bar is defined in isolation, don’t look at the other code. It adds two variables: y, which is the formal parameter to bar, and x. How does x receive its value? It doesn’t. Not inside of bar! But if you look at foo in its entirety, you see that x is the formal parameter to foo. Aha! So the value of x, which is set inside of foo, carries through to the inner function bar.

Can we square this code with the answers quoted earlier? Let’s try.

A function whose free variables have been bound. - A function, in this case bar. Free variables, in this case x. Bound, in this case defined as the formal parameter x to the function foo.

A nested function. - The function bar.

A lexical closure, often referred to just as a closure, is a function that can refer to and alter the values of bindings established by binding forms that textually include the function definition. - A function, in this case bar. That can refer to and alter, in this case bar refers to the variable x. values of bindings, in this case the value of the bound variable x. established by binding forms, in this case the body of the function foo. that textually include the function definition, in this case foo includes the function definition of bar.

So yes, they all make sense. If you understand what it’s all about.

Let’s return to the code example. We now call the function foo with argument 1. As we enter foo, x is bound to 1. We now define the function bar and return it, because that is the return value of foo. So now we have the function bar, which takes one argument. We give it the argument 5. As we enter bar, y is bound to 5. And x? Is it an undefined argument, since it’s not defined inside bar? No, it’s bound *from before*, from when foo was called. So now we add x and y.

In the second call, we call foo with a different argument, thus x inside of bar receives a different value, and once the call to bar is made, this is reflected in the return value.

Well, that was easy. And to think I had to wait so long to clarify such a simple idiom. So what is all the noise about anyway? Think of it as a way to split up the assignment of variables. Suppose you don’t want to assign x and y at the same time, because y is a “more dynamic” variable whose value will be determined later. Meanwhile, x is a variable you can assign early, because you know it’s not going to need to be changed.

So each time you call foo, you get a version of bar that has a value of x already set. In fact, from this point on, for as long as you use this version of bar, you can think of x as a constant that has the value that it was assigned when foo was called. You can now give this version of bar to someone and they can use it by passing in any value for y that they want. But x is already determined and can’t be changed.

Last time I blogged about Firefox (2.x), I was complaining about how much memory it was sucking up. I have read about how many memory leaks were supposed to have been fixed in Firefox 3 but I had to see it to believe it. So after suffering again from low memory while running VMWare and Firefox (whose memory consumption regurarly climbs to 500MB) at the same time, I decided to upgrade and leave Firefox 2.0 for good. Firefox 3.0beta5 is blazingly fast and I have yet to see it use more than 170MB of memory. Thanks to the all the Firefox developers for finally fixing what was probably the biggest problem with Firefox 2.0.

Update: I also installed it on my Linux machine at home and memory usage and performance with many tabs open is far better than Firefox 2.

Nonsense laws are socially divisive

In Saudi Arabia, ownership of a Bible is illegal. If you are found in the possession of a Bible, at the very least you will have it confiscated, you may be given corporal punishment. If you are found in possession of many bibles, then you can be executed (source).

We do not have that law in UK, or any laws like it. Not only is the UK a Christian country, (in name at least), but also because the majority of the people in the UK think that laws against owning books are stupid, only barbarians have such laws. In Britain you can own the Bible, the Koran, Harry Potter, whatever the heck book you want.

No matter what legal arguments you make for a law, if the majority of the public think it is stupid then it won't work.

A lot of countries used to have laws against "Nightwalking", i.e. walking around at night, because people out at night are obviously up to no good. Most countries have abolished such laws. Make laws about soliciting maybe, but wandering around at night? That was just silly.

Silly laws should be abolished. If you just leave silly laws on the statue book, expecting everyone to just ignore these laws, then you are undermining the law itself and making a mockery of the institutions charged with enforcing the law.

How many million people file share?

An study in 2005 claimed than 9.2 million people in Britain were involved in filesharing, which represented an annual 50% increase from the 4.3 million people that were involved in filesharing in 2003. Source 1, Source 2.

What is the number now? If a 50% annual increase was maintained, then it would have been 13.8 million people in 2006, 20.7 million people in 2007, and 31 million by the end of this year, 2008.

By the end of next year, it would be 46.6 million, and by 2010, it would be 70 million. That can't be possible of course as the population of the UK is only 60.6 Million.

So we can be pretty sure the number of people file-sharing is within the range of 10 million to 60.6 million. If you have more accurate figures please let me know. For sake of discussion, lets choose an arbitrary number - 20 million.

Filesharing is normal behaviour

Lets say that 20 million people in the UK have been or are involved in filesharing. With 20 million people, filesharing is not a crime, it is a mandate.

Labour achieved 9.5 million votes at the 2005 general election, for this it received 356 out of 646 possible seats, and became the government.

If 20 million people in the UK are filesharing then it cannot be considered a crime, or a bad act, it is the democratic will of the nation.

The majority of people have decided that previewing a song by downloading it is fair, the majority of people have decided that sharing a file with your friends is not the same as stealing a car.

Therefore, the government must stop trying to make 20 million British people into criminals, but instead should try to understand the cultural changes happening here and then frame the policy agenda accordingly.

The public are not interested in the police spending time in a futile mission to stop kids sharing music with other in order to prop up dying foreign companies. Instead spend the scarce resources on stopping organised drug crime, or on solving murders or on confiscating illegal guns from urban street gangs.

If the government cannot see this, and wants to waste our money on misadventures, then we will throw you lot out and get a government that does represent our values. That is democracy.

The old music companies are not important to the economic future of Britain

Likewise, the music and film industries also need to wake up and smell the coffee, their potential customers like sharing music and film on the web with each other. It is too late, they need to just get over it.

Suing their own customers is not going to help them manage decline. It is not going to turn back the clock. The old companies that represent yesterday's music industry cannot burn down the Internet, however hard it tries. The Internet has become far bigger and far more important economically and socially than the old music or film companies.

Google is one of the biggest web companies, alone it makes $10 billion in annual profit, that is double the profit for the whole music industry. To put it another way, all of these old music companies are worth, economically speaking, half a Google.

Things change, that is life, get on with it. The arrival of electric refrigeration killed off the ice storage companies. Tough luck, no one made a law protecting the old ice storage companies against people using freezers in their homes to make ice.

These old music and film companies if yesterday must be told to just live with it, get on with it, embrace it. Make services that appeal to these people, or cease to exist. Governments must not kill the Internet golden goose for some old dying companies.

The old ice storage companies going to the wall did not end cold drinks in the summer, in fact electronic refrigeration led to a massive increase in the number of cold drinks available.

Likewise if the old music companies are too slow to adapt and go to the wall, it will not be the end of music, or the beginning of the end of music, it will be, as Churchill famously said, the end of the beginning.

Politicians, this is your final warning

If you are a politician, be aware, we are watching you like never before. In the past you might have taken party contributions from special interests and then given them special treatment.

Now we, the public, have our own communication channels and this time, we will punish you for it. You will represent us, the people, or we will remove you.

Discuss this post - Leave a comment

Entry at Digg

As part of my new job im left dealing with a whole series of databases, and one of the regular jobs is to update them from various sources. Normally that involves extracting data as a CSV file from the main database, which then needs to be edited and checking in excel because all the table names need changing so that when you import it into the database system used within the office it matches the fields used there. Also the conversion to csv seems to introduce random characters in fields, so as part of the process you also have to check all the data and make sure it doesnt contain any of these random characters before finally importing it and hoping everything works.

Now incase this didnt sound too painful, while most of the time there are only a few new records, im about to get to the one time of year where we receive in excess of 2500 new records. And while in previous years it may have been accepted that you need to manually check them, i *really* dont want to have to waste a day (and my sanity!) checking all this data.

So we get to my problem(s), not only am i stuck using windows (bad!), but ive also got a crappy CRT monitor (my eyes!), and finally i have to spend half my time working with a custom frontend to an oracle database which is written in java and helpfully launches from inside IE and is so obtuse and badly designed i feel sick everytime i use it!

However, getting away from my severe java allergies :p im hitting a roadblock in terms of automating at least some of this, originally i was going to try out my python skills, till i realised i really dont know anything, so i resorted to using PHP. In both cases though, despite some googling there seem to be very few, if any, decent tutorials/guides to working with CSV files. So my question plea is for any guides/tutorials or any decent documentation for working with CSV files in PHP. Im open to pretty much anything as the best ive found is for pretty much just opening the file and reading it.

Now all i have to do is dust off my PHP skills and hope i can throw something together that works!

Linus Torvalds writes the Linux kernel, he also likes a good mailing list flamewar, not least because he has a very sarcasatic wit. Here he is, writing about various topics.

On fair use:

When you start thinking that you have absolute control over the content or programs you produce, and that the rest of the worlds opinions doesn't matter, you're just _wrong_.

Me, personally, I think the RIAA and the MPAA is a shithouse. They are immoral.

On virtualization:

I think what you're seeing is virtualization proponents being absolutely _desperate_ for any reason to use virtualization.

On userspace binary drivers:

No user-space ass-hattery here.

On turning off interrupt requests:

You cannot have a generic kernel driver that doesn't know about the low-level hardware (not with current hardware - you could make the "shut the f*ck up" a generic thing if you designed hardware properly, but that simply does not exist in general right now).

On those arguing for userpace interrupt request handlers:

You may be a bit simple. But I think it's more polite to call you "special". Or maybe just not very used to how hardware works.

On C++ :

In fact, in Linux we did try C++ once already, back in 1992. It sucks. Trust me...

C++ is a horrible language. It's made more horrible by the fact that a lot of substandard programmers use it, to the point where it's much much easier to generate total and utter crap with it. Quite frankly, even if the choice of C were to do *nothing* but keep the C++ programmers out, that in itself would be a huge reason to use C.

So I'm sorry, but for something like git, where efficiency was a primary objective, the "advantages" of C++ is just a huge mistake. The fact that we also piss off people who cannot see that is just a big additional advantage.

On Linux Kernel version 2.6.19:

It's one of those rare "perfect" kernels. So if it doesn't happen to compile with your config, you can rest easy knowing that it's all your own d*mn fault, and you should just fix your evil ways.

On Intel's inventions:

The fact that ACPI was designed by a group of monkeys high on LSD, and is some of the worst designs in the industry obviously makes running it at _any_ point pretty damn ugly. And the fact that MB vendors don't test it with anything else than Windows (and sometimes you wonder whether they do even that) doesn't help.

EFI is this other Intel brain-damage (the first one being ACPI). It's totally different from a normal BIOS, and was brought on by ia64, which never had a BIOS, of course. Sadly, Apple bought into the whole "BIOS bad, EFI good" hype, so we now have x86 machines with EFI as the native boot protocol.

On Apple OS X:

OS X in some ways is actually worse than Windows to program for. Their file system is complete and utter crap, which is scary.

I always seem to update my blog in bursts. So, while I suggest that no one get hasty and expect large posts the next couple days, I thought I would mention a little something to keep me from feeling totally neglectful.

I bought a HP 2133 ("Mini-note") to play with, the middle configuration with SuSE, to be exact. So that should be exciting. Won’t be here for another couple weeks, though. :( The plan is to use it as essentially a quick companion for work and such, with enough of a desktop environment to get work done and/or SSH into my various sites of interest. The MacBook is still around, too, for any hypothetical bigger or more involved tasks. Not that the HP 2133 is a toy, but a lot of it is about playing around and being a bit less tied to my desktop.

Looking forward to when it gets here, since it would seem that it will also become a minor task to get everything supported in Gentoo… looks like I get to contribute to another page on the Gentoo wiki.

I sit down to emerge -av
It churns through the dependencies
I get frustrated in 2 minutes
pmerge -a is what was amiss

I alias `emerge` to spit out a warning
that a crappy PM is what I'm invoking
Now I'll remember to always use;
pmerge so my time's not abused!

Diverse technical interests

My technical focus has shifted quite a lot over the years.

I was really into computers as a small child, but took my teenage years off it to play guitar, bike outside and meet girls. Then I started to get more heavily into computers (again) in 1998 when I went to University and had a broadband connection.

I started by copy-editing for the web and writing bucket loads of HTML. This led to ASP and then a bit of PHP. I then, around 2002, moved to open source software and got deeper and deeper into Linux/Unix system administration, as well as, of course, making websites and setting up CMSes and so on.

Over the years I had picked up the very basics in many different programming languages, but I decided in 2006 that if I was going to develop my programming (while working and studying part time), then I should concentrate on one language for the moment.

I wanted a language that I could use to make interactive websites, as well as use for system administration, but it also needed to be good for programming for the Linux desktop.

So in the Summer of 2006, I sat in a relative's garden with a beatup ancient laptop and some printed out Python documentation, and never looked back.

These days my favourite computing activity is programming, especially with the Python, as regular readers might have noticed. I am churning out Python at a good rate, some of it for paid work, some of it for my own private learning and a tiny bit ends up in my public Python script directory.

In the coming months the amount of programming will rise still further as I am likely to take on some new roles from May, more of that will be revealed later when/if that happens.

Let's all share our scripts together

So I was asked:

zeth have you got a bzr repo of your python scripts? so people using your little apps could give something back?

An interesting idea. The whole idea of free software/open source is that people can modify the code and share the changes. It might be that no one ever uses it, if so I have lost nothing, it is worth a try.

I only wanted to include the ones I thought were both useful and were relatively self-contained. The focus is on more on developing useful scripts that can be used by others, rather than in old scripts that only fulfilled a need of mine only. So I had a bit of a clear out, and uploaded nine fairly self-contained Python modules to Launchpad. With the idea is that I publish new ones there in the months ahead.

Here is the site. I was going to call it "Zeth's scripts" but since the idea is that it will be a collaborative affair, i.e. you can get involved, it is called Eden.

It is a garden because it is small now but might develop and change over time. Some scripts will grow, some will die.

For example, in the last post here, I made a module which took updates from Twitter and put them in pop-up notifications in the GNOME-desktop. Say someone takes the program and changes it so that Facebook updates are published instead, or makes it so it makes pop-ups for KDE or the Mac instead of for GNOME.

That person could then push their version ('branch' in bzr terminology') back up to the site. They do not need permission from me or anyone else. Launchpad can contain as many branches as you like.

Even more importantly, other people can also submit their own scripts and we can discuss them and modify them. You don't even have to use Python (shock, horror).

Making your own branch

So to get your own branch of the programs.

bzr branch lp:eden

Now lets say you have edited one of the scripts, or you have put new scripts into the directory. You can then share them back to launchpad like this:

bzr push sftp://zeth0@bazaar.launchpad.net/~zeth0/eden/devel

Replace zeth0 both times with whatever your launchpad username is.

If you don't have a launchpad username click here and enter an email address.

As that bloke with the beard says, "Happy Hacking".

Discuss this post - leave a comment

The Basics

% dar -s 600M -c backup-home -R /home/user/foo
-- output stripped --
% ls
backup-home.1.dar backup-home.2.dar

Excluding Files/Directories

% dar -s 600M -c backup-home -R /home/user/foo -X ".thumb*" -P "media/audio" -P "public_html"

Compression

% dar -s 600M -m 256 -y -c backup-home -R /home/user/foo -X ".thumb*" -P "media/audio" -P "public_html" -Z "*.zip" -Z "*.gz" -Z "*.bz2" -Z "*.png"

Batch Operations

## dar config for backup-home

# affects all commands
all:
  # be verbose
  -v

# affects only -c (create archive)
create:
  # don't compress the following files
  -Z "*.gz" -Z "*.bz2" -Z "*.zip" -Z "*.png"
  # only compress files bigger than 256M
  -m 256
  # create slices of 600M size
  -s 600M
  # use bzip2 compression algorithm
  -y
  # directory to backup
  -R /home/user/foo
  # exclude directories
  -P media/audio
  -P public_html
  # exclude files
  -X "*thumb*"
  # run some commands once a slice was created
  -E "echo created slice %b.%n"

% dar -c backup-home -B /home/user/backupcfg

Differential Backups

% dar -c backup-home-diff -B /home/user/backupcfg -A backup-home
-- output stripped --
% ls
backup-home.1.dar backup-home.2.dar backup-home-diff.1.dar

% dar -A backup-home -C backup-home-catalogue
% ls
backup-home.1.dar backup-home.2.dar backup-home-catalogue.1.dar

% dar -c backup-home-diff -B /home/user/backupcfg -A backup-home-catalogue

Restoring Backups

% cd /home/chi/restore
% dar -x /home/chi/backups/backup-home
-- output stripped 
% dar -x /home/chi/backups/backup-home-diff
-- output stripped

This is part two of our look at using Python with the API of the Twitter social networking website. (Part one here)

I am assuming you are using a GNOME-based Unix/Linux system. If you use something else then sorry, you might want to read one of my other posts.

So in Twitter, you can have updates of what your friends are doing appear on the Twitter website (like this), however, it is more fun to make these updates come to you.

So I wrote a little program in Python that makes your Twitter updates pop-up on the desktop, like this:

This uses a feature of the Gnome desktop called 'notify', as explained by Andy W, and then followed up by me two posts ago.

Without further ado, lets get started.

1. Get python-notify

There are three very small but important dependencies. Firstly, you need the Python bindings for libnotify. These are available from your Linux distribution (you might even have them installed them already):

You will need setuptools as well. I am sure you already have that installed, but it cannot hurt to double-check.

The two computers I have here are Ubuntu and Gentoo:

On Ubuntu or Debian:

sudo apt-get install python-notify python-setuptools

On Gentoo:

sudo emerge notify-python setuptools

On your Linux/Unix, you can alter the commands slightly so it matches your package management system.

2. Get json and Twitter bindings

Now we need two more, simplejson and python-twitter

From any Linux Distribution, run:

sudo easy_install simplejson

sudo easy_install python-twitter

3. Twitter account

Next you will need to sign up to a Twitter account and add at least one friend (e.g. zeth0), in the spooky terminology of twitter, to add a friend you "follow them".

4. Get my script

Next you need to download a copy of my script and make the file extension .py. Get it from here, or type:

wget http://zeth.me.uk/python/twitnotify.txt -O twitnotify.py

chmod +x twitnotify.py

Now you need to edit the top of the script so that your username and password are correct. For historical reasons, I'm an Emacs man, but if you don't have a favourite editor, then gedit is pretty easy to use:

gedit twitnotify.py

5. Engage!

Now we are finally ready to run the script:

./twitnotify.py

If you have a lot of friends then there will be lots of updates the first time. The updates and pictures are all cached, so you don't have to download those again.

Six degrees of separation

After I made that script, I tried to be more adventurous and wrote a new script that found other people that shared your interests.

It would spider recursively through your friends, followers and their friends and followers, and their friends and followers, and their friends and followers (and so on), and returned a set of updates based on whatever keywords you specified.

It worked pretty well and the results were really fun. However, what I didn't know was that Twitter has quite sharp rate limits, so when this script stopped working I was combing through the code to find out what had gone wrong. It turned out that my code was fine but that my username had just been suspended from Twitter.

Twitter rate usage

Your username is allowed 70 requests per 60 sixty minute time period, starting from the first request.

Using my original twitnotify.py script posed no problem, the script usually uses three requests each time (one for normal updates, one for replies and one for personal "direct" messages).

So if you update every 10 minutes, then that will be about 18 requests an hour, easily under your 70 maximum.

Getting the photos of your friends is included in the above, but the first time you get a private message from a new person who is not a friend, it will take one request to download the photo.

My second script, the one to find new friends according to your interests, easily ate up 70 requests in one pass. So I had to give that up.

Fortunately, it seems that you are automatically un-suspended from Twitter after a couple of hours, and having your username temporally suspended from the API does not affect logging into the Twitter website.

Discuss this post - leave a comment