Planet Larry

Uninterruptible power supply & suspend2ram/suspend2disk

Using software raids on linux ‘mdadm‘ is problematic if you hit a ‘power outage’. Using a single device without a ’software raid’ is not a problem since it uses a journal – ext3++/xfs for instance are quite stable in this regard. This is probably also true when using LVM – the ‘logical volume manager’ (but don’t quote me there, my LVM ‘use case’ might differ from yours).

Using the same filesystems with a software raid makes the ‘journal’ useless, see [1] and [2]. Let’s quote [2]:

Power Loss and Ensuing Data Corruption

Many beginners think that they can test RAID by starting a disk-access intensive job, and then unplugging the power while it is running. This is usually guaranteed to cause some kind of data corruption, and RAID does nothing to prevent it or to recover the resulting lost data. This kind of data corruption/loss can be avoided by using a journaling file system, and/or a journaling database server (to avoid data loss in a running SQL server when the system goes down). In discussions of journaling, there are typically two types of protection that can be offered: journaled meta-data, and journaled (user’s) data. The term “meta-data” refers to the file name, the file owner, creation date, permissions, etc., whereas “data” is that actual contents of the file. By journaling the meta-data, a journaling file system can guarantee fast system boot times, by avoiding long integrity checks during boot. However, journaling the meta-data does not prevent the contents of the file from getting scrambled. Note that most journaling file systems journal only the meta-data, and not the data. (Ext3fs can be made to journal data, but at a tremendous performance loss). Note that databases have their own unique ways of guaranteeing data integrity in the face of power loss or system crash.

Normally one would use a ‘UPS’ to shut down the system properly. But looking at my two c2c intel archs and my 32bit intel laptop, I wonder how long this scenario will remain to be a ‘default’.

I’d expect suspend2ram or suspend2disk becoming a new way doing things better. Say the power outage is just 3minutes. Your suspend2ram takes 5secs, resume 7secs some users might not even notice your system was down.

But if the power outage was say 1-3hours and the system consumes 3W of power you would need no powerful system. Say you use suspend2disk by default you can completely shutdown the system. A battery would have to stand the peak power usage of 130W for about 7 secs (this is just a guess, say c2c processor, several network cards, 4 3,5″ harddrives (10W) each.

I have two low power gentoo fileservers, both need less than 35W and both use a different technology.

• VIA board with a VIA Nehemiah processor (tested using 2.6.24-tuxonice-r4)
• Intel board with Intel(R) Atom(TM) (untested)

The VIA board was already tested for support of suspend2ram/suspend2disk with great success(and I don’t mean standby). I did not test the Intel board, but I’d expect it to work as well. You might wonder why I tested suspend2disk in the first place – this was not to check for power outage capabilities! I usually have no screen attached to the computer and it is located in a very uncomfortable location. When something goes wrong – for example I shut down the network by fault and I don’t want to shut down the system I can suspend2disk this machine pressing the power button.  Next I pick that box up and put it on a desk and resume it with a screen attached. This works fine – trust me on that.

On the other hand I don’t use a ‘software raid‘ in these systems so ‘a power fail is not a problem at all‘. I just wanted to bring up the idea – maybe someone already uses this the way I proposed?!

[1] http://tldp.org/HOWTO/Software-RAID-0.4x-HOWTO-4.html

[2] http://linas.org/linux/raid.html

Godaddy sucks

I'm in the process of moving all my domains the heck off of Godaddy. I'm trying Namecheap which seems slightly less evil, if the sheer amount of ad banners and upselling bullcrap is any indication. But probably only slightly less evil.

Honestly Godaddy has so many ads I can't even find the button to renew my domains. The process of buying anything takes you through 6 or 7 pages of the most garish, fanatical sleaze-peddling that you are likely to encounter on a website.

Domain registrars are the used car salesmen of the internet.

How to use flex and bison with qmake (my own way)

qmake (sort of) supports lex/yacc, but modern people prefer to use flex/bison. If you google, you can find some tricks like convincing qmake that yacc is spelled ‘bison’ and so on.  Morever qmake supports for lex/yacc has several drawbacks, such as weird naming scheme, and (when using the tricks), dependencies are not building files in the right order.

Finally, I created my own *.pri files, and here they are:

% cat flex.pri

flex.name = Flex ${QMAKE_FILE_IN}

flex.input = FLEXSOURCES

flex.output = ${QMAKE_FILE_PATH}/${QMAKE_FILE_BASE}.lexer.cpp

flex.commands = flex -o ${QMAKE_FILE_PATH}/${QMAKE_FILE_BASE}.lexer.cpp ${QMAKE_FILE_IN}

flex.CONFIG += target_predeps

flex.variable_out = GENERATED_SOURCES

silent:flex.commands = @echo Lex ${QMAKE_FILE_IN} && $$flex.commands

QMAKE_EXTRA_COMPILERS += flex

% cat bison.pri

bison.name = Bison ${QMAKE_FILE_IN}

bison.input = BISONSOURCES

bison.output = ${QMAKE_FILE_PATH}/${QMAKE_FILE_BASE}.parser.cpp

bison.commands = bison -d -o ${QMAKE_FILE_PATH}/${QMAKE_FILE_BASE}.parser.cpp ${QMAKE_FILE_IN}

bison.CONFIG += target_predeps

bison.variable_out = GENERATED_SOURCES

silent:bison.commands = @echo Bison ${QMAKE_FILE_IN} && $$bison.commands

QMAKE_EXTRA_COMPILERS += bison

bison_header.input = BISONSOURCES

bison_header.output = ${QMAKE_FILE_PATH}/${QMAKE_FILE_BASE}.parser.hpp

bison_header.commands = bison -d -o ${QMAKE_FILE_PATH}/${QMAKE_FILE_BASE}.parser.cpp ${QMAKE_FILE_IN}

bison_header.CONFIG += target_predeps no_link

silent:bison_header.commands = @echo Bison ${QMAKE_FILE_IN} && $$bison.commands

QMAKE_EXTRA_COMPILERS += bison_header

Of course, the proper way to use those is to include them and then define FLEXSOURCES / BISONSOURCES as needed. Example:

include(flex.pri)

include(bison.pri)

FLEXSOURCES = myproject.l

BISONSOURCES = myproject.y

Kensington SlimType Keyboard (64365) Review

“Let your fingers do the walking”.

That old Yellow Pages slogan applies good to us that are on the keyboard often. We learn pretty quickly that a good tool makes all the difference. I learned this previously when I got a bargain keyboard that had broken keys originally, and later one had a shift key that quickly became very picky on how it was to be pressed, and then failed resume from sleep (which was a killer to debug). Since I am on the keyboard quite a bit, this was something I began to think about over and over.

The Kensington Slimtype keyboard has a laptop-like keyboard feel. There are two types of keyboards: low-profile, and those that have the high key presses. The best keyboard I use is on an old iBook G3 clamshell. It allows quick keypresses with a soft response that never bunch or conflict. When I saw Brians’ keyboard, I was a bit surprised people still used them. Seemed like a lot of extra work to press the key another half inch. I’m guessing a lot of this is about what a person is used to. People that use the Happy Hacking Keyboard have told me that they have a less likely chance of accidental keypresses using these keyboards. This isn’t true to me, but can accept that each has their own. If you like low profile keyboards, the SlimType does very well.

On the SlimType, the keypresses are nice and responsive. Not quite as low profile as on most laptops but actually the difference provides a good feel. The keys provide obvious click feedback and there is no doubt to when the key has been pressed. However, the bounce response a little too obvious. When I first started typing it felt like it was fighting me – the clicks a little too pronounced and bounces too springy. I found I had to run my hands over the keys a several hundred times (might be a bit of an exaggeration) and I got the feel I like. After I did this I found the typing to be fast and could closely replicate the speed I do on the iBook.

The gripping on the keys is subtle and useful. Kensington puts a nice texture on the keys that prevents slidding and it gives it a good tactile feel. This is my favorite feature on the keyboard but that perspective may be a bit skewed because I never had it before.

The keyboard layout makes the SlimType considerably narrower than a normal keyboard. This is very useful for desktops that have limited space. The reduced width mainly comes from how the keys are laid out, but the keys are a bit slimmer too (about a 1/16th of an inch from A to ;). This isn’t a big difference but I do find myself doing an awkward keypress on a rare occasion. The rest of the keyboard layout does take a bit of time to get used to. The numbers are layered a bit different and the function keys are slim. The Home, PgUp, PgDn are the most difficult to get used to and are on a single column right next the the Enter, Backspace… keys. And when I say right next to them, I mean it. There is no gap between them and I am still trying to get use to them after two weeks – old habits die hard :). If you were pretty loose about the space on the right side of the keyboard before (i.e. the gap between the Enter, Backspace.. and Home, PgUp…) you might find yourself accidentally pushing these keys from time to time. Also the arrow keys are taking some re-learning too being shifted to the left as well.

There are seven multi-media keys on the keyboard that cover the basic uses, unfortunately they don’t have the same quality as the other keys (they click awkwardly and and roll a little). Since they don’t get used often I don’t consider them a huge minus.

The keyboard profile is thin, not even a half-inch high. And allows those of us that rest our wrists on the desktops a lot more comfort. Not the proper position, I know :).

The footing grips very good and the keyboard is a bit heavier than other keyboards I’ve used that helps typing by keeping the keyboard in place.

Pluses

• Low profile keys, can type fast
• Textured keys
• Weight and footing
• 5 year warranty from a company that are good about their warranties
• Thin and narrow
• All the keys work in Linux and suspend! (ok, haven’t tried the Windows key)

Minuses

• Keypositions take some getting used to.
• Arrow keys to the left of what you usually expect.
• Chrome border – ugly and takes up extra space.

Conclusion

Absolutely great for normal typing after the spring had been taken out of them. If you like the laptop keyboard profile, you will very probably like this keyboard. It’s quick, responsive, and has a good tactile feel. For $30 I can’t really be too rough, without the Chrome border and no space between the Enter,Backspace… and Home, PgUp… it would have gotten a 4.5.

4/5 stars.

SpiderOak referral

I've already mentioned before that I chose SpiderOak as my backup utility and provider.

The problem is that although I would be prepared to pay for the package, I can't because I don't have a credit card. This means that I only get the complementary 2 GiB and the possibility to upgrade to max. 5 GiB by friend referrals.

I normally wouldn't do this, but with my laptop dead and my whole data residing on an aging external disk, those three extra gibibytes would mean I can at least have all my documents and the whole inbox backed up.

So, if you have already thought of trying out SpiderOak and would like to do me a favour, please use my referral link when downloading SpiderOak.

hook out >> with terror wathcing the progress bar in SpiderOak slowly, but surely crawling to the end of the free space
<!--break-->

openresolv imported into NetBSD

 openresolv has been imported into NetBSD, which allows more than one daemon to update /etc/resolv.conf sanely and configure local nameservers for enhanced DNS, especially if running on a VPN.  dhcpcd already uses resolvconf when available and dhclient in NetBSD has been patched to use it.

This is important for NetBSD, as many packages support resolvconf, but only when /sbin/resolvconf exists. This meant that a lot of packages that supported resolvconf, failed to work with any resolvconf implementation from pkgsrc.

PPP users who maintain their own scripts are encouraged to try it out :)

linux love, gentoo love!

recently i updated my gentoo box. the computer is a c2c processor with a fast nvidia graphics card. but what is cool about that “you” wonder?

gentoo

actually the new portage kde 4.3.3 seems to be very stable (read usable) for me. the proprietary graphics driver nvidia.ko is doing great! i use suspend2ram about 10 times a day. i leave the pc in suspend2ram for days and it only needs 3W of power! suspend2ram needs 2-3 secs for going down and about 7 secs to come back up! plasma is now at a point that i see it is going to be useful. the expose clone of the kde folks is very very nice to use. dragon player seems to get stable so i can use it with the smb:// kio plugin to watch films from my server. firefox64 selfcompiled now uses a 64bit flash plugin. and finally nearly all applications i’m using regularly work like a charm, which include:

• firefox (even watching flash films, fullscreen still stutters but then usually once can do: mplayer -fs /tmp/FlashAVE3 (name is somehow similar)
• very fast 3d support (nvidia i love you!) in kde and in fullscreen games as well as normal 2d acceleration for movies
• that said, the new kde desktop effects are the best user experience i’ve had for years now (i didn’t test vista/win7) but mac os x
• sound seems to be working now. to be honest i don’t understand why (didn’t test skype tough)
• the game ’spring’ runs with 280fps on my computer (that is a peak value of course) but then with medium details on a map with 5on5 (10 kaik ais) it still seems to be no problem to get decent 60fps!

to conclude: i’m very happy especially since this suspend2ram thing now works. both pcs (i just cloned the gentoo image from the first pc) did work with the same settings altough both use different hardware.

that is really really cool!

currently i’m looking forward getting:

• into kdevelop 3.9.95
• akonadi: using a backend !google for pim (i don’t have a google calender nor do i use google mail)
• getting rid of my laptop with this awful radeon r300 card, i’ll probably buy a “hp elite” with a nvidia card and a 6mb L2 cache (c2c cpu)

to all gentoo readers out there: i love this distribution so much since portage is doing well and the update (although taking long) was a great success.

UPDATE: i forgot to feature the graphics card i use: it is a ‘Gigabyte GeForce 9600 GT Passiv 512MB for 83,90 euro’ (20 nov 2009) which is as silent as you can imagine: 0 dB

The making of Perspective

Perspective is my school’s magazine of which I’ve been the designer last year. Though the old team has moved on and I’m no longer associated much with it, such releases have remained as an archive online. Mainly because I like the open-source ideology I’ve produced the magazine primarily using FOSS tools – such practice is quite rare in the publishing industry due to Adobe’s dominance. However -especially noticeable in the latest release- the quality of results have truly been quite astounding. So of course I was pleased to recently receive an email about it:

Hi Dion,

I was blown away by the layout you did for Perspective using scribus, and I was wondering if you could offer a COMPLETE newbie to scribus some tips on how to layout a simple greeting card. I would really appreciate your help.

Everybody loves seeing their work being appreciated, and so I crafted a reply, and whether or not it helps anybody else, I thought it’d be good to share.

Hello,

Why thank you! I assume you’re talking about the July 09 edition, where I took the time and effort to actually design each page instead of simply lump text and boxes around.

The first step was to use Scribus to split the page into a grid using the rulers. As a magazine, I split the page into three, then created more rulers to allow for padding. I also put rulers at each of the borders to give ample space for page margins. Once this was done I now have a clear idea of what space I have allocated for my content.

I have to admit that the main crux of the design process was done in the GIMP. I would export a blank canvas and then mirror the rulers with GIMP’s rulers to make it so that my designs also were within those borders. The entire page layout was created in GIMP – I would guess how much space an article might take, and then create it in GIMP. Putting the entire article’s text in The GIMP along the side of one column allowed me to get a pretty good guess at how much space it would take – a screenshot showing this can be seen here:

If you look at that screenshot you’d also notice that all of the pictures – eg: the page template as well as the pictures for the article and fancy font used for the title – were done on The GIMP and NOT on Scribus. The problem I found was that because printer’s were unreliable, when given to them in a format that separated text and many images being overlayed above one another, the colours would run and many artifacts would be seen. So what I did was to create all non-text items in one single image, then export that and use it as a background for the page in Scribus. This keeps all the images together and greatly reduces the number of artifacts – it worked so well, in fact that I would be confident to say that there were almost no artifacts at all!

At the same time, when dealing with such a potentially complex document this technique would keep Scribus running extremely fast yet still benefit from a well designed and with pages full-of-effects (eg: shadow, interesting overlays and rotations). Of course this also meant I could use cool fonts for titles and not be worried on whether or not the printer also had those fonts! Another benefit is that it keeps the final file small and extremely fast to export. Less chances of crashes and easy portability!

The final step once all the non-text were finished in The GIMP was to fire up Scribus and put text on top- so on each page, there was only one image, no more, no less. This also made it a lot easier to manage and edit text. I could still make text flow around images by inserting invisible shapes then selecting Shape -> Text flows around image in the properties window (F2). With line and bezier shapes this really gave me the precise control I needed.

The title page was made almost last – I find this helps as firstly you don’t sit around wasting time thinking of a design because you’re doing the easier and repetitive inner pages. Over time you gather a feel of the document’s style and creating the title just flows out.

On this issue I chose a “small” text font size. This does actually make it seem more professional as you really notice the flow around images. Previous issues I refrained from doing this as my editor told me it would strain people’s eyes. Turns out that they were wrong (well, nobody complained).

Release early, release often – share your work’s in progress with your friends. They’ll help point out things that don’t look nice. If at anytime you’re unsure of how to layout a page, stop what you’re doing on the computer, print it out, and start throwing down ideas with a pen(cil).

So in short – basic layout started in Scribus, design and playing around with ideas done in GIMP, inserting and arranging text done in Scribus, and that’s it. I hope that helped – if you’re unsure on the actual design, searching google images and especially design blogs such as SmashingMagazine can really give you some inspiration.

In other news, things are starting to calm down a bit so within a week or so I should be able to continue development on WIPUP and perhaps even do a few small creative projects.

Related posts:

1. Perspective July 2009 Released
2. Perspective Failure
3. Perspective in progress

Changes in Microsoft Windows EULA

Making a Django form field using aggregated values

So another Django post for those who like that sort of thing. In this post we will dynamically create the choices argument for the forms.ChoiceField.

Often you will be making a search form where the objects to be searched have common but dynamic field values. The Django admin app does this kind of thing when you use list_filter, but I haven't yet found an official function in the API for doing this (but Django adds new functions all the time).

To take a simple hypothetical example of a web application for a pet shop, you have a model 'Fish' which has a field called 'species' which is just a normal CharField that the staff enter when they get new stock.

Here is my made up Fish model:

from django.db import models
class Fish(models.Model):
    name = models.CharField(max_length=255)
    species = models.CharField(max_length=255)
    price = models.DecimalField(max_digits=7,
                                decimal_places=2,
                                null = True)
    sold = models.BooleanField(default = False)
    record_created = models.DateTimeField(auto_now_add = True)
    record_updated = models.DateTimeField(auto_now = True)

You want to allow the customers to search fish by species.

How do you do this?

Well the most obvious yet drastic solution would be to completly change the workflow to create a species model that the staff fill in every time they add a new kind of fish. Then you can make the form as follows:

class FishSearchForm(forms.Form):
    species = forms.ModelChoiceField(Species.objects.all())

This is not a complete solution however, as your form will have every species name the shop has ever stocked, not necessarily what they have now. Allowing the customer to search for a south american aquarama when you cannot get them anymore is not giving a good experience, it is just raising expectations and then dashing them straight away. So then you have to make a backwards lookup from the species to the fish and filter out the species that are not in stock and so on.

Instead, the way I have been solving this problem is by dynamically creating the 'choices' field based on what is in the database:

class FishSearchForm(forms.Form):
    species = forms.ChoiceField(choices = valuechoices(Fish, 'species'))

My valuechoices function is below:

def valuechoices(model_name, field_name):
    """Return a choices list based on aggregated values of a model.
    Like the list_filter in the admin.
    (For unaggregated objects, just use a ModelChoiceField)

    For example:
    class FishSearchForm(forms.Form):
        species = forms.ChoiceField(choices = valuechoices(Fish, 'species'))

    """
    from django.template.defaultfilters import slugify
    choice_list = model_name.objects.values_list(field_name).distinct()
    choices = [(slugify(choice[0]), choice[0]) for choice in choice_list]
    choices.sort()
    return choices

It is easy to add a none value, e.g. for advanced forms where there are lots of optional ways to search:

species_choices = valuechoices(Fish, 'species')
species_choices.insert(0, (None, u'All'))
class FishSearchForm(forms.Form):
    species = forms.ChoiceField(choices = species_choices)

Be careful when setting an initially selected value, since obviously the initial value needs to exist in the database for it to work.

if you did set the initial value to a something that is not in 'choices', the current version of Django would probably let you off and display the form anyway, but it is best not to rely on undocumented behaviour. It is quite easy just to check whether the initial value is in the database:

initial_fish_species = 'koi' if Fish.objects.filter(species = 'koi').count() else None
class FishSearchForm(forms.Form):
    species = forms.ChoiceField(choices = valuechoices(Fish, 'species'),
                                        initial = initial_fish_species)

Happy hacking and all that jazz.

Command Line Dictionary Update

I do alot of work from the terminal so I like to have a dictionary available there. Previously if you read about my command line dictionary post you know that I used some html parsing with curl to do this, lately though I’ve learned about Sdcv and it’s absolutely great. Since I’m not always on the internet this option works alot better with the same results. I also added a thesaurus too. You can find it here.

They’re Back! Amarok Finally Reimplements Its Killer Feature!

https://bugs.kde.org/show_bug.cgi?id=168781

--- Comment #13 from Daniel Dewald <TheCrasher gmx de>  2009-11-18 02:56:25 ---
Labels support was added and is available in the GIT Version. First stable
release with label support will be 2.2.2.

Woot!

Nostalgiarama

On a whim today, I decided to dust off a bit of my Jpop (Ayumi Hamasaki, to be precise) and give it a listen. It is amazing what memories are invoked from just the first few tracks off of LOVEppears, which I first started listening to in 2000 — my first total immersion dives into computing (nay, computer appreciation), fleeting loves, the Milwaukee School of Engineering, Gentoo Linux, compiling Gentoo Linux, breaking all of my computers at least once.

Lonely nights in a warm bedroom, surrounded by computers, Ayu blasting through XMMS, heralding my first great interest shift. The daily commutes to MSOE punctuated with long sessions before and after class in the student center, tasting for the first time the sweet nectar of broadband, filling my belly with Japanese culture. My last serious interest in anime, my first serious appreciation for the potpourri of the Internet.

The whiteness of an Ayu wallpaper. My rust bucket truck. The couches in the Cudahy Student Center. The cheeseburgers they served there. Windows file sharing clients, their names long lost, from an age before the luxury of BitTorrent. A certain kind of orange present in my Trillian IM client. Shit, Trillian itself. Conversations never saved, never remembered, no longer the highlight of a lazy afternoon between classes. People not seen in years. Quiet conversations outside, in the cold, revisiting the high school roof.

All from Ayu’s “Trauma". If I had one hope, it was that I never lost these memories. If I had one fear, it would be losing the ability of music to invoke them.

Portage 2.2 and License Management

Portage 2.2 has recently introduced a new feature that allows users to select what licenses they want to allow on their install. The interface still needs some work (for example, Portage needs to explain when it has “ignored” packages due to license during updates) so that users don’t, for example, get confused as to why their system suddenly wants icedtea instead of sun-jdk. Other than these fairly minor issues, it seems to be working well so far.

There’s an article on Gentoo Wiki that explains this new feature from a users point of view and how to tailor it to your individual tastes.

msmtp queueing for offline use

When you start using a small and slow laptop over an unreliable GPRS connection (say while travelling in an ICE at 300km/h) you start noticing certain things that are missing from your current shell-setup. One of these problems manifests itself when you want to send the mail you just typed in mutt. Fortunately there is an easy and elegant fix (if you use mutt with msmtp): msmtpqueue. This gives you three little scripts to enqueue, list and send messages which would otherwise have gone directly into msmtp. Just put them somewhere and then add this to your .muttrc:set sendmail="$HOME/local/bin/msmtp-enqueue -i"
macro index \Cy "!$HOME/local/bin/msmtp-runqueue<enter>"</enter>Now “sending” mail from mutt happens instantly and you can manually push the batch of mails out to your smtp once the train has stopped in a station and your reception is stable ;). msmtp-listqueue shows you the queued mails while msmtp-runqueue mails them (using msmtp of course!). For the downstream-direction of offline mutt-usage there’s offlineimap, but I feel like this deserves an extra post ;)

Get adblocking back for archivum.info

If you have adblock enabled and you try to visit any url of www.archivum.info you will get a really nasty alert saying:

You Are Using Adblock Plus or some other advert blocking software! Archivum.info relies on advertising for revenue. Please add www.archivum.info to your ad blocking whitelist or disable ad blocking when you visit www.archivum.info.

When I first saw this I laughed…and then I tried to find a way to bypass it.
I used curl to see the sites html code:

<code2>$ curl -v www.archivum.info
curl -v www.archivum.info 
* About to connect() to www.archivum.info port 80 (#0)
*   Trying 69.147.224.162... connected
* Connected to www.archivum.info (69.147.224.162) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15 libssh2/1.2
> Host: www.archivum.info
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 17 Nov 2009 11:24:22 GMT
< Server: Apache
< Last-Modified: Mon, 16 Nov 2009 08:41:17 GMT
< Accept-Ranges: bytes
< Content-Length: 9392
< Vary: Accept-Encoding
< Content-Type: text/html
< 
<html>
<head>
<title>archivum.info - The Internet archive.</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<script type="text/javascript">var disabled = false;</script><script type="text/javascript" src="http://www.archivum.info/js/adblocker_probe.js?
site=http://googlead.foobar.tld/"></script><script type="text/javascript">if (disabled == false) { location.replace("http://www.archivum.info/denied");
alert("You Are Using Adblock Plus or some other advert blocking software! Archivum.info relies on advertising
for revenue. Please add www.archivum.info to your ad blocking whitelist or disable ad blocking when you visit
www.archivum.info.");}</script></head>

[snip]</code2>

Here’s how this site blocks Adblockplus: there’s a variable called disabled set to “false” then if a js (http://www.archivum.info/js/adblocker_probe.js) runs it sets disabled to “true” . The hint is that adblockplus blocks urls starting with “googlead.” so it won’t visit “http://www.archivum.info/js/adblocker_probe.js?site=http://googlead.foobar.tld/” and the variable will remain “false“. Then the alert pops up.

The solution is very simple, just add an exception to your local AdblockPlus rules, AdblockPlus Preferences -> Add Filter:
@@|http://www.archivum.info/js/adblocker_probe.js?site=http://googlead.foobar.tld/

So firefox, visits the js url, disabled becomes “true” you are allowed to continue browsing the site and AdblockPlus continues blocking all blockable items.

Countries in Django

Django is my preferred web framework. If you don't know what I am talking about, then you can read my previous posts, e.g. here and here and so on.

Anyway, for the people who do like Django, it comes with a nice set of local field types for inputting post codes and zip codes in different countries as well as Peruvian national identity numbers, Australian territories, Romanian Bank Account Numbers, Icelandic phone numbers and lots of other stuff. This saves you the hassle of implementing fields for these types of data yourself.

Somewhat bizarrely however, Django does not currently contain a field for what country the person or thing is in, which is often a very basic yet important piece of data to collect.

Fortunately, people have started work on this and on a LanguageField, however the patches have been languishing on the bug tracker for some time.

I wanted the CountryField now, so I copied and pasted from the patches and put it in a module, this is on my code page (direct link is here).

It is pretty obvious how you use it, import the field into your models file:

from whatever.countries import CountryField

Then you can add it to a model:

class Whatever(models.Model):
    country_based_in = CountryField(blank = True)

Here the field is shown in the automatically generated admin:

It seems to have all of the countries that one is ever likely to encounter on the web. The module lists countries alphabetically, however lots of online forms put the United States and perhaps the United Kingdom and other large Internet using countries at the top. If you want to do this, then just arrange the tuple in the order that you want, and delete everything from line 276 onwards (i.e. def sorted_countries etc)

This module is beautifully simple, yet it probably saved me 30 to 60 minutes of fiddling about. This is the joy of programming in a friendly open source community, someone has already done the hard work for you already, you just need to dig it out of some bug tracker and adapt it to your needs.

Happy hacking and all that jazz.

Fluxbox, we meet again

I'm sort of tired of KDE4 crashing left and right and Plasma barfing all over me all day. So I decided to check out the current state of lightweight window managers.

Lo and behold, Fluxbox is still going strong. It was the first WM I used way back in 2000-something when I started using Linux full-time. Last time I tried, there were always weird compatibility problems with system tray icons and pagers working properly when running a mix of KDE and Gnome and other apps, but those seem to have cleared up nicely; I have yet to hit any snags. Here's a screenshot.

This took very minimal effort to install and set up. Maybe a couple hours total. I'm using ipager and conky. The wallpaper comes from the UniQ KDE theme. Vim and Emacs themes are my own.

The Fluxbox style is mydefcon_4 from tenr.de which is probably the largest and most thorough set of themes created by one person that I've witnessed. That fellow is motivated.

For all the bells and whistles of KDE4, what features did I actually use regularly?

1. A menu of apps
2. Taskbar + System tray + Clock
3. KWin's good window management.
4. Global keyboard shortcuts galore
5. One widget: current CPU/RAM/Network usage
6. Mouse/keyboard management, background-setting, etc.

Fluxbox gives me all but number 5, and Conky gives me that. Number 6 you can do with xset and feh and such.

And I like being motivated to use keyboard shortcuts for more things. I'm already halfway there. Maybe I can take the plunge eventually and try a tiling window manager. Not sure I've reached that level of nerditude yet though.

And now I can move and resize windows without my graphics card bursting into flames. Maybe when I can afford a few more cores worth of CPU I'll try KDE4 again. Honestly I think I have too much monitor real-estate for my ancient computer to handle smoothly in KDE4.

Not to knock KDE4; it's awesome and I'll probably go back someday. But everyone needs a break now and then.

This Week in Python Stupidity: os.stat, os.utime and Sub-Second Timestamps

The primary design principle behind the Python programming language is to take everything that’s horrible and wrong with Perl and get it horrible and wrong in a completely different and even more hideous way. Today, however, we shall be looking at a particularly egregious case of stupidity the likes of which not even PHP has managed to replicate.

On Unix, timestamps have traditionally been held as an integer number of seconds since the epoch. The modification time for a file is one place such a timestamp has been used. Two groups of system calls are of interest to us here.

First, stat (and its fstat and lstat variants). The stat system call places information about a file into a struct also named stat (which is possible thanks to a lesser case of brain damage in C’s design). To get the mtime of a file, historically we would have used the st_mtime field, which is of type time_t, which is an integer of some kind:

#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char * argv[])
{
    struct stat s;
    if (-1 == stat("timmy", &s))
        return EXIT_FAILURE;

    printf("stat.st_mtime for timmy is %ld\n", s.st_mtime);
    return EXIT_SUCCESS;
}

Sometimes we might want to modify a file, but not affect its mtime. Thus, we need a way to set a file’s mtime to a given value, and to do this we would historically have used a function from the utime family:

#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdlib.h>
#include <utime.h>
#include <fcntl.h>

int main(int argc, char * argv[])
{
    struct stat s;
    if (-1 == stat("timmy", &s))
        return EXIT_FAILURE;

    int fd;
    fd = open("timmy", O_WRONLY, O_TRUNC | O_CREAT);
    if (-1 == fd)
        return EXIT_FAILURE;
    if (0 != close(fd))
        return EXIT_FAILURE;

    struct utimbuf times = { .actime = s.st_atime, .modtime = s.st_mtime };
    if (-1 == utime("timmy", &times))
        return EXIT_FAILURE;

    return EXIT_SUCCESS;
}

(Sidenote: the above almost certainly should be using fstat and futimes instead to avoid race conditions, but this is irrelevant for our examples.)

But all of this operates only on a second-precision basis. For many applications this is no longer sufficient. Fortunately, some kernels and filesystems now support nanosecond-resolution timestamps.

First, for the stat family: rather than using st_mtime, we now use st_mtim, which is a struct timespec:

#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char * argv[])
{
    struct stat s;
    if (-1 == stat("timmy", &s))
        return EXIT_FAILURE;

    printf("stat.st_mtim for timmy is %lds %ldns\n",
            s.st_mtim.tv_sec, s.st_mtim.tv_nsec);
    return EXIT_SUCCESS;
}

And if our filesystem supports it, we get something like:

$ ./mtimens
stat.st_mtim for timmy is 1258321672s 173919603ns

As we can see, running our old utime-using code preserves the seconds but not the nanoseconds:

$ touch timmy
$ ./mtimens
stat.st_mtim for timmy is 1258321978s 62671870ns
$ ./utime
$ ./mtimens
stat.st_mtim for timmy is 1258321978s 0ns

To modify preserving nanoseconds, we use either utimensat or futimens:

#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdlib.h>
#include <utime.h>
#include <fcntl.h>

int main(int argc, char * argv[])
{
    struct stat s;
    if (-1 == stat("timmy", &s))
        return EXIT_FAILURE;

    int fd;
    fd = open("timmy", O_WRONLY, O_TRUNC | O_CREAT);
    if (-1 == fd)
        return EXIT_FAILURE;
    if (0 != close(fd))
        return EXIT_FAILURE;

    struct timespec times[2] = { s.st_atim, s.st_mtim };
    if (-1 == utimensat(AT_FDCWD, "timmy", times, 0))
        return EXIT_FAILURE;

    return EXIT_SUCCESS;
}

And now it works as expected:

$ touch timmy
$ ./mtimens
stat.st_mtim for timmy is 1258322326s 852774523ns
$ ./utimens
$ ./mtimens
stat.st_mtim for timmy is 1258322326s 852774523ns

Incidentally, POSIX.1-2008 considers the non-nanosecond-resolution functions and members to be deprecated, although since the nanosecond resolution functions aren’t universally available yet, a certain amount of autovoodoo is generally required…

Now we shall look at some Python. First, the old way:

import os

s = os.stat("timmy")

f = open("timmy", "w+")
f.close()

os.utime("timmy", (s.st_atime, s.st_mtime))

Now, to see if we can guess how the new way works:

>>> import os
>>> os.stat("timmy").st_mtim
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'posix.stat_result' object has no attribute 'st_mtim'

Mmm, nope. Time to consult the documentation. Nothing under stat, but there’s something interesting called stat_float_times:

stat_float_times([newvalue])

Determine whether stat_result represents time stamps as float objects. If newvalue is True, future calls to stat() return floats, if it is False, future calls return ints. If newvalue is omitted, return the current setting.

Uh oh. This can’t be good. Let’s look more closely at what happens when we run our code that uses stat.st_mtime and os.utime:

$ touch timmy
$ ./mtimens
stat.st_mtim for timmy is 1258324320s 762942258ns
$ python utime.py
$ ./mtimens
stat.st_mtim for timmy is 1258324320s 762942000ns
$ ./utimens 12345678901 111111111
$ ./mtimens
stat.st_mtim for timmy is 12345678901s 111111111ns
$ python utime.py
$ ./mtimens
stat.st_mtim for timmy is 12345678901s 111110000ns

What’s that, Lassie? Timmy has lost several significant digits of its sub-second mtime? Oh noes!

Yup, that’s right, Python’s underlying type for floats is an IEEE 754 double, which is only good for about sixteen decimal digits. With ten digits before the decimal point, that leaves six for sub-second resolutions, which is three short of the range required to preserve POSIX nanosecond-resolution timestamps. With dates after the year 2300 or so, that leaves only five accurate digits, which isn’t even enough to deal with microseconds correctly. Brilliant.

Posted in python Tagged: python

Migrated to Sabayon, in need of a EvroKorpus plasmoid

What I would give for a EvroTerm/EvroKorpus plasmoid right now!!! XD

Also, the stability of the Jamendo plugin in Amarok leaves a lot to be desired.

Sabayon is cool (for a temporary replacement for Gentoo).

hook out >> legal counciling, translating and sipping tea...
<!--break-->

links for 2009-11-15

• briancarper.net :: I'm turning into a Lisp snob
  Brain Carper looks at Google's new Go language from an entrenched Lisp/Clojure perspective and is unimpressed
  (tags: Go, Clojure)
• I think (Algol) is richer, clearer and more expressive (than Go)
  A lua-users mailing list post that compares Go to Algol 68. Hilarious, interesting, and a little depressing
  (tags: Go, Algol)
• Lennart Poettering's 5cents on the OOM Situation
  Lennart Poettering explains how Out-Of-Memory errors should not be detected by NULL returns from malloc but from SIGKILL from the OS, or basically why it's OK to stop checking for NULL's from malloc :)
  (tags: OOM, malloc, Memory Management)

A lovely little emoticon set from KDE.

I am of the opinion that the default emoticons on KDE are disgusting. Over time I’ve grown accustomed to them but lately I decided to do something once and for all. My solution was in the KDE forums – which had a different iconset (shouldn’t they be the same?) Sure, they didn’t have as many emoticons but I’m not an avid user. Anyways, here is a picture for those unaquainted with those iconsets:

… and of course you can download it here. KDE users can install it from System Settings, or even through Get Hot New Stuff. Users on other operating systems, it’s just a regular .tar.gz file so uncompress it and get the pictures inside and import it your own way.

Related posts:

1. What is FTP?

It's Been Awhile...

Installing Go in Ubuntu Jaunty

Processor: x86
Operating System: Ubuntu Linux 9.04 Jaunty Jackalope

1. CONFIGURE THE ENVIRONMENT
Set the necessary variables in your .bashrc that is if you are using bash. Assuming you have a 386-descendant processor and you want to keep the installation clean in an external disk (/media/disk).

# Google Go Programming Language Settings
export GOROOT=/media/disk/go
export GOARCH=386
export GOOS=linux
export GOBIN=/media/disk/go/bin

2. INSTALL MERCURIAL.
Easy.

$ sudo easy_install mercurial

Install process,

Searching for mercurial
Reading http://pypi.python.org/simple/mercurial/
Reading http://www.selenic.com/mercurial
Best match: mercurial 1.3.1
Downloading http://mercurial.selenic.com/release/mercurial-1.3.1.tar.gz
Processing mercurial-1.3.1.tar.gz
Running mercurial-1.3.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-a3YaRd/mercurial-1.3.1/egg-dist-tmp-VyNqd2
zip_safe flag not set; analyzing archive contents...
mercurial.lsprof: module references __file__
mercurial.templater: module references __file__
mercurial.extensions: module references __file__
mercurial.i18n: module references __file__
Adding mercurial 1.3.1 to easy-install.pth file
Installing hg script to /usr/local/bin
	
Installed /usr/local/lib/python2.6/dist-packages/mercurial-1.3.1-py2.6-linux-i686.egg
Processing dependencies for mercurial
Finished processing dependencies for mercurial

3. CHECKOUT GO USING MERCURIAL

$ hg clone -r release https://go.googlecode.com/hg/ $GOROOT

Checkout process,

requesting all changes
adding changesets
adding manifests
adding file changes
added 4016 changesets with 16888 changes to 2931 files
updating working directory
1640 files updated, 0 files merged, 0 files removed, 0 files unresolved

4. BUILD GO

$ cd $GOROOT/src
$ ./all.bash

Build process,

--- cd pkg/exp/ogle
rm -rf *.[568vqo] *.a [568vq].out *.cgo[12].go *.cgo[34].c *.so _obj _test _testmain.go ogle
8g -o _go_.8 abort.go arch.go cmd.go event.go frame.go goroutine.go rruntime.go rtype.go rvalue.go process.go vars.go
rm -f _obj/exp/ogle.a
gopack grc _obj/exp/ogle.a _go_.8
8g -I_obj main.go
8l -L_obj -o ogle main.8
	
real	0m2.264s
user	0m1.496s
sys	0m0.136s
	
--- cd ../doc/progs
	
real	0m4.430s
user	0m3.144s
sys	0m0.444s
	
--- cd ../test/bench
fasta
reverse-complement
nbody
binary-tree
binary-tree-freelist
fannkuch
regex-dna
spectral-norm
k-nucleotide
mandelbrot
meteor-contest
pidigits
threadring
chameneosredux

It should end with the following lines,

--- cd ../test
0 known bugs; 0 unexpected bugs

5. ENJOY!

Aspire One / X.org screen blanking

When using X on my Acer Aspire One (110L) and playing videos with mplayer, X would occasionally blank the screen after a few minutes without any way out but to suspend/resume to get your X back. This didn’t only happen with mplayer, and it didn’t have anything to do with xscreensaver.Anyway, after some digging I found that adding Option "FramebufferCompression" "off"to the Device-section of my xorg.conf resolved the problem.
Another cool thing I found out while going through ssh manpages is that you can silence the motd on login simply by touching ~/.hushlogin.