Posts for Wednesday, July 15, 2015

The Web We Should Transcend

There’s a very popular article being shared right now titled “The Web We Have To Save” written by Hossein Derakhshan who was incarcerated for 6 years by the Iranian Government for his writing, his blog. His well-written and beautifully illustrated article is a warning, a call to arms for each and every Internet user to bring back that old web. The web before the “Stream”, before social media, images and videos. The web in its best form according to Derakhshan. But I fear it’s not my web.

Derakhshan’s article rubs me the wrong way for a bunch of different reasons which I’ll try to outline later but maybe this sentence sums it up best. He writes:

Everybody I linked to would face a sudden and serious jump in traffic: I could empower or embarrass anyone I wanted.

Maybe it’s just a clunky way to phrase things, maybe it’s not his intention to sound like one of the people that communities and platforms such as Reddit suffer from. But he does. I’m all for using one’s standing and position to empower others, especially those with maybe less access, people with less socioeconomic standing, people who usually do not get a voice to speak to the public and be heard. But considering the opportunity to embarrass anyone at whim one of the glorious things about the Internet is seriously tone deaf given the amount of aggressive threats and harassment against a multitude of people.

It’s that libertarian, that sadly very male understanding of the web: The Wild West of ideas and trolls and technology where people shoot not with a revolver but duel each other with words and SWAT teams send each other’s way. A world where the old gatekeepers and authorities made of flesh and iron have been replaced by new gatekeepers and authorities sitting in their own homes while trying to become just as powerful as the old entities of power.

Because that’s his actual complaint.

People used to carefully read my posts and leave lots of relevant comments, and even many of those who strongly disagreed with me still came to read. Other blogs linked to mine to discuss what I was saying. I felt like a king.

It is about power and reach. About being heard within the small elite of people who did invest time and money to access the Internet. To create reputation and relevance in that community. In a way to “win”. But when you obey that very market/capitalist mindset, you can’t plateau, you need to grow. You need to become a gatekeeper.

Those days, I used to keep a list of all blogs in Persian and, for a while, I was the first person any new blogger in Iran would contact, so they could get on the list. That’s why they called me “the blogfather” in my mid-twenties — it was a silly nickname, but at least it hinted at how much I cared.

I remember the time of these lists of links. Carefully curated by individuals with power in small or sometimes bigger communities. Where Derakhshan very vocally complains about algorithms forming and directing attention, he or people like him doing the same is the utopia we lost.

If you print the article and hold it to your ear you can hear a silent reading of the Declaration of the Independence of Cyberspace. The so called home of the mind where it was supposed to be all about ideas and text and high-level philosophical debate. Again a starkly male point of view. But the web got richer. Video is important these days and many creative people experiment with how to not just put forward ideas there but represent themselves and their lives. Instagram is huge and awesome because it (or platforms like it) can depict life in forms that were hidden in the past. Our public common consciousness gets richer by every How to do your Nails tutorial, every kid’s pictures from him or her hanging out with their friends. But that’s not “right” we learn:

Nearly every social network now treats a link as just the same as it treats any other object — the same as a photo, or a piece of text — instead of seeing at as a way to make that text richer.

Yes links are treated as object people share and comment on just as videos, pictures, podcasts, soundtracks, GIFs and whatever. Why should that one way of representing culture be the best one. The one we should prefer before everything else. Is it because that’s the way men used to communicate before kids and women entered sometimes not liking that made up world of supposedly objective and true words and links?

Hm. So what is the world today like online? Channeling the recently deceased German publicist Frank Schirrmacher Derakhshan takes all his old-man-yelling-at-clouds-anger about the modern world and hides it behind a mostly undefined term, in this case: The Stream.

The Stream now dominates the way people receive information on the web. Fewer users are directly checking dedicated webpages, instead getting fed by a never-ending flow of information that’s picked for them by complex –and secretive — algorithms.

Obviously not every stream is algorithmicly filtered: Twitter and Instagram for example just show you the stuff of the people you manually chose to follow. But let’s not nitpick here. The complaint is that we are no longer doing it right. We are not reading things or liking them for their imminent value, their objective content, but:

In many apps, the votes we cast — the likes, the plusses, the stars, the hearts — are actually more related to cute avatars and celebrity status than to the substance of what’s posted. A most brilliant paragraph by some ordinary-looking person can be left outside the Stream, while the silly ramblings of a celebrity gain instant Internet presence.

But is the Internet that’s not showing me cute cat GIFs any better? What kind of weird chauvinism towards any culture he doesn’t like or understand or value is that? My web has twerking videos, cat GIFs, philosophical essays and funny rants as well as whatever the hell people like, whatever makes them happy to make or read or view or share (unless it harasses people then GTFO with your crap).


There is some meat to what Derakhshan writes and I encourage you to read his essay. He makes some valid points about centralization in the way certain Platforms like Facebook try to AOLify the Internet, how certain platforms and services use data about their readers or consumers unfairly and unethically and how it’s getting harder and harder to port data from one platform to another. He has some very very good arguments. But sadly he doesn’t get to the point and instead focuses on attacking things that “kids these days” do that do not support his understanding and vision of the web as a bunch of small kingdoms of digital quasi-warlords.

Dear Hossein Derakhshan.
Thanks for your article, I did enjoy reading it even if I don't agree with everything. The club of people who don't enjoy how our mostly unleashed and rabid form of capitalism ruins everything from human relationships to technology to funny cat videos meets every day down at the docks in the old factory building with the huge red star painted at its wall. Hope to see you there, soon.

The Web Derakhshan wants to save is not the digital Garden of Eden. It sounds like what Ayn Rand and some technocrats would come up with for a western movie. And the days of John Wayne are past. Get over it.

Title Image by sciencefreak (Pixabay)

Flattr this!


Loading CIL modules directly

In a previous post I used the secilc binary to load an additional test policy. Little did I know (and that’s actually embarrassing because it was one of the things I complained about) that you can just use the CIL policy as modules directly.

With this I mean that a CIL policy as mentioned in the previous post can be loaded like a prebuilt .pp module:

~# semodule -i test.cil
~# semodule -l | grep test

That’s all that is to it. Loading the module resulted in the test port to be immediately declared and available:

~# semanage port -l | grep test
test_port_t                    tcp      1440

In hindsight, it makes sense that it is this easy. After all, support for the old-style policy language is done by converting it into CIL when calling semodule so it makes sense to immediately put the module (in CIL code) ready to be taken up.

Posts for Saturday, July 11, 2015

Protecting the Cause

People ain’t perfect. We forget and neglect, actively ignore, don’t care enough, don’t live up to what we’d like to be or know we could be or should be. Now don’t get me wrong, I’m no misanthropist. I believe that people in general are nice and try their best given their own perception of the world in spite of sometimes things ending badly.

In the last weeks I’ve had a bunch of very different reasons to think about a general problem that has infected many human rights/civil liberties activist circles, organizations, subcultures or structures. Maybe even infection isn’t the right word. Maybe it’s more of a problem of our individual views on the world with are … well … too individualistic. But I’m getting ahead of myself.1

I kept on coming back to a song by the British HipHop artist Scroobius Pip called “Waiting for the beat to kick in”2. The song describes the artist’s dream of walking through New York (“but not New York in real life the New York you see in old films”) and meeting different characters telling him their idea of how to live your life.

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="390" src=";rel=1&amp;fs=1&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" type="text/html" width="640"></iframe>

So after meeting a bunch of positive, inspiring people the protagonist’s walk ends with meeting someone named “Walter Nepp” who wants to “bring him back down to earth“:

You may think yourself in general to be a nice guy,
But I’m telling you now – that right there is a lie,
Even the nicest of guys has some nasty within ’em,
You don’t have to be backlit to be the villain,
Whether it be greed lust or just plain vindictiveness,
There’s a level of malevolence inside all of us

I’ve been coming back to those lines over and over again in the last weeks and not just for personal reasons thinking about my own actions. Which is something I need to do. A lot. Sometimes I talk and write faster than I think which lets me end up in situations where I am not just wrong due to ignorance but actually harmful to people, causes or values I claim to support. These things don’t happen. I cause them through my ignorance, stupidity, carelessness or just not giving enough of a shit. I might behave racist, sexist or any other number of -ists in spite of me being an OK guy3. Everybody does these things because nobody is perfect – though some do it way more frequently or more hurtfully than others. Which is no and can be no excuse: The fact that we all behave hurtfully doesn’t make that behavior OK or normal just frequent. A chronic head- and heartache in our collective consciousness.

But I can consider myself lucky. My friends, peers, contacts keep me honest by calling me out on my bullshit. Which is more of a gift than many probably realize. I am living a privileged life and as much as you try, sometimes it is hard to break the veil that position casts on the world. To make a long story short: I need people telling me when I mess up to comprehend not just the situation at hand but also fix my thinking and perception as good as possible. Rinse and repeat. It’s a process.

I am not especially smart so many people do a lot better than I do. And many of them invest tons of their energy, time and life into making the world a better place. They work – often even without payment – in NGOs and activist structures fixing the world. Changing public perception and/or policy. Fighting the good fights on more fronts than I can count (and my Elementary School report card said that I wasn’t half bad at counting!).

As humans relying on the rights or legislations these groups and individuals fight for we are very lucky to have all that on our side. Networks of networks of activists helping each other ( and us) out. Cool. But sadly, very rarely are things as simple as they might seem.

In the last months I keep hearing very similar stories from the inside of many of the (digital) human rights/civil liberties NGOs. Stories contradicting the proposed mission statement. Stories of discrimination, sexism, harassment and communicative violence. Structural failure in organisations completely screwing up providing a safe environment for many trying to contribute. These stories usually have very visible, prominent men within those organisations at their core. But nothing changes.

Obviously there is not one singular cause for the deafening silence the stories drown in.

People might not want to speak up against their (past) employer to protect their present or future which is very understandable. And without witnesses and proof here is no story. It would be easy for me to ask for victims and witnesses to blow the whistle and speak up but what would I do in a similar situation? How can we expect people to destroy their actual lives for some lofty idea of truth, transparency and ethical righteousness?

But I believe there is something that we can change even from the fringes or even outside: We can stop to let people use – explicitly or implicitly – the cause as their shield.

I have seen this mechanic numerous times: Some civil rights dude fucks up but instead of calling him out people keep shtum to not hurt the cause. Few have mastered that mechanic as well as Julian Assange who basically finds new ways to break with the norms of almost everything in any given week, but he’s not alone just probably more intense than the rest. You can see these things in probably any one of the heroes of the scene and as I showed earlier: We all fuck up why shouldn’t they? But with the cause so intimately intertwined with their personas, their public identity any of their personal lapses or transgressions could cause damage for that cause.

Sadly people don’t just shut up but actually target those not shutting up. When questions came up about Assange’s antisemite friends and peers it was quickly labeled a smear campaign against whatever Wikileaks’ cause is4. And it’s not just a Wikileaks thing.

There is a lot to learn from this issue, about the lack of sustainability in chaining a cause or structure to one or very few public superstars. But mostly it’s about understanding that – especially as an NGO/structure fighting for human rights or similar things – you have to be better than most of the people and organisations you stand against.

Recently Wikileaks published a bunch of data from an Italian Cyber(war|security)5 company who seemingly even sold 0-days and exploits to oppressive regimes. Very cool, let’s see what journalists can dig up as stories about who builds these tools used to take other human beings’ rights and sometimes lives. Who sells them. And what we can do. But then I see people – fierce and vocal supporters of privacy as human rights – just randomly digging out personal information about people working for the German secret service from some list published without context (that also included all kinds of random addresses which did not all appear in the emails also leaked). When the data of millions of US government employees leaked people from similar groups were just too happy and quick in dragging out information about individuals and contextualizing them – often without proper research.

Sure. You can use any data you can get your hands on. But you are damaging your own cause, creating contradictions and becoming less credible. And as an activist/NGO credibility is your main asset. So you need to be called out on it. Even if your opponents might pick up that criticism and use it against you. I know it sucks but as an NGO/activist you don’t have the luxury of not caring how you do what you want to do. For example: If you claim privacy to be a human right (which is a popular argument in the civil rights scene) you just can’t republish (and thereby recontextualize) individual’s data on your Twitter feed or whatever. Because that would mean that you either don’t actually believe that privacy is a human right because you don’t grant it to people in certain jobs or that you just don’t respect their rights which would lead to the question why people should listen to you arguing for them.

Fighting for ethics and standards means that you have to live them as well: You can’t fight for transparency while being intransparent, can’t fight for right to privacy while just throwing “evil people”‘s data out there just cause they deserve it for some reason.

And that is where we as a networked subculture fail so often. We don’t call people out for that kind of crap. Because we like them. Because they were or are our friends or peers. Because we are afraid of how it might endanger the cause or the campaign or whatever. Sadly two failures (by the original fucker-up and by us not calling them out on it) don’t make one success. But helping people see their mistakes can make future successes possible.

Maybe it’s just one of these nights, sitting alone in front of a computer thinking, writing instead of anything else but I am just soo tired of hearing digital martyrs and saviors being glorified while seeing them discriminating against people for their gender or religion or acting according to different rules than what they demand for everyone else. I’m tired of all the secret stories being exchanged in the dark between so many members of our communities that they have basically become public knowledge and still nothing getting better.

‘Ethical behavior is doing the right thing when no one else is watching- even when doing the wrong thing is legal.’ Aldo Leopold

Photo by CJS*64 A man with a camera

  1. Excuse my strange stream of consciousness kind of writing here, I’m basically writing while thinking here.
  2. which incidentally shares some sort of stream of consciousness lyrics style with this way clunkier text
  3. Obviously my own tainted opinion 😉
  4. and let’s not even start with the accusations against Assange in Sweden
  5. as if there was a difference really

Flattr this!


Restricting even root access to a folder

In a comment Robert asked how to use SELinux to prevent even root access to a directory. The trivial solution would be not to assign an administrative role to the root account (which is definitely possible, but you want some way to gain administrative access otherwise ;-)

Restricting root is one of the commonly referred features of a MAC (Mandatory Access Control) system. With a well designed user management and sudo environment, it is fairly trivial – but if you need to start from the premise that a user has direct root access, it requires some thought to implement it correctly. The main “issue” is not that it is difficult to implement policy-wise, but that most users will start from a pre-existing policy (such as the reference policy) and build on top of that.

The use of a pre-existing policy means that some roles are already identified and privileges are already granted to users – often these higher privileged roles are assigned to the Linux root user as not to confuse users. But that does mean that restricting root access to a folder means that some additional countermeasures need to be implemented.

The policy

But first things first. Let’s look at a simple policy for restricting access to /etc/private:

policy_module(myprivate, 1.0)

type etc_private_t;

This simple policy introduces a type (etc_private_t) which is allowed to be used for files (it associates with a file system). Do not use the files_type() interface as this would assign a set of attributes that many user roles get read access on.

Now, it is not sufficient to have the type available. If we want to assign it to a type, someone or something needs to have the privileges to change the security context of a file and directory to this type. If we would just load this policy and try to do this from a privileged account, it would fail:

~# chcon -t etc_private_t /etc/private
chcon: failed to change context of '/etc/private' to 'system_u:object_r:etc_private_t:s0': Permission denied

With the following rule, the sysadm_t domain (which I use for system administration) is allowed to change the context to etc_private_t:

allow sysadm_t etc_private_t:{dir file} relabelto;

With this in place, the administrator can label resources as etc_private_t without having read access to these resources afterwards. Also, as long as there are no relabelfrom privileges assigned, the administrator cannot revert the context back to a type that he has read access to.

The countermeasures

But this policy is not sufficient. One way that administrators can easily access the resources is to disable SELinux controls (as in, put the system in permissive mode):

~# cat /etc/private/README
cat: /etc/private/README: Permission denied
~# setenforce 0
~# cat /etc/private/README
Hello World!

To prevent this, enable the secure_mode_policyload SELinux boolean:

~# setsebool secure_mode_policyload on

This will prevent any policy and SELinux state manipulation… including permissive mode, but also including loading additional SELinux policies or changing booleans. Definitely experiment with this setting without persisting (i.e. do not use -P in the above command yet) to make sure it is manageable for you.

Still, this isn’t sufficient. Don’t forget that the administrator is otherwise a full administrator – if he cannot access the /etc/private location directly, then he might be able to access it indirectly:

  • If the resource is on a non-critical file system, he can unmount the file system and remount it with a context= mount option. This will override the file-level contexts. Bind-mounting does not seem to allow overriding the context.
  • If the resource is on a file system that cannot be unmounted, the administrator can still reboot the system in a mode where he can access the file system regardless of SELinux controls (either through editing /etc/selinux/config or by booting with enforcing=0, etc.
  • The administrator can still access the block device files on which the resources are directly. Specialized tools can allow for extracting files and directories without actually (re)mounting the device.

A more extensive list of methods to potentially gain access to such resources is iterated in Limiting file access with SELinux alone.

This set of methods for gaining access is due to the administrative role already assigned by the existing policy. To further mitigate these risks with SELinux (although SELinux will never completely mitigate all risks) the roles assigned to the users need to be carefully revisited. If you grant people administrative access, but you don’t want them to be able to reboot the system, (re)mount file systems, access block devices, etc. then create a user role that does not have these privileges at all.

Creating such user roles does not require leaving behind the policy that is already active. Additional user domains can be created and granted to Linux accounts (including root). But in my experience, when you need to allow a user to log on as the “root” account directly, you probably need him to have true administrative privileges. Otherwise you’d work with personal accounts and a well-designed /etc/sudoers file.

Posts for Thursday, July 9, 2015

Let’s kill “cyborg”

I love a good definition: Very few things in life wield extreme power as elegantly as definitions do. Because every teeny, tiny definition worth its salt grabs the whole universe, all things, all ideas, all stories, feelings, objects and laws and separates them into distinct pieces, into two different sets: The set of things conforming to the definition and the set of things that don’t. If you’ve read enough definitions, super hero comics are basically boring.

Definitions structure our world. They establish borders between different things and allow us to communicate more precisely and clearly. They help us to understand each other better. Obviously many of them are in a constant development, are evolving and changing, adapting to our always fluid experience of our shared world.

And just as they change, sometimes definitions and the concepts they describe just stop being useful. Today I want to encourage you and me to put one definition and concept to rest that has outlived its usefulness: Let’s kill the term “cyborg”.

The term cyborg has been with us since the 1960s and has influenced more than just cheesy science fiction movies: Talking about cyborgs was a way to describe a future where human beings and machines would meld in order to describe our actual current lifestyles. Because for better or worse: We have been hybrid beings of part nature, part technology of sorts for many many decades now.

Still, the idea of “natural humans” put in contrast to machine-augmented humans was useful to elaborate the requirements that we as a society would need to postulate in order to integrate technology into ourselves, our bodies and our mental exoskeletons in a humane way. It has been a good conversation. But lately it really hasn’t been.

“Cyborg” has mostly been a word to single out and alienate “freaks”. It refers to body hackers that do things to their bodies that the mainstream doesn’t really understand but that they just love to watch like one of those strangely popular torture porn movies like Saw. It refers to people with disabilities in a way that does not include them in whatever the mainstream view of society is or that helps them make a case for how to design and engineer things in a way more accessible but as these weird “others”. I can’t count the amount of times that for example Neil Harbisson has been talking on conferences about his perception augmentation allowing him to hear colors with the gist of it in the media reception being mostly: Look how weird!

Instead of helping us to understand ourselves and our decision to intertwine our lives, worlds and bodies with all kinds of different technologies “Cyborg” just creates distance these days. It doesn’t build bridges for fruitful debates but in fact tears them down to look at freaks on the other side of the river without them coming closer.

We are all cyborgs and we have been for quite a while. But when everybody is a cyborg really nobody is. The distinction from whatever “norm” that the word, idea and definition provided is no longer helpful but actually hurtful for so many debates that we have to have in the next few years.

Thank you “cyborg”, it’s been an interesting ride. Rest in peace.

Photo by JD Hancock

Flattr this!

Posts for Sunday, July 5, 2015


Intermediate policies

When developing SELinux policies for new software (or existing ones whose policies I don’t agree with) it is often more difficult to finish the policies so that they are broadly usable. When dealing with personal policies, having them “just work” is often sufficient. To make the policies reusable for distributions (or for the upstream project), a number of things are necessary:

  • Try structuring the policy using the style as suggested by refpolicy or Gentoo
  • Add the role interfaces that are most likely to be used or required, or which are in the current draft implemented differently
  • Refactor some of the policies to use refpolicy/Gentoo style interfaces
  • Remove the comments from the policies (as refpolicy does not want too verbose policies)
  • Change or update the file context definitions for default installations (rather than the custom installations I use)

This often takes quite some effort. Some of these changes (such as the style updates and commenting) are even counterproductive for me personally (in the sense that I don’t gain any value from doing so and would have to start maintaining two different policy files for the same policy), and necessary only for upstreaming policies. As a result, I often finish with policies that I just leave for me personally or somewhere on a public repository (like these Neo4J and Ceph policies), without any activities already scheduled to attempt to upstream those.

But not contributing the policies to a broader public means that the effort is not known, and other contributors might be struggling with creating policies for their favorite (or necessary) technologies. So the majority of policies that I write I still hope to eventually push them out. But I noticed that these last few steps for upstreaming (the ones mentioned above) might only take a few hours of work, but take me over 6 months (or more) to accomplish (as I often find other stuff more interesting to do).

I don’t know yet how to change the process to make it more interesting to use. However, I do have a couple of wishes that might make it easier for me, and perhaps others, to contribute:

  • Instead of reacting on contribution suggestions, work on a common repository together. Just like with a wiki, where we don’t aim for a 100% correct and well designed document from the start, we should use the strength of the community to continuously improve policies (and to allow multiple people to work on the same policy). Right now, policies are a one-man publication with a number of people commenting on the suggested changes and asking the one person to refactor or update the change himself.
  • Document the style guide properly, but don’t disallow contributions if they do not adhere to the style guide completely. Instead, merge and update. On successful wikis there are even people that update styles without content updates, and their help is greatly appreciated by the community.
  • If a naming convention is to be followed (which is the case with policies) make it clear. Too often the name of an interface is something that takes a few days of discussion. That’s not productive for policy development.
  • Find a way to truly create a “core” part of the policy and a modular/serviceable approach to handle additional policies. The idea of the contrib/ repository was like that, but failed to live up to its expectations: the number of people who have commit access to the contrib is almost the same as to the core, a few exceptions notwithstanding, and whenever policies are added to contrib they often require changes on the core as well. Perhaps even support overlay-type approaches to policies so that intermediate policies can be “staged” and tested by a larger audience before they are vetted into the upstream reference policy.
  • Settle on how to deal with networking controls. My suggestion would be to immediately support the TCP/UDP ports as assigned by IANA (or another set of sources) so that additional policies do not need to wait for the base policy to support the ports. Or find and support a way for contributions to declare the port types themselves (we probably need to focus on CIL for this).
  • Document “best practices” on policy development where certain types of policies are documented in more detail. For instance, desktop application profiles, networked daemons, user roles, etc. These best practices should not be mandatory and should in fact support a broad set of privilege isolation. With the latter, I mean that there are policies who cover an entire category of systems (init systems, web servers), a single software package or even the sub-commands and sub-daemons of that package. It would surprise me if this can’t be supported better out-of-the-box (as in, through a well thought-through base policy framework and styleguide).

I believe that this might create a more active community surrounding policy development.

Posts for Thursday, July 2, 2015


Moved blog to hugo, fastly and comma

  • I noticed what a disservice I was doing my readers when I started monitoring my site using litmus. A dynamic website in python on a cheap linode… What do you expect? So I now serve through fastly and use a static site generator.

  • pyblosxom was decent while it lasted. It can generate sites statically, but the project never got a lot of traction and is slowly fading out. There were a bit too many moving parts, so …

  • I now use the hugo static site generator, which is powerful, quite complete and gaining momentum. Fast and simple to use.

  • Should also keep an eye on the caddy webserver since it has some nice things such as git integration which should work well with hugo.

  • Trying to get disqus going was frustrating. Self hosted options like talkatv and isso were too complex, and kaiju is just not there yet and also pretty complex. I wrote comma which is a simple comment server in Go. Everything I need in 100 lines of Go and 50 lines of javascript! Let me know if you see anything funky.

  • migrated all content.

Digitally organic

There’s a new kid in music streaming town: Apple has entered this market of different services who provide all basically the same 30ish million tracks of music for a monthly fee. And there was much rejoicing™.

Some services (like for example Google’s Play Music and obviously Apple’s) derive a market advantage from their tight integration into the respective eco systems: You already use $company’s mail, calendar and mobile operating system, why not just throw in the music subscription, right? But still, as a service trying to win over new clients (and maybe even migrate them to your other offerings) you need a unique selling proposition, something that you have and nobody else. Something that ideally is even hard to clone and copy.

Apple decided to opt for the curation road: Apple’s service provides its customers with playlists or radio stations whose content was artfully crafted, chosen and curated by actual human beings, not algorithms. The selection of DJs is sorta problematic creating a sort of Audible Gentrification as David Banks called it, but the idea itself reasonates with the dominating digital zeitgeist.

On Twitter I casually remarked

but I had the feeling some people didn’t understand it in the way I had intended it so I’ll try to explain it in a little more detail here.

The “organic” movement (for lack of better term) consists of people who are willing to pay a premium for food items (and in extension other things) if they are producted organic: No genetically modified crops, no or very little “chemicals” (as in artificial fertilizer or pesticide) and produced locally if possible. Many of the aspects embodied in that ideology are smart: Why drive milk thousands of kilometers to supermarkets instead of buying local milk? Being more careful and conscious about how to fertilize soil and how to combat dangers to the harvest is also very important to help our environment not collapse. And nobody wants barely tested genetic modifications in our food chain (or any other significant untested changes for that matter).

But the focus on “nature” on the original is something else. It’s a conservative perspective in a way, trying to bring back the “good old days” (a broken sentiment that culminates in extremes such as the Paleo Diet). A way to live better by moving back in time while distancing oneself from the industrialized mainstream. This ideology branches out in many other areas and brought us the renaissance of the artisan.

While mainstream culture celebrates or at least quietly supports the technologically driven “progress”, certain people – more often than not people with a good income/social position – have shifted to conscious consumption of things manually crafted. “Realer” things made by people, not machines and industrial complexes. Things imbued with a vibe from a better, simpler, more honest time without all the complexities and nuances of today’s connected, industrialized lifestyle.

It’s no surprise that Apple would plug into this sentiment: The company whose products signify and embody the digital 1% like nothing else sets itself aside from the competition by using artists, craftsmen and -women to generate their playlists. With algorithms becoming increasingly better and running them becoming cheaper, having people choose your music is a weird throwback. On first sight.

On the second glance it becomes clear how “people not algorithms” can easily become a way for upper classes to distinguish from lower classes: If you can afford it, you have people choose your tunes, if you’re poor you have to live with whatever algorithms come up with. Recommendation algorithms are to industrialized mainstream as curators are to coffee artisans (and whatever else needs to be hipstered up to be sold for a premium).

The focus in the artisan is only very superficially a critique of the industrialized way: Drinking your hand-made artfully crafted coffee is only half the fun without your laptop or tablet computer, right? It’s a way of eating one’s cake and having it, too: Appear critical and reflected without losing the benefits of a globally networked industrial system.

We will see more of these “people, not algorithm” services in the future. Services that will probably cost a premium and provide a way for people to show their economic privilege without looking like a rich bastard but a conscious consumer.

P.S.: I do believe in conscious consumption. I buy local and fair trade if at all possible. This article is about framing. About how products are advertised and to whom.

Photo by asboluv

Flattr this!

Posts for Tuesday, June 30, 2015

removing old backups

Ever had to think about how to remove old backups? Like you want to keep the last 30 backups? Whenever i had to think about a solution to this i though about something with "find -mtime". However, this only works when backups were made constantly on a daily base.

But what happens if a backup fails or an external server doesn't have a connection to the storage? In my case my laptop only sporadically creates backups. If my laptop would be turned off for 30 days all of my previous backups would be deleted with "find -mtime".

Until now i had a huge script which checks for such cases. Just stupid...

Today i found THE solution!
A really easy and nice one-liner to always keep the last 30 backups. It's just so nice :D

Attention: Don't simple copy/paste this one-liner - it can remove files without asking!

find /backup/folder -type f -printf '%T@ %p\n' | sort -k1 -n | head -n-30 | cut -d' ' -f2 | xargs -r rm

I think i don't really have to explain it. It keeps the last 30 backups - doesn't matter how old they are, but they are always the newest one. In case you have multiple backups make sure to keep them in separated directories or filter them with "find -name "*keyword*"". And before using this one-liner i strongly suggest removing the last part (xargs -r rm) to see what would be removed.

Hope someone can find it useful. I've searched hours to find something like this and never found anything. (probably because i searched with the wrong keywords...)

Posts for Saturday, June 13, 2015


Where does CIL play in the SELinux system?

SELinux policy developers already have a number of file formats to work with. Currently, policy code is written in a set of three files:

  • The .te file contains the SELinux policy code (type enforcement rules)
  • The .if file contains functions which turn a set of arguments into blocks of SELinux policy code (interfaces). These functions are called by other interface files or type enforcement files
  • The .fc file contains mappings of file path expressions towards labels (file contexts)

These files are compiled into loadable modules (or a base module) which are then transformed to an active policy. But this is not a single-step approach.

Transforming policy code into policy file

For the Linux kernel SELinux subsystem, only a single file matters – the policy.## file (for instance policy.29). The suffix denotes the binary format used as higher numbers mean that additional SELinux features are supported which require different binary formats for the SELinux code in the Linux kernel.

With the 2.4 userspace, the transformation of the initial files as mentioned above towards a policy file is done as follows:

When a developer builds a policy module, first checkmodule is used to build a .mod intermediate file. This file contains the type enforcement rules with the expanded rules of the various interface files. Next, semodule_package is called which transforms this intermediate file, together with the file context file, into a .pp file.

This .pp file is, in the 2.4 userspace, called a “high level language” file. There is little high-level about it, but the idea is that such high-level language files are then transformed into .cil files (CIL stands for Common Intermediate Language). If at any moment other frameworks come around, they could create high-level languages themselves and provide a transformation engine to convert these HLL files into CIL files.

For the current .pp files, this transformation is supported through the /usr/libexec/selinux/hll/pp binary which, given a .pp file, outputs CIL code.

Finally, all CIL files (together) are compiled into a binary policy.29 file. All the steps coming from a .pp file towards the final binary file are handled by the semodule command. For instance, if an administrator loads an additional .pp file, its (generated) CIL code is added to the other active CIL code and together, a new policy binary file is created.

Adding some CIL code

The SELinux userspace development repository contains a secilc command which can compile CIL code into a binary policy file. As such, it can perform the (very) last step of the file conversions above. However, it is not integrated in the sense that, if additional code is added, the administrator can “play” with it as he would with SELinux policy modules.

Still, that shouldn’t prohibit us from playing around with it to experiment with the CIL language construct. Consider the following CIL SELinux policy code:

; Declare a test_port_t type
(type test_port_t)
; Assign the type to the object_r role
(roletype object_r test_port_t)

; Assign the right set of attributes to the port
(typeattributeset defined_port_type test_port_t)
(typeattributeset port_type test_port_t)

; Declare tcp:1440 as test_port_t
(portcon tcp 1440 (system_u object_r test_port_t ((s0) (s0))))

The code declares a port type (test_port_t) and uses it for the TCP port 1440.

In order to use this code, we have to build a policy file which includes all currently active CIL code, together with the test code:

~$ secilc -c 29 /var/lib/selinux/mcs/active/modules/400/*/cil testport.cil

The result is a policy.29 (the command forces version 29 as the current Linux kernel used on this system does not support version 30) file, which can now be copied to /etc/selinux/mcs/policy. Then, after having copied the file, load the new policy file using load_policy.

And lo and behold, the port type is now available:

~# semanage port -l | grep 1440
test_port_t           tcp      1440

To verify that it really is available and not just parsed by the userspace, let’s connect to it and hope for a nice denial message:

~$ ssh -p 1440 localhost
ssh: connect to host localhost port 1440: Permission denied

~$ sudo ausearch -ts recent
time->Thu Jun 11 19:35:45 2015
type=PROCTITLE msg=audit(1434044145.829:296): proctitle=737368002D700031343430006C6F63616C686F7374
type=SOCKADDR msg=audit(1434044145.829:296): saddr=0A0005A0000000000000000000000000000000000000000100000000
type=SYSCALL msg=audit(1434044145.829:296): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=6d4d1ce050 a2=1c a3=0 items=0 ppid=2005 pid=18045 auid=1001 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=pts0 ses=1 comm="ssh" exe="/usr/bin/ssh" subj=staff_u:staff_r:ssh_t:s0 key=(null)
type=AVC msg=audit(1434044145.829:296): avc:  denied  { name_connect } for  pid=18045 comm="ssh" dest=1440 scontext=staff_u:staff_r:ssh_t:s0 tcontext=system_u:object_r:test_port_t:s0 tclass=tcp_socket permissive=0

Posts for Wednesday, June 10, 2015


Live SELinux userspace ebuilds

In between courses, I pushed out live ebuilds for the SELinux userspace applications: libselinux, policycoreutils, libsemanage, libsepol, sepolgen, checkpolicy and secilc. These live ebuilds (with Gentoo version 9999) pull in the current development code of the SELinux userspace so that developers and contributors can already work with in-progress code developments as well as see how they work on a Gentoo platform.

That being said, I do not recommend using the live ebuilds for anyone else except developers and contributors in development zones (definitely not on production). One of the reasons is that the ebuilds do not apply Gentoo-specific patches to the ebuilds. I would also like to remove the Gentoo-specific manipulations that we do, such as small Makefile adjustments, but let’s start with just ignoring the Gentoo patches.

Dropping the patches makes sure that we track upstream libraries and userspace closely, and allows developers to try and send out patches to the SELinux project to fix Gentoo related build problems. But as not all packages can be deployed successfully on a Gentoo system some patches need to be applied anyway. For this, users can drop the necessary patches inside /etc/portage/patches as all userspace ebuilds use the epatch_user method.

Finally, observant users will notice that “secilc” is also provided. This is a new package, which is probably going to have an official release with a new userspace release. It allows for building CIL-based SELinux policy code, and was one of the drivers for me to create the live ebuilds as I’m experimenting with the CIL constructions. So expect more on that later.

Posts for Monday, June 1, 2015

testing is fun (binpkg-multi-instance)

Since version 2.2.19 (now 2.2.20) portage implemented a feature called binpkg-multi-instance. This is a feature which i was looking for quite some time. In the last days i had some time and i decided to test it.
The feature itself brings the ability to portage to keep multiple versions (with different use settings) of a single package version.
Until now, if you created a binary package, portage could only keep exactly one binary-version of any package. If you build the package again with different use-settings and created a binary package the version prior would had gone.

Now this is probably not something many people were looking for. I was one of those who were really exited about it. When the feature hit git i was already tempted to test it directly from git head.

So why is that so exciting for me? Well, because some time ago i set up a nice test system where this feature helps _alot_ keeping compile times at a minimum.


The idea was simple. How to test many different setups and keep compile times at a minimum?
I wanted a base system which i could clone anytime i want so that i could install and test various package combination's and use already compiled packages as much as possible. That being said, a virtual machine with snapshots did come into mind. However, i had a dedicated hardware which had nothing to-do and thus there was no need for virtualization. Another candidate was btrfs with it's snapshot features. The problem here: What if i want to test another filesystem? ;)

Logically i decided to go with lvm.

The boot partition is shared with every system. Every other partition is on lvm. Every root partition is unique, only /home, /usr/portage and /usr/src are on a separate lvm partitions as those can be shared as well.
First i've created a "base" gentoo system. Basically a stage3 system with some additional programs and a few important settings.
EMERGE_DEFAULT_OPTS is one of the most important in this case. In my case it looks like following:

EMERGE_DEFAULT_OPTS="--binpkg-changed-deps=y --binpkg-respect-use=y --buildpkg --buildpkg-exclude \"virtual/* sys-kernel/*-sources\" -k --jobs=3"

It tells portage to always use, if possible, binary packages and, except for kernel-sources and virtual packages, to always create binary packages. Since this setting is my base system it's in every clone of it. (as long as i don't change anything by hand)

And that's were binpkg-multi-instance comes into mind. Since every system access the same binary package store, but every system might have different use setting for a particular package, every package now only has to build once in any case!

Compiling is really funny right now, cause it looks quite often similar like here:


Sure, the whole setup is of course a bit more complex and while this setup works really great there are a few things to mentioned. For example, the kernel(s) needs a few features in every system (like lvm snapshot, openrc and systemd - if i want to test both, which i do). Also since home is shared with every system, testing various window managers (like kde,gnome,xlqt,...) could mess up their configurations. Also having different arches (x86 and amd64) need adjustments to the base configuration. (but it's working too!)

Besides that i've also wrote a small script which does most of the work. It clones and installs (grub) any system at any moment even with a different file-systems if desired. (plus it can also encrypt a cloned system).
For example, basically all i have to-do is:
./sysed -q
This clones the actual running system with the actual file-system and size and creates an grub entry which is called "${lvm_name}_testing".
The script can also backup, restore, delete and edit my lvm systems.

I'm using this script quite often as it's really simple cloning a whole system in about ~2 minutes. So far i already have 14 amd64 and 2 x86 systems. Below a list of my systems (from lvs).

  gentoo_amd64_acp               vg0  -wi-a----- 10.00g
gentoo_amd64_base vg0 -wi-ao---- 10.00g
gentoo_amd64_base_selinux vg0 -wi-a----- 10.00g
gentoo_amd64_base_systemd vg0 -wi-a----- 10.00g
gentoo_amd64_cinnamon vg0 -wi-a----- 10.00g
gentoo_amd64_enlightenment vg0 -wi-a----- 10.00g
gentoo_amd64_gnome vg0 -wi-a----- 10.00g
gentoo_amd64_kde vg0 -wi-a----- 10.00g
gentoo_amd64_kde_testing vg0 -wi-a----- 10.00g
gentoo_amd64_lxqt vg0 -wi-a----- 10.00g
gentoo_amd64_mate vg0 -wi-a----- 10.00g
gentoo_amd64_qtile_systemd vg0 -wi-a----- 10.00g
gentoo_amd64_sec vg0 -wi-a----- 10.00g
gentoo_amd64_secure vg0 -wi-a----- 10.00g
gentoo_x86_base vg0 -wi-a----- 10.00g
gentoo_x86_kde vg0 -wi-a----- 10.00g

binpkg-multi-instance had an big effect here, especially when trying things like abi_x86_32 or selinux. From now on i won't have to compile any package a second time anymore as long as i already build it once!

Big thx to the gentoo portage team!

Posts for Thursday, May 28, 2015

less portage rsync output

Ever thought how to silence rsync when doing emerge --sync (or eix-sync). Sure, it's nice when we get lots of information. But it's like with compiling packages - The first few years it's amazing looking at the terminal while it's compiling the latest stuff. However, after a while these things become a bit boring.
While we have --quite-build for emerge, rsync per default outputs every single file which gets transferred and deleted. Luckily, recent versions of rsync, which also went already stable, support new ways of progress output and since i use them already on other scripts i decided to modify my portage rsync settings a bit:


The output looks similar like that:

Neat, isn't it?
BTW, in order this works correctly - the remote rsync server need to run a recent version of rsync as well.

Posts for Monday, May 25, 2015


PostgreSQL with central authentication and authorization

I have been running a PostgreSQL cluster for a while as the primary backend for many services. The database system is very robust, well supported by the community and very powerful. In this post, I’m going to show how I use central authentication and authorization with PostgreSQL.

Centralized management is an important principle whenever deployments become very dispersed. For authentication and authorization, having a high-available LDAP is one of the more powerful components in any architecture. It isn’t the only method though – it is also possible to use a distributed approach where the master data is centrally managed, but the proper data is distributed to the various systems that need it. Such a distributed approach allows for high availability without the need for a highly available central infrastructure (user ids, group membership and passwords are distributed to the servers rather than queried centrally). Here, I’m going to focus on a mixture of both methods: central authentication for password verification, and distributed authorization.

PostgreSQL default uses in-database credentials

By default, PostgreSQL uses in-database credentials for the authentication and authorization. When a CREATE ROLE (or CREATE USER) command is issued with a password, it is stored in the pg_catalog.pg_authid table:

postgres# select rolname, rolpassword from pg_catalog.pg_authid;
    rolname     |             rolpassword             
 postgres_admin | 
 dmvsl          | 
 johan          | 
 hdc_owner      | 
 hdc_reader     | 
 hdc_readwrite  | 
 hadoop         | 
 swift          | 
 sean           | 
 hdpreport      | 
 postgres       | md5c127bc9fc185daf0e06e785876e38484

Authorizations are also stored in the database (and unless I’m mistaken, this cannot be moved outside):

postgres# \l db_hadoop
                                   List of databases
   Name    |   Owner   | Encoding |  Collate   |   Ctype    |     Access privileges     
 db_hadoop | hdc_owner | UTF8     | en_US.utf8 | en_US.utf8 | hdc_owner=CTc/hdc_owner  +
           |           |          |            |            | hdc_reader=c/hdc_owner   +
           |           |          |            |            | hdc_readwrite=c/hdc_owner

Furthermore, PostgreSQL has some additional access controls through its pg_hba.conf file, in which the access towards the PostgreSQL service itself can be governed based on context information (such as originating IP address, target database, etc.).

For more information about the standard setups for PostgreSQL, definitely go through the official PostgreSQL documentation as it is well documented and kept up-to-date.

Now, for central management, in-database settings become more difficult to handle.

Using PAM for authentication

The first step to move the management of authentication and authorization outside the database is to look at a way to authenticate users (password verification) outside the database. I tend not to use a distributed password approach (where a central component is responsible for changing passwords on multiple targets), instead relying on a high-available LDAP setup, but with local caching (to catch short-lived network hick-ups) and local password use for last-hope accounts (such as root and admin accounts).

PostgreSQL can be configured to directly interact with an LDAP, but I like to use Linux PAM whenever I can. For my systems, it is a standard way of managing the authentication of many services, so the same goes for PostgreSQL. And with the sys-auth/pam_ldap package integrating multiple services with LDAP is a breeze. So the first step is to have PostgreSQL use PAM for authentication. This is handled through its pg_hba.conf file:

# TYPE  DATABASE        USER    ADDRESS         METHOD          [OPTIONS]
local   all             all                     md5
host    all             all     all             pam             pamservice=postgresql

This will have PostgreSQL use the postgresql PAM service for authentication. The PAM configuration is thus in /etc/pam.d/postgresql. In it, we can either directly use the LDAP PAM modules, or use the SSSD modules and have SSSD work with LDAP.

Yet, this isn’t sufficient. We still need to tell PostgreSQL which users can be authenticated – the users need to be defined in the database (just without password credentials because that is handled externally now). This is done together with the authorization handling.

Users and group membership

Every service on the systems I maintain has dedicated groups in which for instance its administrators are listed. For instance, for the PostgreSQL services:

# getent group gpgsqladmin

A local batch job (ran through cron) queries this group (which I call the masterlist, as well as queries which users in PostgreSQL are assigned the postgres_admin role (which is a superuser role like postgres and is used as the intermediate role to assign to administrators of a PostgreSQL service), known as the slavelist. Delta’s are then used to add the user or remove it.

# Note: membersToAdd / membersToRemove / _psql are custom functions
#       so do not vainly search for them on your system ;-)
for member in $(membersToAdd ${masterlist} ${slavelist}) ; do
  _psql "CREATE USER ${member} LOGIN INHERIT;" postgres
  _psql "GRANT postgres_admin TO ${member};" postgres

for member in $(membersToRemove ${masterlist} ${slavelist}) ; do
  _psql "REVOKE postgres_admin FROM ${member};" postgres
  _psql "DROP USER ${member};" postgres

The postgres_admin role is created whenever I create a PostgreSQL instance. Likewise, for databases, a number of roles are added as well. For instance, for the db_hadoop database, the hdc_owner, hdc_reader and hdc_readwrite roles are created with the right set of privileges. Users are then granted this role if they belong to the right group in the LDAP. For instance:

# getent group gpgsqlhdc_own

With this simple approach, granting users access to a database is a matter of adding the user to the right group (like gpgsqlhdc_ro for read-only access to the Hadoop related database(s)) and either wait for the cron-job to add it, or manually run the authorization synchronization. By standardizing on infrastructural roles (admin, auditor) and data roles (owner, rw, ro) managing multiple databases is a breeze.

Posts for Monday, May 18, 2015


Testing with permissive domains

When testing out new technologies or new setups, not having (proper) SELinux policies can be a nuisance. Not only are the number of SELinux policies that are available through the standard repositories limited, some of these policies are not even written with the same level of confinement that an administrator might expect. Or perhaps the technology to be tested is used in a completely different manner.

Without proper policies, any attempt to start such a daemon or application might or will cause permission violations. In many cases, developers or users tend to disable SELinux enforcing then so that they can continue playing with the new technology. And why not? After all, policy development is to be done after the technology is understood.

But completely putting the system in permissive mode is overshooting. It is much easier to make a very simple policy to start with, and then mark the domain as a permissive domain. What happens is that the software then, after transitioning into the “simple” domain, is not part of the SELinux enforcements anymore whereas the rest of the system remains in SELinux enforcing mode.

For instance, create a minuscule policy like so:

policy_module(testdom, 1.0)

type testdom_t;
type testdom_exec_t;
init_daemon_domain(testdom_t, testdom_exec_t)

Mark the executable for the daemon as testdom_exec_t (after building and loading the minuscule policy):

~# chcon -t testdom_exec_t /opt/something/bin/daemond

Finally, tell SELinux that testdom_t is to be seen as a permissive domain:

~# semanage permissive -a testdom_t

When finished, don’t forget to remove the permissive bit (semanage permissive -d testdom_t) and unload/remove the SELinux policy module.

And that’s it. If the daemon is now started (through a standard init script) it will run as testdom_t and everything it does will be logged, but not enforced by SELinux. That might even help in understanding the application better.

Posts for Sunday, May 10, 2015


Audit buffering and rate limiting

Be it because of SELinux experiments, or through general audit experiments, sometimes you’ll get in touch with a message similar to the following:

audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=44395 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded

The message shows up when certain audit events could not be logged through the audit subsystem. Depending on the system configuration, they might be either ignored, sent through the kernel logging infrastructure or even have the system panic. And if the messages are sent to the kernel log then they might show up, but even that log has its limitations, which can lead to output similar to the following:

__ratelimit: 53 callbacks suppressed

In this post, I want to give some pointers in configuring the audit subsystem as well as understand these messages…

There is auditd and kauditd

If you take a look at the audit processes running on the system, you’ll notice that (assuming Linux auditing is used of course) two processes are running:

# ps -ef | grep audit
root      1483     1  0 10:11 ?        00:00:00 /sbin/auditd
root      1486     2  0 10:11 ?        00:00:00 [kauditd]

The /sbin/auditd daemon is the user-space audit daemon. It registers itself with the Linux kernel audit subsystem (through the audit netlink system), which responds with spawning the kauditd kernel thread/process. The fact that the process is a kernel-level one is why the kauditd is surrounded by brackets in the ps output.

Once this is done, audit messages are communicated through the netlink socket to the user-space audit daemon. For the detail-oriented people amongst you, look for the kauditd_send_skb() method in the kernel/audit.c file. Now, generated audit event messages are not directly relayed to the audit daemon – they are first queued in a sort-of backlog, which is where the backlog-related messages above come from.

Audit backlog queue

In the kernel-level audit subsystem, a socket buffer queue is used to hold audit events. Whenever a new audit event is received, it is logged and prepared to be added to this queue. Adding to this queue can be controlled through a few parameters.

The first parameter is the backlog limit. Be it through a kernel boot parameter (audit_backlog_limit=N) or through a message relayed by the user-space audit daemon (auditctl -b N), this limit will ensure that a queue cannot grow beyond a certain size (expressed in the amount of messages). If an audit event is logged which would grow the queue beyond this limit, then a failure occurs and is handled according to the system configuration (more on that later).

The second parameter is the rate limit. When more audit events are logged within a second than set through this parameter (which can be controlled through a message relayed by the user-space audit system, using auditctl -r N) then those audit events are not added to the queue. Instead, a failure occurs and is handled according to the system configuration.

Only when the limits are not reached is the message added to the queue, allowing the user-space audit daemon to consume those events and log those according to the audit configuration. There are some good resources on audit configuration available on the Internet. I find this SuSe chapter worth reading, but many others exist as well.

There is a useful command related to the subject of the audit backlog queue. It queries the audit subsystem for its current status:

# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=1483 rate_limit=0 backlog_limit=8192 lost=3 backlog=0

The command displays not only the audit state (enabled or not) but also the settings for rate limits (on the audit backlog) and backlog limit. It also shows how many events are currently still waiting in the backlog queue (which is zero in our case, so the audit user-space daemon has properly consumed and logged the audit events).

Failure handling

If an audit event cannot be logged, then this failure needs to be resolved. The Linux audit subsystem can be configured do either silently discard the message, switch to the kernel log subsystem, or panic. This can be configured through the audit user-space (auditctl -f [0..2]), but is usually left at the default (which is 1, being to switch to the kernel log subsystem).

Before that is done, the message is displayed which reveals the cause of the failure handling:

audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=44395 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded

In this case, the backlog queue was set to contain at most 320 entries (which is low for a production system) and more messages were being added (the Linux kernel in certain cases allows to have a few more entries than configured for performance and consistency reasons). The number of events already lost is displayed, as well as the current limitation settings. The message “backlog limit exceeded” can be “rate limit exceeded” if that was the limitation that was triggered.

Now, if the system is not configured to silently discard it, or to panic the system, then the “dropped” messages are sent to the kernel log subsystem. The calls however are also governed through a configurable limitation: it uses a rate limit which can be set through sysctl:

# sysctl -a | grep kernel.printk_rate
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10

In the above example, this system allows one message every 5 seconds, but does allow a burst of up to 10 messages at once. When the rate limitation kicks in, then the kernel will log (at most one per second) the number of suppressed events:

[40676.545099] __ratelimit: 246 callbacks suppressed

Although this limit is kernel-wide, not all kernel log events are governed through it. It is the caller subsystem (in our case, the audit subsystem) which is responsible for having its events governed through this rate limitation or not.

Finishing up

Before waving goodbye, I would like to point out that the backlog queue is a memory queue (and not on disk, Red Hat), just in case it wasn’t obvious. Increasing the queue size can result in more kernel memory consumption. Apparently, a practical size estimate is around 9000 bytes per message. On production systems, it is advised not to make this setting too low. I personally set it to 8192.

Lost audit events might result in difficulties for troubleshooting, which is the case when dealing with new or experimental SELinux policies. It would also result in missing security-important events. It is the audit subsystem, after all. So tune it properly, and enjoy the power of Linux’ audit subsystem.

Posts for Sunday, May 3, 2015

How to conference

(This text evolved over a few weeks in short bursts of work in my free time. Sadly I don’t really have the time to properly fix it, edit it and make it nice right now, I just need to get it out. So I apologize for the bad quality of writing/text.)

I have had the privilege to attend quite a few conferences in my life, some as speaker/presenter, some as participant. After the first few times where these events can wash over you like a tsunami I started realizing how different these things we call conferences really are now wide the spectrum of conferency things actually is.

In this short brainfart I am writing mostly from the perspective of a presenter because that is an area where I think I can actually add a few things that maybe have not been said before by others. In the last few months I have seen quite a few interesting reflections on how to improve the concept and implementation of “conference” especially when it comes to accessibility and respect, I encourage you to for example check out this post by the organizers on how little it actually cost them to make a safer space for the participants and this ttw15 inspired reflection on conferences as a technoligy by Sky Croeser . There is a lot more of that out there.

So let me just quickly state the obvious before I dive into these notes: If you provide any sort of public space (as a conference) you do not only need an anti-harassment policy but also clear mechanisms for victims to report problems and have them taken care of. Many conferences these days have a policy in place but no real mechanics to enforce them, which is ridiculous in this day and age. You also need to think about whether people with disabilities can actually enter and navigate your space, have access to bathrooms. Offering some sort of drinkable water supply and if possible snacks to people with limited mobility on site is also something often overlooked (it can be hard to “just jump over to the next store to buy water if you can’t walk that far). And there is always the question of offering some kind of child care to enable parents to participate. The list of these issues is long and I get that not every conference has the resources to tackle all of them, you should think about them when thinking about organizing one though.

So let’s talk about presenting. I enjoy presenting at conferences. It feels like I contribute as well as have a good reason to attend (apart from my own indulgence). It also forces me to boil an idea or a thought down into a presentable piece which helps me think on. I tend not to do the same talk multiple times because it bores me and I find it a waste of time and limited possible space for ideas to redo a session that already has a publicly available recording. I also try to finish a rough version of the session a few or at least one week in advance to have time to properly prepare the session1.

The usual process to land a presentation at a conference is (if you are not some big shot invited speaker) to submit a session proposal to the Call for Participation. After a while you get a notice of acceptance or rejection and if accepted you write your paper and prepare your presentation. Finally you attend the conference and give your talk. Sounds simple but there are a few things that go wrong a lot or at least don’t work ideally.

Now it’s obvious that different conferences have a very different amount of resources to commit to preparing the program/schedule: Some conferences are very expensive, with goodie bags for attendants and a lot of flashy tech and whatever. Some run on almost no budget with all required time and work provided by voluntaries2. From my experience the level of professionalism in how presenters are prepared does in no way correlate to the amount of money available. But since not every conference can provide everything I write down here I’ll separate this list in two blocks: Mandatory and Desirable.


These are the things that I find completely necessary in order to deliver a presentation that actually fits into the conference.

Write a clear and somewhat specific CfP

Writing a CfP is hard: You don’t want to narrow the scope too much in order to get new and exciting ideas and projects in while still being clear enough on what it is your conference  if about and trying to achieve. I feel that some CfPs err on the side of being less specific which makes it very hard to write a proposal that concisely describes a session. Add direct questions you’d like to see answers to or discussion about at the conference. Also provide examples on what you do not care about. Help potential submitters to understand what you really want: It helps to avoid a lot of wasted work for conference organizers as well as submitters. Obviously you can always add a “catchall” but having precise questions helps to see if that weird and spacey idea really fits or not.

Be transparent about the criteria for acceptance

At some point you will have to make a decision on who to accept and who not to. Those decisions are hard and more often than not you will have to reject proposals that you’d have liked to have because there just isn’t enough room. But you also will have to decide based on whatever you consider to be quality. But proposal quality is not an objective metric. Some people look at the person presenting and if they have experience or a big name, others don’t. Some value implementation or hard data a lot, others like to see a certain familiarity with certain technical terms. Be transparent on what constitutes a “good” proposal for your team.

Deadlines aren’t enough, also commit to a schedule

Every CfP comes with a deadline. Get your proposal in in time or you’re out. That’s fair. Where some conferences have a harder time is to commit to a schedule for when notifications of acceptance will be send to the presenters. “At the end of May” or “a few weeks later” might sound specific enough while giving the organizers enough room to take extra time to evaluate proposals, it can be very problematic for presenters though: You have to keep your schedule open during the conference, you might have to book accommodation, organize a place for your cat or your kids to stay. The more embedded you are in a dependency network of jobs and family etc. “just waiting” can be actually very damaging and problematic. As organization try to commit to a plan of when people will get their responses so presenters know how long to keep their schedule open. Deadlines sometimes can’t be held and that’s fine, be transparent and postpone acceptions/rejections for a week or two, but keep submitters in the loop.

Brief your presenters

The weeks before the actual conference are always full with all kinds of things that need to happen before the event can actually take place. Which is why, after having send out acceptances/rejections, often there is some radio silence between the conference and the presenters. There’s usually another email before the event telling presenters what kind of equipment they can expect to be present and other technical and organizational details (schedule, file format for presentation slides etc), but they often lack a briefing for the presenters about what kind of social environment to expect. What kind of people will probably be there, what age, which level of expertise, which backgrounds, which experiences will they probably have? What kind of language should the presentation use, should jargon be avoided or used? What’s the “culture” of the scene attending the conf3? It’s hard to guess especially when a presenter is attending an event for the fist time. What would you like your presenters to do? Speak for the whole time slot (maybe with a bunch of questions afterwards)? Generate an actual debate/discussion? Tell presenters what to expect and what you expect to help them build a better presentation for that specific situation.


Certain things are desirable but can’t always be done by every conference. Especially smaller, volunteer-run conferences can’t always do this but I still thought I’d mention them.

Feedback for Proposals

Building a schedule and deciding what to put in it is hard, especially saying “no”, but that’s just part of the job. There are many reasons for a proposal being rejected, no all of them very pleasant to the submitter or the organization. Still I’d wish to get some sort of specific feedback instead of a blanket statement like “yeah we had so much good stuff, we#d love to have taken it all but you just weren’t lucky”. There are things to help submitters to put forward a better presentation proposal the next time without being insulting: “You proposal wasn’t clear and specific enough, we couldn’t understand what your point was”, “We felt your proposal was good but didn’t fit into this conference because of $reason. Maybe try $otherConf?”, “We already filled our quota of white dudes. Your presentation proposal was fine, the others by white males were just better for this conference, so better luck next time!”. Help people understand your reasoning, not by sending out 2 pages of comments on a 500 word proposal, but by giving an actual reason that’s not just a feel-good-marketing term. Still: If possible stay polite, especially if it’s people’s first time very negative feedback can be devastating.

Describe dress code

This one might sound weird but it’s been a problem for me a few times already: Tell presenters about the dress code at your conference. Yeah “there’s none”. Sure. That just means that there is no explicit but an implicit code. Especially when you haven’t been at an event before it’s sometimes really hard to estimate what kind of clothing is expected and being overdressed can be a problem just as being underdressed. Give presenters a description of the spectrum of acceptable clothing especially when you are not sure if they have the same background as the other people at the event.

Drop the motto if it doesn’t mean anything

Oh conference mottos, how I hate thee. At some point many conferences, even those not run by marketing people, adopted the policy of having a different motto each year. These mottos are supposed to represent the vibe, the general topic or impetus of a conference for that specific year. But mostly they don’t mean anything. Which you realize when looking at the program which really has nothing to do with it.

Trying to write a good proposal for a conference is hard so I try to understand the conference as good as possible before deciding how to present a proposal or an idea. So if there is a motto, I try to find something connected to it to put forward my idea. It actually makes it a lot harder to present your session because you have to look at it through the motto lense. And it really sucks when you see afterwards that all that work was for nothing. If you can’t drop the motto, put somewhere in the CfP that it’s not really relevant for the conference to save everybody some time and work.

These are all things that I’d like more conferences to do strictly speaking from a presenter’s perspective. But conferences as a technology, as a thing have many more stakeholders and getting all their needs met especially with limited resources is far from easy. I have a few thoughts on the general format dominant at conferences (a hour of someone lecturing) that’s far from the silver bullet it’s often made out to be but that might be for a different post.

Also: These are just a bunch of notes I wrote down and lack some editing etc. but sadly I am very pressed for time right now so I apologize again for this somewhat random stream of thoughts.

Photo by Cydcor

  1. Having the presentation look as if the presenter(s) are seeing it the first time and “improvise” actually infuriates me as an attendant. It’s the most obvious signal of disrespect for your audience’s time you can send.
  2. We could go very deep into the whole self-exploitation angle of the conference circus here – especially when it comes to some hacker/tech/internet conferences but we’ll leave that for another time
  3. At ttw15 for example some people read their pre-written presentation from paper. On the conferences in the tech context I usually go to, that would be a huge no-no, but certain domains seem to do it that way.

Flattr this!

Posts for Thursday, April 30, 2015


Use change management when you are using SELinux to its fullest

If you are using SELinux on production systems (with which I mean systems that you offer services with towards customers or other parties beyond you, yourself and your ego), please consider proper change management if you don’t do already. SELinux is a very sensitive security subsystem – not in the sense that it easily fails, but because it is very fine-grained and as such can easily stop applications from running when their behavior changes just a tiny bit.

Sensitivity of SELinux

SELinux is a wonderful security measure for Linux systems that can prevent successful exploitation of vulnerabilities or misconfigurations. Of course, it is not the sole security measure that systems should take. Proper secure configuration of services, least privilege accounts, kernel-level mitigations such as grSecurity and more are other measures that certainly need to be taken if you really find system security to be a worthy goal to attain. But I’m not going to talk about those others right now. What I am going to focus on is SELinux, and how sensitive it is to changes.

An important functionality of SELinux to understand is that it segregates the security control system itself (the SELinux subsystem) from its configuration (the policy). The security control system itself is relatively small, and focuses on enforcement of the policy and logging (either because the policy asks to log something, or because something is prevented, or because an error occurred). The most difficult part of handling SELinux on a system is not enabling or interacting with it. No, it is its policy.

The policy is also what makes SELinux so darn sensitive for small system changes (or behavior that is not either normal, or at least not allowed through the existing policy). Let me explain with a small situation that I recently had.

Case in point: Switching an IP address

A case that beautifully shows how sensitive SELinux can be is an IP address change. My systems all obtain their IP address (at least for IPv4) from a DHCP system. This is of course acceptable behavior as otherwise my systems would never be able to boot up successfully anyway. The SELinux policy that I run also allows this without any hindrance. So that was not a problem.

Yet recently I had to switch an IP address for a system in production. All the services I run are set up in a dual-active mode, so I started with the change by draining the services to the second system, shutting down the service and then (after reconfiguring the DHCP system to now provide a different IP address) reload the network configuration. And then it happened – the DHCP client just stalled.

As the change failed, I updated the DHCP system again to deliver the old IP address and then reloaded the network configuration on the client. Again, it failed. Dumbstruck, I looked at the AVC denials and lo and behold, I notice a dig process running in a DHCP client related domain that is trying to do UDP binds, which the policy (at that time) did not allow. But why now suddenly, after all – this system was running happily for more than a year already (and with occasional reboots for kernel updates).

I won’t bore you with the investigation. It boils down to the fact that the DHCP client detected a change compared to previous startups, and was configured to run a few hooks as additional steps in the IP lease setup. As these hooks were never ran previously, the policy was never challenged to face this. And since the address change occurred a revert to the previous situation didn’t work either (as its previous state information was already deleted).

I was able to revert the client (which is a virtual guest in KVM) to the situation right before the change (thank you savevm and loadvm functionality) so that I could work on the policy first in a non-production environment so that the next change attempt was successful.

Change management

The previous situation might be “solved” by temporarily putting the DHCP client domain in permissive mode just for the change and then back. But that is ignoring the issue, and unless you have perfect operational documentation that you always read before making system or configuration changes, I doubt that you’ll remember this for the next time.

The case is also a good example on the sensitivity of SELinux. It is not just when software is being upgraded. Every change (be it in configuration, behavior or operational activity) might result in a situation that is new for the loaded SELinux policy. As the default action in SELinux is to deny everything, this will result in unexpected results on the system. Sometimes very visible (no IP address obtained), sometimes hidden behind some weird behavior (hostname correctly set but not the domainname) or perhaps not even noticed until far later. Compare it to the firewall rule configurations: you might be able to easily confirm that standard flows are still passed through, but how are you certain that fallback flows or one-in-a-month connection setups are not suddenly prevented from happening.

A somewhat better solution than just temporarily disabling SELinux access controls for a domain is to look into proper change management. Whenever a change has to be done, make sure that you

  • can easily revert the change back to the previous situation (backups!)
  • have tested the change on a non-vital (preproduction) system first

These two principles are pretty vital when you are serious about using SELinux in production. I’m not talking about a system that hardly has any fine-grained policies, like where most of the system’s services are running in “unconfined” domains (although that’s still better than not running with SELinux at all), but where you are truly trying to put a least privilege policy in place for all processes and services.

Being able to revert a change allows you to quickly get a service up and running again so that customers are not affected by the change (and potential issues) for long time. First fix the service, then fix the problem. If you are an engineer like me, you might rather focus on the problem (and a permanent, correct solution) first. But that’s wrong – always first make sure that the customers are not affected by it. Revert and put the service back up, and then investigate so that the next change attempt will not go wrong anymore.

Having a multi-master setup might give some more leeway into investigating issues (as the service itself is not disrupted) so in the case mentioned above I would probably have tried fixing the issue immediately anyway if it wasn’t policy-based. But most users do not have truly multi-master service setups.

Being able to test (and retest) changes in non-production also allows you to focus on automation (so that changes can be done faster and in a repeated, predictable and qualitative manner), regression testing as well as change accumulation testing.

You don’t have time for that?

Be honest with yourself. If you support services for others (be it in a paid-for manner or because you support an organization in your free time) you’ll quickly learn that service availability is one of the most qualitative aspects of what you do. No matter what mess is behind it, most users don’t see all that. All they see is the service itself (and its performance / features). If a change you wanted to make made a service unavailable for hours, users will notice. And if the change wasn’t communicated up front or it is the n-th time that this downtime occurs, they will start asking questions you rather not hear.

Using a non-production environment is not that much of an issue if the infrastructure you work with supports bare metal restores, or snapshot/cloning (in case of VMs). After doing those a couple of times, you’ll easily find that you can create a non-production environment from the production one. Or, you can go for a permanent non-production environment (although you’ll need to take care that this environment is at all times representative for the production systems).

And regarding qualitative changes, I really recommend to use a configuration management system. I recently switched from Puppet to Saltstack and have yet to use the latter to its fullest set (most of what I do is still scripted), but it is growing on me and I’m pretty convinced that I’ll have the majority of my change management scripts removed by the end of this year towards Saltstack-based configurations. And that’ll allow me to automate changes and thus provide a more qualitative service offering.

With SELinux, of course.

Posts for Monday, April 27, 2015


Moving closer to 2.4 stabilization

The SELinux userspace project has released version 2.4 in february this year, after release candidates have been tested for half a year. After its release, we at the Gentoo Hardened project have been working hard to integrate it within Gentoo. This effort has been made a bit more difficult due to the migration of the policy store from one location to another while at the same time switching to HLL- and CIL based builds.

Lately, 2.4 itself has been pretty stable, and we’re focusing on the proper migration from 2.3 to 2.4. The SELinux policy has been adjusted to allow the migrations to work, and a few final fixes are being tested so that we can safely transition our stable users from 2.3 to 2.4. Hopefully we’ll be able to stabilize the userspace this month or beginning of next month.


Practical fault detection on timeseries part 2: first macros and templates

In the previous fault detection article, we saw how we can cover a lot of ground in fault detection with simple methods and technology that is available today. It had an example of a simple but effective approach to find sudden spikes (peaks and drops) within fluctuating time series. This post explains the continuation of that work and provides you the means to implement this yourself with minimal effort. I'm sharing with you:
  • Bosun macros which detect our most common not-trivially-detectable symptoms of problems
  • Bosun notification template which provides a decent amount of information
  • Grafana and Graph-Explorer dashboards and integration for further troubleshooting
We reuse this stuff for a variety of cases where the data behaves similarly and I suspect that you will be able to apply this to a bunch of your monitoring targets as well.

Target use case

As in the previous article, we focus on the specific category of timeseries metrics driven by user activity. Those series are expected to fluctuate in at least some kind of (usually daily) pattern, but is expected to have a certain smoothness to it. Think web requests per second or uploads per minute. There are a few characteristics that are considered faulty or at least worth our attention:
looks good
consistent pattern
consistent smoothness
sudden deviation (spike)
Almost always something broke or choked.
could also be pointing up. ~ peaks and valleys
increased erraticness
Sometimes natural
often result of performance issues
lower values than usual (in the third cycle)
Often caused by changes in code or config, sometimes innocent. But best to alert operator in any case [*]

[*] Note that some regular patterns can look like this as well. For example weekend traffic lower than weekdays, etc. We see this a lot.
The illustrations don't portray this for simplicity. But the alerting logic below supports this just fine by comparing to same day last week instead of yesterday, etc.

Introducing the new approach

The previous article demonstrated using graphite to compute standard deviation. This let us alert on the erraticness of the series in general and as a particularly interesting side-effect, on spikes up and down. The new approach is more refined and concrete by leveraging some of bosun's and Grafana's strengths. We can't always detect the last case above via erraticness checking (a lower amount may be introduced gradually, not via a sudden drop) so now we monitor for that as well, covering all cases above. We use
  • Bosun macros which encapsulate all the querying and processing
  • Bosun template for notifications
  • A generic Grafana dashboard which aids in troubleshooting
We can then leverage this for various use cases, as long as the expectations of the data are as outlined above. We use this for web traffic, volume of log messages, uploads, telemetry traffic, etc. For each case we simply define the graphite queries and some parameters and leverage the existing mentioned Bosun and Grafana configuration.

The best way to introduce this is probably by showing how a notification looks like:

(image redacted to hide confidential information
the numbers are not accurate and for demonstration purposes only)

As you can tell by the sections, we look at some global data (for example "all web traffic", "all log messages", etc), and also by data segregated by a particular dimension (for example web traffic by country, log messages by key, etc)
To cover all problematic cases outlined above, we do 3 different checks: (note, everything is parametrized so you can tune it, see further down)

  • Global volume: comparing the median value of the last 60 minutes or so against the corresponding 60 minutes last week and expressing it as a "strength ratio". Anything below a given threshold such as 0.8 is alerted on
  • Global erraticness. To find all forms of erraticness (increased deviation), we use a refined formula. See details below. A graph of the input data is included so you can visually verify the series
  • On the segregated data: compare current (hour or so) median against median derived from the corresponding hours during the past few weeks, and only allow a certain amount of standard deviations difference
If any, or multiple of these conditions are in warning or critical state, we get 1 alert that gives us all the information we need.
Note the various links to GE (Graph-Explorer) and Grafana for timeshifts. The Graph-Explorer links are just standard GEQL queries, I usually use this if i want to be easily manage what I'm viewing (compare against other countries, adjust time interval, etc) because that's what GE is really good at. The timeshift view is a Grafana dashboard that takes in a Graphite expression as a template variable, and can hence be set via a GET parameter by using the url
It shows the current past week as red dots, and the past weeks before that as timeshifts in various shades of blue representing the age of the data. (darker is older).

This allows us to easily spot when traffic becomes too low, overly erratic, etc as this example shows:

Getting started

Note: I Won't explain the details of the bosun configuration. Familiarity with bosun is assumed. The bosun documentation is pretty complete.

Gist with bosun macro, template, example use, and Grafana dashboard definition. Load the bosun stuff in your bosun.conf and import the dashboard in Grafana.

The pieces fit together like so:
  • The alert is where we define the graphite queries, the name of the dimension segregated by (used in template), how long the periods are, what the various thresholds are and the expressions to be fed into Grafana and Graph-Explorer.
    It also lets you set an importance which controls the sorting of the segregated entries in the notification (see screenshot). By default it is based on the historical median of the values but you could override this. For example for a particular alert we maintain a lookup table with custom importance values.
  • The macros are split in two:
    1. dm-load loads all the initial data based on your queries and computes a bunch of the numbers.
    2. dm-logic does some final computations and evaluates the warning and critical state expressions.
    They are split so that your alerting rule can leverage the returned tags from the queries in dm-load to use a lookup table to set the importance variable or other thresholds, such as s_min_med_diff on a case-by-case basis, before calling dm-logic.
    We warn if one or more segregated items didn't meet their median requirements, and if erraticness exceeds its threshold (note that the latter can be disabled).
    Critical is when more than the specified number of segregated items didn't meet their median requirements, the global volume didn't meet the strength ratio, or if erraticness is enabled and above the critical threshold.
  • The template is evaluated and generates the notification like shown above
  • Links to Grafana (timeshift) and GE are generated in the notification to make it easy to start troubleshooting

Erraticness formula refinements

You may notice that the formula has changed to
(deviation-now * median-historical) /
((deviation-historical * median-now) + 0.01)
  • Current deviation is compared to an automatically chosen historical deviation value (so no more need to manually set this)
  • Accounts for difference in volume: for example if traffic at any point is much higher, we can also expect the deviation to be higher. With the previous formula we would have cases where in the past the numbers were very low, and naturally the deviation then was low and not a reasonable standard to be held against when traffic is higher, resulting in trigger happy alerting with false positives.
    Now we give a fair weight to the deviation ratio by making it inversely proportional to the median ratio
  • The + 0.01 is to avoid division by zero

Still far from perfect

While this has been very helpful to us, I want to highlight a few things that could be improved.

In conclusion

I know many people are struggling with poor alerting rules (static thresholds?)
As I explained in the previous article I fondly believe that the commonly cited solutions (anomaly detection via machine learning) are a very difficult endeavor and results can be achieved much quicker and simpler.
While this only focuses on one class of timeseries (it won't work on diskspace metrics for example) I found this class to be in the most dire need of better fault detection. Hopefully this is useful to you. Good luck and let me know how it goes!

Posts for Tuesday, April 7, 2015

Smartwatches and the “veil of the analogue”

When I was about 5 my father gave me my first watch. It was very cool, had a colorful watch band and had a football player1 on its watch face. And it was a manual watch that I had to wind up every week which I found utterly fascinating: Just a few turns of the small wheel and it “magically” worked for a long time (when you’re 4 or 5 a week is long!).

I don’t know where that watch ended up or how long I used it. I went through some more or less cheap ass digital watches during my time in school. Ugly, plasticy devices that had one thing going for them: They were digital. Precise. You got the exact time with a short glance and not just a rough estimation. Which felt “better” in a weird way, like the future as it was shown in the 70ies SciFi movies, with clear and precise, sometimes curved lines. It felt right in a way. A little bit of the promise of a more developed future. Star Trek on my wrist. In some small and maybe not even meaningful way it felt powerful. As if that quartz driven precision would somehow make the constant flow of time more manageable.

When I went to study computer science I always had some form of cheap watch on my arm to tell the time since I refused to get a cell phone for the longest time (for a bunch of mostly stupid and pretentious reasons). But when some software development gig that me and a friend worked on finished with a bonus payment I got a “real” watch.



I wore that thing for a few years and loved it. Not just because it reminded me of Gir from Invader Zim (even though that was a big part of it). The fallback to an analog watch felt like a nice contrast to the digital spaces I spend more and more time in. The two watch faces were so small that you could hardly read a precise time on them and it had two of them. It was a strangely out-of-time anchor to the past while I dove head on into what I perceived to be “the future“.

A cellphone came, then a feature phone and finally smartphones and at some point I stopped replacing the batteries in my watch. I carried my phone around with me anyways so why add another device to the mix that had only that one feature? It felt pointless especially with me being sort of the prototypical nerd back then explicitly not “caring” about “looks and style” and all that stuff while sticking very closely to the codices and fashions of that subculture I identified with back then. If you gather from this that I was probably just another insecure dude in the beginning of his twenties a little too full of his own bullshit you would probably be correct.

But about 6 months ago things changed and I got another watch. A “smart” one even (we’ll come back to that word later). Here’s some kind of “review” or a summary of the things I learned from wearing one for those few months. But don’t be afraid, this isn’t a techbloggy text. Nobody cares about pixel counts and the megahertzs of whatever core processors. Given the state of the art the answer to most inquiries about features and performance is usually “enough” or “more than enough”. It’s also not about the different software platforms and if Google’s is better than Apple’s (which very few people have even used for more than a few minutes if at all) because given the relative newness of the form factor and class of device it will look very different in a few years anyways. And for most interesting questions about the devices the technical configurations just don’t matter regardless of what the producer’s advertising is trying to tell you.

I’ll write about how a smartwatch can change your relationship to your phone, your interactions with people and how smartwatches cast a light on where tech might be in a few years. Not from a business, startup, crowdfunding perspective but as a few loosely connected thoughts on humans and their personal devices as sociotechnical systems.

I should also add a short disclaimer. I understand that me posting about jewelry and rather expensive devices could be read as classist. I live a very privileged life with – at least at the moment – enough income to fund frivolities such as a way more expensive than necessary smart watch, a smart phone etc. but I do believe that my experience with these device can help understanding and modeling different uses of technology, their social context and their meaning. These devices will grow cheaper including more and more people (at least in certain parts of the world). But I am aware of the slightly messed up position I write this from.

Let’s dive in (finally!).

So. What is a smartwatch, really?

Wearables are a very hot topic right now2. Many wearables are … well let’s not say stupid, let’s call them very narrowly focused. Step counters are very popular these days for example turning your daily movement into a few numbers to store and compare and optimize. Some of these things try to measure heart rates similar low hanging fruits as well.

On the other end of the spectrum we have our smart phones and tablet computers or maybe even laptops which we carry around each day. Many might not consider their phone a wearable because it resides in your pocket or satchel but in the end it is more than just some object you schlep around all day but – for many if not most of us – an integral piece of our mental exoskeleton. Just ask people whose phone needs a repair longer than a few hours.

Smartwatches are somewhere between those two extremes. Many modern examples of this class of devices include a few of the sensors typically associated with dumb wearables: A heartrate monitor or a pedometer (fancytalk for step counter) for example. But smart watches can do more can install apps and provide features that make them feel very capable … unless you forget your phone.

Because in the end a smartwatch is just a different view into your smartphone. A smaller screen attached to a somewhat more convenient location of your body. Sure, there are great apps for smartwatches. I got one that makes my hand movements give me Jedi force powers turning certain movements into configurable actions. Another app is just a very simple recorder for audio memos. There are calculators, dice rolling apps and many more but their usefulness is usually very limited. No, let’s again say focused. And that is a good thing.

Without a phone connected my watch falls back to one of its dumbest and surprisingly most useful feature: It shows me the time and my agenda.

You can imagine the sort of look that my wife gave me when I proclaimed this fundamental understanding of my new plaything. “So the killer feature of that smart device is showing the time? ” she asked jokingly. But it’s a little more complex. My smartwatch allows me to check certain things (time, agenda, certain notifications from apps I authorized) without picking up my phone which can – and all too often does – pull your attention in like a black hole. You just wanted to check the time but there’s this notification and someone retweeted you and what’s going on on Facebook … what was I supposed to do here?

I’m not a critic of our networked culture in general, not one of the neo-luddites trying to frame being “offline” in some way as the better or more fulfilling mode of life. But the simplicity that the small form factor and screen size of smartwatches enforces, the reduction to very simple interactions can help to stay focused when you want to.

Most apps on my watch are mere extensions of the apps running on my phone. And that’s actually precisely what I want and not the drawback it’s sometimes made out to be. I get a message pushed to my wrist and can react on it with a few prepared response templates or by using voice recognition (with all the funny problems that come with it). But again: I can stay focused on whatever I am doing now (such as riding my bike in traffic) while still being able to tell the person I am meeting that I’ll be there in 5 minutes. The app I use to record my running shows me certain stats on my wrist, I can switch to the next podcast or music track in my queue while still keeping my attention on the road.

I don’t know when I printed my last map screenshot or route description. When smartphones and navigation became widely available there was not longer the need to reduce a place you were going to to a set of a handful of predefined railways to created for yourself in order to  not get lost. You drop out of the train or plane and the unknown city opens up to you like the inviting and fascinating place it probably is. You can just start walking since you know that even if you don’t speak the language perfectly you’ll find your way. My smartwatch does that while allowing me to keep my phone in my pocket allowing me to look less like a tourist or target. When the little black thing on my arm vibrates I check it to see where to turn, apart from that I just keep walking.

Sure, there will be apps coming that use these watches in more creative and useful ways. That thrive not in spite of but because of the tiny form factor. But that’s mostly a bonus and if it doesn’t happen I’d be fine as well. Because the watch as a simplified, ultra-reduced, ultra-focused remote to my mobile digital brain is feature enough. Where digital watches used to give me an intangible feeling of control of time the smart-ish watch does actually help me feel augmented by my devices in a way that doesn’t try to capture as much of my attention as smartphones tend to do. The watch is not a smaller smartphone but your phone’s little helper. The small and agile Robin to the somewhat clunky Batman in your pocket3.


Any new technology has to carve out its niche and fight for acceptance. And some don’t and die for a plethora of reasons (Minidisk, I always liked you). There are many reasons why people, mostly “experts” of some sort, don’t believe that smartwatches will gain any traction.

You have to recharge them every night, my watch runs for weeks, months, years!” Yeah. And on Tuesday it’s darker than at night. Oh, we weren’t doing the whole wrong comparison thing? Damn. Just as people learned to charge their phones every night they’ll get used to throwing their watch on a charger at night. My watch gets charged wirelessly with a Qi standard charger that sets you back about 10 bucks. It’s a non-issue.

But it doesn’t do a lot without a phone! It needs its own camera, internet connection, coffee maker and washing machine!” Nope. Simplicity and reduction is what makes that class of devices interesting and useful. I don’t need a half-assed smartphone on my arm when I have a good one in my pocket. I need something that helps my use my actual device better. Another device means all kinds of annoyances. Just think about synchronization of data.

I am in the lucky position not to have to deal with tech writers and pundits in all of the facets of my live. What I learned from interacting with non-techy people and the watch is actually not that surprising if you think about it: A smart watch is a lot less irritating and invasive than a smart phone.

There are friends where I know I can just look at my phone while we hang out and they’ll not consider it an insult or affront. They might enjoy the break from talking, might want to check a few things themselves or just relax for a second without having to entertain me or the others in the room. But not everybody feels that way (and why should they, it’s not like submerging yourself in the digital is the only right way to live). In those situations the look at the watch is an established and accepted practice mostly unless you check your watch every minute.

Some tech people tend to ignore the social. They might try to press it into services and data but often seem to overlook any sort of information a social act transports apart from the obvious. In pre-digital worlds checking your watch every few minutes was sometimes considered rude or would be read as a signal to leave your hosts etc. But where the glance at the watch is merely the acknowledgement of the existence of time and a world outside of the current situation, getting out your smartphone puts the outside world into focus making the people you share a physical space with just a blur in the corner of your vision.

Of course it’s your right to check your phone whenever you want just as people can be insulted or at least irritated by it. The way a smart watch can serve as a proxy of your access to your digital identity and network from your physical location and context it can help you communicate that you value the moment without feeling disconnected. Especially since neither being very digitally connected nor valuing physical meetings more highly is “better”, having this sort of a reduced stub of the digital that closely on you can serve as a good compromise for these situations.

A smartwatch is accepted because it is a watch. And we as a culture know watches. Sure, some very techy, clunky, funky looking devices break that “veil of the analogue” by screaming “I AM TECHNOLOGY, FEAR ME” through their design. But the more simple versions that avoid the plasticy look of Casio watches on LSD are often even overlooked and not even perceived as technology (and therefore as an irritation or even dangerous) by people who are sceptical of technology. That’s the problem devices such as Google’s Glass project have who also have very interesting and potentially beneficial use cases but look so undeniably alien that everyone expects a laser gun to appear. And that’s where smart watches embed themselves into existing social norms and practices. by looking like the past and not screaming FUTURE all too loud.

Body Area Network and the Future

What does this mean for the Future(tm)?  The idea of the Body Area Network and the Personal Area Network do already exist: We are more and more turning into digital cyborgs4 creating our own personal “cloud” and network of data and services along the axes of and around our physical bodies.

Right now Smartphones seem to be some sort of Hub we carry around. The little monolith containing our data, internet access and main mobile interface to our digital self. Other devices connect to the hub, exchange data and use the services it provides (such as Internet connectivity or a camera). But looking at things like Google’s project Ara a different idea emerges.

Ara is a module smartphone platform that allows you to add, remove and change the hardware modules of your phone at runtime. While it’s mostly framed as a way for people to buy their devices in parts upgrading it when the personal financial situation allows it the modular approach also has different trajectories influencing how our BANs and PANs might look in a few years.

Changing a phone can be annoying and/or time consuming. The backup software might have failed or forgotten something valuable. Maybe an app isn’t available on the new system or the newer version is incompatible to the data structure the old version left in your backup. We suffer through it because many of us rely on our personal information hubs making us potentially more capable (or at least giving us the feeling of being that).

Understanding smart watches as reduced, minimal, simplified interfaces to our data, looking at wearables as very specific data gathering or displaying devices it seems to make sense to centralize your data on one device that your other devices just connect to. These days we work around that issue with tools such as Dropbox and other similar cloud sync services trying to keep all our devices up-to-data and sometimes failing horribly. But what if every new device just integrates into your BAN/PAN, connects to your data store and either contributes to it or gives you a different view on it? In that world wearables could become even “dumber” while still appearing to the user very “smart” (and we know that to the user, the interface is the product).

The smartphones that we use are built with healthy people in mind with nimble fingers and good eyesight. Smart watches illustrate quite effectively that the idea of the one device for every situation has overstayed its welcome somewhat. That different social or even personal circumstances require or or benefit from different styles and types of interfaces. Making it easier for people to find the right interfaces for their needs, for the situations they find themselves in will be the challenge of  the next few years. Watches might not always look like something we’d call a watch today. Maybe they’ll evolve into gloves, or just rings. Maybe the piercing some wear in their upper lip will contain an antenna to amplify the connectivity of the BAN/PAN.

Where Ara tries making phones more modular, wearables – when done right – show that we can benefit a lot from modularizing the mobile access to our digital self. Which will create new subtle but powerful signals: Leaving certain types of interfaces at home or disabled on the table to communicate an ephemeral quality of a situation, only using interfaces focused on shared experience of the self and the other when being with another person creating a new kind of intimacy.


But right now it’s just a watch. With some extras. Useful extras though. You wouldn’t believe how often the app projecting me the video from my smartphone camera to my wrist has been useful to find something that has fallen behind the furniture. But none of them really, honestly legitimizing the price of the devices.

But the price will fall and new wearables will pop up. If you have the opportunity, try them out for a while. Not by fiddling around on a tiny display playing around with flashy but ultimately useless apps but by integrating them into your day for a few weeks. Don’t believe any review written with less than a few weeks of actual use.

  1. Football in the meaning most of the world uses it. The one where you play by kicking a ball around into goals.
  2. It’s so hot that my bullshit-o-meter has reached a new peak while reading the term “wearable clothing” somewhere recently.
  3. that sounded dirtier that it was supposed to
  4. we have always been cyborgs, beings combined from biology and culture and technology so that isn’t actually surprising

Flattr this!

Posts for Saturday, April 4, 2015

Paludis 2.4.0 Released

Paludis 2.4.0 has been released:

  • Bug fixes.
  • We now use Ruby 2.2, unless –with-ruby-version is specified.

Filed under: paludis releases

Posts for Sunday, March 29, 2015

A wise choice? Github as infrastructure

So more and more projects are using github as infrastructure. One of the biggest cases I’ve seen is the Go programming language which allows you to specify “imports” directly hosted on code sharing sites like github and “go get” to get them all before compilation, but also lots of other projects are adopting it like Vim’s Vundle plugin manage which also allows fetching and updating of plugins directly from github. Also I wouldn’t be surprised if one or more other languages’ package managers from pip to npm do this too. I know it’s pretty easy and now cool to do this but…

It isn’t actually infrastructure grade. And that is hilighted well in event’s like this week when they are suffering continuals outages from a massive DDOS attack that some news sources are suspecting might be nation-state based.

How much fun is your ops having deploying your new service when half it’s dependencies are being pulled directly from github which is unavailable? Bit of a strange blocker hm?

Posts for Wednesday, March 25, 2015

HowTo: Permanently redirect a request with parameter consideration in Tengine/NginX

Well, this one gave me a super hard time. I looked everywhere and found nothing. There is a lot of misinformation.

As usual, the Nginx and Funtoo communities helped me. Thanks to:

  • MTecknology in #nginx @ Freenode
  • Tracerneo in #funtoo @ Freenode

So, how do we do this? Easy, we use a map:

    # get ready for long redirects
    map_hash_bucket_size 256;
    map_hash_max_size 4092;

    # create the map
    map $request_uri $newuri {
        default 0;

        /index.php?test=1 /yes;

    server {
        listen *;
        root /srv/www/php/test/public;

        # permanent redirect
        if ($newuri) {
            return 301 $newuri;

        index index.php index.html;
        autoindex on;

        include include.d/php.conf;

        access_log /var/log/tengine/php-access.log;
        error_log /var/log/tengine/php-error.log;

So, basically, you want to use the $request_uri in order to catch the the uri with it’s parameters. I wasted all day figuring out why $uri didn’t have this. It turns out it discards the parameters… anyway.

This one was a hard one to find. Please, share and improve!


Posts for Friday, March 6, 2015


Trying out Pelican, part one

One of the goals I’ve set myself to do this year (not as a new year resolution though, I *really* want to accomplish this ;-) is to move my blog from WordPress to a statically built website. And Pelican looks to be a good solution to do so. It’s based on Python, which is readily available and supported on Gentoo, and is quite readable. Also, it looks to be very active in development and support. And also: it supports taking data from an existing WordPress installation, so that none of the posts are lost (with some rounding error that’s inherit to such migrations of course).

Before getting Pelican ready (which is available through Gentoo btw) I also needed to install pandoc, and that became more troublesome than expected. While installing pandoc I got hit by its massive amount of dependencies towards dev-haskell/* packages, and many of those packages really failed to install. It does some internal dependency checking and fails, informing me to run haskell-updater. Sadly, multiple re-runs of said command did not resolve the issue. In fact, it wasn’t until I hit a forum post about the same issue that a first step to a working solution was found.

It turns out that the ~arch versions of the haskell packages are better working. So I enabled dev-haskell/* in my package.accept_keywords file. And then started updating the packages… which also failed. Then I ran haskell-updater multiple times, but that also failed. After a while, I had to run the following set of commands (in random order) just to get everything to build fine:

~# emerge -u $(qlist -IC dev-haskell) --keep-going
~# for n in $(qlist -IC dev-haskell); do emerge -u $n; done

It took quite some reruns, but it finally got through. I never thought I had this much Haskell-related packages installed on my system (89 packages here to be exact), as I never intended to do any Haskell development since I left the university. Still, I finally got pandoc to work. So, on to the migration of my WordPress site… I thought.

This is a good time to ask for stabilization requests (I’ll look into it myself as well of course) but also to see if you can help out our arch testing teams to support the stabilization requests on Gentoo! We need you!

I started with the official docs on importing. Looks promising, but it didn’t turn out too well for me. Importing was okay, but then immediately building the site again resulted in issues about wrong arguments (file names being interpreted as an argument name or function when an underscore was used) and interpretation of code inside the posts. Then I found Jason Antman’s converting wordpress posts to pelican markdown post to inform me I had to try using markdown instead of restructured text. And lo and behold – that’s much better.

The first builds look promising. Of all the posts that I made on WordPress, only one gives a build failure. The next thing to investigate is theming, as well as seeing how good the migration goes (it isn’t because there are no errors otherwise that the migration is successful of course) so that I know how much manual labor I have to take into consideration when I finally switch (right now, I’m still running WordPress).

Planet Larry is not officially affiliated with Gentoo Linux. Original artwork and logos copyright Gentoo Foundation. Yadda, yadda, yadda.