The Foursquare Blog

"We're so plugged into the consumer entertainment industry that we don't know what's going on in the world. We know all about Johnny Depp and Angelina Jolie and U2 and The Family Guy, but we don't know the laws that are being passed in our countries. We don't know about the injustices of our social systems. We don't know where our food comes from, or the suffering required to get food onto our table. We don't know about the people in our neighbourhoods who are writing books and making movies and putting on plays and singing in choirs and running garage bands. Just like the citizens of The Matrix, we are mostly oblivious to reality, in part because somebody has convinced us that consuming entertainment is a worthwhile pursuit."

- Paul Nijjar, Adjacency Matrix

vicious writes with annoyance about Gnome's git setup that requires tags to have a GPG signature.

I think the reasoning should be viewed as less of a security measure for the source tree itself, and more of a best practice. Consider it a kindness extended to your end users.

Developers who commit to a tree are often familiar with the code. Those that are truly involved in a project may read every commit that comes in. They know what the code is supposed to do, and how it works. They know who the other contributors are, and they are in the best position to spot defects, bugs, or security holes.

The end user has none of this advantage. In most cases, he doesn't even know where the source code is. He wouldn't know how to read it if he did. And he may not know a good patch from a bad one, nor have the time to figure it out.

How do you bridge this chasm? The most practical way is to have a string of GPG signatures from the developer to the end user, and make it easy for the end user to verify this.

So, the developer signs his release with his GPG key. The distro packager checks this key, builds a package, and signs that with his own GPG key. And the end user downloads the updated software with the packaging tool, which automatically verifies the package signature. A very friendly verification chain from developer to end user.

So the question is: if you're not willing to sign a tagged release of your software, should you be in charge of releasing the code to end users at all?

Today

A quick heads-up to those who haven't registered online yet. Richard Weait announced that today is the closing day for the $40 advanced registration. If you miss today, it's $60 at the door.

You can see an updated list of presentations on the website. This year's fest is packed full of FLOSS goodness.

Hope to see you there.

A quick update to point everyone to the growing list of presentation topics for this year's Ontario Linux Fest. The list is happily large, and will get bigger.

If you see any topics that grab your interest, remember sign up online to get the early bird rate.

Hope to see you there!

Registration is open! It has been open for a few weeks now, and I've just gotten around to announcing it in my blog here now. Apologies.

This year's event is being held on Saturday, October 24. Just 3 weeks away! If you register in advance at the website the cost is only $40. If you show up at the door, it is $60. Earlier is better.

If you would like your name in lights, and would like to help sponsor the event, you can become an individual sponsor as well. More details on the registration page.

As usual, there's a welcome party on Friday night, and an after party Saturday night. Let us know if you can join us by checking the party options during your registration.

You can see a partial list of speakers here. The final list and schedule is still in progress, and if you want to submit a presentation topic proposal, there's still time to get it in under the wire.

The official Twitter feed for the Ontario Linux Fest is @OntarioLinux. We're using the group tag #oglf09. The 'g' is for GNU.

You can also follow me personally on Identi.ca as @cdfrey. I may post there from time to time as I work on the show guide.

Hope to see you at the Fest! I'll update this blog more regularly as more news becomes available.

Git has the ability to specify ranges of commits, with flexible variations.

But with that flexibility comes complexity. Have you ever wondered what the difference was between "commit1..commit2" and "commit1...commit2"?

If so, then this mini howto is for you.

(If you want to follow along, use the sample setup commands at the end of this document)

1. What is a branch?

It is helpful to remember that git thinks of a branch as any name or tag that refers to a SHA1 commit ID. The branch represented by that commit includes that commit itself along with all the ancestors that make it up.

For example:

        git log master

will show you the master commit, with all the parent commits that belong to it. If you say:

        git log master desktop

it will list all the ancestor commits of both master and desktop.

2. Commit walking

The command git-rev-list takes a list of commits and prints the SHA1 commit IDs of all the ancestors for those commits. This is a low level git command and is the basis for specifying commit ranges.

For example, on a repository I have on my hard disk, I have the following branch structure:

               /---o---o master
              /
        o----o (base)
              \
               \---o---o desktop

The following command:

        git rev-list master | wc -l

shows 3 commits. If I include the desktop branch, which has 2 unrelated commits in it, and does not have the 2 commits from master:

        git rev-list master desktop | wc -l

it shows 5 commits.

This shows that specifying multiple branch names is inclusive. All commits from all branches are included in the resulting list of ancestors.

3. Limiting the list

If I want to see all commits that are in master but not in desktop, I can invert one of the branches:

        git rev-list master ^desktop | wc -l

This shows 2. The two commits on the master branch. I can verify this with:

        git log master ^desktop

which shows only the patches on the master branch.

I can reverse this as well, showing only the 2 commits on the desktop branch:

        git log ^master desktop

The shorthand for this is the 2 dot ellipsis. The following are identical:

        git log ^master desktop
        git log master..desktop

Again, reversing, the following are identical:

        git log master ^desktop
        git log desktop..master

4. Finding the branch point

Git does not store branch divergence points. This can be calculated on the fly by looking at the heads of two branches and finding the first common commit in the list of ancestors.

This job is done by git-merge-base. If we used it on the above branch structure, we would get a SHA1 commit ID of the point marked (base) above.

        git merge-base master desktop

This returns c79f6f286e720b39976531b7a5b713b87308b576 in my repo. It will be different in yours, since you have different author information.

5. Listing all changes

Using what we know, we can now list all the changes on both branches that are not in the main repo:

        git log master desktop ^c79f6f

This shows 4 commits, from both branches.

The shortcut to this, is the triple ellipsis:

        git log master...desktop

Basically, this means show all the commits from both branches that happened since they diverged.

6. Diff: turn the world upside down

This is all fine and good, but wait! The git-diff command turns this inside out.

Git-diff takes two branches and shows a unified diff patch between the _endpoints_ that they represent. The usual format of the command is as follows:

        git diff master desktop

This will create a patch which, if applied to master, will turn it into desktop. i.e. it removes the 2 changes done on master and adds the 2 changes done on desktop.

The reverse will do as expected:

        git diff desktop master

This is the same as having two trees checked out, and running a manual diff -ru on them from the command line.

The double ellipsis is used in git-diff as well, but has no special meaning. It is an alias for the above usage. The following are identical:

        git diff master desktop
        git diff master..desktop

The difference in git-diff is that the triple ellipsis uses the same logic as git-log does when finding the merge-base, but the result is the difference between the base and _one_ of the branches.

For example:

        git diff master...desktop

This will find the branch point of master and desktop, and then generate a diff between that base commit and the commit of desktop.

In other words, the diff will contain only the 2 changes on the desktop branch, and is effectively the same as:

        git diff c79f6f desktop
or
        git diff $(git merge-base master desktop) desktop

Reversing the options will do the same for master. The following commands are identical:

        git diff desktop...master
        git diff c79f6f master

Now, the diff will contain only the 2 changes on the master branch.

The parsing logic of the triple ellipsis range is the same, but the result is the opposite. With git-log, all 4 commits are shown, with git-diff, only 2 commits from one side of the branch are shown.

7. Play

Pasting the following commands into a shell will create a sample branch with which you can experiment with these range specifiers.

mkdir play
cd play
git init
echo data > file.txt
git add file.txt
git commit -m "Initial commit"
echo data > README
git add README
git commit -m "Added README"
echo data > license
git add license
git commit -m "Added license file"
git checkout -b desktop master^^
echo data > main.c
git add main.c
git commit -m "New source code"
echo data >> main.c
git commit -a -m "Bug fix"
git checkout master

Enjoy.

Thanks to Ilari on #git irc.freenode.net for the discussion that lead to enlightenment.

badger:

This is where the --onto switch comes in, with git-rebase.

For example:

git checkout -b staging master
[make changes]
git checkout -b base_url staging
[make changes]
git rebase --onto master staging

This does: finds all the commits between the current branch (base_url) and staging, switches base_url's HEAD to master, and then reapplies the above found commits. You end up with a base_url branch that looks like it was only branched from master originally.

Paul Graham on the Manager Schedule vs. the Maker Schedule.

The planning stage of the Ontario Linux Fest is well under way. This year it will be held on Saturday, October 24, at the same great venue as last year: the Days Hotel and Conference Centre in Toronto.

We are looking for speakers to present on the following general topics, but of course, it is not limited to these:

• Welcome to (y)our World - Starting out with Linux
• Business and Linux
• Embedded Technologies
• Community: Building and Defending

If you have a presentation you would like to present at the Ontario Linux Fest, please add your proposal using the web form.

Our sponsor page is also open for proposals. If you or your company would like to help sponsor the event and get some publicity as well, please check it out. There are sponsorship levels for every kind of organization.

I'm currently working on finishing the registration pages. I'll blog again when it goes live.

If you have any questions, or if you run into problems on the website, please email me. Stick "onlinux" in the subject line to make sure I see it.

A little while ago, I wrote a summary on Linux sound. Since then, the Insane Coding blog posted a new summary of its own, which I'm linking for completeness.

After much reading of PulseAudio on Wikipedia and the interviews linked there and the blog posts linked from there (2009/04/30) and the main PulseAudio site and threads on OSS4 on linux-kernel, I've come to the conclusion that PulseAudio has the potential to be a good thing (using less power on a laptop, for example), and also the potential to be a bloated pain if not handled properly. Its tendrils reach many places, admitted by the developer himself, and it is hard to integrate into a distro without doing your homework.

I'm partly willing to give PulseAudio the benefit of the doubt though, if only for the power saving potential. I find it disappointing that an entire daemon and library system has to be built on top of ALSA and OSS to achieve this, but since Linux has decided that mixing belongs in userspace, and that no floating point is allowed in the kernel (understandable), then mixing has to be done somewhere, and PulseAudio looks like a good attempt by a sane developer, who is saddled with working with the pre-existing sound mess that is Linux.

The library situation looks interesting... if you want to just program with a library and forget the low level, use libao or libao2 (libao2 comes from the mplayer guys). Libao is crossplatform.

Another library called libsydney is also intended to be crossplatform (Linux, Windows DirectSound, Mac, etc). I haven't looked closely into that, but it is probably worth a look.

See http://blogs.adobe.com/penguin.swf/linuxaudio.png for a graphic of other sound APIs and systems. As a programmer, I would aim for either programming directly to OSS, which is portable crossplatform, or use libao2. Maybe libsydney if it doesn't have too many dependencies. My needs as a sound programmer would not be anywhere near the heavy-duty, so these options would work for me.

As for PulseAudio's state of readiness, I think there is a definite reason why it ain't at version 1.0 yet! :-) But that's not a bad thing. For stable-loving users like myself, it is probably worth waiting for a few distro release cycles before the bugs are worked out to a satisfactory degree. Newer PulseAudio releases will even stress ALSA drivers in new and interesting ways, so I expect there will be a strain on the sound system for a good while yet, from the applications right on down to the kernel.

If you don't need sound mixing, don't use PulseAudio yet. If you don't need power savings, don't use PulseAudio yet. If you use a laptop on battery then PulseAudio may be very useful, but you may need a good chunk of memory to support it, and it will pay to stay as up to date as possible, with kernel, distro, and PulseAudio.

From a power standpoint, there are reports of PulseAudio not even showing up on powertop, even while playing music, which is a good thing. I don't know how mpg123 + ALSA or OSS would compare.

There's a lot of flameage out there regarding ALSA and OSS3 and OSS4. It is easy to get caught up in it, and yes, I was caught up in it too... but my style of getting caught up in something usually involves me wasting many hours on research and reading to get the facts, and at the end of all this, I'm feeling less harsh with everyone, even though my mpg123 + ALSA configuration sometimes uses 40% CPU on a P4. (grrr!) :-) I can see the history and the reasons why Linux sound has evolved the way it has, and while some things still look unfortunate, they are understandable, and people continue to work on improving the situation.

I must say, though, that a piece of software going from "open" to closed, such as OSS3 did, can cause much disruption in the Free Software community. Even KDE and Gnome were arguably split due to licensing issues, even though they evolved in quite different technical directions as well. I think it would behoove the Free Software community to be more watchful of such situations, and guard against such collateral damage. The side effects can last for decades.

badger, those pull and push settings are configurable. Usually when I do a git-fetch, it grabs everything. Check how your remotes are setup in .git/config, and edit to taste.

Note that pull and push are not two halves of the same coin. For that, use fetch and push. Pull is a mix of fetch and merge, and it doesn't make sense to merge using multiple branch targets all at once. In fact, that is impossible as far as I know. Whenever you do a merge, you are always merging some other branch into your current branch.

chalst and ncm: I confess that I still read the recentlog, nearly every day, without logging in. So a feature that tracked my reading habits would be slightly incorrect.

This might brand me as a heretic, but I'll say it anyway.

I like brittle software.

Now, what I mean by that deserves some explanation, but whenever I think of ideal software that does a job for me, I think of it as a harmonious blend of the robust and the brittle.

First, the robust. Software needs to check its buffer sizes, check OS error codes, behave defensively and not crash even if given complete garbage as input. This side of the equation is very strongly linked to security. The software should be impossible to use for malicious purposes due to a bug in the code.

Another aspect of robustness is flexibility. It should put the user in control, while defending the user from possible mistakes he might make while learning to use the program, or in just everyday use. It should be hard to use wrong.

The brittle side of software also shows up in its error detection, and is a critical part of putting the user in control. If there is an error, I want to know about it. If the program is not absolutely certain that my data is safe, I want to know. If an attacker is trying to use the program to harm me, I want it to complain loudly and often.

I would rather have the program stop running than harm my data.

Not every program can achieve these lofty goals, but this is the nebulous image my mind creates when I think of ideal software. Some of this comes from my background in writing industrial network firmware, where it was better for firmware to halt completely and go to a known safe state, than to assume it knew what you meant and just let that piston stay on your coworker's arm.

So bringing Postel's Law into the equation, there are places where I definitely do not want my software to be liberal in what it accepts. Take a git repository, for example. If there is any remote possibility of data corruption, I want to know, and I want git to refuse to proceed. Luckily, git does a fantastic job of data integrity, which is one of the reasons I like it so much.

Postel's Law was originally about TCP, and so pertains to communication. Taking TCP as a specific example, and trying to apply Postel's law, let's say that a packet arrives with the entire TCP header shifted by 1 bit. I am no TCP stack expert, but I would be surprised if any TCP stack would be able to parse such a packet correctly. The only logical way of dealing with such a packet would be to look at the existing bits with the header format in mind, try to make sense of the non-sensical data, and send back an error if possible, or drop it on the floor.

How does Postel's Law fix a situation where the very format of the data is not respected?

I can understand if there is a TCP flag that is out of order, or if some unambiguous packet abnormality exists in an incoming packet. But there are only a limited number of such possible error combinations. Things have actually progressed in the opposite direction: TCP stacks have gotten more strict to deal with various attacks, and firewalls are used to strictly guard inbound and outbound traffic.

Being too liberal has a cost. And some costs are so high, they are seen as a detriment to society. Just look at the once-popular open SMTP relay.

Even in places where it would seem obvious that being liberal in what you accept is a good thing, upon further reflection, it turns out to be not so clean cut.

Take for example, a corrupt OpenOffice document. The user definitely wants to open it. Maybe some invalid XML is getting in the way, or maybe the file is only half there. What should OpenOffice do? It should make a best effort at retrieving what data it can, and it should open the file read only. It should also warn the user loudly that the file was corrupt, invalid, and inaccurate. The user needs to know this. This may be a file from an outside source, and so the error is not important. But it may also be the first error the user receives about a dying hard disk, or a bad network, or a corrupted filesystem, and restoring from backup is the next item on the agenda.

So in my frame of reference, Postel's Law, which is also called the Robustness Principle, fits right in, but only to half of my ideal software equation. Yes, be robust in what you accept. Be able to process complete garbage input without crashing. I've heard of people who fed a program's own EXE into itself as test input. It should be safe to use input data as a baseball bat and bludgeon the program without it falling over.

Yet the Brittle half of ideal software doesn't change. It is still as loud as before, warning the user of potential pitfalls and trouble on the horizon, and will refuse to proceed if it can't guarantee his data's safety or determine his intent without ambiguity.

In the end, Postel's Law is too ambiguous to be a useful guide, let alone a law. Those that accept it are too liberal, and those that don't are too strict. :-)

apenwarr has written an excellent rebuttal to my original rebuttal. I'd like to clarify my different viewpoint.

I think the crux of the argument boils down to these two statements:

Strict receiver-side validation doesn't actually improve interoperability, ever.

and

If you didn't catch it, the precise error in cdfrey's argument is this: You don't create a new file format by parsing wrong. You create a new file format by producing wrong.

I obviously disagree with both of these statements, but I understand how you could think they were true.

On the surface, parsing doesn't seem to create a new format, but even in Avery's own example, the majority of browsers accepting an incorrectly quoted option have indeed created a new format. It isn't a documented format. It is actually an anti-documented format, because the spec says it is wrong. But anyone writing a browser today would not be able to merely follow the specs and produce a functional browser... they would have to follow the behaviour of every other browser in existence as well. Just ask the developers of the now long-defunct, but exciting, Project Mnemonic.

Now, obviously this makes it easier for Average Joe to write his own webpage, and it probably did help advance the popularity of HTML and the rise of the web. But there is a defacto HTML standard out there because parsers were not strict enough. I don't know how you can deny that. (Part of the problem was that browsers were developed alongside the spec, so that contributed a lot too. The poor spec didn't stand a chance.) :-)

And strict receiver-side validation does improve interoperability. Can you imagine if the average C++ compiler allowed a relaxed syntax? Suddenly compiling code that "works" on one platform would not work on another. I admit that this is already a problem to a smaller extent than HTML, but differences in validating the C++ language spec between compilers is seen as a bug in the compiler, and rightly so.

This raises an interesting Option 4, to add to Avery's list: let the parser be forgiving with unambiguous syntax, but warn loudly.

This would be a huge improvement to what we have today. We need more web browsers that report the number of HTML errors in a page, by default, in the status bar. And it should be hard to disable, so that a site's non-compliance is widely seen and scorned. (And yes, some of my web pages would be scorned too.)

I believe Opera has a feature like this, if memory serves.

As for the side note claiming that parsing is not the problem, but the rendering is, my argument was based on XML as a data exchange format, and less as a way to display content in a browser. For example, the opensync project uses XML for data interchange, and plugin config. These formats are defined in strict schemas, which are tested and used via libraries like libxml2, which I assume falls into Avery's Big XML Parser category. If these schemas were not correct, I would consider it a bug detrimental to future interoperatility, and something that should be fixed.

The web itself is already so goofy that trying to apply XML to it now is like nailing jello to the wall. So in that respect, I can understand Avery's pain. It just bothered me that someone was boasting about an incomplete parser and claiming they interoperate with XML better than the big libraries. It seemed to me that he was discounting the long-range goals at work in XML in order to avoid some short-term pain. Of course, in the real world of making the customer happy, such shortcuts are often needed, but it's something to hide away in that closet of programming hacks, not published as an example on the web. That sounds harsher than I mean it, but I like those long-range goals, and XML is a solid technical achievement in its own right, even if rather cumbersome. Hey, I'm a C++ programmer... I like strict syntax. :-) I believe strict syntax promotes accuracy, and that accuracy helps you down the road when projects get larger and more complex.

In some ways, you could say that the inaccuracy found all over the web results in an unstable foundation that is generally holding the web back from greater things.

I hope it is also clear that I'm not a fan of XML. (Referring to it as a monstrosity was probably a good hint.) It has its place, but I think a lot can be accomplished by drastically simpler documented formats, and I'm quite willing to hack up my own simple file format if I think it is appropriate. I just don't call it XML.

I have some thoughts percolating in my head about Postel's Law, but that will have to wait for another post.

movement: Yes, I was partly expecting this response. But I'm sure it's because my lawn is so pristine that you kids keep wanting to mess it up. :-)

Seriously though, I have no problem with Javascript as a language that people might want to use to get things done on the desktop. The problem is that, in almost all current implementations of Javascript, it is setup to run any random code from the internet that the user clicks on... or even code he doesn't click on, in some cases.

In order for me to consider using a web-enabled Gnome desktop, I need to be confident that I have the power to enforce this strict separation of church and state. My PC is the church, and the internet is the state. :-)

I need to be able to flip a setting that makes it impossible to run any javascript that comes from outside my machine, whether it be through email, the web, or various files left over in /tmp or .webgnome or /home/cdfrey/Desktop, and only run javascript that I've installed and authorized, such as through apt-get or /usr/local.

This is where my confidence in Gnome's security design falls apart, because history seems to show that it is always more tempting to enable the new shiny web than it is to lock it down securely.

Dear apenwarr,

I'm sure you know all these things, but I'm afraid I'm as compelled to respond to you as you were compelled to write your own incompatible XML parser.

I do believe you are missing the point of XML. Yes, it is a horrendous pile of textual complexity, piled high to the sky with syntax and nested markers. Yes, the available API's, in order to deliver the promise of XML to the end user via the application programmer, are complex enough to fall out of your head as soon as you've finished implementing the feature.

But if you're going to do XML, please do it right.

Interoperability is hard. Anyone can write their own parsers. And everyone has. That's why the monstrosity called XML was invented in the first place.

It all starts with someone writing a quick and dirty parser, thereby creating their own unique file format whether they realize it or not. And since they probably don't realize it, they don't document it. So the next person comes along, and either has to reverse engineer the parser code, or worse, guess at the format from existing examples. This is then documented, either in Engrish, or in yet more code, which is likely to have at least one bug in it that makes it incompatible with the original format. This bug means that the second programmer has, whether they realize it or not, created their own unique file format.....

Oh dear.

The DTD is the documentation. XML kinda forces you to create it. The nice thing about DTD is that the computer can check it, which means it is less ambiguous than the pages of documentation that came with the old format. And with DTD + XML data, the computer can verify both, guaranteeing that the next programmer who gets the data can parse it the way it was meant to be parsed.

If your customer had used this XML process, you could have used the Big Fancy Professional XML Library. But instead, your customer is using their own UnknownXML, and you've created ApenwarrXML. The next programmer to come along may be forced to create yet another ThirdXML version, and the pain keeps propagating.

While I'm in the mood to make incendiary diary entries (insert good natured chuckle here), I'd like to write about Advogato etiquette.

I think the addition of syndicated blog entries, while bringing a chunk of good traffic, also carries a significant drawback. To some degree, it seems like people setup external blogs, turn on the Advogato syndication option, and then forget about Advogato.

How many people actually read the recent log after they have enabled syndication for their own external blog?

When I reply to posts that other people make, I used to be able to have a conversation in the recent log. This seems to be a more remote possibility now.

Is it just my own laziness in not wanting to add a trackback entry in an ever-increasing list of external blog sites? Should I get off my backside and implement automatic email notifications in mod_virgule whenever someone posts a response? Probably.

But it also seems to me that anyone who avails themselves of the Advogato community, and adds to their own readership by hitching their ride to Advogato's star, also has a responsibility to participate more thoroughly than just tossing their output over the wall.

company writes:

Note that I’m not talking about a web browser. I’m talking about a full featured framework that allows every application to display (partial) web pages, execute XHRs, upload photos to flickr, run user-downloaded scripts or otherwise integrate with the web. In fact, even the browser situation is worse than ever as GNOME is currently trying to switch to a different framework.

So how are we gonna marry GNOME with the web?

Maybe I'm an old foggie, yelling "get off my lawn!" here, but it is my opinion that this is the direction that caused a lot of Linux people to laugh at Microsoft.

I don't want the web everywhere on my desktop. I don't want my desktop married to the web, downloading and running scripts in the background for my convenience, or even by an accidental mouse click which is sure to happen. I don't even run javascript in my browser. Why would I want it anywhere near my web-connected desktop?

Remember back when it was impossible to get a virus via email? Then Microsoft came along, and made it default to automatically open your email in an Outlook subwindow and excute its HTML? I don't want my desktop to go in the same direction.

Perhaps you think that Gnome developers are smarter and they won't fall into the same traps that Microsoft developers did. Maybe Gnome developers are special? Perhaps. But I fear that the Linux desktop is making the same mistakes that Microsoft did, only more slowly.

Registration

The Ontario Linux Fest for 2008 is being held this coming Saturday 25th in Toronto. The day's schedule has been posted, as well as the detailed table of presentations and topics. Please check it out... there are some pretty interesting topics this year again, from the return of Open Moko, to human-machine interfaces, to legal matters and intellectual property, to OpenStreetMap.

If you haven't registered yet, there is still one more day.... today! Online registration closes on Thursday morning (the 23rd), which means if you want to get the lower rate of $40 (as opposed to $60 at the door), you need to do it today. You can register and pay for the Fest here.

With 27 presentations, birds-of-a-feather sessions, and labs, sandwiched between a Friday welcome party, and a Saturday night reception party, it'll be a fun-packed day of Linux and Free Software.

Hope to see you there!

At long last, the list of presentations at this year's Ontario Linux Fest is now available!

Please have a peek at the list, and then head over to the registration page to sign up in advance.

The Days Hotel, where the Fest will be held, has a special arrangement for the attendees. When you mention Ontario Linux Fest, the room price is only $89. I hear there are plenty of rooms available, but it never hurts to book in advance.

There is a welcome party on the Friday night before the Fest, and an after-party on Saturday night. Both will be held at the Days Hotel as well, so if you're attending the Fest from out of town, you may want to stay for both nights, and meet some of the presenters face to face.

If you or your organization were pondering being a sponsor for the Fest, the deadline for the show guide is in 1 day! The show guide includes your logo and ad, depending on your sponsorship choice. There's still time, but you need to signup quickly, and email me your desired logo and ad right away.

That's all the announcements I have for tonight. Hope to see you there!

For those of you who want to help spread the word of this year's Ontario Linux Fest, we have a poster available in PDF format, in both black-and-white and colour.

There are other ways you can help spread the word too. Here I'll swipe the suggested list from a recent email that John Van Ostrand sent out about the Fest:

• Tell your friends, family and co-workers
• Post on suitable forums, groups and mailling lists
• Tell your fellow students, alumni and professors
• Post flyers at your regular haunts: the water cooler, break room, server room door, classroom or favourite computer store
• Tell your favourite Open Source friendly company about it
• For all those writers, blog it
• For Facebook or LinkedIn users, join the "Ontario Linux Fest" group to show your support
• For users of other social networking sites why not create an Ontario Linux Fest group?
• And if you're hard core about it, add it to your email signature

Fest 2008

It is high time I caught everyone up with our progress at the Ontario Linux Fest, so here's a quick status update. As you know, this year it is being held in Toronto, Canada on Saturday, October 25, 2008, at the Days Hotel and Conference Centre.

The two keynote speakers this year are Jeremy Allison and Jon 'maddog' Hall.

We are working on getting the remaining speakers on the main website, so expect more updates over the next few days and weeks.

Registration is open! It involves creating a Drupal account and filling out the contact information and survey. Tickets are $40 in advance, and $60 at the door, so it pays to sign up early! Remember there's the welcome party on the Friday night, and the reception party Saturday night, where you can rub shoulders with the presenters and attendees.

It should be another fun and informative event, and I'm looking forward to seeing you all there.

Till the next update...

I started writing a little simulator for fun, and while trying to compile, make tells me:

make: *** No rule to make target `lender.o', needed by `economy'. Stop.

I feel the need to post about this issue in the hope that similar problems can be avoided in the future.

My initial disclaimer is that I'm not a package maintainer for any of the major distros, so I'm not intimately familiar with the stresses or workloads that they may face everyday. I am, though, the lead developer on a project that I hope one day will be included in major distros.

Whenever I get some interest from potential distro maintainers, I try to stress my keen interest in getting any downstream patches. This is to hopefully lighten their workload as well as to improve the software for everyone.

Unfortunately, it appears to me that the patch that caused the trouble in Debian recently was not fed back to the upstream developers, and if it had, it may have been caught much earlier.

What can be done from an upstream developer's point of view to encourage these upstream patches to keep flowing?

And is it not almost a duty for all downstream package maintainers to send patches upstream whenever possible?

Perhaps in some cases, the upstream packages themselves are not actively maintained, in which case being a distro package maintainer is even harder. But OpenSSL is not such a case.

I've run into 3 cases so far where a bad patch to the libtar library has sneaked into various distros and caused trouble for people trying to compile Barry on their systems. Would it not be better for these distro-specific patches to be fed upstream, and get rejected with a proper reason? Would it not be better for all distro maintainers of a particular package to be subscribed to its development mailing list, and see these issues first hand?

Obviously I think so, but I'd like to hear your thoughts on it. I think it is an issue that needs to be discussed, and now's the perfect time.

The planning stage for the 2008 Ontario Linux Fest is well under way. Richard Weait recently posted an announcement on the KWLUG mailing list, which you can read in detail in the archives. I'll summarize briefly here:

When:

Saturday 25 October 2008

Where:

Days Hotel and Conference Centre - Toronto Airport East
1677 Wilson Avenue
Toronto, ON M3L 1A5

The venue is at "The Days". There is a shuttle from the airport, and it is an easy connection to the Wilson subway. If there is enough interest, a shuttle from the subway to the Fest can be arranged, so let us know. Since the venue is both a hotel and conference centre, there are special deals on rooms, and ample parking on-site.

Who:

The call for papers is now open. See the Papers page to submit your topic. There are spots open for formal presentations, for lightning talks, for Birds of a Feather sessions, for Demo room sessions, and for Vendor room sessions.

How:

There are also various levels of sponsorships open as well, for FLOSS-friendly companies and groups. See the Call for Sponsors page if you or someone you know is interested, from multi-nationals to family businesses to Linux oriented groups.

How To:

If you have a suggestion, idea, desire, or criticism, we want to know!

Mark your calendars. I hope to see you there!

salimma,

I ran into that performance problem too, with my simple newmail.cc mbox summary program (see update_mbox_cache()). You can work around it by replacing std::getline() with the normal .getline() member of the io class, and using a non-std::string buffer.

It's still processing things line-by-line, which when compared to something like grep, is inefficient. Grep reads large blocks of the file, I think 16k or more at a time, and does its own very fast processing. But replacing std::string with a char buffer made the performance acceptable for me.

I believe the main cause of performance issues with std::string stem from the way iostreams insert a character at a time, and when using std::string, it is not very smart when growing its buffer.

I also find Boost regex to be rather slow at times, but perhaps I'm not using them correctly.

As you may know, the Ontario Linux Fest is being held in Toronto on this coming Saturday, October 13. Advance registrations are closing tomorrow (the 10th) at noon, so if you plan on being at the event, register now and pay through PayPal. It's all handled through the onlinux.ca site linked above.

It's going to be a day packed full of Linux goodness. I look forward to seeing you there... I'll be the big guy helping out at the sign-in area.

kelly, I pretty much agree with you about PHP. It is useful in some ways, but once I start programming something of any real size, I start to miss simple things like how the compiler is forced to check the syntax of all code, whereas an interpreted language can let you write some invalid code in if statement, and you may miss it until it runs.

As for operator overloading in C++, that's a bit like saying you won't use C because it has some bloated feature like printf() instead of write(). :-) You don't have to use operator overloading, but it sure is handy when it makes sense to use, such as when creating fundamental types like money values or complex numbers.

Shameless plug time. I've volunteered to help out with the upcoming Ontario Linux Fest which is to be held on October 13 in Toronto. Just returned from a planning meeting and things are moving along nicely.

There are still some slots open for presenters, so if you have a topic you'd like to present, send your proposal via one of the methods on the contact page, for best response. We have some great topics and presenters already lined up, including the likes of Theodore Ts'o and John "Maddog" Hall.

I'm personally looking forward to the OpenMoko presentation. I hadn't heard of this project until recently, but it sounds like everything a cell phone should be.

This Fest is planned in the same spirit as the Ohio Linux Fest: a grassroots knowledge conference instead of a product-based computer show. If you plan to be in the Toronto area in October, do sign up. Registration is cheaper if you do it in advance (only $40, and $20 for students).

Even if you can't make it, you can help by spreading the word to anyone you think would be interested. Help us pack the place.

I hope to see some fellow Advogators there! Drop me an email if you want to meet up.

For those of you with BlackBerry devices plugged into your Linux machines, you've probably noticed the frustrating fact that no matter what you do, it only asks for 100mA, and then complains that it doesn't have enough power.

And yet in Windows, it all works seamlessly.

Debugging this was a challenge, because I could not seem to find anything special happening on the USB level, when I used either USBSnoop on Windows, or usbfs_snoop on Linux with VMware.

Turns out there was a bug in the usbfs_snoop logging code, which you can fix with this patch (or wait for the next kernel release).

With this patch, the full proprietary USB conversation is exposed, and quickly duplicated for Linux. You can download the juicy results at the Barry project site. Look for bcharge in the SourceForge file downloads section.

Happy New Year!

My list of mailing lists is getting so large that I now have a class of mboxes that are more archive and reference than something I'm actively interested in.

Here's an update for those following my newmail.cc mailbox reporter program.

I added the capability of listing archive mboxes in your .newmailrc with a value of -2. Then you can selectively hide/show this second-tier list of mboxes with the -H and -S switches.

I also added an environmental getopt() that behaves the same way as getopt(), but prepends the command line options from an environment variable ahead of the argv[] options.

env_getopt() seemed like something that somebody would have implemented before, but I didn't see anything like it in my web search. I may have reinvented the wheel. Anyway, it is now implemented as GPLv2 in case others need the same functionality.

[Edited to fix link]

fxn:

Advogato also has XMLRPC, with which you can do programmatic access to your diary. I do this in the hope that it saves advogato bandwidth.

Below is a small python beginner script that I cooked up just for this task. It checks the dates of the posts and saves the new ones as individual files.

If someone finds it useful, please feel free to use it.

 #!/usr/bin/python

 import xmlrpclib
import os
import difflib

 def Download(filename, entry):
        print "Downloading: " + filename
        out = open(filename, "w")
        create, update = server.diary.getDates("cdfrey", entry)
        out.write("%s\n" % update)
        out.write(server.diary.get("cdfrey", entry))
        out.close()

 def GetTimestamp(filename):
        inf = open(filename, "r")
        s = inf.readline()
        inf.close()
        return s[0:len(s)-1]

 def Update(filename, entry):
        print "Updating: " + filename
        filetime = GetTimestamp(filename)
        webcreated, webupdated = server.diary.getDates("cdfrey", entry)
        if( filetime != webupdated ):
                print "Entry %d is out of date" % entry
                if os.access(filename + ".bak", 0):
                        os.unlink(filename + ".bak")
                os.rename(filename, filename + ".bak")
                Download(filename, entry)

                 oldf = open(filename + ".bak", "r")
                newf = open(filename, "r")
                oldl = oldf.readlines()
                newl = newf.readlines()
                print ''.join(difflib.unified_diff(oldl, newl))


 path = "/home/cdfrey/text/advogato/posts/"
server = xmlrpclib.Server("http://www.advogato.org/XMLRPC")

 entryCount = server.diary.len("cdfrey")

 for entry in range(entryCount):
        filename = path + "advogato.%03d" % entry
        if os.access(filename, 0):
                Update(filename, entry)
        else:
                Download(filename, entry)

 

fzort:

Thanks for sharing the link to those chess problems. It reminds me of a paperback book I picked up last year called Bobby Fischer Teaches Chess. It's full of problems like that, taking the reader from beginning backrank mates to more complex combinations. It's a very fun book to read, especially the second half, as you hold the book upside down.

The thing I've found is that chess problems, and the Bobby Fischer book, only helped my end game, and the planning, if I remember to do that. It is the opening that can make or break a game, and I've had many games broken by a bad opening.

I'm experimenting with lightening games lately, and while I lose almost all of them on time, I find it is a good way to practice openings, and to force my brain to work faster.

Maybe I'll see you on freechess.org. I have the same handle there. I'm still very much an amateur.

The last month has been kind of a mental lull for me. Slow progress everywhere.

But alas, I'm finally back on the Barry project after a few months' detour on other critical issues.

The libusb project has undergone an API redesign over the last few months too. It's all in the devel tree. If you're following libusb, and wondering why things are not being updated, make sure you're looking at the SVN repository, not CVS as its website says. :-)

Since Barry depends on the devel tree of libusb, there has been some porting needed in Barry, and some hacking needed on libusb. The last little while I've been reading through the libusb code, and adding libusb_wait(), libusb_poll(), and libusb_abort() support, which are critical if you want to use async calls and still avoid a threading library.

Still lots of work needed in Barry. It will be nice to have a working libusb and test setup again soon.

[Edited to fix broken link]

dwmw2 mentioned that he can't live without a tree view of mail folders showing which have new messages.

I currently use a combination of procmail, mutt, and my own scripts to sort and read mail. The procmail script sorts the mail into organized mbox files in ~/Mail/. Then I run newmail to get a list of mailboxes with new messages in them. Then I use

mutt -f Mail/mboxfile

to read.

Here's an example of my "newmail" output:

 Total     New  Mbox                                                          
------  ------  ----------------------------------------------                
    55       1  barry
   158      21 *bochs-dev (1)
  2096    1405  boost
   446     444 *bugtraq (3)
    80      78  bugtraq-generic
   402     144  c++
  4072       3  canada-dmca-opponents
  1200     157 *cdfrey (1)
   166     120  debian
   246      67  gentoo-announce
    41      15 *gentoo-desktop (1)
  1848    1022  gentoo-dev
    38       6  gentoo-gwn
    82      28  gentoo-hardened
   355     237 *gentoo-portage-dev (1)
   605      62  gentoo-security
   276     247 *gentoo-server (4)
  1217    1217  git
    11      10  gnupg-announce
    91      90  gnupg-devel
   423     331  gnupg-users
   129      82  kt
  1162       8  kwlug
   364     133  libusb
 10023   10022  linux-kernel
   601     344  linux-thinkpad
   144     144 *mailer-daemon (5)
   551     314  mplayer-users
   411     401  open-graphics
  1404     164  plusplus
   541      35  plusplus-commits
    63      34  risks
   211     189  slashdot
    47       2  spca50x-devs
     8       1  xboard

Yes, I'm a little behind. :-) This shows the total messages in the mbox file, the total unread messages, which mboxes have had new mail since I last checked (*), and how many new messages arrived since I last checked.

It's worked pretty well, not only as a mail system but as a spam whitelist too. I only use the code personally, so the code isn't polished for release. It's a single .cc file that can be compiled standalone, so it shouldn't be too hard to play with if you want.

Update: Found a bug where two uninitialized variables cause a new mail count to be out of whack the first time you run it. Once the mailbox data is cached, behaviour is normal.

Grab the file again if this matters to you.

jvic, Just run another X server.

If you have a BlackBerry, and wish to retrieve data from it on your Linux system, consider paying a visit to the Barry Project page and giving the latest release whirl.

This is of course a very early alpha release, but so far it has the capability to download calendar items, email messages (dates not supported yet), and address book entries. It attempts to convert the address book entries into LDAP LDIF format, and the email into mbox format.

You can also use the command line tool to capture protocol data during database retrieval, in case you wish to lend a hand in the reverse-engineering process.

The library and tool are written in C++. The ultimate goal of this project is to create a fully functional syncing mechanism on Linux. My goal is to have the API easily available in Python, perhaps via SWIG, so GUI folks can use it to integrate with their application of choice; but I haven't gotten that far. People with C++-to-Python experience are extremely welcome right now, even to just give advice on some of the pitfalls to avoid on the C++ side, so when the Python integration time comes, it will be as painless as possible.

I hope to see you on the mailing list if you are interested in this project.

One of the projects currently on the front burner for me is a library and application set for syncing the USB model of the RIM Blackberry handheld on Linux. Net Direct is the company behind this project, with full plans to open source it when it's ready. I may put it on SourceForge early if there is interest, but there's not a lot of code to hack on yet.

The first major task, of course, is to figure out the binary protocol that these devices use. I know about the documentation for the serial protocol at the Cassis project, but unfortunately, the USB protocol isn't the same.

There are rumours that the USB protocol may have been reverse engineered already, but my Google searches come up empty. If anyone knows of such a document, please email me. It would bring this project to fruition much faster.

While I'm on the topic, let me plug two fine USB tools:

• usbsnoop - a Windows filter with source code that intercepts the USB conversation between application and device.

• libusb - an LGPL cross-platform library that lets you work with the low level USB interface in userspace, without developing a kernel driver first.

To all the folks that replied to my previous diary entry, thanks! I apologise for being so busy I couldn't respond in depth as I wanted to.

Thanks to all the folks that bumped me up to Apprentice, which lets me comment on articles. Much appreciated.

ncm:

Monotone

What made you pick Monotone? I haven't yet made the time to evaluate all the distributed source control systems yet to find which one I like best, so I'm always eager to hear other people's reasons for their choices.

The ever present C++ debate

What worries me about the indulgence of template based generic programming over OO inheritance and virtual functions is that with templates, you are duplicating code for each type you wish to code for. This doesn't happen with OO, albeit with a slight performance hit of virtual functions.

This code duplication caused by templates is fine for code that you'd normally write anyway. For example, for_each instead of for loops, or any similar small piece of code.

But the larger your template functions become, the more each instance costs every time you use it.

Take the mozilla string hierarchy currently in discussion. How would you use templates to provide the same string functionality without the code duplication of templates? Mozilla is plenty big enough as it is.

Creating template versions of the mozilla string class, perhaps parameterized with the various functionality as template arguments, would cause each function that needed a general string argument to be a template as well, to accept whatever the user passed in. Am I missing something obvious here?

I must be, because there are a lot of C++ gurus that keep saying inheritance is bad, but tend not to go into detail. :-)

I finally got around to adding some features to the GNU split command, which I felt were sadly lacking ever since I thought of them. :-)

I often use split to break huge files into multi-CD images, such as for backups or archiving. At the same time, I often find I'm low on disk space, so I would like to process each split file chunk as they appear instead of doing it all at once at the end.

I wrote a patch that adds exec and pause functionality with the following arguments:

  -e, --exec=CMD          run CMD after each output file is closed
  -w, --exec-wait=CMD     run CMD after each output file is closed
                          and wait for the child to exit
  -p, --pause             pause for keypress after each output file is closed

I find these features very useful, so I hope they make it into the official coreutils tarball.

If anyone has any tips on how to handle recovering a tty-based STDIN (see the split source code), please let me know. I'm currently just open()ing the tty on STDOUT, but that is not ideal.

Spent most of today reading through the various websites for for C++ wrappers of SQL databases. The best ones, in my opinion, are the official ones: mysql++ and libpqxx (was libpq++, but official now (2025) is libpqxx). Unfortunately, these are both tied strictly to their specific databases and are not database independent.

Also in the mix are DBConnect and GQL. Both are rather pointer-happy, but at least DBConnect encourages use of it's own smart pointer to avoid copious use of delete. On the plus side, they are database-independent.

Only mysql++ has the SSQLS feature. What is that you ask? It is a mechanism whereby you can easily create structures that mirror your database tables, and then go around using them as members of container objects, and doing things like:

query.insert(row_data);

This makes the library code extremely hairy (so hairy that it motivated a fork of the code a few years ago), but gives the advantage of enabling the compiler to check your field names, localizing your C++-to-SQL linkage in one place, and letting the library take care of generating properly escaped SQL code.

So far, my research confirms that the free software community still lacks a proper database independent C++ SQL library that shields the programmer as much as possible from quoting issues. As always, please email me if you know better.

Cool Program of the Day

Unpaste is a program by Jeroen Vermeulen that attempts to find repeated code in your C, C++, and Java programs. It is a work in progress, but already useful. It scans all the files you give it, and detects duplication even with name changes or differences in const.

[Edited to fix and remove broken links]

boog, your Snippets project is cool, but it's not a new idea. It reminds me of a similar project with the same name which I ran across back in the days I used to be a regular FidoNet user. It was run by a fellow named Bob Stout. Here is his original Snippets archive (archive.org) (github).

The license of each of the contributions varies with the contributor, but they are all freely available for use last time I checked.

You might consider joining forces with Bob Stout and combining your efforts to produce one large repository (perhaps mirrored on both sites). Not sure if you tried this already, but it would appear that the original Snippets site could use a little help.

Fun

While perusing the site, I found a link to The Parable Of Two Programmers (alternate link). Excellent read.

C++ and the Web

After writing a dynamic website for a client in PHP, I was struck by the lack of help that the language and interpreted nature of PHP gives a programmer for dealing with large sites. Also, I'm amazed at how many XSS and SQL injection attacks I see for PHP websites posted on Bugtraq. It reminds me of the arguments against C I used hear from anti-C people, who used to blame buffer overflows on the language. At the time, I thought it wasn't right to blame the language when it was the programmer's fault, but now I'm on the other side of the fence, taking a step backward (in my opinion) from the safety of C++ to the wild west of PHP.

And the problems with C pointed out in the past by security-conscious language bigots :-) are all "fixed" in PHP. There are no buffer overflows. There are no memory leaks. There are no real number size issues.

Instead, I'm now on the bigot side, and really missing strict type checks, and really missing a compiler to check all syntax. Wondering how people manage with this.

So I got to thinking, and I'm trying to design a PHP clone in C++. I'm writing about it now so that people can head me off at the pass if something like this already exists. My goals are to make it nearly impossible, or at least harder, for a programmer to put XSS or SQL injection bugs into their websites. Make the compiler check for these things. That's what C++ is for anyway.

Perhaps C++ can give the same structure and organization to a website that it gives to a large local application.

Some of these ideas were inspired by Joel Spolsky's article referenced by ncm's recent post, which referenced titus's recent post. I agree with Nathan, but Joel's article did get me thinking. His type method should fit nicely into C++, where everything is a type, and instead of the programmer doing the checking by reading "wrong" code, the compiler will do it, and enforce it.

So that's my idea, which I plan to call CPPHP. I hope to have some code someday soon that actually works. Right now I'm just prototyping my design. If this already exists somewhere, or if you just want to send feedback, please email me.

Edited to add:

I just want to plug John Torjo's win32gui library, which has some ambitious aims for programming GUI's with C++. If that could be done for the web, that would a wonderful advance.

[Edited to fix some broken links]

In the hopes that record company executives are reading my blog, I'll post some free advice which I hope they take to heart.

A few months ago, I was driving home from work when I heard a really cool song. This was not just any ordinary song. It was one of those impulse buy songs, and I don't buy CDs very often (I already have plenty).

So while still in the car, I drove straight to nearest music store, and start searching for the CD. I asked the checkout girl about it, and she searched in vain in the computer system. Then the answer appeared.

The CD was to be released in about a month.

Dear record executives: the song playing on the radio is your best advertisement (if it is a good song). Don't waste that momentum by separating the radio release from the CD release.

By now, my impulse to buy is long past, and while the CD is surely in the stores by now, I don't hear it on the radio anymore.

[edited]

fzort: Thanks for the link to Herb Sutter's concurrency article. It was quite the interesting road of travel to the end.

Here's a link link to Andrei Alexandrescu's presentation on lock-free programming, which reference's Herlihy's 1991 paper on Wait-Free Synchronization, in case anyone else is interested in blowing a few hours.

Read over last year's ACCU conference events and wished I'd been there. Would be very cool if I could make this year's.