The Foursquare Blog

redi, thanks for the pointer to timegm(). Both GNU and BSD, you say? Nice. I did not know that function existed.

It would still be nice if timegm() was in a library of its own instead of libc. But that's probably asking too much.

Good to know. Thanks!

In the C library, there is a very handy function called mktime() which takes a struct tm pointer, pointing to time data in the local timezone, and converts it to a unix time_t.

There is also a very handy set of functions that goes in the other direction. The function gmtime() takes a time_t and converts it to a struct tm in the UTC timezone. And localtime() does the reverse of mktime(), converting a time_t to a struct tm in the local timezone.

The one function missing is the reverse of gmtime().

One way to do it, via standard library function calls, is to adjust the TZ environment variable to "UTC" and then call mktime(), which will then use it as the current timezone. But that gets messy. Not only do you have to save the old TZ setting to restore it afterward, if your program uses threads, messing about with the global TZ variable is hardly safe.

Another way to do it is to use gmtime() as a reference point, and loop while adjusting the original struct tm until mktime() + gmtime() gives you the result you started with. This is brute force and inelegant, but it manages to do the job without fiddling with the global environment.

When parsing ISO timestamp strings, this problem is hit pretty quick. An ISO timestamp can contain a time in either the local timezone or UTC, depending on the trailing 'Z' flag. In my research, it does not appear that more specific timezones can be specified in the timestamp format, so at least we only have two states to worry about.

After much searching, I've ended up with something that I find suitable enough to release. I tackled this problem back in 2007 for the OpenSync project (you can see similar code in the opensync_time.c file), but looking back on it now, the code is not clean enough for me to reuse very easily. And even though there's a vtime2unix converter function, it leaves the burden of timezones to the user.

I took a look at Boost's date_time library as well, and while huge and comprehensive, and though it breaks the problem of time, dates, and durations into nice manageable chunks, it seemed to reverse the status of the C library: it made UTC conversions easy, and local timezone conversions hard. And it reads timezone information out of CSV files... I don't want to have to maintain my own timezone database when the OS does it for me.

So, if I don't want to write my own conversion routines, and if I don't want to maintain my own timezone database, I'm stuck with the C library method, and the first two solutions. Might as well make it easy to use.

The source files for TzWrapper contain the following utilities:

• iso_to_tm() - simple ISO timestamp converter
• utc_mktime() - the brute force UTC to time_t converter
• TzWrapper - C++ wrapper class for setting and restoring TZ

Using TzWrapper, it's now possible to do things like:

	struct tm pacific_tm = { ... };
	time_t t = TzWrapper("Canada/Pacific").mktime(&pacific_tm);
	cout << TzWrapper().Set("Australia/Brisbane").ctime(&t);
 
	const char *iso = "20100430T231500Z";
	time_t stamp = TzWrapper::iso_mktime(iso);

If you have any further tips to make this code better, or pointers to better libraries that make this code obsolete, please let me know!

My list of mailing lists is getting so large that I now have a class of mboxes that are more archive and reference than something I'm actively interested in.

Here's an update for those following my newmail.cc mailbox reporter program.

I added the capability of listing archive mboxes in your .newmailrc with a value of -2. Then you can selectively hide/show this second-tier list of mboxes with the -H and -S switches.

I also added an environmental getopt() that behaves the same way as getopt(), but prepends the command line options from an environment variable ahead of the argv[] options.

env_getopt() seemed like something that somebody would have implemented before, but I didn't see anything like it in my web search. I may have reinvented the wheel. Anyway, it is now implemented as GPLv2 in case others need the same functionality.

[Edited to fix link]

fxn:

Advogato also has XMLRPC, with which you can do programmatic access to your diary. I do this in the hope that it saves advogato bandwidth.

Below is a small python beginner script that I cooked up just for this task. It checks the dates of the posts and saves the new ones as individual files.

If someone finds it useful, please feel free to use it.

 #!/usr/bin/python

 import xmlrpclib
import os
import difflib

 def Download(filename, entry):
        print "Downloading: " + filename
        out = open(filename, "w")
        create, update = server.diary.getDates("cdfrey", entry)
        out.write("%s\n" % update)
        out.write(server.diary.get("cdfrey", entry))
        out.close()

 def GetTimestamp(filename):
        inf = open(filename, "r")
        s = inf.readline()
        inf.close()
        return s[0:len(s)-1]

 def Update(filename, entry):
        print "Updating: " + filename
        filetime = GetTimestamp(filename)
        webcreated, webupdated = server.diary.getDates("cdfrey", entry)
        if( filetime != webupdated ):
                print "Entry %d is out of date" % entry
                if os.access(filename + ".bak", 0):
                        os.unlink(filename + ".bak")
                os.rename(filename, filename + ".bak")
                Download(filename, entry)

                 oldf = open(filename + ".bak", "r")
                newf = open(filename, "r")
                oldl = oldf.readlines()
                newl = newf.readlines()
                print ''.join(difflib.unified_diff(oldl, newl))


 path = "/home/cdfrey/text/advogato/posts/"
server = xmlrpclib.Server("http://www.advogato.org/XMLRPC")

 entryCount = server.diary.len("cdfrey")

 for entry in range(entryCount):
        filename = path + "advogato.%03d" % entry
        if os.access(filename, 0):
                Update(filename, entry)
        else:
                Download(filename, entry)

 

dwmw2 mentioned that he can't live without a tree view of mail folders showing which have new messages.

I currently use a combination of procmail, mutt, and my own scripts to sort and read mail. The procmail script sorts the mail into organized mbox files in ~/Mail/. Then I run newmail to get a list of mailboxes with new messages in them. Then I use

mutt -f Mail/mboxfile

to read.

Here's an example of my "newmail" output:

 Total     New  Mbox                                                          
------  ------  ----------------------------------------------                
    55       1  barry
   158      21 *bochs-dev (1)
  2096    1405  boost
   446     444 *bugtraq (3)
    80      78  bugtraq-generic
   402     144  c++
  4072       3  canada-dmca-opponents
  1200     157 *cdfrey (1)
   166     120  debian
   246      67  gentoo-announce
    41      15 *gentoo-desktop (1)
  1848    1022  gentoo-dev
    38       6  gentoo-gwn
    82      28  gentoo-hardened
   355     237 *gentoo-portage-dev (1)
   605      62  gentoo-security
   276     247 *gentoo-server (4)
  1217    1217  git
    11      10  gnupg-announce
    91      90  gnupg-devel
   423     331  gnupg-users
   129      82  kt
  1162       8  kwlug
   364     133  libusb
 10023   10022  linux-kernel
   601     344  linux-thinkpad
   144     144 *mailer-daemon (5)
   551     314  mplayer-users
   411     401  open-graphics
  1404     164  plusplus
   541      35  plusplus-commits
    63      34  risks
   211     189  slashdot
    47       2  spca50x-devs
     8       1  xboard

Yes, I'm a little behind. :-) This shows the total messages in the mbox file, the total unread messages, which mboxes have had new mail since I last checked (*), and how many new messages arrived since I last checked.

It's worked pretty well, not only as a mail system but as a spam whitelist too. I only use the code personally, so the code isn't polished for release. It's a single .cc file that can be compiled standalone, so it shouldn't be too hard to play with if you want.

Update: Found a bug where two uninitialized variables cause a new mail count to be out of whack the first time you run it. Once the mailbox data is cached, behaviour is normal.

Grab the file again if this matters to you.

I finally got around to adding some features to the GNU split command, which I felt were sadly lacking ever since I thought of them. :-)

I often use split to break huge files into multi-CD images, such as for backups or archiving. At the same time, I often find I'm low on disk space, so I would like to process each split file chunk as they appear instead of doing it all at once at the end.

I wrote a patch that adds exec and pause functionality with the following arguments:

  -e, --exec=CMD          run CMD after each output file is closed
  -w, --exec-wait=CMD     run CMD after each output file is closed
                          and wait for the child to exit
  -p, --pause             pause for keypress after each output file is closed

I find these features very useful, so I hope they make it into the official coreutils tarball.

If anyone has any tips on how to handle recovering a tty-based STDIN (see the split source code), please let me know. I'm currently just open()ing the tty on STDOUT, but that is not ideal.