Geek Drivel

The head of Project Badger (at the much larger company that bought it a few years ago) recently moved on to other duties in the company. We talked briefly just before he did, and he said that he’d learned a lot from me.

I was startled, but I figured that was just a stock corporate-speak phrase he was using with everyone. It didn’t seem there was anything that I (primarily a hobbyist-turned-professional software developer) knew that would be useful to someone like him, a corporate manager. But later I got to thinking about it.

When we first sold Project Badger to that company, I had to adapt quite a bit. I’d never dealt with a large corporation before, and didn’t understand the culture. I thought the Dilbert comic strip was an exaggeration. It is, but not by much.

For example, soon after we sold it to them, they asked us to embed their help-line 800 number into one of the error messages. Several months later, a customer who got the message actually called the number and got a big surprise — it was a phone-sex hotline. It turns out that the phone number they’d given me was wrong in one digit.

It was a simple and innocent mistake, and one that was easily corrected (though not quickly — there were a lot of products incorporating it by that time). But I watched in amazement as they assumed that it was deliberate attack on them, and paranoidly tried to find some conspiracy. No doubt they discussed whether we were working against them too, though they didn’t let us know that. I hadn’t noted which one of them had given me the number, and after several months I couldn’t remember for sure, so they were looking cross-eyed at each other too. I know corporations have to be concerned about that kind of thing, but they took it to a ridiculous extent. My feelings are plainly read on my face (I’m no good at poker), so I’m sure my stunned amazement showed; maybe it helped show them how badly they were overreacting.

As another example, different people would load a different project onto a single person at each meeting, and a few days later ask why this task or that one wasn’t done. It was just like college, where each professor assigned you enough work to keep you busy until the next class, deliberately ignoring the other classes you were taking. It was ludicrous, and no one seemed to be able to see that it was a problem, let alone do anything about it.

I can only concentrate on one thing at a time, so I came up with an aikido-style solution: whenever someone tried to give me a new task, I would agree to it, then immediately ask that person to clarify which one was the priority, the new one or the one I was working on already that was assigned by person X. If they felt that they had the authority to override person X, they could give their project priority; if they didn’t, then they would realize that their project was on the back burner. Whenever anyone started playing the blame game, or confronted me about not having something done, I could give a logical and air-tight reason for it, and one they couldn’t argue with. I think that earned me some grudging respect.

Then there’s e-mail. When someone asks for my analysis of a situation, I assume they want to know it, so I try to give them the whole thing. I quickly learned that when I did so, the management people didn’t bother reading it — they wanted a quick yes or no answer. So after writing my full analysis, I went back to the top and added a one- or two-sentence “executive summary”:

Subject: Preventing the sharing of licenses

Executive summary: don’t do it, it would cost us more, irritate our legitimate customers, and do nothing to stop the abuse.

Long answer:

> This is an old subject that for some is unpopular, however this
> addresses the problem of customers sharing their keys with contractors
> and associates despite the fact that our EULA specifically forbids it,
> and I think it’s time this was addressed. [...]

Sorry to do this, but I have to disagree.

[Long analysis with example follows.]

The management people could quickly see the results, and members of the team who were interested in the reasons behind it could see that too.

I’m not even going to dig into the culture of having hour-long meetings about even the most minor things, and requiring the developers to attend them instead of getting any development work done. I’m sure I stepped on more than a few toes while getting that straightened out, but I can’t apologize for it; it’s a ridiculous practice, and a waste of valuable programming time.

So in the end I think he was telling the truth: he did learn a lot from me, and I just didn’t realize it at the time.

At the end of my last iPod Touch entry, I mentioned that I was going to try jailbreaking the Touch so I could load music onto it from Linux, instead of relying on a VMware Windows machine.

I did so. Everything seemed to work with no problem, but after I loaded any song into it, the Touch claimed that it had no music on it at all, and iTunes said that I had to restore it to factory settings before it would even look at it again (which involves rebooting into the full version of Windows, and about 45 minutes of restoring various things). After the fourth failed attempt, I stumbled across a note on the Ubuntu forums that explained the problem: Apple has changed the hash algorithm they’re using for the database. Again.

Why? I don’t understand the purpose behind it, since it does nothing but cause problems for people (their customers!) who are trying to load music onto the device without going through iTunes, and pretty much the only people who want to do that are people using Linux. If it were Microsoft doing it, I’d understand it — discriminating against anything non-Windows is part of their culture — but Apple already supports Windows as well, so it can’t be a matter of OS snobbery. I know Apple is a control-freak about some things, but that doesn’t sufficiently explain it.

In any case, someone will figure out the new algorithm soon, and update the Linux software to account for it. Until then, I’ve found a way to move my playlists from RhythmBox (my Linux music player) to iTunes* to get them onto the Touch, so I suppose I can keep using iTunes for now. It’s just an extra (and unnecessary) step.

[* It requires exporting the playlists to m3u format, running sed over them to change the filenames to Windows drive-letter format, then importing them into iTunes, which is using the same on-disk music library.]

If you have any sympathy at all for Microsoft, you’ve gotta wince at this story. It’s not the first time something like that has happened either… on a trip to Toronto a few years ago, I saw an absolutely HUGE electronic billboard — which was showing a gigantic Windows error message-box instead of the ads that it was supposed to. Not a Blue Screen of Death, but just as embarrassing to the owners, and to Microsoft.

So, having learned that the iPod Touch probably could replace my slowly-dying Palm TX, I’ve picked one up a few days ago. The box said that it required Windows or Mac OS X, which is an irritation but not a problem; I have this system set up to dual-boot between Ubuntu Linux and Windows XP, as well as having a VMware virtual Windows XP box under Linux.

I pulled it out, carefully unwrapped it, and turned it on. It showed a screen that seemed to be saying it had to be plugged into my computer, so I did so. Then, reading the quick-start manual, I saw that I had to load iTunes. Grumbling, I downloaded and installed it.

iTunes says that my brand-spankin’-new iPod Touch has an old version of the OS, and would I like to upgrade to 2.0 for an additional $10? Grumbling further (but not really surprised — I’d read up on it before I bought it), I agreed, paid my ten bucks, and downloaded the update. iTunes wiped the old OS, put the Touch into recovery mode, and then… said that there was an error.

It seems that you can do nearly everything with the Touch via Windows XP running in a VMware virtual machine, but upgrading the OS requires Windows to be running on the bare metal.

After taking care of that little problem (via dual-boot), I settled down to moving my data.

My contact list isn’t all that large — about 100 people — and it had ten years worth of accumulated cruft and outdated information, so I decided to move that manually, updating each one as I did. That took a while, but had no major surprises. I decided to sync it with my GMail account as well, so I’d have an online backup; no problems there either.

The 120+ e-books that I’ve purchased for my Palm over the years were all bought from eReader.com. The eReader program was available in the iTunes Application store (for free, of course), so I loaded it up. I dredged up the password for eReader.com, entered it, and was pleased to discover that my e-books were all there, waiting to be downloaded for the iPod. I downloaded one old favorite and tried to unlock it. Tried being the operative word… somehow, even after entering the proper unlocking information, it wouldn’t work. I had to sign onto the site via a web browser and update my information, then re-download the book, before it would work. It should work for all of them now though.

The To-Do List was… interesting. The iPod Touch doesn’t have one built in, so I checked out their Application Store. There were three free to-do list applications that I downloaded and tested. The first two (I don’t remember their names) were too simple to be useful for me; neither one could handle more than a single list, for instance. The third one was Dobot Todos, and it didn’t take long to realize that I had a winner. Unfortunately the Apple store is two revisions behind, and is dragging its collective feet about updates… I’m sure they’ll correct that eventually, but I wish they’d do it more quickly.

Next up: my calendar information. This is the single most-used function of my Palm TX, and I really really didn’t want to have to re-enter all of the data for it by hand. But from everything I could see, the only direct way to get my information moved over (without a Mac computer to go through) was to go through Microsoft Outlook.

Yuck.

But I did have a long-since-retired copy of Office XP (with Outlook 2002) around here somewhere, so I started excavating my CD-ROM collection and finally found it. It installed with no problem, and with a little setup, it synced to the iPod easily — the only trouble was that two of my contacts somehow got duplicates. I carefully deleted the duplicate entries, re-synced, and made sure the originals were still in both places (they were) and wouldn’t re-duplicate themselves on later syncs (they didn’t), then I was ready for my Palm data.

Unfortunately, my Palm doesn’t seem to want to retire, and it and the rest of my electronics fought me all the way.

First I had to dig out the CD that it originally came with (which meant another excavation), then RE-install the Palm Desktop software (for some reason, the installation I already had on that machine wasn’t good enough for it), then tell that to sync to Outlook instead (carefully telling it to overwrite the Palm’s contact information with the desktop’s). Then the virtual machine wouldn’t make a wired connection with the Palm, no matter what I tried — I never did figure that one out. Then the network connection couldn’t find my XP virtual machine (VMware’s bridged network connection was hooked to a different network card than the one that I’m currently using). I finally got everything set up properly though, and the Palm reluctantly turned over its information to Outlook, which then sent it on to the iPod. A few manual tweaks (because the iPod doesn’t seem to like the “zero minutes ahead” alarm setting), and all seems well now.

My next project is getting a few audio-books (that I purchased from Audible.com) onto it. That shouldn’t be a problem… I hope. Then, once WinPwn comes out with an update for the 2.0.1 firmware, I’ll jailbreak the iPod so I can load my music onto it from Linux. Then the configuration stage should be complete.

Lovely. Okay developers, time to get moving — add public-key code-signing stuff, so that your programs can tell whether they’re getting a legitimate update or not. Don’t know how, and don’t have time to learn? Try the GnuPG Made Easy (GPGME) library.

I’m happy to say that Ubuntu Linux isn’t affected by this, because it already utilizes code signing for all of its updates. I’m not sure whether Windows does as well, but if it didn’t, I suspect it would be the first thing in the title’s list.

As mentioned previously, my company sold the rights to one of our previous Windows software products to a much larger company a few years ago. The much larger company keeps my company on retainer to advise them on the product’s continued development.

The product (let’s call it Project Badger) was originally written more than ten years ago now, with me as the sole original developer. At the time, I treated C++ as simply an improved version of C. Microsoft Visual C 6.0, my compiler of (reluctant) choice at the time, barely supported the newly-standardized C++ Standard Template Library (STL); the Boost library didn’t exist yet; and I hated the non-standard and non-portable Microsoft Foundation Classes with a passion. That didn’t leave much choice: I had to hand-code all the algorithms and data structures myself, and use the painfully primitive C string functions and the raw Win32 API.

(Yes, and walk barefoot in the snow to get to school everyday. Uphill. Both ways. But that’s another story.)

Project Badger has grown a great deal since then, but it was still using the hand-coded data structures and algorithms that I originally came up with, and the C string functions and raw Win32 API. Although they still worked, it was a very creaky and increasingly ugly infrastructure… grafting support for Unicode onto it (as our customers demanded) was a nightmare that required a lot of hand-crafted code, and it was only supported in certain places. And we couldn’t move to Unicode completely, because we still had to support Win9x (Windows 95, 98, and Millennium), for reasons I won’t go into, and they didn’t support Unicode. (Using the UNICOWS DLL, which is Microsoft’s way of retrofitting Unicode support into Win9x systems, was deemed unacceptable in this case, again for reasons I won’t go into.)

This all came to a head a few weeks ago, when a customer reported a problem that should have been easy to fix, but that our bastardized infrastructure made nearly impossible. For a few of its abilities, Project Badger has to open a copy of its EXE file and read some data from a table tacked onto the end; this customer reported that these features wouldn’t work if the EXE file were placed in a path with Unicode characters. The reason was easy to track down: we were using the GetModuleHandleA function to find the filename of the EXE, and CreateFileA to open it. (The ‘A’ on the end means that they’re the ASCII versions of those functions, rather than the Unicode versions, which would be denoted by a ‘W’ there.) But fixing it, in a way that wouldn’t break Win9x compatibility… that was more interesting.

After one of our developers spent a frustrating couple days trying to work around the problem using the “short filenames” (legacy of DOS and the FAT12/FAT16 file system) — to no avail — I proposed overhauling the entire program, replacing many hand-coded parts of the program with components from the STL and Boost libraries. Since I was the only one on the team that had much experience with both, I volunteered to do the initial conversion.

It was a large undertaking, even larger than I’d anticipated. It took me ten straight days, working twelve to sixteen hours a day, to finish it (I’d estimated seven days, for a broader scope of changes). But I think the result was worth the effort.

A key component to the overhaul was a way to support Unicode strings and functions similar to the way that the UNICOWS library does: on platforms where it’s available, dynamically load the Unicode version of the API function and pass the Unicode parameters directly to it. On Win9x platforms (where the Unicode functions aren’t available), fall back on the ASCII versions of the functions, translate all strings to ASCII before passing them in, and translate any results to Unicode when passing them back. To support this, I put together a specialized String class, using the Boost::Variant library. Here’s a simplified version of its declaration (the full version has conversion support for a few program-specific types as well):

namespace os {
class String {
public:
String(const std::string& init): mData(init) { };
String(const std::wstring& init): mData(init) { };
String(const char* init);
String(const wchar_t* init);

Data mData;
}
}

(Again, my apologies for the lack of proper indentation; it’s there, but the blog software doesn’t display it. The code should be easy enough for a minimally-experienced C++ developer to follow despite that.)

Note that the boost::variant typedef Data can accept a std::string (a standard ASCII string), a std::wstring (a Unicode string), or a raw ASCII or Unicode string pointer. The pointer option was necessary because some Win32 API functions allow you to pass in an invalid string pointer, encoding (usually via the MAKEINTRESOURCE macro) some specialized information instead of a standard string.

With that in place, and using a set of simple template classes I wrote to handle the dynamic loading, I could write functions that would take either ASCII or Unicode strings and pointers (or even a combination of different types) and do any necessary conversion on the fly:

namespace os {
HANDLE CreateFile(const String& filename, DWORD acc,
DWORD share, LPSECURITY_ATTRIBUTES sec,
DWORD create, DWORD flags, HANDLE tpl)
{
static StdFn7<HANDLE, LPCWSTR, DWORD, DWORD,
LPSECURITY_ATTRIBUTES, DWORD, DWORD, HANDLE>
fn("CreateFileW", cKernel32);

}
}
}

It’s a good solution, though not a perfect one. For one example, it’s not as fast as the raw calls. That doesn’t matter in our case, because most such calls are in user-interface code (where even the slowest machine is fast enough that much more inefficient code than that wouldn’t be noticeable), and the rest are in one-time operations where the speed isn’t critical.

Another limitation is that you can’t pass in a zero or NULL for one of the os::String parameters, even when it might be supported (as in the “title” parameter to the MessageBox function, which defaults to the localized “error” string if passed a NULL). I got around that by defining

const os::String NULLSTRING(static_cast<char*>(NULL));

in the source file, and then putting

extern const os::String NULLSTRING;

in the header. Then, whenever I needed to pass in a NULL, I just changed it to NULLSTRING, and everything worked as normal. I could probably have also defined an os::String constructor that accepted an int, but that seemed like overkill, and would have worked against type-safety.

The end result: minimal changes to the existing code, while adding maximum flexibility. I’m very happy with it.

If you run your own website, you might have been following the brouhaha over the new LinkScanner feature of GriSoft’s AVG virus scanner.

I can’t find this text on the GriSoft website, but it was quoted in a comment on another blog I was reading today:

Following is AVG’s official response to LinkScanner concerns:

We’d like to thank our web community for bringing these challenges to our attention, as building community trust and protecting all of our users is critical to us. We have modified the Search-Shield component of LinkScanner to only notify users of malicious sites; this modified version will be rolled out on July 9th 2008. As of this date. Search-Shield will no longer scan each search result online for new exploits, which was causing the spikes that webmasters addressed with us. However, it is important to note that AVG still offers full protection against potential exploits through the Active Surf-Shield component of our product, which checks every page for malicious content as it is visited but before it is opened.

Hopefully that’s really from GriSoft, and it’s the last thing we’ll hear about the problem.

Interesting Terms of Service on Google Gears… not only does it explicitly exclude open-source products from the usual disassembly prohibitions, but it enjoins you (if you develop code that uses it) to “protect the privacy and legal rights of those users” (section 5.5). Not the sort of thing you generally see in a software license, but one I heartily endorse.

As mentioned previously, I’ve been using the Avant Window Navigator (AWN) program as my main launcher for the last little bit. But I’ve taken advantage of this holiday weekend to figure out another program I’ve heard really good things about, and after using it for a couple days, I’ve completely replaced AWN.

The program is GNOME-Do, and it’s a much better fit for the way I work. Graphical menus are indispensable for general use, but I was raised on command line systems, and I still prefer the keyboard over the mouse for a lot of things. Once I know the name of the program I want, having to navigate through menus to find it each and every time I want to start it is irritatingly slow.

GNOME-Do, apparently inspired by the Mac’s Quicksilver program (which I’ve never seen), is called up by a hot-key combination (the Windows key plus the space-bar, by default). When it appears, you just type in a few letters for the item you’re looking for, and it shows you the closest match. If that’s not the one you want, hit the down-arrow and you’ll get a list of the next-closest matches. Once you get the one you’re looking for, the second pane shows the most likely action for it (such as, in the case of a program, “Run”; press Tab to move to that pane, and you’ve got a pull-down menu of other options). Press Enter, the Do window vanishes, and that action is done.

You don’t have to type the first letters of the name either, the program is smart enough to match letters that are within the name, even non-sequential ones. For example, if I type “mssen” (accidentally skipping the first ‘e’), “Pidgin Internet Messenger” is the third item on the pull-down list. Instead of having to go back and correct my mistyping, I can just press the down-arrow a few times to get to the entry I want, and Enter to start it!

And it learns about the way you work, too. If you make that mistake a few more times, it will notice, and start offering Pidgin as the default item when you type “mssen”. Or, if you like, you can explicitly assign “mssen” (or “im”, or anything else you want) as an alias for Pidgin.

Do isn’t just a program launcher, either. Want to quickly open a web page? Type the URL into Do, it’ll automatically recognize it and offer to open it in your web browser. With the Firefox plug-in activated, it will search through your Firefox bookmarks as well. Turn on the “Files and Folders” and “Locate Files” plug-in, and you can type part of the name of a file, and have Do find, copy, move, or delete it, as well as just running it. Want to send an instant message to a particular person, or call them via Skype? Turn on the Pidgin or Skype plug-ins, and Do will recognize the names of your contacts from those programs and let you instantly start a call or IM session with them, or even change your status, without having to open the program separately. Want to play a particular song file you’ve got on your computer? Turn on the Amarok or Rhythmbox plug-ins, and you can start playing it with just a few keystrokes.

I couldn’t figure out the attraction of Do at first, because the version currently in Ubuntu’s repositories (0.4.1, I believe?) isn’t as useful. There’s no configuration panel in it (as there is with 0.5), and apparently many of the things that are optional plug-ins in 0.5 are always active in the earlier version, which can really clutter up the pull-down menu with items that aren’t relevant to you. The Launchpad version, which can easily be installed, is much better.

GNOME-Do is Linux-only, of course (apparently there’s another Linux program called Katapult if you prefer KDE-based solutions, though it’s said that Do will run under KDE as well), but if you’re using Windows, I hear that there’s a free (and also open-source) program called Launchy that does the same thing there. Give one of them a try, you might just find that you like the keyboard better than the mouse too.

I generally agree with Joel Spolsky on most things computer-related, but I’ve been thinking about this for a few days, and I just have to disagree. I see where he’s coming from, but it seems to me that there’s a better way to let the user know why an option is disabled (maybe a mouse-over tooltip?), rather than leaving it enabled but useless.