As I’ve mentioned before on this blog, my previous company was purchased by a larger company (I’ll call it BigCo for the rest of this entry) several years ago, and they still keep me on the payroll for consulting and the more intricate development work on Project Badger.
BigCo is not a software company, it’s a web company that just happened to have a need for Project Badger. As such, it doesn’t have very advanced software development tools, or any training for them, something that I’ve griped about before. The specific tool that I’m talking about today is version-control software, which they use CVS for.
CVS was a very decent tool for its time, but its time was a quarter-century ago. More recently, and particularly in the last ten years, much better version-control tools have arisen. And with the introduction of Mercurial and Git, distributed version control tools — which are a much better fit for groups of developers that aren’t all in one building — have finally become mainstream.
The way they have to use CVS isn’t very efficient either. It requires a lot of manual labor to vet and merge changes, for instance. But the biggest pain-point recently has been that I don’t have access to it at all, because I’m not within BigCo’s internal corporate network. There have been several cases recently that required some coding from me, which just exacerbated the problem… I don’t always have the latest source code to work on, and have no way to know about it, or to get it except having it manually sent to me (heavily encrypted, of course) by e-mail or download. It’s too big to pack up the whole thing, so that requires manual intervention by someone inside BigCo too. And then when I’m done, I have to figure out what files I’ve changed, pack them into a zip file, encrypt them, and send them back, where they’re manually merged into the main body and checked into CVS.
There is a better way, and I spent about fourteen hours on Saturday researching it and setting it up.
As I’ve mentioned before, I use Git (a distributed version-control system) for my own development. Git has the wonderful ability to update remote repositories via the secure network protocol SSH. It also has a set of scripts, known as Gitosis, to make it very simple to set up a secure Linux-based Git server. I’ve already got an SSH server set up in my network (so I can get to it when I’m on the road), and setting up a Git server as well didn’t require too much additional work. (Getting it set up securely, and in such a way that the repositories are secure even if the hardware gets stolen, and are still automatically backed up properly… that’s what required the vast majority of the fourteen hours. But I digress.) Now, once we set up his end on Tuesday, the lead developer on Project Badger in BigCo has direct access to my development repository, and can update it or pull changes from it with ease. He still has to deal with CVS on his end, but this should make things a lot easier for both of us.
Now comes the problem.
It’s well known that there are a number of people trawling the Internet for SSH servers and trying to break into them using dictionary attacks on account passwords. One of this blog’s regular readers, who shall remain nameless (cough Ploni cough ๐ ), was hacked that way a few years ago. My existing SSH server doesn’t allow password-based logins at all because of that threat (only public-key authentication), and is on a non-standard port as well. But since I had to put the new SSH server on a different machine, I decided to try an experiment: still set it up for public-key authentication only, but leave it on the standard SSH port, and see what happens.
Normally, with the setup I used, I think this would work just fine. But I’m running the new SSH server on a Linux virtual machine, running under VMware Fusion on a MacBook Pro. What happened was that last night, a couple hours after opening the firewall for it, the system slowed to a crawl. It cleared itself up after about half an hour, with no evidence as to why it had happened, but I had my suspicions. This morning I wasn’t able to contact the system at all, so I visited the office, only to find that I couldn’t even get it to come up — it wasn’t locked up, but it was so slow that after five minutes, it still hadn’t managed to pop up the password prompt. Big Red Switch time.
Circumstantial evidence suggests that Mac OS X doesn’t handle being bombarded with failed SSH logins very well.
So I switched this one to a non-standard port too. I’m rather disappointed that I had to, but hey, whatever works. We’ll see whether it suffers from the same problems. I don’t think it will.
Was it OS X or Fusion that doesn’t handle the bombardment well? Maybe a virtualized server in a consumer-grade VM is an answer to OS X’s swiss cheese security, but it might not perform too well under load, as you’ve seen….
From what I’ve heard of OS X’s security, and the fact that VMware’s products are used in a lot of high-traffic network applications with no problem, I’m willing to bet that OS X is responsible.
Pingback: Geek Drivel ยป Blog Archive ยป Git, Gitosis, and SSH
VMWare Fusion is not one of their server-oriented products, according to someone who’s an OS X Server admin (which he now hates, but can’t get his Apple-fanboy boss to switch ๐ ) that I talked with on IRC. Of course, then again, it could be OS X, but OS X’s network stack is lifted from FreeBSD and that’s pretty solid.
Hm. I don’t know which is to blame then. Not that it matters in the least, since I’ve already switched the port.
I ran a very low traffic ssh server, on the standard port, for tunneling VNC, under OS X on my Mac mini and didn’t have those problems. It used a password instead of a key at that, which was perhaps a bit reckless given my previous experience with Debian and a weak root password, but I don’t think the VNC client on my PalmOS Centro phone supported anything else. ๐
I used a little script that detected repeated login attempts and would block the offending IPs using hosts.deny, for security reasons, maybe that’s why I didn’t have problems.
Yes, that little script would stop dictionary attacks cold. A distributed dictionary attack might get through it, but they’re very uncommon at this point.
I don’t think that would solve the problem I had though. If it is an OS X or VMware Fusion problem, it would happen before the script had a chance to do anything.