Musings of an anonymous geek

November 13, 2007

I’ve Been Added

Filed under: Me stuff,Productivity,Python,Sysadmin,Technology — m0j0 @ 8:08 am

Friday was my last day working as a contractor for the GFDL. I had friends there before I ever worked there, so I’m sad to be leaving them, but I’m happy to be moving on to a really exciting opportunity. I’m now an employee of

I’ll be able to focus on architecture, and how to scale out an internet-based service, which is quite different from scaling an internal IT infrastructure, if only because the growth is exponentially faster. I’ll still be working with databases and virtualization, web servers and DNS, but this opportunity also gives me the chance to do quite a bit of development work, which I’ve been wanting to do.

Best of all – none of the development work I’ll be doing involves designing interfaces, and the back end stuff I’m working on is mostly Python. I’m happiest when I’m not coding stuff that has to render aesthetically pleasing results. That tier is best left to people who specialize in that. I just munge data and code business logic. 🙂

So, at some point, I have to make a choice as well, regarding my blog, because I’m not going to work for and then *not* use the product (and truthfully, I *want* to use the product). Just having social bookmark links is ok, but being able to view statistics related not only to what eyeballs landed on the page, but to what people found interesting enough to bookmark is valuable. has been good to me, but I’ve been feeling the pain of not being able to add really *any* per post goodies for some time. For a while I was manually adding social bookmark links to each post, and manually adding technorati tags to each post. Know what happens when you do that? It makes blogging more of a chore, and something I’m less likely to do. So do I host the blog myself somewhere else, or do I find another non-wordpress solution altogether? I had a nightmare of a time with Blogger – has it gotten any better? Does it support trackbacks yet? Can a blog published to a url have labels?

What about other blog services? Are there any that provide the niceties of the WordPress software, but without the limitations of the service? I really like categories, I like (but don’t need) the GUI blogging interface, and I *really* like that I don’t have to host it myself and maintain the blog software. Ideas are hereby solicited.

October 25, 2007

Recovering deleted files from an svn repository

Filed under: Sysadmin,Technology — m0j0 @ 9:13 pm

I know I’m going to forget how to do this, because I only ever need to do it once a year or something, so I’ll put it here for safe keeping:

To recover a file from svn that you deleted from your local repository, it’s first necessary to get the proper name of the file, and the revision of the repository it last existed in.  To do that (assuming you don’t know, because if you do you have bigger issues), you go to the directory it was in (or as close as you can get to the directory it was in) and run:

> svn log --verbose

You should be able to find the file you’re looking for and the revision you need in the output of that command. Assuming your file’s name is ‘file.txt’ and it was in revision 250, you run the following to recover it:

> svn up -r 250 file.txt

Done. It’s there waiting for you. Enjoy. I had been fumbling around with ‘svn co’ syntax until a digital buddy of mine corrected me. Thanks, Nivex!

October 8, 2007

With Great Funding Comes Great Responsibility

Filed under: Big Ideas,Sysadmin,Technology — m0j0 @ 9:28 pm

For the past ~6 weeks, I’ve been talking to people, getting buy-in, educating users and administrators, and generating copious amounts of project proposal and six sigma documentation presentig VMware Infrastructure as an infrastructure building and management tool.

There’s a whole manifesto behind this, but I’ll try to boil it down. Basically, this client has three sites, and the infrastructure needs to be consistent at all three sites. Also, ideally it would be overseen and generally managed from one site (there are obvious limits to this, but you get the idea). My thinking is that I have three choices:

  1. Order/rack/setup/test hardware and software, stage system, install stuff, ship to site, where someone else racks machine and turns it on.
  2. Assume and require that there is a senior enough admin at each site already to take care of all of that.
  3. Decouple the OS image from the hardware altogether and just build an infrastructure server “factory” at the main site, and ship (read, scp or similar) to the VMware servers at the other sites.

I chose option three – but this is oversimplification and doesn’t go into all of the benefits.

So, I just found out today that my bill has made it through Congress, and my project now has legs (read: funding)!! When the project is complete (the first phase is to migrate the main site using this methodology, and replication to other sites is a later phase of the project), I’ll try to give a talk on it or something.

In the meantime, if anyone has thoughts on virtualized infrastructure, or if you’re doing something cool with this technology, please post your comments. I value your insight!

September 23, 2007

Sometimes it’s just about attitude

Filed under: Me stuff,Sysadmin,Technology — m0j0 @ 10:33 pm

Today I spent the day completely breaking down and replacing the entire above-ground portion of my in-ground pool’s plumbing. I had some problems crop up with my multi-port valve, and I had to take a few things apart to troubleshoot and remove the pump as the source of a pressure problem (namely, there wasn’t any).

Without going into too much gory detail about the troubleshooting, suffice to say that, over the course of the past few weeks, I’ve spent a great deal of time in the pump house. More than I ever had to before. While I was in there I had some time to closely inspect the plumbing. While taking a break and sipping a home brewed scotch ale, I just sorta stared at each component in the line and thought about the logic employed in the overall project. By the end of my 5-minute break, I was not happy.

This plumbing was put in place by a contractor that was hired by the previous owner of this house. Thinking about that got me really steamed for two reasons;

1. I know the previous owner of this house, and he’s a great guy.

2. The contractor completely fleeced the previous owner :-/

The pipe coming out of the ground is 1 1/4″. It went downhill from there. Connected to the black pipe coming from the ground a 1 1/4 barb/thread adapter, and held in (in part) by a worm clamp. Actually – two worm clamps. And guess what screwed into there? A 1 1/4″ thread/barb adapter! Shoved onto *that* with two worm clamps was another length of the black pipe, which went to a very sharp, and *iron* threaded elbow joint which had a 1 1/4 thread/barb bit attached to connect the pipe. Yes, more worm clamps. From there there was an enormous, old, and broken brass or iron valve, and then we see our first bit of actual PVC – a union going into the pump.

Everything in that last paragraph has been completely replaced with PVC.

From the pump to the multi-port valve, there’s a PVC elbow that’s still in tact, but it had a simple barbed adapter, to which was attached 1 1/2″ vinyl tubing…. with worm clamps. The vinyl tubing connected in the same exact manner on the multi-port valve side.

Everything above was replaced with PVC, and the connections to the valve and pump are now either threaded or glued – but not barb fittings, and there are no worm clamps.

The return line (from the multi-port valve back to the pool), was, if you can imagine, even worse. A barb stuck out of the multi-port valve, and attached to that was the 1 1/2″ vinyl tubing – using worm clamps. The tubing attached to another barbed fitting on the other side that was screwed into another enormous brass or iron valve that was so broken it was not ever used. The valve actually sat on a milk crate, and I never really had any reason (or time) to consider why this is: it’s because instead of the valve being attached to some kind of rigid pipe with an elbow, it just sat on a milk crate and was connected to the ground pipe using….. vacuum hose.

Yes, that vacuum hose. If you had a pool growing up, you might’ve swung lengths of this stuff around over your head to hear the noise it would make. Or you might’ve tied up your little brother with it. The point is, it’s a toy. It’s not meant for this kind of application. It’s the wrong size, but of course, worm clamps to the rescue!

I know this is all kinda hard to believe, so I took a few pics of the “before” version of the plumbing during the takedown/troubleshooting phase of my project.


So above, Note that big valve – that’s coming from the pool into the pump. See all that black electrical tape? Yeah – that’s bad to see in a plumbing design. There was also some shrink wrap stuff on parts of the pipe that are under this table. You can also see here the infamous milk crate :-/


Here’s the other half – vinyl tubing, barbs and worm clamps. Oh my! In its defense, it sorta worked, and had the nice side effect of being able to see the water moving through the tubing. I guess that’s cool. Of course, my pump has a clear lid so you can see what’s going on, and there’s that nifty pressure gauge on the multi-port valve so….


This is a shot of my ‘testing’ setup. It’s a closed loop – the water comes from the bucket, up the hose on the left to the pump, and out of the pump back into the bucket. This worked wonderfully and allowed me to prove to myself that I didn’t need a new $x000 pump 😉

The funnier part here is that you can now see clearly in the background that the return valve is connected (with a worm clamp!) to the black and white vacuum hose! 😛

I told you all of that so I could tell you this

Nobody ever questioned this monstrosity of a setup. Somebody came in and set this up, and somebody else paid for it, and was happy to not have to think about it. It was clearly shoved together with whatever parts the guy had lying around. And “shoved” is really the perfect word here. The bottom line is that this was a fiasco. An adventure. It was not taken seriously at all. I’m starting to see things like this more and more in both my professional and non-professional work. People don’t take the work they do seriously. They don’t think much about their work. They’re here today, gone tomorrow, and they’ll get paid on Friday, and that’s all that matters.

This whole plumbing system is just that – a system. Systems require thought. A well-functioning system is almost always the result of some amount of thought and design and perhaps (gasp!) discussion. Just because they’re locked away in a place nobody ever goes doesn’t mean they’re not important. In fact, most things that are put out of the way are put there specifically because they *are* important.

If you build systems, please take the building of those systems seriously. If you’re inheriting or coming into an existing system, or managing the building of a system, question it. Question everything. Ask why. Ask what the alternative solutions are. Ask what is compromised by doing it one way as opposed to another. Ask was is commonly done in your scenario and why. Be curious and skeptical. Look for red flags. Should $.10 tin worm clamps be used in a plumbing system? Won’t they rust to the point where they are unusable? Certainly this can’t be the optimal solution? What are some alternatives? Odd – I’ve never seen vacuum hose used in that way, can’t we use something else?

I know this is goofy, but this is how my brain works: every single thing you can learn is likely to be analogous to either a) something you already know or b) something you can more readily wrap your brain around. The good thing about that is that it means you can pretty much learn every single thing 😉

In this case, my pool plumbing system is analogous to a computing system infrastructure, inasmuch as both are systems, made up of many and varied components – and each component represents a decision that must be made, and a potential source of a future problem. Does the component fit? Is it made fo this application? Does it integrate well into the rest of the system? I could be asking those questions of almost any component of almost any system whether it moves bits or water.

Well, enough rambling for one night. There was a message in that somewhere, but it’s late, and I’ve been working with PVC all day. 😉

September 17, 2007

Two ssh-isms I forgot today

Filed under: Sysadmin,Technology — m0j0 @ 1:27 pm

1. To remove an offending known_hosts key without opening the file or using sed or whatever, you can just run ssh-keygen -R <hostname>, which will remove any and all keys for the given hostname (see man ssh-keygen for other options like this).

2. To copy files from a local directory with spaces in its name to a remote one with spaces also in *its* name, escape the space in the destination directory *twice*. I was moving VMware virtual machines from one server to another today and it bit me. Here’s the right way to do it:

scp -P 2222 *.vmdk “localhost:/var/lib/vmware/Virtual\\ Machines/ks/.”

Note that I’m also working in an environment where everything I do requires various ssh tunnels and other workarounds, which necessitates the -P argument, and explains why it looks like I’m ssh’ing to localhost 😉

(Thanks to deryni for reminding me about ssh-keygen – I was looking for an ssh flag for this)

UNIX mtime vs. ctime

Filed under: Linux,Sysadmin,Technology — m0j0 @ 8:27 am

Sometimes I get questions from people about stuff that I’ve long since taken for granted. One is “what’s the difference between mtime and ctime?”

The answer is simple, but I wanted to post it here in case it can help anyone. In UNIX:

mtime is “modification time”, and it is the time at which the last modification *TO THE CONTENT* of the file was made.

ctime is “change time”, and it is the time at which the last modification *TO THE PROPERTIES* of the file was made. By “properties” I mean things like ownership and permissions.  You can think of this as “chtime” – the last time a “ch*” command like chown, chmod, or chgrp was performed on the file.

September 7, 2007

New Job!

Filed under: Linux,Me stuff,Python,Scripting,Sysadmin,Technology — m0j0 @ 7:32 am

I started a new job about 6 weeks ago. I’m now doing infrastructure architecture at

GFDL stands for Geophysical Fluid Dynamics Lab. It’s a NOAA site that supports atmospheric and climatology research. So in other words, the work I do supports research into things ranging from global warming to what the atmosphere on Mars is like to the weather here on Earth to simulations of the shape and movement of Katrina. I think of it as sort of an Institute for Advanced Study devoted to climatology research. Great minds in the field are here.

The research actually takes place at three different sites, DC, Boulder and Princeton, and affiliations with academic institutions flourish as well. In fact, I knew at least 4 people who worked here because of interactions between this site and, my former employer.

My job, as it’s been described to me, is to provide a vision as to the design and direction of the infrastructure which supports the rather enormous high performance compute (HPC) cluster. This involves something of a learning curve to understand what’s here, how the systems are used, what the needs are, what people like and hate, where the redundancies and inefficiencies exist, etc. It also involves having meetings and coordinating with people who manage the network, the facilities (power & cooling, etc), the security policy, etc. I’ll be grilled on my ideas, and create prototypes and demos to get my ideas across. Lots of communication.

An aspect of my job will also involve getting my hands on the HPC clusters themselves as well, which are also at each site. All of the clusters are on last time I looked. Just go through the pages and search for GFDL and/or NOAA.

The systems here are all Linux. Even the standard-issue workstations are running Linux.  Scripting is done in Perl and shell, but Python is everywhere, so I’ll be doing either Perl or Python if I have the choice (because “shell” == “csh” here, which I never took well too, honestly). Some aspects of the environment are pretty fascinating. For example, how exactly do you store (*and* easily retrieve, on the fly) 9 PETABYTES of data? How do you back that up? How do you recover from hiccups? How do you instrument systems consisting of thousands of CPUs,  to pinpoint problems and get them fixed? And, by the way, how’s the best way to tune a system’s network stack to use a 50MBps pipe (that’s Mega *bytes*) efficiently enough to move multiple terabytes of data every day between collaborators at different sites? How, exactly, do you consolidate services and provide failover across geographically dispersed sites?

So that’s it for now 🙂  It’s too early to tell how things are going, really. It’s certainly not the cushy environment that Princeton U. was, but there are bigger challenges and problems to be solved here, and that’s the part I’m looking forward to.

July 6, 2007

On the demise of SysAdmin Magazine

Filed under: Big Ideas,Sysadmin,Technology — m0j0 @ 12:57 pm

CMP recently announced that they will cut 200 jobs, and shut down more than one magazine in the process, folding their content into other existing magazines. There has been a lot of buzz in the sysadmin community (which I know largely as a loose collection of people who belong to LOPSA, SAGE, USENIX, or local groups of various kinds) about how sad and unexpected this is and how it’s a sign of the times or something.

You Can’t Sell a Generalist’s Mag to a Market of Specialists

I actually think SysAdmin’s demise was a long time coming – in part because, well, I’m involved in the publication of a couple of magazines, and read lots of others, and know people who work in publishing on other magazines as well as large online media outlets (some of whom I also write/edit/consult with). The plain fact of the matter is that it is extremely difficult to cover a topic like system administration in a generalist sort of way when your audience no longer consists mostly of generalists.

For example, the last two issues of SysAdmin magazine I remember getting were about Database Management, and Linux, respectively. The database management issue talked about Oracle and MySQL, and then had 3 or 4 articles on things not really directly related to databases at all, if memory serves. The Linux issue is probably of no interest to the admin who was all revved up for the Oracle articles, because that admin is probably not so much a sysadmin as an “Oracle admin”. Meanwhile, the average Linux administrator is probably uninterested in the Oracle RAC Primer.

Most admins aren’t generalists anymore unless they work in academia, research, or a company small enough that there are only a couple of people to handle the entire infrastructure. Even people who would like to be more general aren’t doing generalist things in production. I know several people who work *only* on {Oracle, sendmail, websphere, whatever-other-service-you-like} at work, and nothing else, but they run Apache, Bind, Postfix, and a few other services at home. That knowledge is nice to have, but it’s hardly something you can use to market yourself as a production generalist administrator.

So the market is flooded with mail administrators, backup administrators, storage administrators, cluster administrators, network administrators, database administrators, websphere administrators, exchange administrators, desktop administrators… the list goes on and on and on. The magazine market has mostly followed suit. There are magazines about cluster computing, DB2, AIX, SQL Server, Linux servers, Windows servers, and lots of other specialized areas. Note in that list that there are two separate magazines for two separate database products, but no “DBA’s Journal”. In killing SysAdmin Magazine, CMP is just following along with market trends.

Programming has mostly gone in the same direction. Dr. Dobb’s Journal is no longer available on your local bookshelf, but you can find php|architect, a magazine about nothing but PHP, doing well. Others exist for .Net, Cold Fusion, C/C++, Java, and now even Python. I’m sure Ruby and Lua aren’t far behind.

“I know, let’s start our own magazine!”

There are lots of naysayers out there who seem to think you need a huge staff, tons of money, and loads of other resources to start a magazine. Not true. You need a few dedicated, motivated people, and a small amount of seed money, and some time and hard labor.

An example is php|architect. It started with two guys: the publisher (who was a software development consultant as well), and his business partner, who also knew Quark pretty well – plenty well enough to do the layout. Then I came on board. So after the first issue came out, the rest were all tech and copy edited by the publisher and I, and the layout was done by the other guy. That’s really minimizing the amount of work we did, but the point is you don’t need an army.

When you’re as big as CMP and you’re looking at cutting a million bucks or something from your operating expenses, you go ahead and cut away! A small outfit isn’t playing with these kinds of numbers. A full-time editor at a company like CMP probably makes a 6-figure salary. A guy like me who edits as a side job makes something like… far less than that. At that rate, you can add another editor, a tech editor, a couple of columnists, all part time, and the advertising revenue will still cover the costs.

So if you want to see a sysadmin magazine, get some dedicated people who have half a brain and go do it!

Good luck.

Technorati Tags: , ,

Social Bookmarks:

May 18, 2007

Regular Expressions with Python’s “re” Module

Filed under: Python,Scripting,Sysadmin,Technology — m0j0 @ 3:11 pm

If you’re moving over from PHP, Perl, Ruby or something similar, don’t be intimidated by all the Python regular expression documentation. It doesn’t really have to be complicated or even all that much different from Perl (though it can be, if you want to go there).

Here’s a search and replace I ripped out of a Perl script for use in a Python script that replaces it. It insures that any MAC address fed to it has two digits in every field. So, for example, this would change “0:c:e:fe:d0:ae” to “00:0c:0e:fe:d0:ae”. This is good if you need to insert the value into a PostgreSQL column of type ‘macaddr’, or you just want to be consistent.

Perl: $macaddr =~ s/\b([0-9a-f])\b/0\1/ig

Python: macaddr = re.sub(r'(?i)\b([0-9a-f])\b', r'0\1', macaddr)

There are a few differences when moving to Python. First, there’s only one assignment operator in Python (to my knowledge – comment to correct me if I’m wrong) – so we’re calling a function instead of using “=~”. That’s fine with me. Less cryptic symbols are better.

Second, part of calling a function also means that the operation is explicit: we’re doing substitution using the “sub” method. There’s no “s/” like there is in Perl.

Third, there’s also no “/ig” in Python like at the end of the Perl example. The “i” means “ignore case”, and in Python, that indication (the “(?i)”) goes next to the pattern in question instead of at the end of the line. That’s easier for my brain to parse. I like to read what I’m doing in my native language (English), and if you think in that context, then reading regexes in Perl is kinda like reading in German, not English.

Finally, calling a function also means that the pattern and the thing you want to apply it to are separate arguments to the function instead of things that are delimited by more “/” characters. In fact, in Python, the only slashes of any kind appear only in the regular expression syntax. None of the actual language syntax contains a slash.

Though there are lots of differences in just this one very very simple example, I’ll also note that the actual regex syntax itself (the parts inside quotes for the Python example), are not different at all except for the addition in the Python example of the “ignore case” operator “(?i)”!

Technorati Tags: , , , , , ,

Social Bookmarks:

May 16, 2007

Using TRUNCATE to empty a PostgreSQL database

Filed under: Database,Me stuff,Sysadmin,Technology — m0j0 @ 10:22 am

This is not something that’s any big hack or secret, but emptying a database of all data without dropping the structure along with it is one of those tasks that I do just often enough in my development work to be annoying. If you ask me, there should just be a big ol’ “EMPTY” statement you can apply to an entire database.

You *can* empty a database table in PostgreSQL using an unqualified DELETE statement, by the way – but it takes longer because it does a full scan of each table. TRUNCATE just nukes everything – and if you feed it the ‘CASCADE’ keyword, it’ll nuke everything in its path as well. This is nice, because I have a bunch of tables in my database, but I know that there are a relatively small collection of tables that everything else links to, so I can pass about 10 table names to TRUNCATE, and giving it the ‘CASCADE’ argument will wipe out about 2 dozen tables.

While I love writing code that creates stuff, writing code to do demolition is somehow amazingly satisfying as well.

Technorati Tags: , , , , ,

Social Bookmarks:
Next Page »

Blog at