Musings of an anonymous geek

October 21, 2007

“For the Community”

Filed under: Big Ideas,Technology — m0j0 @ 10:08 pm

Sometimes people claim they’re doing things for the good of the community, but I guess that doesn’t necessarily mean they intend to involve the community in the effort :-/

A group of open source/free software users in New Jersey (where I currently reside) learned that the hard way when the maintainer of a web site that advertised it was “For the Free and Open Source Software Communities of New Jersey” posted a shut down notice.

The biggest slap in the face to the community the site was allegedly for was the text of the shutdown notice itself. For example:

“Maintaining GnuJersey has been mostly fun, but I want to prune the list of blogs I read daily, and I can’t do that while I maintain a website featuring some blogs I don’t want to read.”

So… this is a site “for the community” whose shut down notice contains 5 instances of the word “I” in the single sentence that is supposed to give us some clue as to why this is happening.

But wait! There’s more!

“[The site being taken down] is not up for transfer and I will not use DNS to point to a successor blog aggregator.”

Sweet. Not only is he not entertaining the idea of maintaining the site himself, he’s also eliminating the possibility that the site will be maintained “for the community”, by anyone, at *all*.

He does offer to link to a successor site, but insists that we get permission from the syndicated bloggers (and presumably, that we prove that we have said permission), if we expect him to link to us. So, we shouldn’t have any expectations of him to live up to his word and maintain the site “for the community”, *or* to let the community maintain the site for the community, but we should honor his request to prove that we have permission from all of the authors involved to put a successor site in place.

Rich, ain’t it?

Well, I’ve cloned the site here for whoever wants to continue to keep up with their friends and colleagues involved in open source and free software in New Jersey.

Advertisements

October 17, 2007

For my next pet project…

Filed under: Me stuff,Productivity,Python,Scripting,Technology — m0j0 @ 9:24 am

Stand back!

running install_egg_info
Writing /usr/lib/python2.5/site-packages/gdata.py-1.0.9.egg-info
brj@dawg:~/working/gdata.py-1.0.9$ python
Python 2.5.1 (r251:54863, May  2 2007, 16:27:44)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdata
>>> print 'yay!'
yay!
>>>

No good can come of this! ;-P

Seriously, though – I really really strongly dislike spreadsheet interfaces. I hate resizing cells so I can see what’s going on, I hate cell selection, copy/pasting, and doing anything in those little cells. However, I really *need* to use one to handle some administrivia at Python Magazine, because it’s already being used by some back end processes/departments, and I don’t have time to write code and overhaul that whole process, and I don’t want to rock the boat anyway – what they have works – I just hate spreadsheets. It’s my problem, not theirs πŸ˜‰

The good news is they use Google Docs, and there’s a Google Data client library for Python. So I’m creating a command line interface to the spreadsheet πŸ™‚

October 10, 2007

The Technology Behind Python Magazine

Filed under: Big Ideas,Python,Technology — m0j0 @ 9:04 am

Hi all,

I mentioned to a buddy (who is also an editor) that we used subversion in our editorial process. He didn’t know what that was, and said that they used either this big nasty home grown system, or email attachments, to coordinate the editorial process.Β  He was incredibly curious about how we used subversion and what else we were using.

I started writing this kind of long email and then figured that others might be curious as well about the various technologies we use (or are moving to, etc) at Python Magazine, so here’s a quick list of tools we’re currently using:

  • Subversion – of course, I’ve mentioned this. We view every email attachment as a problem to be solved. Email is a communication tool. It is not a file transfer protocol (no, really – it isn’t), and it is certainly not a collaboration tool. We have a very simple directory hierarchy on the server representing the various stages in the editorial process, from the initial, original submission as received by the author, all the way to the final PDF rendering of the entire magazine, and all parts in between. The final review of the magazine even happens in SVN. We have a ‘corrections.txt’ file that we all add to as we review the PDF, and when that file is empty, the PDF is moved to the directory representing “go to press!”
  • Plain text – sometimes less is more. I’ve edited and authored using Word, OpenOffice, LaTeX, and a few other tools. In the end, plain text with extremely simple and minimalist formatting tags win the day by a long shot. Authors aren’t forced to use any particular tool or platform to write their articles, editors don’t have to wonder which version authors have, which language setting they were using, etc. We don’t have to wonder if our version control system will handle a binary format properly, and the files are smaller. It’s also easier to run scripts against them to do things like strip formatting, or selectively apply it given a regex or something.
  • Google Calendar – We are notified the night before any article deadline, and the calendar is shared among the editors. Theoretically, the same calendar could be used to indicate that an editor is going to be unavailable or a tech reviewer is going to receive an article, but so far, it mainly reminds us of upcoming deadlines.
  • IRC/Google Talk – We actually don’t send very much email to each other. Sometimes we talk on IRC about emails we received or need to be added to the cc list of, etc. Almost everything we do involves either IRC or Google Talk. Of the 50 or so people on the authors mailing list for Python Magazine, at least 40 have gmail.com email addresses, and so do all of the editors here, so even some of the author/editor communication is email-free. In addition, the Python Magazine IRC channel is irc.freenode.net/#pymag, and you can talk to editors and authors there. The only email that gets sent is:
    • subversion server updates,
    • users who need to mail info at pythonmagazine to ask subscription questions,
    • authors sending to editors at pythonmagazine to submit article ideas (we don’t take them on irc),
    • replies to threads, usually initiated by one of the above actions.
  • PHP – yes, believe it or not, the main site is written in PHP. The publishing company (MTA) was originally formed around php|architect Magazine, which is a magazine about PHP. That was in 2002. Today, there are two language-based magazines. Some day there may be five language-based magazines. Certainly, we’re not going to maintain websites using 5 different languages! O’Reilly doesn’t do it, and they publish entire *books* on different languages (and platforms! and databases!) I was impressed by the Python community’s understanding in this matter. Lesser communities would’ve sent lots of hate mail.
  • Python – Doug Hellmann (our tech editor) and myself (to a lesser degree, because Doug is far better at it) write any little tools and scripts we need using Python. Sometimes I think about writing Python scripts just to make Doug laugh. Don’t forget, I launched this magazine not because I professed any deep knowledge of Python. On the contrary – it was because I figured there were neophytes like myself who would like to know more, and advanced coders who would like to look into areas of Python outside their immediate area of expertise within the language.
  • Adobe InDesign – InDesign is the main layout tool. Layout is like some spooky ethereal realm to me. I imagine other tools are used during the layout/design process, but I don’t honestly know what they are. I’m sure the layout team prefer it that way. It’s probably better if I just say “I’d rather see the title moved up and to the right” than to start trying to tell them how to use their tools.

Those are the tools I can think of off the top of my head aside from back end things like a relatively standard LAMP stack that runs the web sites, and which I also don’t have much of a role in maintaining. Of course, there’s also one big element of all of this technology that blows them all away: the people. Every single person is technical in some way. Me, the layout folks, all of the editors for the whole company that I’m aware of, and even our fearless leader are all technical people. Technology is a common thread that runs through the entire organization, and ties all of us together. It makes an enormous difference, and I’m proud to be a part of the team.

October 8, 2007

With Great Funding Comes Great Responsibility

Filed under: Big Ideas,Sysadmin,Technology — m0j0 @ 9:28 pm
Tags:

For the past ~6 weeks, I’ve been talking to people, getting buy-in, educating users and administrators, and generating copious amounts of project proposal and six sigma documentation presentig VMware Infrastructure as an infrastructure building and management tool.

There’s a whole manifesto behind this, but I’ll try to boil it down. Basically, this client has three sites, and the infrastructure needs to be consistent at all three sites. Also, ideally it would be overseen and generally managed from one site (there are obvious limits to this, but you get the idea). My thinking is that I have three choices:

  1. Order/rack/setup/test hardware and software, stage system, install stuff, ship to site, where someone else racks machine and turns it on.
  2. Assume and require that there is a senior enough admin at each site already to take care of all of that.
  3. Decouple the OS image from the hardware altogether and just build an infrastructure server “factory” at the main site, and ship (read, scp or similar) to the VMware servers at the other sites.

I chose option three – but this is oversimplification and doesn’t go into all of the benefits.

So, I just found out today that my bill has made it through Congress, and my project now has legs (read: funding)!! When the project is complete (the first phase is to migrate the main site using this methodology, and replication to other sites is a later phase of the project), I’ll try to give a talk on it or something.

In the meantime, if anyone has thoughts on virtualized infrastructure, or if you’re doing something cool with this technology, please post your comments. I value your insight!

October 5, 2007

Python Magazine Status Update

Filed under: Python,Technology — m0j0 @ 8:59 am

First, and most important, Python Magazine’s premiere issue has been unleashed!

I love this business. Doing all of the negotiations, the communications with authors, coordinating with layout and contract people, web administrators, tech editors, and the like, can get pretty chaotic. It’s sort of like managing…. a tornado. And, like a tornado, one second you’re not sure you’re going to make it, and the next everything is just fine. Finally, at some time after midnight yesterday, I looked up at my TODO list and realized that Python Magazine’s premiere issue was complete.

I didn’t know what to do. “Do I just… go to bed or something?” When you do what we at MTA like to call “marathon editing”, the moment you stop you get this weird sensation – like the one you get if you go roller-blading for two hours and then take a walk immediately afterward. Your body has to get used to *not* roller-blading.

About the First Issue

So, the biggest news associated with this first issue is that it is completely, 100% free. That’s right! You can go download the PDF at will, sans payment of any kind. This is big news for a couple of reasons.

First, the magazine costs money to produce. We pay our authors very competitively, and there are also editors, layout/design folk, and other people involved in the production. The business plan for producing a magazine where people get paid for what they do involves selling that magazine to (at least!) cover the costs. The idea that we’re giving the first issue away for free is, if nothing else, a testament to our commitment to and confidence in the product, as well as our stability.

Second, giving away the first issue means anyone can get their hands on it, read it, share it, print it, leave it in the coffee area at work, and do pretty much whatever they want with it. This means more people will see it, and see what we’re going after, and give us feedback so we can make it better.

Third, of course, we hope it means more subscribers, so that we can have the support we need to make the magazine better, and to give more back to the community by providing more services and whatever else we can.

Participate!!

Please send feedback! I want feedback! We all want feedback! Send it to editors atΒ  pythonmagazine dot com

Also, write for us! Never wrote before? Don’t know where to start? No problem – drop us a line (editors at pythonmagazine.com) and let us know what your thoughts are, what you’re doing with Python, how it’s helped you, etc. We’re happy to help you develop the article idea, and get it ready for publication. If you’re intimidated by writing – don’t be. There are lots of authors who have already submitted articles to us who are doing really cool things with Python, have never written for a publication before, and their articles are being published – and they’re great articles!

Anyway, go get the first issue. Let me know your thoughts. πŸ™‚

September 24, 2007

My New iPod

Filed under: Apple,Me stuff — m0j0 @ 7:06 am

My wife and I don’t really celebrate Valentine’s Day. I’m not sure why. I think it started because, for the first couple of years we dated, we were a few hundred miles apart, so we couldn’t. Now it’s just tradition. Anyway, to make up for it, we go all out on our anniversaries. We had several anniversaries before we were married, but this is our second wedding anniversary, and my wife got me at 160GB black iPod Classic πŸ˜€

This is the exact model I wanted. I did not want the iPod Touch. I think the iPod Touch is an insult to my intelligence to tell you the truth. I don’t know why Apple thinks I’m stupid enough to not realize that the iPod Touch is going to magically grow a phone over the next year or two. I don’t know why they think I should spend more money to get 1/10 the storage capacity of the iPod Classic.

The real kicker is that the iPod Touch, which has 1/10 the storage capacity of the iPod Classic, has a direct line to the iTunes Music Store, but the iPod Classic – the social icon around which people build their identities these days, and which has 160GB capacity, doesn’t. Seems like Apple could just put a wi-fi radio in the iPod classic, give it a direct connection to the music store as well, and watch the kids stand around on the playground listening to each other’s tunes and downloading a copy of their own for $.99. Hell, Apple could probably eat 75% of the cost of the data connection, further subsidize it through deals with the ISP, perhaps a small fee built into the initial cost of the iPod, or a $.05 surcharge on songs downloaded directly to the iPod or something, charge the customer *once*, for the iPod instead of a monthly data transmission fee or whatever, and still make money.

For $50 more than the 16GB Touch, you can get the iPhone. It only has 8GB capacity, but replaces two gadgets you may carry around all the time. Your phone, and your point-n-shoot digital camera. I’m not a fan of Apple’s stance toward users who would like to run software not made by Apple on the hardware they allegedly own, but if we’re just talking value and comparing the devices in their line, I have to believe the iPhone is a slightly better value than the iPod Touch.

Anyway, this was really meant to be a glowing review of my iPod Classic. It works wonderfully. I’ve already loaded a few months worth of home brewing podcasts, all the music that I care about (for now – more coming), a bunch of photos of my wife and daughter and stuff, and I’m all set to go. My wife must truly love me. At some point I hope to figure out why. πŸ™‚

September 23, 2007

Sometimes it’s just about attitude

Filed under: Me stuff,Sysadmin,Technology — m0j0 @ 10:33 pm

Today I spent the day completely breaking down and replacing the entire above-ground portion of my in-ground pool’s plumbing. I had some problems crop up with my multi-port valve, and I had to take a few things apart to troubleshoot and remove the pump as the source of a pressure problem (namely, there wasn’t any).

Without going into too much gory detail about the troubleshooting, suffice to say that, over the course of the past few weeks, I’ve spent a great deal of time in the pump house. More than I ever had to before. While I was in there I had some time to closely inspect the plumbing. While taking a break and sipping a home brewed scotch ale, I just sorta stared at each component in the line and thought about the logic employed in the overall project. By the end of my 5-minute break, I was not happy.

This plumbing was put in place by a contractor that was hired by the previous owner of this house. Thinking about that got me really steamed for two reasons;

1. I know the previous owner of this house, and he’s a great guy.

2. The contractor completely fleeced the previous owner :-/

The pipe coming out of the ground is 1 1/4″. It went downhill from there. Connected to the black pipe coming from the ground a 1 1/4 barb/thread adapter, and held in (in part) by a worm clamp. Actually – two worm clamps. And guess what screwed into there? A 1 1/4″ thread/barb adapter! Shoved onto *that* with two worm clamps was another length of the black pipe, which went to a very sharp, and *iron* threaded elbow joint which had a 1 1/4 thread/barb bit attached to connect the pipe. Yes, more worm clamps. From there there was an enormous, old, and broken brass or iron valve, and then we see our first bit of actual PVC – a union going into the pump.

Everything in that last paragraph has been completely replaced with PVC.

From the pump to the multi-port valve, there’s a PVC elbow that’s still in tact, but it had a simple barbed adapter, to which was attached 1 1/2″ vinyl tubing…. with worm clamps. The vinyl tubing connected in the same exact manner on the multi-port valve side.

Everything above was replaced with PVC, and the connections to the valve and pump are now either threaded or glued – but not barb fittings, and there are no worm clamps.

The return line (from the multi-port valve back to the pool), was, if you can imagine, even worse. A barb stuck out of the multi-port valve, and attached to that was the 1 1/2″ vinyl tubing – using worm clamps. The tubing attached to another barbed fitting on the other side that was screwed into another enormous brass or iron valve that was so broken it was not ever used. The valve actually sat on a milk crate, and I never really had any reason (or time) to consider why this is: it’s because instead of the valve being attached to some kind of rigid pipe with an elbow, it just sat on a milk crate and was connected to the ground pipe using….. vacuum hose.

Yes, that vacuum hose. If you had a pool growing up, you might’ve swung lengths of this stuff around over your head to hear the noise it would make. Or you might’ve tied up your little brother with it. The point is, it’s a toy. It’s not meant for this kind of application. It’s the wrong size, but of course, worm clamps to the rescue!

I know this is all kinda hard to believe, so I took a few pics of the “before” version of the plumbing during the takedown/troubleshooting phase of my project.

IMG_1708.JPG

So above, Note that big valve – that’s coming from the pool into the pump. See all that black electrical tape? Yeah – that’s bad to see in a plumbing design. There was also some shrink wrap stuff on parts of the pipe that are under this table. You can also see here the infamous milk crate :-/

IMG_1709.JPG

Here’s the other half – vinyl tubing, barbs and worm clamps. Oh my! In its defense, it sorta worked, and had the nice side effect of being able to see the water moving through the tubing. I guess that’s cool. Of course, my pump has a clear lid so you can see what’s going on, and there’s that nifty pressure gauge on the multi-port valve so….

IMG_1713.JPG

This is a shot of my ‘testing’ setup. It’s a closed loop – the water comes from the bucket, up the hose on the left to the pump, and out of the pump back into the bucket. This worked wonderfully and allowed me to prove to myself that I didn’t need a new $x000 pump πŸ˜‰

The funnier part here is that you can now see clearly in the background that the return valve is connected (with a worm clamp!) to the black and white vacuum hose! πŸ˜›

I told you all of that so I could tell you this

Nobody ever questioned this monstrosity of a setup. Somebody came in and set this up, and somebody else paid for it, and was happy to not have to think about it. It was clearly shoved together with whatever parts the guy had lying around. And “shoved” is really the perfect word here. The bottom line is that this was a fiasco. An adventure. It was not taken seriously at all. I’m starting to see things like this more and more in both my professional and non-professional work. People don’t take the work they do seriously. They don’t think much about their work. They’re here today, gone tomorrow, and they’ll get paid on Friday, and that’s all that matters.

This whole plumbing system is just that – a system. Systems require thought. A well-functioning system is almost always the result of some amount of thought and design and perhaps (gasp!) discussion. Just because they’re locked away in a place nobody ever goes doesn’t mean they’re not important. In fact, most things that are put out of the way are put there specifically because they *are* important.

If you build systems, please take the building of those systems seriously. If you’re inheriting or coming into an existing system, or managing the building of a system, question it. Question everything. Ask why. Ask what the alternative solutions are. Ask what is compromised by doing it one way as opposed to another. Ask was is commonly done in your scenario and why. Be curious and skeptical. Look for red flags. Should $.10 tin worm clamps be used in a plumbing system? Won’t they rust to the point where they are unusable? Certainly this can’t be the optimal solution? What are some alternatives? Odd – I’ve never seen vacuum hose used in that way, can’t we use something else?

I know this is goofy, but this is how my brain works: every single thing you can learn is likely to be analogous to either a) something you already know or b) something you can more readily wrap your brain around. The good thing about that is that it means you can pretty much learn every single thing πŸ˜‰

In this case, my pool plumbing system is analogous to a computing system infrastructure, inasmuch as both are systems, made up of many and varied components – and each component represents a decision that must be made, and a potential source of a future problem. Does the component fit? Is it made fo this application? Does it integrate well into the rest of the system? I could be asking those questions of almost any component of almost any system whether it moves bits or water.

Well, enough rambling for one night. There was a message in that somewhere, but it’s late, and I’ve been working with PVC all day. πŸ˜‰

September 17, 2007

Two ssh-isms I forgot today

Filed under: Sysadmin,Technology — m0j0 @ 1:27 pm

1. To remove an offending known_hosts key without opening the file or using sed or whatever, you can just run ssh-keygen -R <hostname>, which will remove any and all keys for the given hostname (see man ssh-keygen for other options like this).

2. To copy files from a local directory with spaces in its name to a remote one with spaces also in *its* name, escape the space in the destination directory *twice*. I was moving VMware virtual machines from one server to another today and it bit me. Here’s the right way to do it:

scp -P 2222 *.vmdk “localhost:/var/lib/vmware/Virtual\\ Machines/ks/.”

Note that I’m also working in an environment where everything I do requires various ssh tunnels and other workarounds, which necessitates the -P argument, and explains why it looks like I’m ssh’ing to localhost πŸ˜‰

(Thanks to deryni for reminding me about ssh-keygen – I was looking for an ssh flag for this)

UNIX mtime vs. ctime

Filed under: Linux,Sysadmin,Technology — m0j0 @ 8:27 am

Sometimes I get questions from people about stuff that I’ve long since taken for granted. One is “what’s the difference between mtime and ctime?”

The answer is simple, but I wanted to post it here in case it can help anyone. In UNIX:

mtime is “modification time”, and it is the time at which the last modification *TO THE CONTENT* of the file was made.

ctime is “change time”, and it is the time at which the last modification *TO THE PROPERTIES* of the file was made. By “properties” I mean things like ownership and permissions.Β  You can think of this as “chtime” – the last time a “ch*” command like chown, chmod, or chgrp was performed on the file.

September 7, 2007

New Job!

Filed under: Linux,Me stuff,Python,Scripting,Sysadmin,Technology — m0j0 @ 7:32 am

I started a new job about 6 weeks ago. I’m now doing infrastructure architecture at http://gfdl.noaa.gov

GFDL stands for Geophysical Fluid Dynamics Lab. It’s a NOAA site that supports atmospheric and climatology research. So in other words, the work I do supports research into things ranging from global warming to what the atmosphere on Mars is like to the weather here on Earth to simulations of the shape and movement of Katrina. I think of it as sort of an Institute for Advanced Study devoted to climatology research. Great minds in the field are here.

The research actually takes place at three different sites, DC, Boulder and Princeton, and affiliations with academic institutions flourish as well. In fact, I knew at least 4 people who worked here because of interactions between this site and cs.princeton.edu, my former employer.

My job, as it’s been described to me, is to provide a vision as to the design and direction of the infrastructure which supports the rather enormous high performance compute (HPC) cluster. This involves something of a learning curve to understand what’s here, how the systems are used, what the needs are, what people like and hate, where the redundancies and inefficiencies exist, etc. It also involves having meetings and coordinating with people who manage the network, the facilities (power & cooling, etc), the security policy, etc. I’ll be grilled on my ideas, and create prototypes and demos to get my ideas across. Lots of communication.

An aspect of my job will also involve getting my hands on the HPC clusters themselves as well, which are also at each site. All of the clusters are on top500.org last time I looked. Just go through the pages and search for GFDL and/or NOAA.

The systems here are all Linux. Even the standard-issue workstations are running Linux.Β  Scripting is done in Perl and shell, but Python is everywhere, so I’ll be doing either Perl or Python if I have the choice (because “shell” == “csh” here, which I never took well too, honestly). Some aspects of the environment are pretty fascinating. For example, how exactly do you store (*and* easily retrieve, on the fly) 9 PETABYTES of data? How do you back that up? How do you recover from hiccups? How do you instrument systems consisting of thousands of CPUs,Β  to pinpoint problems and get them fixed? And, by the way, how’s the best way to tune a system’s network stack to use a 50MBps pipe (that’s Mega *bytes*) efficiently enough to move multiple terabytes of data every day between collaborators at different sites? How, exactly, do you consolidate services and provide failover across geographically dispersed sites?

So that’s it for now πŸ™‚Β  It’s too early to tell how things are going, really. It’s certainly not the cushy environment that Princeton U. was, but there are bigger challenges and problems to be solved here, and that’s the part I’m looking forward to.

« Previous PageNext Page »

Blog at WordPress.com.