HPC on the (relative) cheap using public cloud providers

For the past several years, I’ve been working on leveraging high-performance computing techniques for high-throughput data intensive processing on desktop computers for stuff like image and video processing. Its been fun tracking what the multi-processing end of HPC has been doing, where the top 100 super-computer list has been very competitive and very active. Countries, IHVs and universities vie for who can generate more teraflops; spending millions and millions of dollars on the cooling plants alone for their dedicated data centers. These super computers exist to solve the BIG PROBLEMS of computing, and aren’t really useful beyond that.

At the same time, I’ve been following the public computing clouds like Amazon’s EC2, Google’s App Engine and Rack Space’s Public Cloud. These have been interesting for providing compute on the other end of the spectrum, occasional compute tasks, or higher average workloads with the occasional spike capability (like web apps). The public clouds are made up of thousands of servers and certainly rival or best the super computers in numbers of cores and raw compute power, but they exist for a different purpose.

This article in The Register really got me excited. Especially when I read this:

Stowe tells El Reg that during December last year, Cycle Computing set up increasingly large clusters on behalf of customers to start testing the limits. First, it did a 2,000-core cluster in early December, and then a 4,096-core cluster in late December. The 10,000-core cluster that Cycle Computing set up and ran for eight hours on behalf of Genentech would have ranked at 114 on the Top 500 computing list from last November (the most current ranking), so it was not exactly a toy even if the cluster was ephemeral.

The cost of running this world-class super computer?

Genentech loaded up its code and ran the job for eight hours at a total cost of $8,480, including EC2 compute and S3 storage capacity charges from Amazon and the fee for using the Cycle Computing tools as a service.

Real world HPC is now coming into price points where it is accessible to even small companies or research groups. This seems like a ripe opportunity for companies who can apply HPC-techniques to solve real problems for others, and for tools vendors who can make using these ephemeral clouds easier for companies who want to take advantage of them without having to build up high-end expertise in-house.

On Test-Driven Development

I was having a conversation with someone the other day about unit testing. OK, actually I was interviewing someone for a Quality Engineering position on my team. We were talking about the difference between white-box tests that quality engineers write and tests that developers write.

I suggested that good white-box testers test the functionality and the failure cases (the intent of the function) and developers test the code that they’ve written (the function as coded). This then lead me to a new revelation around test-first development methodologies (or possibly reminded me of something I had forgotten).

I have been a proponent of writing tests first, since I first started doing Extreme Programming and read Kent Beck’s original book, Extreme Programming Explained: Embrace Change while working at Bootleg Networks (thanks Carmine for making me do that, by the way). Although admittedly, like many developers, I haven’t always been that rigorous at following that rule.

What I like about writing the tests before the function is that it clarifies my thinking about what the function should do, it alerts me to the corner cases, it gives me reasons to consider if the function is doing too much, and it gives me a way to instantly know if the function works once it is written. Writing the tests first also makes sure that the tests are written at all. Once the function is coded, it sometimes gets tempting to move on to the next bit of coding work with the intention of filling in the tests later.

What I hadn’t considered about writing the tests before the code is that it puts me into a quality mindset without having any bias to the code as I’d written it. I’m divorced from my own blind-spots around my coding. This actually leads me to writing better tests because I have no assumptions about how the code should work or fail. I’m testing the functionality, not the code.

Maybe I’d thought about this before, but I hadn’t really considered that benefit recently until that moment. Now, when I start to get lazy about writing my unit tests before my implementation, I’ll have a better reason to keep up my discipline.

A Couple New Pixel Bender Links

Frequent contributor to the Pixel Bender forums, Royi Avital, has released a new set of After Effects and Photoshop plug-ins written with Pixel Bender under the name Flixel Plugins. The first three are now available on aescripts.com

Flixel Plugins on aescripts.com

ApexVJ is a really beautiful Flash-based music visualizer that uses Pixel Bender

Simo Santavirta, the creator, wrote an article on his blog about it.

Moving on…

After many happy and productive years working on Pixel Bender and the Adobe Image Foundation, I’ve decided to take on some new challenges. I’m still at Adobe, but I’m now building a new team and launching a brand new product in the Photoshop family. I can’t say too much yet, but I will have news soon. I’ll still be posting about Pixel Bender stuff here (I’m still a very enthusiastic user!), but for the newest news, you should now also watch the official Pixel Bender blog.

What I’m trying to do in Outlook

So, in my work e-mail, I get around 200 messages a day. I periodically get myself back down to inbox zero, but if I take a day or two off, I immediately get behind. I recently decided on a new mechanism for sorting my incoming mail. First off, would be to divert any mail not sent directly to me (where my name isn’t on the to or cc line) into a separate folder. This would be the stuff I would get to when I had time. Next would be to divert mail where I’m CC’d into a separate folder (this is the mail I’d read after reading my inbox), all mail with me on the TO line would be left to filter to my inbox. This way, I think I could make sure that I’m not losing the important messages in the noise of the stuff that I don’t need to read (but will when I have time).

Unfortunately, Outlook’s rules don’t let me do this. I can create a rule for messages where my name isn’t on the “To” line, and I can create a rule for messages where my name is on the CC line, but then messages where I’m in the CC line get put into two different folders because they aren’t mutually exclusive. Since the rules in Outlook are more or less fixed, there doesn’t seem to be a way to do what I want here.

Any suggestions (other than get a real mail program)?

A simple tip for prospective job seekers

If you are graduating with a degree in Computer Science and you are applying to companies in the software field; it is probably not a good idea to say something like “I’m looking for a job where I don’t have to sit in front of a computer all day” in your summary statement.

Upcoming talks and events

(all times/dates are PST)

I have a few conference talks and such in the next couple weeks, so I thought I’d send out some pointers.

If you are attending the Intel Developer Forum in San Francisco, I’ll be speaking on a panel discussing how to educate the next generation of computer scientists for the new paradigms of parallel processing. The Panel is called “Parallelism and Education: Navigating Through a Sea of Cores”, the session is on Monday 9/13 at 11am, right after the keynote. I’ve written about this session last month.

On Tuesday, September 14th, around noon, I’ll be appearing live on Intel Software Network TV, you can watch here.

Later that evening, I’ll be hosting a Pixel Bender Meetup at 6pm at the Mars Bar in San Francisco. All Pixel Bender developers are welcome to join me and talk Pixel Bender. More info and directions here.

The next week, I’ll be speaking at the NVidia Graphics Technology Conference in San Jose. My session is on Thursday, September 23rd at 11am and it is called “GPGPU in Commercial Software: Lessons From Three Cycles of the Adobe Creative Suite.” More information here.

If you attending IDF or GTC or you’ll be in San Francisco on the 14th, come by and say “hi!” Otherwise, I’m hoping that I’ll be able to post video or slides from my sessions soon after.

Of course, I’ll be at MAX as well and may have some surprising things there, but I can’t talk about that yet 🙂

Dear Alaska Airlines and Apple Computer…

Yes, I’m an idiot. I left a valuable piece of my property on your plane, Alaska. I know that you are not responsible for items left behind, it’s my fault, not yours.

However, that was the last flight of the day and the item in question was out of sight of other passengers, but my newspaper was sticking out, which meant that it was likely that the seat pocket would be examined in cleaning. It’s extremely likely that my property was picked up by one of your employees either after the flight or in the morning before the next one. I immediately registered with your web site for the item in question, identifying not only the flight, but the seat (thanks for not posting a phone number, but only an impersonal web page. It really shows how much you care). So, given that this kind of thing happens all the time, I did hope that there was a chance I would get it back. It was a long shot, to be sure, but not a non-zero one.

Now you Apple, thank you for creating the mechanism so that I could track my device, send a message to it, lock it and wipe it. In this case, I can’t locate it or send a message to it as it was on airplane mode. So, most likely, whoever has it has either wiped it, failed the login and had it wipe itself, or is trying to hack it (or fence it). Since, by your own definition, the device is now stolen property, maybe you could let me know when someone else tries to use it after it has been reset? No, of course, you can’t do that. Doing that would mean that you might be liable in all sorts of ways, right? There are so many ways that some mechanism could go wrong like that, that your corporate council decided it was best not to get involved. Even though you could actually immediately notify me the minute that the device registered to me with the unique ID you use to track your customer’s devices popped up again. This actually works out better for you, because now you’ll have sold me the same device twice.

So, yup, I can now troll the net looking for the device myself in a time-consuming, frustrating and fruitless attempt while constantly reminding myself of my own mistake. I wasn’t that fond of it in the first place, but I actually need it for a project, so I’m not left with much choice. I’ll need to file this as just another case of crappy customer service.

Thanks for nothing guys!

A Kettle of Fish (Vancouver BC)

Reading the reviews of this restaurant, I was unsure what to expect. It sounded like it was decorated like an antique store. The food decent, but not amazing; uneven and over-priced.

I have to say I was very pleasantly surprised. The atmosphere was indeed distinctive and unusual, but not overbearing. The dining room was open, light and inviting, the bar cozy and dark. There were certainly some antique touches and lots of big plants, but it felt more like the Palm Court at the Plaza in New York or the Laurel Court at the Fairmont in San Francisco rather than your great grandma’s dining room.

The food was also better than expected. My seafood fritter appetizer had been allowed to go cold in the pass-through and was not impressive, but my dining companions’ fried oysters were hot and quite good. However, my halibut main course was cooked perfectly; the inside tender and the outside with a nice thin crust. The others had the Salmon and also enjoyed their meals. For desert, I had a coconut pudding that was a bit sweet but which I had no problem finishing.

The service was good, attentive, but not overbearing. The staff friendly, but not obsequious. The prices were a bit high, but not unreasonable.

I think that this would be a good restaurant to try when looking for something a little different, something unique.

A Kettle of Fish on Urbanspoon