Speaking on the “Teach Parallel” show on IntelTV tomorrow

[crosspost from my adobe.com blog]

Tomorrow morning, I’ll be speaking with Paul Steinberg of Intel and Tom Murphy of Contra Costa college about the criticality of understanding parallel programming techniques for industry.

In my previous role on the Adobe Image Foundation, it was an obvious requirement for our hiring candidates. We were building tools for a insanely parallel problem, image and video processing. Now that I’m working on a new product, it would maybe seem that it would not be as important. In fact, our threading models are even more complicated than in my previous group. My expectations around threading knowledge for incoming candidates are just as high.

Even the most modest mobile hardware is going (or has gone) parallel. In addition, the expectations from a user perspective around interactivity with their applications is never higher. A laggy touch interface is death to an application (or a platform). Going to get coffee while your image renders on a desktop is a thing of the past. User’s expectations of the software we write is higher than ever and it is nearly impossible to get this interactivity without taking advantage of multi-threading on today’s multi-core processors.

The tools continue to improve, but the threading models continue to evolve. A fundamental understanding of multi-threading is critical for anyone moving into Software Engineering or looking to stay current in their field.

I always enjoy talking with Paul and Tom, and expect that we’ll have a lively conversation.

Tune in live on May 17, 10:00 AM PDT

Here is Paul’s post on the subject.

Apple’s way or the highway

The Daily Show With Jon Stewart Mon – Thurs 11p / 10c
Appholes
www.thedailyshow.com
Daily Show Full Episodes Political Humor Tea Party

Is Apple jumping on the crazy train?

I liked this article on Newsweek: Apple vs. Everybody: The company’s epic battle over a missing iPhone is only the latest in series of contretemps.

It’s the kind of attention that Apple, long a media darling, isn’t used to. Apple’s control-freak nature didn’t matter as much when it was a plucky underdog. Yes, Jobs was a demanding boss and a finicky perfectionist—but he created great products. We rooted for Apple, and wanted it to survive. Apple seemed like the anti-Microsoft, a company that was on our side. But this year Apple will do nearly $60 billion in sales, and its market value stands at $240 billion—the third-largest in the United States, bigger than Coca-Cola and Pepsi combined. Any company that big can seem a little scary. So when police start breaking down doors over a lost phone, it’s a PR disaster, especially for Apple. The company works hard to cultivate a counterculture image, with ads that have featured Gandhi and John Lennon, not to mention the “I’m a Mac” hipster. Yet lately Apple has started to look like the big bully of the tech industry, the kid who doesn’t play well others. Over the long haul, that can put customers off.

It definitely can put developers off (including this one), and when your platform has a lot of competitors gunning for it and a slim percentage of the desktop market, putting off developers is not really a very good idea. Apple is betting the company on their new strategy of a tightly controlled ecosystem where they make a small amount off of every transaction and act as intermediaries between content producers and developers and their customers. It will either be fantastically successful or Apple will crash and burn in a spectacular fashion. Only time will tell.

Moving to scrum: changes

So we started this product cycle with two geographically-dispersed mixed-discipline scrum teams on two week sprints. Specifically, the geographical co-mingling was designed to break up some of the silos that had built up on the team due to the existing functional teams being in different cities. By the third sprint, we’ve now switched back to scrum teams that mimic the original functional and geographic splits and are now on four week cycles. This has made cross-team communication more difficult, but has made everyone a lot happier by keeping the scrum meetings more relevant to everyone on the team. I can definitely see why people like this more, it lets people focus on what they are doing, but I do think we are losing something by not spreading the knowledge of the different parts of the project around the team. Also, not having multiple functional areas represented in the scrum meetings means that some fresh perspectives are lost in any of the technical discussions that come up. Over the next few sprints we’ll try this out and see if this is working. I think this will probably be what we stick with, and we’ll need to address the cross-team communication issues separately.

Switching to four week sprints was mostly to reduce the amount of time spent in the scrum meetings themselves relative to the amount of working time. Between planning poker, retrospectives and sprint planning, a lot of time is lost to the process itself in a two week sprint. For a team used to working on 6 week sprints, fitting stories into 2 week chunks was also difficult. I like the idea of two week sprints, but in practice, it was difficult for a team that is new to scrum. We’ll maybe try 2 week sprints in a future product cycle.

We still haven’t switched to our planned internally created scrum tool which itself makes things difficult. Our backlog is an excel spreadsheet and our tracking is all on Wikis. Hopefully, we’ll be converted over to the tool by the next sprint, which should make life a bit easier on me and the scrum master.

Section 3.3.1 is not new behaviour from Apple

[disclaimer: I am an Adobe employee and an Adobe and Apple shareholder, my opinions are my own and not those of my employer.]

Like the rest of the software industry, I’ve been pondering what the effect section 3.3.1 of the iPhone 4.0 SDK will have. I had fully been planning to make an iPhone application at some point. I had planned to do the initial version with Flex to prototype, but then also spend time doing a Cocoa version to better learn that SDK for myself. This iPhone 4.0 SDK announcement honestly has me questioning if I do really want to develop for the iPhone. Not just because of a higher-minded sense of indignity at Apple’s lack of openness of their platform, but rather because of that combined with their somewhat arbitrary and opaque app store approval process. Could I spend months of my spare time learning ObjectiveC and working on an iPhone application only to have that time be a complete waste if the App store reviewers decide that they don’t want that app in the store?

Thinking about it this morning, I realized that not only was Apple’s move to lock in developers nothing new, but that I’d already written about it before (in fact, I’ve been blogging about it since almost the day I started doing professional development for the Macintosh): iPhone SDK: The carrot for Cocoa, the stick for Flash, The difference between being an Apple developer and a Microsoft developer, Developers Developers Developers Developers.

Gruber had the motivation right, I think, but I also think he got the ramifications wrong. Since Steve returned to Apple, they have been applying the screws tighter and tighter to their developers, trying to get them to lock in. It was somewhat indirect at first, but the long term implication was clear: “We’ll tell you how to develop for our platform, if you do as we say, then you’ll be fine. If you don’t do it the way we tell you, your life will be a never-ending stream of headaches.” The move to Intel (forcing all developers onto X-Code and a big rewrite of any PPC-assembly) was step one, the move to 64-bit (dropping support for Carbon after promising it) was step two. The iPhone 4.0 SDK is just the most obvious move in this process because it basically spells it out. You no longer have a choice: it is Apple’s way or the highway. The problem is the App store. On the Mac, I control my own distribution. On the iPhone platform, Apple does. That means that they no longer have to negotiate with their developers, they can now finally dictate to them.

As a developer, this makes the iPhone platform a lot less attractive because I also can’t be sure that they won’t change the terms again. Once I’m locked in, I’m locked in. Apple can do whatever they want and I’m forced to rewrite my apps or get forced out. As someone who writes software for a living, this scares the crap outta me.

Here are some other blog posts that I thought were good reading around this:
The iPad isn’t a computer, it’s a distribution channel (O’Reilly Radar)
Five rational arguments against Apple’s 3.3.1 policy (37 Signals blog)

Moving to scrum: breakin’ the law

Well, part of the whole idea is that scrum isn’t a hard-and-fast set of rules right? I had some meetings at another office last week which necessitated moving out our first sprint review and retrospective a couple days. So if one of the guiding principals of scrum (and agile in general) is that the dates don’t change, everything else does, that got blown right out of the gate.

Some new stories came up as part of the sprint wrap-up which means that we’ll need to do a new planning poker session. Also, some other things came up which mean that there is a bit of housekeeping to do before we start the second sprint. We’re now in some uncharted zone between two sprints while the product manager and scrum master do some cleanup while the scrum teams do some off-book work for a couple days. Yeah, not ideal, but we’re trying to get it back on track. This sprint was going to be a throw-away anyway as it involved a lot of transition and getting used to the framework.

The retrospective brought up some really good issues and some innovative suggestions from the team on how to handle them. If the suggestions work out, I’ll post them here eventually.

There are still some folks unclear on what we expect to gain by transitioning to a new development process, but most of the team is embracing it as a good experiment and chance to try something new. For the most part, everyone seems to have an open mind and that definitely makes a huge difference as we all try to figure this out together. It hasn’t been the smoothest transition, but so far so good, I think. We’re definitely not as productive as we were before (so far). However, I think that the discipline provided by the product backlog means that we are actually getting the most important things done first. That makes a huge difference.

Converting from agile-mumble to Scrum: the introduction

I’ve been a convert to Agile for almost a decade now. I was first exposed to it when I was working at a start-up called Bootleg Networks as a Development Lead (don’t bother looking it up, it is long lost to time). I’d just come from six years at Microsoft where Waterfall was the only methodology and it was followed religiously. I’d seen all the worst aspects of Waterfall and seen nearly none of the best, but I wasn’t up on any other methodology besides Ad-hoc (also known as “play it by ear”). We’d hired Carmine Mangione as our software architect and he came in spouting all this stuff about “Agile” and “XP.” While I was definitely ready to try a different style of development, it all sounded a bit unstructured to me. Carmine convinced me to try eXtreme Programming, promising that if it didn’t work, we’d drop it. We bought a stack of Kent Beck’s book for the team and jumped into it. I went from Development Lead to XP Coach and quickly became a convert.

We achieved a ton of stuff using XP at Bootleg very quickly with a relatively small team. That short stint really changed my mind on how professional software development could be done. Ever since then, I have found ways to incorporate as many aspects of agile methodologies into any place I’ve worked.

I ended up back at Microsoft a couple years later. The situation there had changed quite a bit. While most of the company was still doing waterfall, more developers were aware of Agile and were curious about it (even if their managers weren’t). I took on a risky new project. It was fixing an ancient set of code written originally by Microsoft, but completely eroded over the years by a series of contractors. It was a mess: full of bugs. Customers didn’t like it. Rather than try to fix it, I convinced them to let me and a small team rewrite it from scratch to clean it up and get the new functionality needed. Program Management was skeptical, Quality Engineering was dead-set against it. I put my neck out and promised that we’d bring it in on-time with a high-level of quality that would be instantly verifiable with a battery of developer-written unit tests. We got some initial buy in and we went to work. We followed as much of XP as we could with an undersized (4 person) development team. Needless to say, we got it done and it shipped as part of WinCE 5.0.

I left Microsoft to join Adobe and was really happy to see that Agile was not only acknowledged, but embraced across many groups in the company. There were many different flavors being used, including several custom ones. Scrum was fairly common though.

I was certainly aware of Scrum, but didn’t have too high an opinion of it. Everything I’d read and heard about it made it seem like it had nearly no process or structure. Looking at how different teams “followed” scrum practices, it seemed like it was just an excuse for Cowboy Coding with a tiny amount of coordination involved. It seemed like a parody of Agile to me at first.

At Adobe, I was lucky enough to be able to build a completely new team. For various reasons, the team ended up being split between two different offices. While I wanted us to follow an Agile methodology, after a lot of research, I couldn’t identify one that would really work for us wholesale. Instead, I took the pieces that made sense and incorporated them as I could. We ended up with a unique process that subsumed bits of Rational (short sprints at first, longer sprints in the middle and short sprints again at the end; co-located functional teams), XP (test-driven development, occasional paired programming, be able to deliver at the end of each milestone, constant refactoring and narrow focus) and Scrum (demo-able at end of each sprint). That was the goal process. Of course, we didn’t always get there. We were pretty open and made a lot of changes along the way to incorporate new best practices and dump the parts that didn’t work. This evolving process worked fairly well and we have had three successful product cycles over several years using it.

For all the good of our existing cobbled-together development framework, we were always finding the seams, the edges that didn’t quite meet. There were other issues too. Having co-located functional teams meant that each team had good velocity, but it also meant that cross-site communication was always a problem and it made collaboration difficult as well. The team was getting too silo’d in my opinion. I wanted to really mix things up for our next cycle and try something totally new.

Over time, I’d gotten over my initial knee-jerk reaction to Scrum as I encountered more teams using it in a more deliberate and successful way. One weakness I still saw was the difficulty of using Scrum in non-co-located teams, but I met teams that came up with ways to manage this.

I decided that we would try Scrum for our next cycle. If it worked for us we would continue to use it for future cycles. If it didn’t, we would incorporate whatever worked for us back into our old process. Luckily, Adobe has a full-time Scrum trainer in the company. We did our scrum training a couple weeks ago, and we start our first Sprint tomorrow.

I’ve decided to document this on my blog for a few reasons:

  • It is something about Software Engineering that I can write about without fear of divulging corporate secrets
  • It might be interesting for other folks in similar circumstances
  • It is too long to tweet 🙂

My plan is to document our transition as much as I think might be interesting to others. Our situation is a little different from most since we are transitioning from a different Agile process to Scrum instead of from a non-Agile process. My team is also somewhat unusual since most of our clients are internal to the company who each use a different methodology (most Scrum, but some waterfall and some other Agile), but we also deliver our technology straight to Adobe’s customers on both long and short cycles. So we have a lot of impedance matches to account for. Some things may need to wait until the cycle is over for fear of disclosing too much, but by focusing on the process, I hope to be pretty open.

I’m not going to go too far into the mechanics of Scrum here since there are a zillion better places for that and because I’m still learning it myself. If I use a term you aren’t aware of and you can’t find a good answer on Google, ask me in the comments and I’ll try to answer it for you.

A bit on the details: My team doesn’t have a “Product Manager,” so I will be taking on the role of Product Owner even though I am the Engineering Manager. I’m not planning to act as a Scrum Master, that role will be taken by the QE manager of the team. If I have some free cycles I can use for development, I will act as a Scrum Team member, but I will not participate in Planning Poker as an estimator. We will have two scrum teams, each team incorporating developers and QEs from both of our sites. We’re going to do short sprints so that we can make changes quickly to our process.

I hope to create a little series here if I have time to keep it up to date.

Stay tuned.