Agile at AstrumU: A Case Study

Introduction

This article is meant to present how we organize our work at AstrumU, a startup based in the US in Seattle with remote offices for sales and some of the development team. Our development team has grown significantly over the last six months. The company is about a year old.

Our product is composed of multiple web-based front-end applications backed by a steadily increasing number of microservices.

My Biases

I’m going to state some of my biases up-front. I firmly ascribe to the dictum that you don’t let your tools dictate your process. I am also a firm believer in using a physical board for a team to organize work.

In my experience, a physical board is only useful with a co-located team. I have managed distributed agile teams for many years. I have not found a way to avoid using a digital board when the team doesn’t physically sit together.

I have used many of the agile digital tools over the years. I am not a strong advocate for any particular one. When I joined AstrumU, we were already using Jira, and I saw no reason to switch to a different tool. There are some workflows that we’ve built in Jira that might be useful for other teams, so I include them here.

Jira is such a catch-all tool that its’ complexity makes it difficult for teams to adopt. I’m hoping that our workflows might show other teams some useful things that Jira can do.

Our Agile Process

At AstrumU we use a simple Kanban process, with some of the ceremonies from Scrum. Some call this “Scrumban.”

We have a daily standup at 9 am in the US. On Mondays, we follow the stand-up with a retrospective or a planning meeting (alternating weeks). The retrospectives and standup are similar to the traditional Scrum ceremonies so I won’t describe them here. Our planning meeting is different. I describe it later in a later section.

Projects and How We Use Them

Given that we build multiple products and a variety of independent supporting services, it makes sense not to have one project for all of our development or one project per team to track work. Instead, we use multiple projects that are each specific to a product or supporting service. We currently have 14 different projects in Jira covering everything from an application front-end to our cross-service security work.

We try to keep the projects clear enough that it is obvious where a work-item should go, but it can still be sometimes confusing to identify the right project for a new task. We’re working to make this more apparent.

A benefit of this structure is that we can leverage pull requests in our GitHub projects to transition stories to the “done” and “released” states automatically as the code moves from a feature branch to the develop branch to the main line.

To track the overall work of the team, we have a master AstrumU project that consolidates and tracks all the work in the other projects. The AstrumU project is the only one in Jira that has a Kanban view and is the single source of truth for prioritization and work-in-progress. This project is also where work items that span multiple projects, such as infrastructure or general documentation, are added.

this is how we include the other projects in our master project

Taxonomy of our work items

We have four different packages of work: Epics, Stories, Tasks, and Bugs. We use the Jira built-in types for these. Each has a different use, scope, and meaning in our workflow.

Epics

The product team works with the development and data teams to create Jira Epics in the AstrumU project to capture substantial efforts such as an MVP of a new feature. Epics are sized to be a reasonable amount of work to complete within a few weeks or less of dedicated development time (i.e., “Students Can Register On the Site” not “Build the Student App MVP”). The Epic ticket includes as much context as possible: links to user research, UI designs, and product concept documents, for example. The Epic also includes acceptance criteria (a.k.a. the definition of done).

Stories

The development and data teams break the Epics down into Stories to track the implementation and design work. We size Stories for completion within a day or two at most (i.e., “Route <url> to new service in Traefik” not “Create the Student API service”). The work for a story includes time to write tests and validate that the code works. The work also includes things like adding telemetry or monitoring as appropriate. The majority of Stories link to an Epic, but that is not a requirement. Stories without Epics tend to be one-off maintenance efforts, small incremental improvements to a feature or refactoring or other technical debt.

Stories should always contain acceptance and enough context that any developer who picks up the task has all the information needed to complete the work. The context is especially critical because of the distributed nature of the team. We have found problems when a story is missing this information because a developer cannot complete it without having to wait for standup to get the missing context. We have also had stories done incorrectly because of missing context.

Tasks

If, as part of defining or working on a Story, we come across a small separable effort for someone else to take over and work on independently, we create a Task and link it to the Story.

Scoping for a Task is in the range of a few hours. If a Task is enough work to be a day or longer, it should be a Story instead. If the Task is simple one-off maintenance or a hygiene effort, we do not link it to a Story or Epic.

Bugs

Bugs are indicators of something broken in existing code. Bugs are not used to specify new or incremental feature work. Refactoring code or reorganizing repositories is not bug work. A page rendering incorrectly or an API failing are examples of bugs.

We file Bugs against the most appropriate Jira project for the issue. We encourage anyone in the company to file a bug when they find it. For folks who don’t know what Project to file the bug against, they file it against the AstrumU project, and it is moved to the appropriate project later.

Organizing and Tracking the Work

We have two Kanban Boards in the AstrumU project. One tracks Epics only and the other tracks Stories, Tasks, and Bugs.

The Epic Kanban Board

Since our epics track the significant efforts in progress, the Epic board is an evident view of the work that the teams are doing and what is next. For those who do not need to follow the details of the work, this is an excellent view-at-a glance of the state of the world. If you want more detail about an Epic, you can open the Epic to see what work is complete and what work remains.

Our Kanban board for Epics has three columns: To Do, In Progress, and Done. The simplicity of the columns makes sense for an Epic workflow where the primary goal is transparency and managing the amount of work in progress. We also maintain a backlog for Epics.

The ordering of Epics in the columns denotes priority, but there is no strict enforcement of having the stories in the other board match the epic prioritization exactly. If there is a significant disparity, that in itself would signal some potential issues in our process.

An Epic moves from the To Do to the In Progress column when it is ready for work by the whole team if there are people available to work on it. There may be some Stories that we start on in Epics that are still in the To Do column. That is almost always cards for the UX or Product Managers to prepare the Epic for the rest of the team.

Epics move from In Progress to Done when all the stories, tasks, and bugs attached to the Epic are complete, and the Product, UX, and Engineers sign off on the Acceptance Criteria (this is very informal).

Epics move from the Done column off of the board after the bi-weekly planning meeting (described later) if all code elements from the Epic are now running in Production.

The Detail Board

The detail Kanban board

The board that the development and UX teams most interact with is the Story/Task/Bug Kanban board. This board contains five columns: To Do, Blocked, In Progress, In Review and Done. Without context, this board can look very chaotic with all the stories from different teams, different projects and different epics. In reality, the team likes it because it shows very clearly what things are complete, what is in progress, and what is next.

Most of the time we have a single swim lane, but when we have any time-critical cards, we use a separate Expedite swim lane to track them.

The cards’ position in the column denotes priority. Developers are expected to take their next work item from as near the top of the To Do column as they can. Because each card that is part of an Epic has the title and color of that Epic on the card, it is straightforward to see if the prioritization of the cards aligns with the priorities of the Epics.

Cards move from To Do to In Progress when a developer is free. We do not let a single developer have more than one card In Progress. When a developer starts work on a card, if they realize that the scope of the work is too big for a story, they break down the card into smaller stories and tasks. They can keep moving forward on their work. We discuss the breakdown in the next day’s standup. If the team agrees on the new stories and tasks, those cards get prioritized in the To Do column.

If a Developer is working on a card and finds that a dependency on another card is blocking their work, they link the two cards and then move the blocked card to the Blocked column. A card only moves to the Blocked column if the blocking dependency is In Progress. If the dependency is the To Do column, then the developer adds comments about what they have done, they push their in-progress branch to Github and then put the card back in To Do and start on something else.

Once the work for a card is done and tested locally, the developer submits a Pull Request for their feature branch and moves the card to the In Review column. Each code change requires two other developers to review and approve the change. When the Pull Request merges into the Development branch, the card moves from In Review to the Done column automatically.

When a release is created in GitHub as part of our semantic versioning scheme and the code moves from our Development cluster to our Production cluster, there is a parallel release done in Jira, and the stories move off of the Kanban board.

I have been considering doing a separate column for UX Review on the Kanban board and may add that in future.

The Planning Meeting

Every other week after the Monday standup meeting we have our planning meeting. The agenda of the meeting is: Review the completed Epics from the last two weeks; review the Epics that are currently in progress and review any new epics that may be moved from To Do to In Progress in the next two weeks.

For the In Progress Epics, we discuss the work remaining with an eye towards making sure the remaining stories satisfy the acceptance criteria of the Epic. If not, we may need to add additional stories.

For the upcoming Epics, we discuss the product, UX and business context of the Epic so that the teams understand why we are working on this Epic next and why it is relevant to our business. We make sure the acceptance criteria is understood. For feature work, the Product Manager and UX designer discuss the rationale behind the epic and the initial UX designs. The development team then reviews the cards associated with the Epic to make sure they are correct and make suggestions of things to change to keep the work in scope.

Initially, we tried generating the stories as part of the Planning meeting, but that proved too cumbersome. The stories and tasks are now generated beforehand by the engineering leadership. Generating the initial stories in this way is a temporary solution. Ideally, the team should generate the stories and tasks themselves.

Challenges

Our current process is the result of iteration and continuous improvement. There are still some challenges to resolve.

One of the biggest challenges we need to resolve has to do with the time difference between the Seattle and remote teams. While a good backlog and prioritization in any agile process require ongoing grooming, having the teams working hours off of each other means that there is a lot of daily grooming work. Especially since new cards are being added every day by developers breaking down stories, Product Managers adding incremental changes, or bugs coming in. If there are a couple of days without dedicated effort on the backlog, developers can find themselves not being sure what to work on next. Our best solution to the problem so far is to empower the lead developer, who is remote, to be able to update the board to unblock developers there as needed.

Another challenge is making sure that all teams are using the process consistently. We’ve had a few issues with the shared board contained too many stories in the In Progress or To Do columns because one team wasn’t using the same criteria as the rest of the organization. Doing some internal documentation and training has mostly addressed this problem.

Our most frequent issue is that the Done column on the detailed Kanban board can get full when we are working on a new feature or service. Since creating and deploying a new release is what moves the stories off the board, the Done column can get long at times. Having an overly-full column on the Kanban board makes it harder to understand the current state of the world (also in Jira it can mean much scrolling). We’re working to get our feature-flagging architecture going which will allow us to release new functionality before exposing it to our customers. As a side effect, this will help us clear out this column more frequently.

The last challenge is that there is still manual work to create releases for the projects that aren’t directly tied to a repository to move the cards from those projects off of the Kanban board. Since there are not usually very many of these stories, a periodic manual release for each of these projects takes care of the issue. Eventually, we will automate this process.

Summary

At AstrumU, we are using a simple Kanban agile process along with some of the Scrum ceremonies to help us organize, prioritize and track our work. We continue to iterate upon this process, but in its current state, it does a good job keeping the distributed product development and data teams informed and coordinated while making priorities, plans and completed work transparent to the rest of the company.

Acknowledgments

I want to give credit to Fedya Skitsko who developed a lot of the early Kanban process and Jira structure that is the basis of our current process and structure.

Answering some questions about Agile Transformation

I was given a set of questions from a consultant working with a company about to begin a transformation to Agile. They asked if I record my answers for their kick-off meeting. That video is above, but I had written my thoughts down as well for clarity, so I am including that text as well.

How hard it can be to implement an agile model in a company where the old model was more hierarchical and conservative?
It can be extremely challenging if only part of the organization is interested in making the change. If the rest of the company are expecting detailed plans and delivery date commitments and the product development team is working with a more iterative approach, that will create a lot of organizational friction. For any agile transformation to be successful, the whole company has to be supportive and committed.

I don’t think that company hierarchy is necessarily an impediment to a successful agile transformation. As long as the responsibilities and expectations of leadership adapt to the new way of working and that leadership is also committed to the agile transformation. Many organizations with more traditional hierarchies build their products successfully with agile methodologies.

What would be your advice for this team to successfully implement the model? What should they be aware of? Basically, the DOs and DONTs.
Do commit to making the transformation, and understand that it won’t be easy. This will be a culture change for your company. Any culture change follows a path where the excitement of making the change is followed by a period where the individuals and teams struggle to understand how to be productive in the new model. During this time (sometimes called the valley of despair), it seems like the best idea would be to go back to the way things used to be. Push through this time and don’t give up. Bit by bit, things will improve, people will figure out how to operate in the new world and you will end up in a much better place.

One of the ways that teams make the transition to agile is to use a known structured methodology like Scrum. At first, the processes and ceremonies will feel strange and not what you understood agile was supposed to be like. Stick with it. As your teams get better at agile thinking, you can start to decide which elements make sense for you and which you may want to change or drop altogether. Each of these things has a purpose, and understanding the purpose and the value when it works well is important before you decide not to do it. Teams that abandon the parts of the process that they don’t like early on often end up with a very poor understanding of agile. They gain very few of the benefits and may be a lot less efficient.

What are the foundational measures they should follow in your opinion?
Like any organizational culture transformation, there should be some time spent by the whole organization understanding why there is a need to make the change, what the expected outcome from the change is and what the plan is. Time should be spent to make sure that all parts of the organization (especially the teams dependent on the team making the change) are committed.

If there is a smaller team that is mostly independent, that team might try to pilot the switch to agile first, to develop some expertise ahead of the rest of the organization and learn from their experience.

What should they anticipate to succeed?
Anticipate that this may be a longer process than they expected, but the effort is worth it! Anticipate that the change may too big for some people to make, and they may choose to leave or try to prevent the change from happening. Anticipate that it will get progressively easier over time.

Other relevant points you might find useful.
I have been working Agile exclusively for almost 20 years after having spent the first 8 years working in a more traditional way. The reason that I have continued to work agile is that I have seen no better way to deliver software efficiently. I am inherently pragmatic. If I saw a better way to work, I would switch immediately. I haven’t found any yet.

The hardest part of adopting agile is learning the agile mindset and understanding that it doesn’t mean abandoning quality, accountability, documentation, process, planning or tracking to deliverables. It is about finding the right amount of each of those things for the project and no more.

In the end adopting agile is adopting a culture of continuous improvement. A culture of always looking for better ways of doing what you are doing. The way we practice agile today is very different from the way that we did it five years ago. Its adaptability is part of its strengths. It’s fluidity also makes it very difficult to learn. It is absolutely worth the effort though.

I wish you the best of luck on your journey!








Slides from my talk on Distributed Teams

Compare the Market was nice enough to invite me to speak at their tech managers’ off-site about distributed teams. This talk reflects my own experience leading distributed teams.

I was presenting to them over video. Their meeting included people in two different offices and also folks dialing in from home. Ironically, in the middle of my talk, I got disconnected from the video conference. Because I was sharing my slides full-screen and had my speaker notes on my second monitor, I didn’t notice. So I spoke to myself for about 15 minutes before I realized what happened and dialed back into the meeting. It was a bit mortifying, but the folks in the UK were extremely nice about it. I can’t think of a better example though of the challenges around working with teams who have to communicate over electronic means constantly, so it was a good illustration of the issues I raised. 🙂








A diversity challenge: tech start-ups have a great opportunity

For decades we’ve been complaining about the lack of diversity in the technology industry. We’ve worked on the pipeline problem. We’ve worked on reducing bias. We’ve worked on the sourcing problem. We’ve worked on the retention problem. The net result thus far is that we’ve barely moved the needle.

Most of the companies that are investing in diversity programs are the larger companies. For them, their continuing lack of diversity is a public embarrassment.

At scale though, it is a far greater challenge for a company like Google, Microsoft, or Facebook to get to any percentage of tech workforce that mirrors their customer base. The numbers are too large to move the needle. It’s far easier for startups.

A critical part of building an inclusive culture that supports diversity is reducing the “otherness.” Inclusiveness is also much harder to do in a large company. If Google hired 1000 developers of color across all their offices, those individuals might never encounter another person like themselves on a daily or weekly basis. They may still be the only person of color that their peers see at work. They will be spread too thinly across the population.

Large technology companies should still work consistently to improve their diversity, but startups are much better suited to solve the diversity problem for the industry as a whole.

A startup with a development team of ten, four of them being women, has a ratio of 40% female developers. Any woman who interviews with the company will see that they are welcome. Any man that interviews will understand that they would be joining a company that takes diversity seriously and will be expected to conduct themselves appropriately. This would be the same for any other underrepresented group. If the company is serious about building a diverse workforce, they will find it easier to continue to be diverse as they grow.

Bringing in a diverse workforce at the early stages of a company will also mean leadership opportunities for those employees as the company grows. It will help address the lack of diversity in industry leadership, which further helps build minority representation. It will also eventually mean more startups started by underrepresented industry groups, which will continue to fuel diversity in the industry. Some of these startups may be acquired, putting their leadership into the leadership of other companies and increasing diversity in those companies as well.

According to most surveys, startup founders’ biggest challenge is hiring development talent. Meanwhile, there are ever-larger numbers of coding schools and boot camps graduating eager junior developers, willing to work hard, and coming from largely underrepresented populations in the industry. There are also many experienced minority developers at the larger companies who would be interested in being in an environment that lets them feel free to be themselves.

Unfortunately, most startups neglect the critical cultural aspects of building their company as they chase product/market fit, funding or customers. What many of them haven’t considered is that building a diverse company will help them find the right product for mainstream audiences, that sources of capital are increasingly valuing diversity in their funding decisions, and that diverse teams build better products that attract more customers.

So, I call on my fellow startup CTOs and CEOs to take on this challenge. If we succeed, we will not only build a better industry; we will also create better companies for our shareholders, our employees, and ourselves.








Talks: 1st half 2017

I’ve been remiss in posting since I’ve been back in Seattle. Readjusting to life in the states, and adjusting to the new role has kept me busy. I hope to rectify that in the future. I have been lucky enough to be invited to speak several times this winter and spring in the US and Europe. If you’ll be there or live nearby and want to meet up, drop me a line on twitter and let me know!








Articles I’ve liked recently (stuff I’ve been reading lately #3)

Why Apple Music Is So Bad When the iPhone Is So Good
On April 28, 2003, Apple launched the iTunes Music Store, saving the music industry from the scourge of piracy while creating a large and steady source of…
http://www.newyorker.com/business/currency/why-apple-music-is-so-bad-when-the-iphone-is-so-good

Scaling Lean — Lean Startup Co. Blog — Medium
Scaling Lean Written by Jennifer Maerz of Lean Startup Co. How can you really measure the effectiveness of your company’s business model? The process goes much…
https://medium.com/lean-startup-co/scaling-lean-d46a52a06fb2#.32bm465po

Silicon Valley’s Scapegoat Complex — Thinkpiece Dot Club
Tech’s Scapegoat Complex T he recent revelation that tech investor Peter Thiel provided the funding for a series of lawsuits against Gawker Media is one of…
https://medium.com/thinkpiece-dot-club/techs-scapegoat-complex-38a4bfb37f22#.b1ckt8ub7

Keep an eye on Norway: Its startup scene is about to go huge
I spent a lot of my formative years in Norway, and have been periodically checking in on the Norwegian startup scene. The first time I went and had a look…
https://techcrunch.com/2016/06/20/norwegian-startup-scene/

‘The Life of Pablo’ and the Death of the Traditional Album
As a genuine certified old person, one who generally doesn’t stream music, who remembers having racks and racks of CDs, and who keeps much of his vinyl sorted…
http://www.complex.com/music/2016/06/kanye-west-life-of-pablo-death-of-traditional-album

If Your CFO Hasn’t Already Told You to Control AWS Costs, He’s About To
You made the obvious move and migrated to Amazon Web Services (AWS). Months later, the attractive glow of the move from CapEx to OpEx spend has been dimmed by…
https://www.linkedin.com/pulse/your-cfo-hasnt-already-told-you-control-aws-costs-hes-jay-chapel

Can Netflix Survive in the New World It Created? – NYTimes.com
One night in early January, a little after 9 o’clock, a dozen Netflix employees gathered in the cavernous Palazzo ballroom of the Venetian in Las Vegas. They…
http://www.nytimes.com/2016/06/19/magazine/can-netflix-survive-in-the-new-world-it-created.html?nytmobile=0&_r=0








Stuff I’ve been reading lately #2

12 Powerful habits I have stolen from ultra-successful people
http://observer.com/2016/03/12-powerful-habits-i-have-stolen-from-ultra-successful-people/ (tip to kate matsudaira)

12 Lessons of Waking Up at 4:30 a.m. for 21 Days — Life Hacks for Business
12 Lessons of Waking Up at 4:30 a.m. for 21 Days If it works for me, maybe it works for you!
https://medium.com/life-hacks-for-business/12-lessons-of-waking-up-at-4-30-a-m-for-21-days-90d1053c3634

Practical Networking Tips for Introverts
Networking was such a foreign concept to me. And I really do mean foreign — we don’t do the organized networking thing in El Salvador. I knew that building…
https://www.ellevatenetwork.com/articles/7282-practical-networking-tips-for-introverts

Curing Our Slack Addiction
I love Slack. I really really do. So much so I would call it an addiction at this point. Slowly but surely this addiction has been killing my sanity and sapping…
https://blog.agilebits.com/2016/04/19/curing-our-slack-addiction/

Official Google Blog – This year’s Founders’ Letter
Every year, Larry and Sergey write a Founders’ Letter to our stockholders updating them with some of our recent highlights and sharing our vision for the…
https://googleblog.blogspot.com/2016/04/this-years-founders-letter.html

The Choice Explosion
Lansing, W.Va. — A few years ago, the social psychologist Sheena Iyengar asked 100 American and Japanese college students to take a piece of paper. On one side,…
http://www.nytimes.com/2016/05/03/opinion/the-choice-explosion.html

Doing a TED Talk: The Full Story – Wait But Why
You’ve probably heard this Seinfeld joke: According to most studies, people’s number one fear is public speaking. Number two is death. Death is number two. Does…
http://waitbutwhy.com/2016/03/doing-a-ted-talk-the-full-story.html

Dropbox cut a bunch of perks and told employees to save more as Silicon Valley startups brace for the cold
You have successfully emailed the post. 20h 8,000 REUTERS/Stringer Winter’s coming in Silicon Valley. When Dropbox employees walked into their new office on…
http://uk.businessinsider.com/cost-cutting-at-dropbox-and-silicon-valley-startups-2016-5

Thoughts on Take Home Interviews
There is a movement now in tech to really think about what it would take to improve our interview process. This is a movement a long time coming. White board…
http://www.elidedbranches.com/2016/05/brief-thoughts-on-take-home-interviews.html