Things We Learned Creating Technology Career Steps

This is a repost from https://labs.spotify.com/2016/02/22/things-we-learned-creating-technology-career-steps/

This is part three of a three-part series on how we created a career path framework for the individual contributors at Spotify. Part one discussed the process we used to formulate the framework. Part two contained version 1.0 of our framework. In this segment, I’ll talk about the lessons we learned rolling out the framework to the technology organization. If you haven’t read the first two parts, I suggest that you read those first before proceeding with this one.

The Launch to the Organization

We launched the request for comments (RFC) version of Career Steps at a special town hall for Spotify’s entire technology department in December, 2014. Leading up to the town hall we’d done several reviews of earlier iterations with increasingly larger groups from the organization, and we’d given trainings in Steps and conversations around Steps for every manager in Technology. In the presentation, we left plenty of time for questions, which was good. We prompted people to read the document, ask questions in the document, via a mailing list or a slack channel that we set up, or to ask their manager. In retrospect, we should have followed up with some more opportunities to talk to our working group face-to-face. Most of the organization would have been reading the document for the first time after the town hall, and some had only read earlier drafts.

The Relationship between Steps and Compensation

There was one area that was still incomplete when we launched Steps, and this turned out to be an issue that challenged us for a long time. This was the connection between your step and your salary. We had asserted that there was a connection, but at the point that we launched the framework, our Compensation and Benefits team had not finished their review of how this should work. We knew the generalities, but not the details. This left us in a very challenging situation, since we had given Steps a critical connection to the individuals in the organization, but we couldn’t yet explain it. We knew that there needed to be a connection between Steps and compensation. If we were saying that Steps embodied what we valued from the members of our organization, but the salary of those members was determined in a completely separate way; we were essentially contradicting ourselves and undercutting the effectiveness of the framework.

The lack of clarity around compensation created some tension in the adoption of Steps with some individuals in the organization. Unfortunately, the connection between salary and step wasn’t resolved until nearly a year later. This meant that the first salary review after launching Steps wasn’t able to use the framework. On one level this was a failure, but in some ways this was also positive. Since Steps were very new, having people get a step and then immediately have it affect their salary might have lead to a lot bad feelings. By delaying the tie to compensation people had time to adjust to the system and potentially change their step before their first salary review.

Our inability to talk concretely about Steps and salary also created misperceptions around how this would work. We had to spend a lot of effort working through these only to have them re-appear in mail threads and discussions again and again. The most persistent misperception was that the only way to change your salary was by changing your step. This naturally caused much concern that was difficult to dispel until the Compensation and Benefits team had completed their work. Our C&B team did a stellar job on creating a fair and very progressive system that allowed good overlap in salary ranges between the steps and a lot of headroom. Since that has been completed, we hopefully have put many of the concerns to rest; especially as we now using Steps as part of our current salary review.

Behavior Versus Achievements

We made an explicit decision around Career Steps that career growth was characterized by your behaviors and not by your achievements. This is counter to the way career advancement works at many other companies. It was something that we not only felt strongly about; it was something that we were proud of. In a culture that encourages innovation, failure (and learning from failure) is a natural occurrence. If we only rewarded success, then we were consequently punishing failure and discouraging risk-taking. We also wanted to encourage real personal growth, and not a culture of checking off achievements on a list and expecting a promotion. This naturally created some ambiguity around the requirements for each step. The working group embraced this ambiguity as a way of giving managers some room to make decisions and also to encourage discussion between the individual and their manager.

In retrospect, our approach was a bit naive in a few ways. While we as an organization are very comfortable with ambiguity in many areas, this was something that was personal, and had personal consequences. Some individuals were very uncomfortable with this ambiguity, and would have preferred more concrete requirements around advancement. There was another group that thought even the examples given were too much of a checklist. This is still something that we are working through.

We are looking for ways to support those in the organization who would like more clarity without being too prescriptive. We have discussed creating lists of (anonymized) examples of the behaviors. The concern is that this would lead to people treating the examples as achievements to be checked off instead.

This has been a particularly thorny path to traverse with some very vocal minorities, but the majority of the organization seems to have understood the concept.

Having Greater Impact Means Not Working on a Team?

There was one aspect of the Steps Framework that we hadn’t anticipated being controversial, and didn’t come up often in our earlier reviews of the document. This was the aspect of Steps being a reflection of your sphere of influence. The core idea around this was that as you increase your professional maturity, you naturally will become a resource for ever larger parts of the organization; either through your technical leadership, your deep technical skills, or your general problem solving ability. This increasing sphere of influence brings new, broader, responsibilities with it. This was well aligned with our own personal experiences in the working group at Spotify and other companies. For some individuals, the idea that they couldn’t move to a Tribe/Guild step without working outside their squads was a concern.

At Spotify, we point to the squad as the central place that work gets done, it is the top of our inverted servant-leadership hierarchy. The question we heard repeatedly was: why couldn’t someone just be better at what they do and get “promoted” for doing the exact same role if they were adding value to their team? I think that this was a case where the we could have presented the reasoning of this decision in a better way in the document.

Not Enough Chances For Advancement?

When deciding on the number of steps to create, we had decided to keep the number relatively small. The thought being that it was easier to add more than it was to remove some. Four steps is not many changes in a potentially forty-plus year career. The expectation was that people would potentially stay at a step for a very long time. This turned out to be very discouraging for some people who felt that they would never be “promoted” in their Spotify career.

Here we missed the understanding that a segment of our organization had been wanting the opportunity for recognition on increasing advancement. We had not been doing this before in any respect, so we hadn’t considered that it was something that a group really felt the absence of. It turns out that we were wrong about that and with Steps we weren’t giving that group enough opportunity to get it. This was a big oversight since we knew that part of what had been encouraging people to make career changes to management or product was this lack of recognition. This is an active area of discussion for next iterations of the framework.

Assigning Steps

After the town hall presentation introducing Steps, we gave people some time to respond to the RFC version of the document and feedback to the working group. In January, we started the process of making sure every individual contributor in the technology organization had a step. Initially, we left the process up to each tribe. This was a definite mistake. The tribes needed some more guidance on a good way to make this process work. Luckily, at Spotify, we are quite good at figuring these things out and sharing good ideas. The Infrastructure and Operations Tribe came up with the suggestion: each individual should use the Steps document to make a self-assessment of which step they should be on, then have a discussion with their manager about where they agreed and disagreed. This was quickly adopted as a good process across most of the organization.

Once these discussions were completed in each tribe, the tribe would meet to make sure that each manager was assigning steps to their employees in a similar way. Additionally, functional managers in different parts of the organization also met to do a similar exercise. These synchronizations helped make sure that we were being fair and consistent across the entire organization. Before the steps for individuals were finalized, the Tribe Leads and CTO met to go over all the recommendations for the Tribe/Guild and Tech/Company steps. This ended up serving two very valuable purposes: it assured consistency across the entire organization for the individuals who were on these steps, and it made the senior technical leadership aware of who these individuals were and what their manager thought they contributed to the organization.

The process of assigning steps to individuals completed in March, but here we also had a problem. The effort of adding steps into our HR systems ended up being a bigger challenge than our HRIS team had anticipated. So, the steps were assigned in the spring, but weren’t easily visible to employees, their managers or HR until the fall. This also meant we had difficulties tracking which teams were behind in submitting steps for their employees. We couldn’t easily run reports to see how steps were distributed as well.

Once this was finally worked out, we found that we were missing a lot of data. Some employees had never been assigned a step. Some employees had transitioned between teams or roles and their step hadn’t been communicated. Steps hadn’t been integrated into the onboarding process, so many new employees had never gotten their step. There were a bunch of other issues as well. All of these had to be rectified before we could do our salary review. The lack of visibility also contributed to an “out of sight, out of mind” issue where some people or managers didn’t do much with the framework after the initial discussion.

In retrospect, the working group should have taken ownership of this until the issues with the HR system were handled. Unfortunately, there was some poor communication around the timeline for the fixes and we were always thinking that the issue was nearly solved.

Positive Results

Once the Steps Framework was launched, we started getting strong positive feedback from the organization in addition to the concerns noted above. Many of our line managers (Chapter Leads) were previously individual contributors; they especially appreciated having a structure to help frame personal development discussions. Also many individuals told us how it was good to have some structure and understanding about how to grow at Spotify. The working group collected feedback in multiple ways including doing interviews with employees in different offices.

We had wanted to have some real data in addition to our anecdotal evidence to have greater confidence we were actually adding value. Some strong supporting data came from our yearly Great Place to Work survey, where in the Technology Organization, the “Management makes its expectations clear” measurement increased by 4 percentage points year over year, and the “I am offered training or development to further myself professionally” measurement increased by 6 percentage points. There were several things that could have impacted these measurements, but we had reasonable belief that Steps had a part in these increases.

While we had started our effort with the desire of being data-driven, as we went to measure our impact it was clear that we were not as rigorous as we should have been. We should have started the entire effort with some qualitative numbers on how the organization felt about the support they had for personal development driven by a survey or poll. This would have given us better metrics to measure ourselves against, and would help us to guide future iterations as well. This is something that the working group is actively pursuing.

Loss of Focus

Once the steps had been assigned, the working group started moving into a less active phase. There was a strong sense of accomplishment, but also weariness. From meeting twice a week and working on the document in between the meetings, meeting with individuals and teams to discuss steps, training the managers, launching the effort and supporting the initial rollout, there had been a tremendous amount of work done. However, this wasn’t anyone’s main job; we were all moonlighting to do this. Also during this time many of us were involved in a large project that also was demanding our focus. It felt like we had completed our mission and that it was time to move onto new things. Unfortunately, we were wrong.

In this post-launch phase is where we made some of our biggest mistakes. We’d launched the framework, but our support for it was largely put on hold. We hadn’t put the necessary structures in place to make sure that new employees were being trained in Steps. We stopped sending updates and communicating about what was happening around Steps. This lead to a lot of questions and confusion in the organization around what was going on with the effort.

Eventually, we realized our error. At that point we had to do some cleanup and rebuilding work to get the program back on track. Luckily during this time, the HRIS team had done their thing, as had Comp and Benefits. We also had our first Steps-enabled salary review to do. We were also very fortunate that parts of the organization had really started to embrace Steps and were using it in several ways to support individual growth through structured trainings and formal or informal mentoring. Having these efforts emerge helped keep Steps from completely being forgotten in the technology team.

The working group had been inactive too long at this point. Most of the members had new commitments. For a time we continued moving the effort forward with the participation we had, but it was clear that we needed to move into a new phase, which would require a new group.

Current Status

Currently, the Working Group is reforming with new members. The current members are engaged with our Human Resources team on training in Steps for Managers and Employees. We’re also starting working on the next iteration of Steps addressing some of the lessons learned and feedback now that Technology has been living with the framework for a while. The new group will have a few members from the first version, which will give some continuity to the effort and provide some history, but the rest of the group will be new, which should inject some new ideas and approaches to the work.

The leadership of the technology organization has been doing a lot to support the framework. We are actively giving more responsibility and opportunities to our Tribe/Guild step employees and we are also looking to grow our Squad/Chapter step employees to the next step if that is what they are interested in.

We’ve had several promotions between steps during the year; which is a serious measure of validation. If people weren’t moving between steps that would be a major issue.

Additionally, in my own organization, two managers who were also extremely skilled technologists moved back to being individual contributors. They were excited by the possibility of being able to be technical leaders without being people managers.

Many tribes are now starting to consider their Tribe/Guild step people as incremental members of their squads so that when they are helping outside their squad they don’t adversely affect the ability for the squad to get its work done.

Lessons

We learned many lessons in the creation of Steps. If there was one thing that I think was paramount, it was that this ended up being a lot more like Culture Change than we expected it would be. Our view was that we were creating something that would support and reinforce our company culture. In fact, we were specifying something where there had been only people’s own conceptions before. So, while we feel that we did create something generally well aligned and supportive of the Spotify culture, it was still a change for many individuals in the organization. If we had realized that, we would have treated the rollout much more like a Culture Change effort, and used more of those techniques in our rollout plans.

Another major lesson that I didn’t capture above was that we didn’t really have our long term support plan in place. We did put together a plan around supporting the effort, but that plan was created after the launch of Steps. As I mentioned above, the working group was pretty tired from the effort of getting the framework to that point. It would have been much better to think through the long term support needs of the work closer to the beginning of the effort and then adjust them later as we learned more.

I think that the process of using increasingly large groups to give feedback was quite good. It definitely resulted in a much better result than if we had just generated the framework in a smaller, more isolated, group. We primarily used the managers and leadership of the organization to collect feedback though. We would have benefited from including groups of individual contributors as well, especially if we could include the same people in multiple reviews of the document, just as we did with the coaches and leadership.

There was something that I neglected to mention in the first post that I thought was quite valuable. Once the Steps Framework was starting to solidify, we did a “simulation” to see what a potential distribution of steps was in the organization might be once we rolled it out. The committee had what we thought was an ideal distribution based on Spotify’s hiring strategy and what would make sense for an organization of our size. We asked each manager to estimate (in a non-binding way) which step each of their employees was on. We then combined these to get a master histogram of what the distribution might be. Luckily for us, we got a distribution fairly close to what we were hoping for. So, it was a good validation of the Framework at that point. This also had the effect of making each manager have to apply the Framework in a concrete way, which also generated more good feedback.

In addition to the lessons in the rest of this document, I want to reinforce the issues we encountered by underestimating the efforts required from Compensation and Benefits as well as our back-office systems to support this work. While HR was involved from the onset, we didn’t really take into account the timelines for the structural support to make the whole effort work. We should have involved those groups from the beginning, which would have avoided difficulties later.

Conclusions

In the process of leading the effort around career pathing in the technology organization at Spotify, I learned a tremendous amount. While I obviously spent more time thinking about how to motivate and incentivize employees than I ever had before, I also learned a lot more about my company and my coworkers than I had expected to. I was also reminded clearly how different my path had been than any of my fellow members of Technology. Things that I had learned experientially, I had to put reasoning and explanation behind and that was incredibly valuable. Many of our employees had never been at a company with career path support at all before. Those with experience at other companies that had career path frameworks, had not seen anything like the working group had created. As I talked with people individually and in small groups, I found better and better ways to articulate the reasoning behind the things that had just seemed obvious to us as we wrote the document. This helped us improve the document immeasurably and hopefully also has helped clarify the things that I’ve written about in these posts.

The effort of creating this framework also raised some issues in the organization that we’ll need to address as a group. The culture of Spotify is characterized by the Swedish word “lagom.” We try to see each other always as equals. We encourage people to challenge their leaders if they don’t think they are right. As is reflected in the Steps document, we put higher value on strong teams than on strong individuals. We also imbue our teams and our individuals with autonomy in how they do their work. We tried to incorporate all of these ideas in our Career Steps. When people in the organization wish for more recognition and status to be reflected in our career pathing, is this counter to our culture, or is this indicative of the culture changing? What is the role of career pathing in enforcing a desired culture rather than supporting it? These are questions we’ll continue to look at as we evolve Career Steps.

Spotify is a unique company, with a singular culture. The specifics of what we created and the lessons that we learned may or may not apply to your company. In aggregate, hopefully you will find our experiences and learnings valuable as you think through how you want to do similar programs at your own company.

Video of my talk “Apportioning Monoliths”

This was my talk at the Daho.am conference. Listening back to it now, I am struck by how often I said “many, many.” And I cursed! I usually try not to do that. So, it’s a bit of a looser take on this presentation. Luckily the audience had beer (this was in Bavaria, after all), so all were fine with it. I had flown in from Stockholm that morning, so I might have been a bit more tired than I thought…

I was really impressed by the lineup of speakers and the content of the presentations. A really good day. The Stylight engineering and event teams did a great job.

The Spotify model: how to create, dissolve, and remix teams to be more dynamic and more innovative

This post was originally written by request and posted on popforms.com. Special thanks to Kate Stull for requesting the article, and helping me with editing.

 

One of the most challenging parts of managing a traditional, hierarchical, organization is being responsive to new opportunities; especially those that require leveraging skillsets outside your own team. At Spotify, our organizational model allows us to create, dissolve, and remix teams with a minimal disruption to individuals or managers. This gives us tremendous abilities to address both temporary and long term opportunities.

How it used to be

As a manager at Microsoft and Adobe, I was always challenged when there was a problem or opportunity that required repurposing a team or adding on additional scope to an existing team.

This kind of thing comes up all the time: a business development opportunity, or integration with another product. Often, this would require small efforts from multiple specialized teams.

It would cause disruption as those teams had to change their current plans and had to coordinate around a new challenge while still making progress towards their existing goals. Given that people and resources were managed within the team, and managers were still responsible for delivery of their existing commitments, often it would be hard to motivate them towards supporting this new effort.

Creating a new “tiger” team is often the solution in these situations, but that isn’t always an adequate solution for long-term or permanent projects since it essentially punishes the managers of the existing teams and requires finding a new temporary manager for the new team.

Another problem in existing organizations is figuring out what to do with a team whose project has been cancelled.

If the team is a high-performing team you may try to turn the team onto a new problem, which may or may not be a good fit for their skills and experience. You may instead dissolve the team, assigning the members to new teams based on the needs of those teams rather than the preferences of the individuals. You may leave it up to the individuals to find new roles in the company or face layoffs if they are unsuccessful.

These solutions end up punishing both the individuals on the teams and their managers, often for reasons beyond their control. In an organization seeking to innovate (which requires some amount of failure), it sends a counter message to one of experimenting and taking chances.

How we remix teams at Spotify

At Spotify, we wanted to create an organization that allowed us to be dynamic around our staffing, and adaptable in our teams.

We embrace failure as being important to learning and innovation, so we didn’t want dissolving a team to be a punishment. We put this new organizational model into effect over two years ago and have been working with it since. In that time, the technical organization has grown from 250 to over 600 people. We went from having three engineering offices to five, and from having 30 teams to over 70.

We focused on building full-stack, autonomous teams, built around a single, clear, mission. The expectation is that once the team’s mission has been fulfilled that it will dissolve.

To this end, new teams are constantly being created and old teams dissolving, with their members building new teams or moving into existing teams if they need additional staffing. Rather than create a formal manager role for these teams, we decided instead to make the teams collectively responsible for fulfilling their mission.

With this model, changing teams does not mean changing your manager, and dissolving a team doesn’t leave a manager looking for a new role.

We do have a strong belief in role of the manager as mentor to their reports, so we have a strong managerial culture; it just is manifested in a matrix, rather than hierarchical model.

Why Chapter Leads work better than traditional managers

Our technical managers are called Chapter Leads. They are usually responsible for managing a narrow range of developer disciplines within their larger organizations, for example: mobile developers, or backend developers. A Chapter Lead usually has direct reports in multiple teams in the organization.

For an individual, it is common to change teams, but it is less common to change managers. As each team is responsible for their full stack and all platforms, a team may include members from several chapters.

An example is the search team in my organization. Its members come from five different chapters: the backend chapter, the mobile chapter, the keyboard and mouse (desktop and web) chapter, the agile coach chapter, and the test chapter. Additionally, there is a product owner and a UX designer, both of whom are part of the product organization (which is organized more traditionally).

The Chapter Leads are not responsible for deliverables directly. Instead, the Chapter Leads are responsible for staffing the teams appropriately; for working with the individuals in the team to help them grow; and for working with the Product Owner and the Agile Coach to make sure that the team is performing well together.

Since the Chapter Lead has visibility into multiple teams, they can often identify short or long-term skill set needs and are empowered to resolve them.

Sometimes, this means switching two developers in two teams temporarily for a skill set need. Sometimes this means moving a developer into a different team to address a short term staffing need. This also means that if there is a new mission to be addressed, the chapter leads can work together to staff a new team to address that mission out of the existing teams in the organization.

A benefit of this model for an individual is that there are many opportunities for them to work on new projects or develop new skillsets since there are new projects spinning up on a regular clip.

When and how we remix and dissolve teams

This remixing is not constant throughout the technology team. We do have several very long-lived teams that are focused on features in the product, but even those teams will shift people between each other based on short or long-term needs. In some parts of the organization, specifically the infrastructure teams, they tend to be focused on short-term projects and are creating new teams more often. Those teams dissolve when they have completed their project.

We will also dissolve teams if we believe that their mission is no longer necessary. Usually this is the result of the team invalidating their mission themselves. We celebrate these conclusions just as much as the successful completion of the project, since we value the lessons from a “failed” project. Celebrating your failures as valuable lessons encourages risk taking, experimentation and innovation.

By striving towards a model that gives the individual consistency (their manager, and their Chapter) while still giving the organization fluidity and adaptability, we’ve found a happy balance that lets us extend our agile-first values beyond the work that a team performs to the organization as a whole. This has allowed us to focus on innovation and leverage opportunities that slower-moving organizations would have difficulty addressing.

Several companies have attempted to adapt our model but there is something critical to understand. Our organization model itself is fluid and continues to change and evolve to support the needs of the organization. The specifics of our implementation are less important than the underlying values and ideals that created it.

If you want the benefits of a dynamic organization, you will need to build something that is suited to the values of your own organization. I would argue that a central requirement is endowing teams with autonomy and decision-making authority. If you cannot support this, then you should look instead to adapt your existing model to remove impediments and bottlenecks instead.

The video of my talk from QCon Shanghai is live

Had a great time at QCon, met a ton of interesting folks, and the organizers were awesome. The video and slides are now live on their site: http://www.infoq.com/cn/presentations/building-strong-engineer-culture

There are a couple issues to be aware of. They did the slides separately, so they don’t always line up with what I’m saying. Also, since I was being simultaneously translated, I talk more slowly and pause more often than I would normally to let the translator catch up.

I think that this will be the last time I give this talk publicly, now that it is recorded fairly well, and available world-wide.

My slides from ProductTank Sthlm

I was asked to speak about how we align around the product while we maintain autonomous teams. Given that this was a talk to product people rather than engineers or coaches, I tried to keep it focused around product definition and prioritization. As usual, I don’t like to put many words in the slides, so hopefully you get the gist. There is a recording being made, so hopefully that will get posted and then I will link to it here.

The Myth of the Startup in a Large Company

I was reading this post from John Gruber, which had this paragraph in an ad for the iOS development team at Google:

My thanks to Google — that’s right, Google (kind of awesome, right?) — for sponsoring this week’s DF RSS feed. They’re hiring developers and designers for their iOS app teams, which operate like a start-up within the walls of Google.

and I thought about the number of times I’d heard that line: “Operates like a startup within insert large company name here” or “Operates like a startup, but without the risk.” I’ve heard that line so many times from recruiters, from friends… To be honest, I’ve even said it myself a few times, trying to sell a prospective candidate who I was trying to woo away from a startup.

That notion, of working like you are in a startup, but being part of a much larger organization, is a myth. Anyone who says it is naive, disingenuous or just plain wrong. Large companies that try to build those kinds of teams; be it “innovation lab”, “startup experiment” or “corporate startup incubator” usually fail to achieve the innovation or energy they sought. The result is usually a whole bunch of wasted money and angry employees who felt like they were promised a bill of goods.

Stewart Butterfield, discussing his experience selling Flickr to Yahoo:

They sold out to Yahoo assuming that they’d be backstroking in rivers of money and terabytes of memory. Instead they had to fight for everything: servers, people, time.

This is talking about the inverse problem, but it comes down to the central crux of the issue at large companies: resource contention. This is beyond innovator’s dilemma.

In a startup: all of your attention is spent on finding the right product/market fit, finding customers, finding a flow of income, and/or finding investment. You will make the trade-offs you need to get your product off the ground. Often this may mean choosing poor technologies in the short term to help you get going more quickly. Your resources are limited, you need to make do to get going. Maybe you will take some short-cuts in other areas just to get the product launched. You are fighting for your life and your income, and you will do whatever it takes to get there. Why do you do it? Because you love the energy, or because you are looking for the fiscal payoff. No risk, no reward.

In a big company, you don’t have to make those trade-offs. There is probably a very mature infrastructure to build on; there is a brand to build off of; there is the promise of a paycheck no matter what the outcome. It is these conditions that destroy the innovation.

There is a mature infrastructure, but maybe it is not a great fit for what you are trying to build. Maybe you just need some small tweaks, but the infrastructure team is primarily focused with serving the existing teams that bring in the revenue; it will be hard to get your needs prioritized. Maybe you can even prototype or launch with your own skunkworks infrastructure; that won’t last for long. The corporate infrastructure is vetted, financed, regulatory compliant, and they own their turf and don’t appreciate someone jury-rigging something else.

There is a brand, but that brand is well known and highly controlled. You can’t launch just anything using that brand. It needs to be vetted. This means that instead of focusing on building your product, you are instead focusing on getting internal support. Maybe you launch under some new secret brand. This may work for a little while, but if you are successful, there will be increasing pressure to join the fold. And in any case, launching under a secret brand basically kills the benefit of the being part of the parent company.

The lack of risk is its own deterrent. Knowing that you get the paycheck is nice, but it is also understanding that you have very little ownership in the outcome. It isn’t “your” product, it is your corporations’ product, you are just one of the people on the team. While you may still put in startup hours for the joy of it, eventually you will realize that you aren’t getting the startup reward for all your hard work, and that is pretty demoralizing.

The general problem is that even if you have the deep pockets of a large corporation backing you, you don’t have the ability to do what it takes to survive. From the minute the project is launched, you are on a clock. Because you are part of a larger (presumably already profitable) parent, you will be restricted from certain business models: it’s hard to justify spending a few years taking substantial losses to scale your business quickly when you are seen as a drain on the profits of your parent company. Amazon, with their don’t-ask-us-about-profits model could never have been created as a division of Microsoft. The shareholders would have rebelled.

If you don’t succeed quickly, your team’s resources will be coveted by the teams around you. They are like vultures waiting for you to fail, and they will rush to declare you a failure as early as possible if they think they can benefit from it.

If you are successful and start to grow, you have the same problem. Teams will attempt to co-opt your mission, or take over your team, or switch you onto the “official” technology stack, or just flood you with resources trying to get some of your “startup” energy.

If you are successful by startup standards, that may not be seen as much of a success in a larger parent company where there is an established business. Being slightly profitable is a huge win for a startup; being slightly profitable is a major loss for an established corporation.

So, how can you create a startup in a large company? I think the university model is an interesting approach. Say you are Company X, a large multi-national technology company, and you are constantly challenged by your inability to move at startup speed or innovate. Instead of creating a startup team inside some division; instead create an actual startup.

Solicit pitches from your entrepreneurial employees. Pick one or more, and fund them as independent companies. Give the founders an equity stake in the new venture, but Company X will own a significant stake as well, plus some non-exclusive licenses on the IP. Allow them to recruit from your company, but they will no longer be employees of Company X. If the venture fails, they may be able to interview to rejoin Company X, and they may get to get some of their benefits back, but there is no guarantee of employment. Also, get them out of your building. Allow them to raise outside investment if they need.

By sponsoring your own employees, they are likely to build in a compatible way with the way your company works, given that it is their training. They will know your industry. They won’t be complacent because they can’t afford to be. They will also be invested because they will directly benefit from their success. In the end, you will get the innovation you want, and probably cheaper than if you tried to fund it from within your own cost structure with all of its overhead.

[This post was updated on August 23, 2014. Nothing was removed, but I added some more thoughts about business model limitations and startup success levels not matching the expectations of a more established company]

Appowr interview from Budapest

A bit out of context: “Think it, Build it, Ship it, Tweak It”, “Go Big or Go Home” and “Play Fair” are some of Spotify’s core values, but you get the idea.

It was really great meeting Jacqueline Do and the Appowr and Layer Gloss crew in Budapest.

Thoughts on emulating Spotify’s matrix organization in other companies

I was in San Francisco in December for a conference. While I was there, I ended up connecting with a couple different companies who have been inspired by Henrik Kniberg’s whitepaper on Scaling Agile at Spotify, and who have been trying to implement some of those ideas in their own companies.

I think Henrik’s paper does an excellent job on describing the what and how, but it seems that the “why”, and some of the critical ideas can get lost when others read it.

If you haven’t read Henrik’s white paper, I’d suggest that you read that before reading the rest of the blog post. I will do a quick recap here though.

Spotify’s engineering and product organization (now over 600 people) is split into several large groups called Tribes. Each Tribe is responsible for a set of related features or engineering functions. For example, our largest tribe is the Infrastructure and Operations Tribe, whose name is pretty self-explanatory. I am the Tribe Lead of the Music Player Tribe. We handle importing audio from our label and distribution partners, storing and streaming the music, search, collection and playlists, artist pages, music metadata and the music knowledge graph that supports things like the above, but also ads, discover, radio and the like.

While the whole company works on the same product, Spotify, each tribe is set up so that it can work as independently as possible. As you will see below, a critical aspect of our organizational model is to give autonomy at every level. This helps remove decision-making bottlenecks and unnecessary dependencies, which improves velocity.

Each tribe is composed of squads. A squad is a team that is responsible for a single feature or component. For example, there is a squad that is responsible for search, a squad responsible for the AB test infrastructure, etc. As each tribe is set up to be as autonomous as possible, each squad is also set up to be autonomous. In the context of a feature development team, this means that each team is a full-stack team. A full-stack team is responsible for both backend implementation as well as the user interface implementation, on all platforms.

A typical feature squad would have web service engineers, iOS, Android, web and desktop engineers as well as testers, an agile coach, a product owner and UX designer. With this staffing, the squad has everything they need to implement anything related to their feature. They don’t have to wait on another team to implement the pieces they need. They also have autonomy and local decision-making ability, so there are few impediments on their speed of execution.

To this point with Tribes and Squads described only, Spotify may seem like a traditional, hierarchical engineering organization, but this is where the similarity ends. Unlike a traditional organization, a squad does not have a single engineering leader whom everyone on the team reports to. In fact there is not a single leader for the squad. The Product Owner and UX Designer work with the engineers and testers collectively to make decisions about their features.

Spotify is not a “no manager” culture though. We feel strongly that managers have a role in supporting the people who work for them. Managers have an important role to play as technical and career mentors and organizational communication conduits. Rather than have management hierarchies follow organizational ones (creating a de facto command-and-control structure), we instead have first level managers responsible for technical functional areas across multiple squads.

We call these reporting and functional groupings “chapters.” Again, as an example, reporting to me, the tribe lead, are Chapter Leads. In my tribe, there are currently three backend (services) development chapters, two front-end development chapters (including all the UI developers), a core library chapter, and a test chapter.  These seven Chapters span eight different squads. Almost every chapter lead has reports in 2 squads, and a few of them have reports in three squads. Almost all chapter leads work within a squad in some capacity as well, either as developer or technical lead, and not necessarily within a squad that has members of their chapter.

This chapters/squads matrix organization is critical to our organizational agility. It allows the squads and the tribe to be more fluid. We can spin up a new squad to take advantage of an opportunity or handle an issue without worry about changing reporting structures. If a squad completes its goals and has no reason to exist anymore, we can dissolve it without punishing a manager. This is a very important difference to a traditional hierarchy, because it gives us a lot of flexibility and helps us avoid the old political issues around empire building and resource contention.

In addition to our Tribes, Squads and Chapters, we also have virtual organizations called Guilds. Guilds are cross-tribe organizations centered on different technical or interest areas and their membership is voluntary. The guilds serve as ways to promote cross-tribe collaboration and communication, especially around things like best practices. For example, we have guilds for Web Development, Agile Practices, Leadership, Test Automation, etc. The guilds foster developer-to-developer communication, which is one of the ways that we keep all these autonomous teams from all going off in completely different directions.

From Henrik’s paper, this diagram illustrates the organizational structure I discuss above:

Screen Shot 2013-11-09 at 7.30.08 AM

I’d like to give some more background around why we have implemented this organizational model at Spotify; elaborate on our goals for implementing it, and discuss the aspects of our culture, which have been critical to its success. It is really great that other companies have been inspired by the way we work, but I think if you implement only parts of the model or try to impose it on a very different corporate culture; you will have a difficult time achieving the same level of success with it that we have had.

If you are considering using the Spotify organizational model within your company, there are a few things that will be critical to your success:

Our organization model assumes that engineering is doing development with agile methodologies. Our goals for autonomy mean that we do not prescribe any particular development framework our squads must subscribe to. However, all of squads use agile development methodologies. While we do our best to minimize dependencies between squads and tribes, there will always be some since we are all working on the same product. Any individual squad choosing to build using a traditional waterfall or other non-agile process would not be able to keep up with the rapidly changing teams around them. If they tried to impose some sort of process on other teams so that they could follow a longer-term development plan, they would start slowing down the rest of the organization.

A critical requirement in making our organization model work well is that the entire company works with and understands agile practices and processes. While our legal team isn’t doing scrum or kanban, they are used to working with engineering teams that use agile processes. Having the entire corporation understand and agree with agile means that no line of business area becomes an impediment to the speed of implementation. Think of this in terms of Amdahl’s law, applied to a development organization. If your development teams are working quickly in parallel, but marketing or legal is not supportive of an agile approach, they will become a bottleneck that will slow down the overall speed of the company.

Similarly, implementing this with just one team as a test in a larger engineering organization will be prone to issues. A traditional engineering organization is not usually set up for autonomy. Adding a single autonomous team within that web of dependencies is likely to hamper and frustrate the team and skew the results of the experiment.

While I’ve mentioned autonomy in several places already, I cannot understate its criticality. Each squad must be empowered to make their own decisions, not only on features, but also on development model, infrastructure, and implementation. Every decision that has to be approved outside the team means a delay that slows development. Each dictated implementation or infrastructure decision means that a technology that doesn’t fit to the way the team works or something new that must be learned before the team can build. This is a challenge to coordination, but in practice it isn’t as bad as it might seem. Best practices and technologies do spread from team to team through avenues like guilds. Teams adopt these practices and technologies on their own schedule or pioneer new ways of working if it makes it easier for them to deliver value to our customers and then spread their learnings to the other teams.

Trying to layer the tribe and squads model over a traditional reporting hierarchy would be very problematic. While we have many long-lived squads at Spotify, we are constantly creating and disbanding squads as new needs arise or missions are fulfilled. Squad membership will also ebb and flow as required by the needs of a squad’s mission. Traditional hierarchical organizations are self-perpetuating and restructuring them is very disruptive both to the management chains as well as the individual team members. You would gain some of the benefits of the Spotify model by building full-stack teams in a traditional organizational hierarchy, but you would lose a lot of the overall speed benefits that we leverage with our matrix organization.

In conclusion, if you are looking to improve the speed of your development and are inspired by Spotify’s organizational model, there are a few things that you need to understand. Our model works because it is layered on top of our corporate culture. Our culture values autonomy, agile processes, democratic teams, and servant leadership, amongst other things. You can certainly take some of the ideas from the way we work and apply them in your organization, but without the cultural underpinnings you may not get the same returns.

Some other references worth checking out are Henrik Kniberg’s keynote at the Paris Scrum Gathering and my keynote at the { develop: BBC } conference.