How we approach failure is critical in any industry, but it is especially crucial in building software.
The answer is simple: invention requires failure.
We don’t acknowledge that fact enough as an industry. Not broadly. It is something we should recognize and understand more. As technologists, we are continually looking for ways to transform existing businesses or build new products. We are an industry that grows on innovation and invention.
Real innovation is creating something uniquely new. If you can create something genuinely novel without failing a few times along the way, it probably isn’t very innovative. Albert Einstein expressed this as “Anyone who has never made a mistake has never tried anything new.”
Filmmaker Kevin Smith says, “failure is success training.” I like that sentiment. It frames failure as leading to success.
Failure teaches you the things you need to know to succeed. Stated more strongly: failure is a requirement for success.
Creating a fail-safe environment
To achieve success, what’s important isn’t how to avoid failure; it’s how to handle failure when it comes. The handling of failure makes the difference between eventual success and never succeeding. Creating conditions conducive to learning from failure means creating a fail-safe environment.
In the software industry, we define a fail-safe environment as setting up processes to avoid failure. Instead, we should ensure that when the inevitable failure happens, we handle it well and reduce its impact. We want to fail smart.
When I was at Spotify, a company that worked hard to create a fail-smart environment, we described this as “minimizing the blast radius.” This quote from Mikael Krantz, the head architect at Spotify during that time, sums up the idea nicely: “we want to be an internal combustion engine, not a fuel-air bomb. Many small, controlled explosions, propelling us in a generally ok direction, not a huge blast leveling half the city.”
So, let us plan for failure. Let’s embrace the mistakes that are going to come in the smartest way possible. We can use those failures to move us forward and make sure that they are small enough not to take out the company. I like the combustion engine analogy because it embraces that failure, well-handled, pushes us in the right direction. If we anticipate, we can course correct and continue to move forward.
One way you can create these small, controlled explosions is to fail fast. Find the fastest, most straightforward path to learning. Can you validate your idea quickly? Can you reduce the concept down so that you can get it in front of real people immediately and get feedback before investing in a bunch of work? Failing fast is one of the critical elements of the Lean Startup methodology.
A side benefit of small failures is that they are easier to understand. You can identify what happened and learn from it. With a big failure, you must unpack and dig in to know where things went wrong.
I worked at Microsoft when the company created Office Assistant. Although I didn’t work on that team, I knew a few people who did.
It is easy to think that the Office Assistant was a horrible idea created by a group of poor-performing developers and product people, but that couldn’t be farther from the truth. Extremely talented developers, product leads, researchers with fantastic track records, and PhDs from top-tier universities built Clippy. People who thought they understood the market and their users. These world-class people were working on one of (if not THE) most successful software products of all-time at the apex of its popularity. Microsoft spent millions of dollars and multiple person-years on the development of Clippy.
So, what happened?
What happened is that those brilliant people were wrong. Very wrong, as all of us are from time to time. How could they have found their mistake before releasing widely? It wasn’t easy at the time to test product assumptions. It was much harder to validate hypotheses about users and their needs.
How we used to release software
Way back before we could assume high-bandwidth internet connections, we wrote and shipped software in a very different way.
Software products were manufactured, transcribed onto plastic and foil discs. For a release like Microsoft Office, those discs were manufactured in countries worldwide, put into boxes, then put onto trucks and trains and shipped to warehouses, like TV sets. From there, trucks would take them to stores where people would purchase them in person, take them home and spend an afternoon swapping the discs in and out of their computers, installing the software.
With a release like Office, Microsoft would need massive disc pressing capability. It required dozens of CD/DVD plants across the world to work simultaneously. That capability had to be booked years in advance. Microsoft would pay massive sums of money to take over the entire CD/DVD pressing industry essentially. This monopolization of disc manufacturing required a fixed duration. Moving or growing that window was monstrously expensive.
It was challenging to validate a new feature in that atmosphere, peculiarly if that feature was a significant part of a release that you didn’t want to leak to the press.
That was then; this is now.
Today, the world is very different. There is no excuse for not validating your ideas.
You can now deploy your website every time you hit save in your editor. You can ship your mobile app multiple times per week. You can try ideas almost as fast as you can think of them. You can try and fail and learn from the failure and make your product better continuously.
Thomas J Watson, the CEO of IBM from 1914 until 1956, said, “If you want to increase your success rate, double your failure rate.” If it takes you years and millions of dollars to fail and you want to double that, your company will not survive to see the eventual success. Failing Fast minimizes the impact of your failure by reducing the cost and delay in learning.
I worked at an IBM research lab a long time ago. I was a developer on a project building early versions of synchronized streaming media. After over a year of effort, we arranged to publish our work. As we prepared, we learned there were two other labs at IBM working on the same problems. We were done, it was too late to collaborate. At the time, it seemed to me like big-company stupidity, not realizing that three different teams were working on the same thing. Later I realized that this was a deliberate choice. It was how IBM failed fast. Since it took too long to fail serially, IBM had become good at failing in parallel.
Recently, I have been thinking about the role of the executive in a scaling startup.
As a senior leader in a growing company, you need to be scaling faster than the organization. You grow by scaling yourself and the leaders in your team more quickly than the business. This fact is well known and is covered excellently in such books as Zero to One and The Hard Thing About Hard Things.
Even if you are aware of this fundamental requirement, it is still challenging to recognize when you are starting to fall behind on that scaling. The people on your team, the people that got you to where you are today, who are working as hard as ever, should be doing better than they are. You may start seeing the signs: teams falling behind, tensions between groups or functions, team leaders beginning to struggle with their work, and increasing responsibilities.
You might not know what these scaling problems look like because you haven’t seen them before. Maybe you do recognize them, but your loyalty to your team lets them go on longer than they should. You can get away with that for a while.
Eventually, your boss (the CEO, the board) or your peers start to recognize the growing gaps in your organization between where you are and where you should be. In a company with a good culture, they will let you know. In a company with a less-open culture, your peers may notice but not feel like it is their place to say.
By the time the problems are apparent outside your team, it will be nearly too late.
When these problems first arise, you need to put together a plan. If you missed the early signs and the challenges are visible outside your team, you need to act immediately.
You need to bring in new talent who can help close that gap. It will take time to do that. If you choose to re-double your efforts to mentor the existing folks, you will only fall further behind. Either you missed your window to mentor, your leaders need more mentorship than you can provide, or they are not yet ready to take on the new responsibilities in their role even with mentorship.
Replacing people who have historically done well in their roles can seem cruel, and this is why it is hard. It feels disloyal to the people that have been loyal to your company and have helped to build it along with you. It is not their fault.
If you don’t make those hard choices, though, they will be made for you by the person whom your boss or the board hire to replace you.
It doesn’t have to be this way.
We have an assumption that in a growing company, people will remain in the roles they have had, and newer employees will come in below them. This assumption is one of the exciting incentives of joining a startup. It can be a career accelerator. Indeed, there are many stories of early employees at startups remaining in their senior leadership roles through rapid growth and past the point of going public. Very few people are capable of this kind of personal development, however.
Instead, we should be explicit about this challenge of growing a company. We should build a culture that acknowledges and celebrates this fundamental fact. Let people your hire know that you will support their growth, but be honest that if the company is scaling faster than they are, they may need to help hire the person who will help with the next phase.
Reid Hoffman talks about these ideas in his book The Alliance. I think Netflix has done well being explicit around the Tour of Duty in their culture. I do think Netflix is a bit too employer-focused in its attitude towards these ideas. This approach works for them because they favor hiring experienced developers and do not invest much in training their employees relative to other companies. That is another definitive decision of their culture.
I advocate for a more balanced and sustainable approach for companies, one that encourages employee development and business realities. Startups that are willing to hire at all levels of experience and support employee growth can hire and retain better. Even those companies face challenges at their scaling inflection points when company leadership changes by the new business reality’s necessities.
Suppose your company builds the concept of succession for scale into its culture. In that case, hiring your successor should be expressed as an opportunity for further mentorship and growth and not as a demotion or failure. Celebrate it as a rite of passage. Challenge the leaders in your team (and give them the tools) to recognize when this time has come, and praise their self-awareness.
Build succession for scale into your compensation structure and leadership career pathing. Ensure that the newly hired leaders train the people they have replaced to assume the role once again. If the position opens up in the future, the person may now have the skills to step back into it.
People leave jobs. If you are a manager, people will leave your team, just as someday you will leave your team.
When this happens it’s an opportunity, a chance to re-evaluate. While you might want to immediately pull out the job description that you used when hiring for the role last time, instead, take some time to think.
A chance to learn
When someone that works for you tells you that they are resigning, it can feel personal: ‘they don’t like working for me.’ It can hurt. You might immediately look for any reason why it isn’t your fault. You may obsess about everything that you could have done differently. It is natural to want to move on as quickly as possible.
Instead, after an employee gives you notice, take a day or two to process and get some distance. Recenter. Come back to them with an open mind. Do not look to assign blame and let them know that you are working to improve the team for the people that are still here. Ask what was not working for them and what they will miss, aiming not to assign any extra meaning to what they say. Take notes. Thank them.
Take some more time to distance, then come back again and think about the leaver’s words. Try to understand from their perspective what they experienced. If they are taking a more senior role elsewhere, was there an opportunity like that in your company that you could have helped them get? If they are joining another company to learn a different technology, was it a technology they could have explored in your organization? Was there another team in your company that they could have joined instead?
Your goal is to understand their unmet needs. Were there signs that you missed? Were there opportunities in your company or in your team that could have addressed their needs?
Once again, the goal is not to assign blame, and the goal is not to get the employee to change their mind either. The goal is to learn from this experience.
So moving forward, how can you approach your role in a better way?
Consider this process to be a personal retrospective, and just like in an Agile team retrospective, you may want to emerge with a list of things to keep doing, a list of things to start doing, and a list of things to stop doing.
A chance to change
As teams evolve they shift and mature. If the leaver has been in the group for a long time, they may have accumulated an unusual set of responsibilities and they may have influenced the technical decisions around their strengths.
While it may seem like the obvious decision is to look for someone with the same skill set, that is just ‘role inertia’ (credit to Omosola Odetunde for introducing that phrase to me). Instead think of this as an opportunity to re-evaluate and make a change without impacting someone.
Consider your technical vision for the team and the skill sets of others in the group. Is there something missing that could help you today or in the future? Is this role still needed? Should you repurpose the position into a different one based on the team’s long-term needs?
It is critical to be thinking about long-term needs and not short-term ones. A mistake tech leads often make is that they hire someone because of near-term demand. They assume that there will be a headcount later to cover the long-term need, but too often that headcount doesn’t appear and now the team is missing a crucial skill set.
Potentially your team is out of balance, where you have too many (or too few) senior folks. This opportunity means you can now rebalance the levels within the group. Maybe this role is no longer necessary and you can give a headcount to another team that needs it more, or is there someone on the team who is looking for a new challenge and can step into the role?
If you are in a position where you manage multiple teams then this might be an opportunity to re-evaluate the team structure, especially if the leaver is a manager. A way to approach this exercise would be to imagine that the person leaving was never on the team. Your manager has given you a brand new headcount and asked you to figure out how you want to use it.
Once you have a plan, you can then write the job description and look to fill the role, as you may decide that you need to replace the person with someone who has a similar skill set. If so, you can move forward confidently knowing that you have thought it through, and if you have also taken the time to learn you will hopefully retain your new hire for a long time.
I’ve been playing bass since I was 15. I play other instruments as well, but I have always been primarily a bass player.
Music has always been not only a joy to me but also a salve. Writing software and leading technology organizations is such an “in your head” endeavor. Playing music for me is much more about intuition and feeling. I can do it for pure pleasure, and if I stumble on something I like, I can go deeper, or just hope I find it again in the future. No stress.
I was recently talking to another technology leadership friend about playing bass, and it made me realize how many things those two pursuits share.
While you can play bass alone, it is not a solo instrument. You need a band. Similarly, you can’t do much as a leader unless you are part of a team.
A good bass player may move to the front from time to time, but usually, they are in the back, keeping everything on track. A bass player keeps the groove going, pushes the song forward, but isn’t necessarily the one that everyone is looking at. If the bass player isn’t there, though, the band is missing a critical element. A lead is a vital element of a development team, but a lot of the value they add isn’t always visible.
While I always appreciated and admired the well-known quick-fingered, super-complex players like Geddy Lee, Flea, Les Claypool, and Mark King, the bass players who most influenced my playing are people like Peter Hook, Paul McCartney, Carol Kaye, and James Jamerson who excelled with elegant simplicity. A worthy engineering lead is not about flash, but about substance. Not interested in complexity for complexity’s sake, but in doing what the team needs and no more. As the Swedes say, “lagom.”
As part of the rhythm section, the bass player works with the drummer to keep time, but also to modulate and push things when needed. As a bass player, you might be helping an over-caffeinated drummer not push the tempo, or you might be conspiring with the drummer to give the song a bit more energy if you think that is what the audience needs. The lead of the team needs to be aware of the team’s dynamics and maintain a good pace, but also be mindful of the customer, and the business and push the team when needed.
While the bass is a melodic instrument, it isn’t necessarily carrying the melody. It supports the melody, tracking the chord changes. The bass player keeps the structure of the song, which allows the other instruments to take chances, embellish, or step into the spotlight to solo. Similarly, the engineering lead maintains the team’s vision, architecture, and the big picture so that the members of the team can shine or try out new ideas without fear of losing the thread of what is essential.
In a recent Lifehack article, Joseph Jo identified “8 Desirable Dating Qualities Of A Bass Player.”
I thought that six of the eight also are desirable qualities of an engineering lead:
They Love to be Connected
They Are Content Regardless of the Lack of Attention
They are Passively Creative
They are Considerate
They Tune in with People
They are the Artists of Adaptation
So, if you want to be a better engineering lead, you don’t need to buy a bass and join a band, but you might want to start trying to think more like a bass player.
I was inspired by this tweet from Dave Nicolette to talk a bit about what I look for when hiring Agile professionals.
Understanding the value of Agile coaches
While I have been working exclusively with Agile techniques since we adopted Extreme Programming at a start-up where I was the development lead in 2000, I had never encountered a team-aligned full-time Agile professional before I joined Spotify in 2013. My prior experience with Agile was always that the team was responsible for it.
As a development lead, I was the XP coach when we did Extreme Programming. When my teams chose Scrum, I might take the role of Scrum Master, or it was the Program Manager, someone else on the team, or float between multiple people.
When I came to Spotify and found that I had three Agile coaches in my tribe, I was first a bit skeptical about the role. The coaches I worked with were not program managers, not scrum masters. They didn’t “lead” Agile in the teams with whom they worked. I wasn’t sure what their purpose was.
I first came to understand their value when one of them went on an extended vacation a few months after I started. At Spotify, I had found the most advanced and mature implementation of Agile/Lean product development at a scale that I have ever seen. I knew the coaches helped this, but I wasn’t sure how.
The coach went on their vacation, and everything kept going on as usual for a while. I would visit the stand-ups, and teams were adding stories and tracking them across the boards. One day I sat in on a squad’s stand-up and noticed that they had added a couple of swim lanes to their Kanban board. They now had more swim lanes than developers—a big red flag.
Over the weeks the coach was gone, the teams slowly slid into some bad habits. Velocity started to slip. I did my best to make them aware of this and get them back onto better paths, but I couldn’t be with each team enough.
The coach came back from vacation, and within a week or so, things were back to their high levels of performance. I wanted to see how he did it, and so I watched the ceremonies when I could. He didn’t cajole or quote Agile texts at them. He gently reminded them what good looked like, lessons they had learned in the past. He asked them many questions around why they thought what they were doing was a good idea. He didn’t “fix” them. He got them to fix themselves—a true coach.
Now I understood the value of the Agile coach role.
Good coach, bad coach
As Spotify grew and the number of Agile coaches in the company swelled, I also got to see some challenges with the role. Some coaches were highly effective and some less so. I had been lucky to start with three excellent coaches in my team. Some of my peers struggled with the coaches in their organizations.
As I came to understand the characteristics of the coaches that I found successful, I started to look for those qualities as we hired into our team. I have continued to look for those qualities as I have created those roles at companies in the US and UK where the role of Agile Coach (versus Scrum Master, Delivery Manager, or Agile Project Manager) is still novel.
Before I enumerate those characteristics, I want to make one point about careers as an Agile professional.
It is a tough job.
In many parts of the world, full-time Agile roles are very hard to come by. Mostly, companies hire Agile folks on a contractual basis. So, most Agile people have to string together six- or twelve-month stints at various companies trying to earn a living.
After reviewing hundreds of CVs in the US and Europe, the same companies show up often. These companies are always the ones who are in year X of a one-year Agile transformation program. Those are soul-crushing gigs.
The stringing together of short-term jobs can lead to a consultant mindset. These folks have the wisdom from having to jump into hostile environments, trying to survive. They have seen many mistakes that companies have made. Few have held the more extended roles where they have not only got teams functioning in an Agile way, but also helped them to evolve to a much better level. Their experience is broad, but not deep.
It is vital to keep that in mind as you review applicants, you need to understand their world and watch out for folks who have gotten stuck in that short-term mindset.
What I look for when I hire Agile coaches
A product development background. It isn’t critical which specific history the person has as a developer, tester, product manager, UX designer, engineering manager… I want to see that they had direct experience shipping a product. Agile roles have been around long enough that people can be trained in them in school and go right into the profession. From my experience, Agile people without experience building products can have a hard time making the trade-offs that are sometimes necessary. They may focus too much on the “how” without understanding the “why,” “what,” or “when.”
Broad knowledge of Agile frameworks and techniques. While the core of Agile thinking has been around for many years, new practices and methods continue to evolve. Like any profession, I look for a candidate to demonstrate that they are not only keeping up but are interested in what is happening in their field.
Experience growing a teams’ proficiency over time. As I mentioned above, many Agile professionals get stuck in an endless series of Agile transformations at different companies. While this is a valuable experience for an Agile consultant, it isn’t that practical for someone coming into a long-term role.
Pragmatic, not pedantic. Pragmatism is something I look for in everyone I hire. I would not expect this to be an issue for an Agile professional, but I have interviewed people whose definition of what was or was not correct was defined by a single book.
Knowing what good looks like. The characteristics of a high-performing Agile team are incredibly context-dependent. There is no single way to be an effectual team. So how do you convince teams to invest in improvement? You need to give them the vision of what they can be, which means that you need to know what “good” looks like.
Knowing what bad looks like. The converse of knowing what good looks like is knowing what bad looks like. I want to hear what the candidate identifies as harmful patterns in a team. The patterns they identify, help me understand how they look at teams. I also want to listen to their techniques for breaking teams out of these patterns. I want to hear what has worked and not worked for them.
A desire to build something bigger than themselves. I want to see some ambition in a coach. Not just to get a group working well, but to redefine what a group can achieve with the right support. If a candidate thinks their job is complete when the team has regular ceremonies, a groomed backlog, and a good flow of tickets, they probably aren’t what I am looking for.
Experience working with cross-functional stakeholders. Too many people view Agile as a software development thing, with defined boundaries aligned to the engineering team. Successful Agile organizations interface with the whole company, even if those functions do not choose to work in an Agile way.
Building an Agile coaching practice in your organization
If you want to build a new Agile coaching practice within your organization, it is best to start slow. Hire one coach, work with them to establish what the role means within your company. When the organization demands more time from them than they have to give, it will be time to hire a second coach, and so on.
Each coach should be able to support multiple teams, especially if you want the teams to own their practices instead of the coach (this is one reason why the coach should not be the scrum master for the groups they work with). Working with multiple groups also helps give some visibility across the organization about the quality of Agile practices and is an excellent conduit for best practice sharing.
You may have over-hired on coaches when the Agile coaches end up driving their deliverables and organizing their work as a function. That may mean that the coach to team ratio is off.
If you are serious about evolving your Agile practice as an organization and improving the quality, efficiency, and happiness of your teams, hire an Agile coach.
One of my family’s quarantine projects is re-assembling all my daughter’s old Lego sets. The pieces from the sets are in several large storage totes, mixed at random from years of building and taking things apart. As I was digging through a box today looking for some specific piece, I started noticing the system I had started to use.
As I looked for a piece, I would start to collect identical pieces and join them up. Joining pieces allows me later to find those pieces later more efficiently, even if I put them back into the box. It also reduced the number of pieces I would have to sort through to find anything. I do this unconsciously because I have done this ever since I was a kid.
Today I realized that this was a perfect metaphor for paying down technical debt.
Grouping the Legos as you are building means that you take a little bit longer on the sets you make at the beginning, but each successive set gets faster. Not only are there fewer Legos to sort through, but the Legos that are there are becoming more and more organized.
When working in a code base that has accumulated a lot of technical or architectural debt, cleaning things up as you go means that your velocity increases over time. Ignoring technical debt is like adding a few random Legos to the box as you take pieces out. Not only does it not get simpler or faster. It gets slower. Eventually, you have to go to the store to buy a new set because it is just easier than finding the pieces for the old one. Or worse, you have to go to eBay and pay twice as much for the same set because Lego stopped manufacturing it. (I am probably abusing the metaphor here.)
I’ve also been thinking about the difference between building a set by pulling out Legos from a big box versus building a brand-new set.
When you build a new set, the pieces come in smaller bags. Lego numbers the bags, so you only need to open one at a time to find the parts you need. Bigger sets may have multiple instruction books, also ordered by number.
The grouping of Lego pieces into bags is a metaphor for Agile software development.
By narrowing the scope and limiting the options, you make the work go faster, even when the problem is involved (like one of their expert models).
The next time you are trying to explain to your product manager (or anyone) why you need to add more tech-debt stories into the backlog even though it means a feature will take longer to deliver, bring in a big box of Legos as a teaching tool. If it doesn’t work, you’ll at least have a fun team meeting…
I was the Chief Technology Officer at Avvo from 2016 until 2018, when the company was acquired. One of the proudest achievements of my tenure there was building a more diverse and inclusive technology team.
While I have spoken a bit about our efforts before in talks, I haven’t written about it. Even though we had done well, I hadn’t achieved the ambitious goals I had set when I joined the company. I was always hoping to do better at my current company, and then use the wisdom from building more diverse organizations multiple times to provide higher-value insight.
Given current events, I hear from friends at tech companies that they are re-evaluating their lack of diversity. They want to improve. While companies have tried different tactics for years, we haven’t made much progress as an industry. If you are trying to address diversity for the first time, it is a daunting task. It is easy to try some things, make no progress, and then give up.
This article isn’t a prescription for improving diversity at any technology company. It is a case study for how we grew diversity at Avvo. The different strategies we employed may be useful for your efforts. They may just inspire you to try new things that are appropriate for your company and culture.
Avvo is an online legal marketplace connecting consumers and attorneys in the US. The company overall was fairly diverse from both a gender and a race perspective. When I joined in 2016, the technology team was a different story.
At the start of my tenure, the organization was not very diverse, as displayed in the charts below. Additionally, there was no one from any under-represented groups with a management title in the Technology team.
The team’s status quo was not the result of overt discrimination, but it did demonstrate a lack of prioritization of diversity, inclusion, and reducing unconscious bias.
My vision for the company was to make it an exemplar of diversity and inclusion within the region. I wanted to not only make Avvo’s technology team diverse to prove that it was possible. I also wanted to show that a diverse group is more than capable. I knew that we could use our diversity as part of our employer branding, helping us stand out in a very crowded market for talent.
Making Improving Diversity a Priority for Technology
I set the tone I intended to take as part of my interview process for the role. I didn’t want any potential for disputes between the rest of the senior leadership and me.
Once I had accepted the job, I had a multiple-month notice period with my former company. I used the time to talk to the Director of Engineering and head of recruiting to prioritize diversity in hiring, set expectations, and start building a better pipeline.
Building a Coalition and Setting Expectations
If you are working to build a more diverse team, you have to get recruiting as your partners on the effort. While it may seem natural, it is not. Recruiting (especially external recruiters) are incentivized or measured on their ability to close roles as quickly as possible. Most recruiters I have worked with are delighted to help find diverse candidates, but it is critical to let them know you understand that it means it will take the whole process longer. It is also essential than their managers understand this.
If you have hiring managers reporting to you, you need to get them on board and excited about this effort as well. One measure of a manager is their ability to hire well. Being serious about diversity will likely mean that they will take longer to fill roles for their team.
You may need to assure them that you will take this into account if they miss their KPIs. You cannot set-up or hold conflicting goals for your managers. If you cannot prioritize building diverse teams over hitting KPIs, you will likely fail at both.
One way to handle the increased time to hire is to start sourcing for a role much earlier than scheduled. Spend the time before the position was supposed to open only looking at under-represented candidates. Create an agreement with your manager that you will be able to hire them if you find a suitable candidate early. If you reach the planned opening date for the role and you haven’t located anyone, you can open the pool to your normal channels without losing time.
Your manager will also need to understand the potential impact of a diversity hiring focus on your KPIs.
With your manager’s support, create a reasonable KPI or OKR as your goal to track your efforts. The OKR should give you not only a target but also some cover if there are challenges to some of the tradeoffs you may need to make because you are hiring more slowly. Share your goal with your team. Update them on your progress regularly. Transparency is essential but also helps to enlist the rest of your team doing their part to help achieve the goal.
At Avvo, my diversity goals were part of my OKRs. I shared our progress with a monthly update to the organization. This repeated communication demonstrated the level of importance I placed on this goal and showed our improvement in real-time.
The good news is that as you make progress towards your goal, your time-to-hire will come down while maintaining or increasing your diversity. It does get easier. You will soon stop making tradeoffs as you continue to improve.
Fixing the Interview Process
When I joined Avvo, the hiring process involved multiple pre-interviews, a do-at-home coding challenge, a review meeting, an in-person interview loop, and a follow-up meeting. The process may have taken a single candidate over a month to get from the recruiter’s first e-mail to an offer. Even without taking diversity into account, it was not an efficient process for a company that wanted to grow.
The Avvo process kept inappropriate candidates out by design. It did not find appropriate candidates, however. In machine learning parlance, the false rejection rate was too high to avoid raising the false acceptance rate. If you are trying to screen out, you miss a lot of good people. Often, biases are part of the filters. Those biases reinforced the lack of diversity.
The state of the current process gave me a license to make some drastic changes. I had the support of the recruiting team and many of the managers. They all had stories of losing good candidates because of the length of the process.
There were some employees and managers who defended the status quo hiring process as required to maintain the quality of the development team. Luckily, they were in the minority. After hearing their position and discussing it with them, I decided to move forward, knowing that I had the support of the majority of the group.
Removing the Coding Challenge
Many companies believe in the at-home coding challenge as a critical way to establish a candidates bona-fides before investing too much time in them. Many sources discuss the problems with coding challenges from a diversity perspective. The arguments against the practice that resonated with me were the following:
Candidates will often take much longer to complete them than companies expect them to. The Avvo guidance was that the candidate should spend no more than two hours on the challenge. One candidate estimated that he spent 12 hours on it. He wanted to make sure that he did well.
A coding challenge is unpaid labor with no value for the results. The demand sends a message that the company places no worth on the candidates’ time. I do know that some companies pay candidates a nominal fee for completing the challenge. I encourage this, but eliminating the challenge also solves this problem.
Not all candidates have sufficient time to work on coding challenges. If a candidate is on a job search, they may get challenges from multiple companies, each with an expectation of hours of work. A candidate may already work at a stressful job or multiple jobs. They may be a caretaker to children or other relatives or have a long commute so that they can live in an affordable home. The candidate may have other issues that require their time. The challenge can be an unreasonable burden for those people.
A coding challenge requires that the candidate have the equipment and connectivity to complete the project. This requirement may not seem like an unreasonable expectation for a software developer, but not every developer may have the economic or personal safety to make this happen.
The combination of all of the above creates a bias towards people who do not have significant constraints on their time and finances. Those biases reinforce and perpetuate the lack of diversity in the industry.
As part of the hiring process, it is still critical to understand a candidate’s technical maturity. We replaced the coding challenge with multiple in-person challenges during the interview loop described later in this article. To not overwhelm our interviewers, we focused on making sure that the recruiter and hiring manager screens established a high culture and technical bar. This expectation on the screening required training the hiring managers on questions to ask for candidates at different levels. As we got comfortable with this process, the new screening process achieved an equivalent level of success identifying candidates while increasing diversity in the candidate pool and significantly decreasing the time-to-hire.
The Interview Loop
There are many articles on reducing bias in the interview loop. We combined the ideas that we felt resonated the best with our culture as well as some of the best ideas I had experienced from prior companies.
We required that anyone participating in the interview loop participate in two training sessions:
An overall interviewer training that myself and the head of recruiting for my team presented. This training included much of the standard content of a traditional interview training effort.
An implicit bias training led by someone from Ada Developers Academy (a Seattle developer training program focused on women and gender-diverse people). We had hired interns from Ada, and their mentors were required to take this course. It was so valuable that we made a deal with Ada to present that class at Avvo from time to time to help with our interviewer training.
We also required that anyone participating in interviews got qualified on the interview questions.
A group of developers for each of the different disciplines got together to create two or three technical problems to solve reflective of the type of work they did every day. Each question had a set of representative answers in a 3×3 grid. One axis represented the candidate’s experience level. The other axis was unacceptable, acceptable, and excellent solutions.
We asked the same questions to any candidate regardless of the level of the role or their experience level. Over time, we added to the set of representative answers as we heard new answers from candidates.
To train on the question, ideally, the interviewer would attempt to answer the problem themselves. Then they would observe a trained interviewer working with a candidate on the challenge twice. Then a trained interviewer would watch them working with a candidate on the question themselves.
Only once all those steps were completed could the person interview candidates. Any trained interview could interview for any team hiring for a role in that discipline. There was no level required for an interviewer.
The Interviewer Panel
We made sure that there was at least one person from an underrepresented group on any interviewing panel. This requirement was challenging at first. Some of the interviewers ended up having many interviews to do.
We felt it was important for two reasons. If we were interviewing someone from an underrepresented group, they would see someone like themselves and would be able to ask questions (if they wanted) about their experience. If we were interviewing someone, not from an underrepresented group, we would potentially catch any red flags that indicated that they might have challenges in a diverse environment.
This practice turned out to be very valuable.
As the interview training required shadowing, almost every interview would have two people from the interview panel. One person worked with the candidate, and one observed the discussion. This pairing meant that the hiring manager got two interpretations of what happened in the meeting and how the candidate responded to different questions. Multiple perspectives were beneficial in reducing the effect of any interviewers’ biases.
We offered to let candidates bring their tools if they chose so that they could be comfortable rather than presenting the candidates with an abstract problem and having them solve it on a whiteboard or a company laptop. The interviewer and the candidate solved the technical challenges collaboratively. This approach mirrored the way that many teams at Avvo worked in a pair-programming style. The goal was to recreate, as much as possible, the actual working environment of our teams.
Gaining Credibility as an Inclusive Technology Employer
Improving our interview process alone wouldn’t make a significant difference if Avvo was not visible in the market as a company serious about diversity and inclusion.
The team already had multiple people volunteering at the Ada Developers Academy but had never hired an intern. We hired our first two interns from Ada in their next cohort.
We followed that by joining the new Washington Technology Associate Apprenti program (a registered technical apprentice program focusing on underrepresented groups and veterans). We hired two apprentices from the first cohort of that program as well.
We always hired interns and apprentices in pairs because we wanted to make sure that they would have someone else going through a similar experience that they could use for mutual support in addition to their Avvo mentors.
Working with these programs let us partner with people deeply enmeshed in the community. We wanted to learn and listen. Working with them also introduced us to their volunteers, often experienced developers from other companies who felt strongly about increasing representation in technology.
Along with our partnership with training programs, we made sure we attended the local meetups for underrepresented groups. Either one of our recruiters, engineering managers, developers, or I would attend these meetups to listen and understand the challenges that these groups faced in the industry. We networked as well, but only tentatively at first. We wanted to establish credibility and not just come to a single meeting and disappear.
The sustained efforts to become a part of the community taught us a lot and raised awareness of what we were trying to do at Avvo. Candidates would apply and mention the different people that they had already met at the company. Avvo was already hosting meetups for different technologies. Those meetings eventually started to become more diverse as candidates from underrepresented groups visited to see our offices.
Evolving the Culture
The focus on increasing diversity did not have universal support. When I joined, the technology team culture was not very inclusive, and some were resistant to making any changes.
I continued to educate and discuss, but I also continued to push firmly forward.
Our agile coaches and I taught new, more inclusive facilitation techniques for meetings and discussions. One of the senior managers started a mentoring program for everyone in the organization designed to support everyone, but especially those who might feel impostor’s syndrome.
Eventually, those who were not interested in the new culture decided to find other opportunities. In the end, this was a relatively small percentage of the team. Most were interested in being part of the new culture.
In under two years, we increased the percentage of women in the technology team from 17% to 27%. We increased the percentage of Black, LatinX, and multi-racial people from 2% to 11%. Our age diversity also improved significantly. We did this while increasing the size of the team by nearly 50%. We also significantly improved our employee net promoter scores during this time.
If one thing stands out to me about what we accomplished during this time, it is a question asked by one of our developers during a town hall right after our acquisition. The town hall was hosted by the acquiring company’s executive team to answer questions from their new employees. The Avvo developer said, “we are very proud of the diversity of our technology team at Avvo. What efforts have you put in place to make sure that your team is diverse?”
There are several people mentioned in this article by title. I do want to acknowledge them here as this was always a team effort and would not have been successful without their involvement and leadership. Specifically: LaQuita Hester, a fantastic recruiter dedicated to improving diversity in the technology industry; Hunter Davis, Director of Engineering and the creator and guide for our mentoring program; Justin Weiss, Director of Engineering; and Leslie Zavisca, Engineering Manager.
Many, many others were instrumental in ways large and small. Every developer, tester, data scientist, and manager helped.
I also want to acknowledge my executive team peers who had built an excellent company and supported me as I brought my team up to a level of diversity closer to what they had created. My executive peers were Mark Britton, Eric Dahlberg, Monica Williams, Bhavani Murugiah, Sachin Bhatia, Kelly McGill, and Jason Moss.
If you are lucky in your career, you will have a few good bosses. They are people who inspire you and teach you how to be a better developer, manager, or person.
Manny Vellon was my first boss at Microsoft. Since leaving college, I had a string of good jobs, but not the best managers. I was a bit raw and somewhat guarded by my experiences.
I was the third person to join the team. There was our Director, Manny as Development Manager and me, so for the first few years, I got to work very closely with him. He had already been at Microsoft for several years in the Developer Tools team, so he had survived and thrived in a callous and competitive culture.
At first, I just respected his programming skill and knowledge. We were building the initial code together. I was amazed at the effortless way he would jump down into the assembly when he needed to understand why some bug was happening.
Once we started making more progress and started meeting with other teams, I was blown away by how he handled the often-tense situations.
Microsoft in the mid-90s was still in its heyday of competitive culture. Disagreements were handled by being louder, making threats, or sneaky political moves to undercut other teams.
In these settings, Manny was the vision of calm confidence, transparency, and good humor. If this didn’t diffuse the situation, he would calmly take apart whatever PM or DM was threatening our team or pounding their fist on the table. They would be left trying to maintain their dignity and backtrack as quickly as they could. He wasn’t cruel or mean. He was firm, he was interested in what was right and would accept no less.
As soon as one of these meetings ended, Manny would be right back to his jovial, wise self.
He was transparent, but not in an obvious way. It was just who he was. He didn’t feel the need to guard information. He knew that I could do my job better if I had the complete picture.
He pushed me to be better, to be more ambitious in my goals. He modeled those expectations himself. If we had a deadline, he was always there, with the rest of the team. Doing whatever he could to push us to hit our commitment. If I got something done but could have done it better, he would challenge me to take it to the next level. Always with humor. He made me feel like it was important to him that I grow. He considered that responsibility as my manager seriously.
A lot of who I am as a leader today comes from the lessons he taught me and what I learned from watching him work. Anyone that has worked with me since then has heard me tell a Manny story or three.
Manny Vellon died on May 27th while hiking.
I had lunch with him a couple of years ago, and I told him how much he meant to me. I am very grateful that I did that. I wish I had kept in better touch with him over the years. I know that there was a lot more I could have learned from him as he moved from Microsoft into starting his own companies and being a CTO.
My deepest condolences go out to his family and friends. There has been a massive outpouring of stories and emotions from the people he touched over the years. My own experience is hardly unique.
In the Seven Habits of Highly Effective People, by Stephen Covey, he asks you to imagine your funeral. What will people say about you? What would you hope that they would say? I hope that Manny would see how he positively touched the lives of so many and be content.
My intent with writing this is not just to tell you about a beautiful and inspiring person but also to charge you with that kind of influence on others.
If you are a manager or leader, the behavior you model, and the lessons you impart can change the direction of the people around you. Positively and negatively. What are you modeling? What are you teaching?
If there was someone like this in your life, a teacher, manager, mentor, or friend, tell them. You will be glad you did, and it will mean a lot to them.
If your company is starting to plan for getting back into your office, now is a good time for you to schedule a personal strategy day.
In the whirlwind of the day-to-day work, it is often hard to carve out time for something that does not have a specific deliverable. Working from home can be even more challenging because of the additional pressures of helping your family. In the tech lead training program I am doing at Onfido, we are currently discussing how to be more strategic leaders. A familiar dictum of strategic thinking is the criticality of taking the time to think and plan.
The Personal Strategy Day
Liminal spaces are the transitionary spaces between things. In the liminal space between moving from working at home to being back fully in the office environment, you have an opportunity to try something new—a personal strategy day.
Block out a day in your calendar as people start moving back to the office. There will be some turbulence then as everyone is transitioning. It is the perfect time to start thinking about what you want to accomplish for yourself and your team before the whirlwind starts up again.
The Personal Off-site
Companies schedule off-sites to get away from the distractions of the office. If you can find a “third place” (a place that is not your office or your home) where you can focus, that is ideal. If not, book a secluded conference room in your office, or find an empty desk far from where you usually sit. If you are still at home, let your family know you are working, and need some blocks of focused time.
Don’t read your e-mail. Don’t take any meetings. Your job on that day is to think and plan. To help me focus, I will usually print out whatever supporting materials I think I need and work on paper to avoid the lure of the notifications and other electronic distractions. Think of this as a gift of time and focus that you are giving yourself.
What to consider
Think of the time before the lockdowns began. You may have just been starting on your 2020 commitments and deliverables. What was going well then? What were your concerns? Now think through what you have learned about your team, the work, and yourself during the lockdown. What do you want to keep, and what do you want to discard as you go back to the office? What about the industry you are in, and the world at large has changed? What new pitfalls and opportunities are there? Now think ahead to the rest of the year. What new goals should you have for yourself and your team?
Take notes. Write things down. You will want to refer to them as you go through the year to revisit or remind yourself of your thinking.
Now make a plan. Not too detailed because the world is going to keep changing. Detailed enough that you feel as if you have something against which to execute. Write that down also. With milestones, if possible.
Finally, think about the process you just completed. Did it work for you? Would you do it again? If so, what would you change next time?
Retrospect and Revisit
This whole process may take anywhere from a couple of hours for you to an entire day. Don’t be too preoccupied with the length of time you spend. You are starting to build a practice of taking time for strategic thinking, so this is about taking more time than you have before.
If you found this valuable, you may want to plan your next strategy day for six or twelve months from now. Again, block it out in the calendar early and protect that time!
You may also find that it is good to schedule smaller blocks of time, weekly or monthly, to revisit your plan, track progress, and make adjustments. These blocks of time are strategic thinking too!
If you struggle to keep time for strategy, building a structured approach like a personal strategy day may help you create a practice. Using the liminal times like transitioning from lockdown back to whatever it is that comes after is an opportunity to block and protect time. The liminal times are also crucial for strategy because your past assumptions are likely no longer correct, and there may be some opportunities or challenges that weren’t visible before.
I’ve spent the last several years building a structured personal strategy process for myself. Many people have inspired my practice. Their suggestions may inspire you as well.
The planners from the Ink+Volt team helped me build some structure around my strategy, and also regular time to revisit and update it. Kate Matsudaira (who is also a very established technology leader) provides useful structured planning and strategic thinking documents in her newsletter and on their blog. https://inkandvolt.com/blogs/articles
In September of 2019, I gave a talk “Developing Your Developers: Constructing Career Paths For Your Technologists” at the Honeypot Hive Conference in Berlin, Germany. The slides from the talk are below. There were many questions from the audience that I didn’t have time to answer. The questions and answers are here. The talk was recorded. I’m hoping that the video will be posted in the near future.
How the organizations you worked in were going about diversity recruiting? What kinds of initiatives were being taken?
In companies where we have had made good progress on diversity recruiting, we used a few different initiatives:
Made it clear to the organization that it was a priority by regular communication, and updates on our progress
Engaged the recruiting team to ensure that hiring pipelines included a reasonable slate of candidates from underrepresented groups, even if that meant taking longer to fill the role.
Incentivized external recruiters with bonuses when we hired candidates from underrepresented groups
Actively participated with external groups supporting underrepresented people in the industry
Ensured that every hiring pipeline included a diverse panel
Can you give an example of what you retroactively changed in a career framework without being too intrusive?
At Spotify, we created a new framework where there had been none. At Avvo, we completely replaced a very traditional structure with one more suited to our culture. At Onfido, we are currently evolving our framework. We are making conscious deltas from the previous career-pathing model to support how our culture has evolved. We have clear goals and will follow the path described in my talk to make sure that the team understands and supports the changes.
How did you test the distribution?
We tested the distribution by doing an exercise with all the managers where we did a preliminary slotting of each person into what level we thought they would go into, including doing a calibration across teams. We then compared the resulting distribution with the one we expected to have when we designed the framework. This simulation also served as part of the training for the managers in the organization.
How do you roll out an updated growth framework in a company where you have something in place? Do you then re-map existing people within new levels?
You should roll out a new framework in a similar way to creating a new structure from scratch, but you should make the process of how you will map from the existing framework to the new one explicitly part of the plan. The mapping will be unique to the old and new pathing models, but you should do a calibration step to make sure that people get mapped fairly.
A lot of companies do give you more money only when you threaten to leave. Should people be rewarded for staying also when they don’t get promoted with a title?
If the only way for an employee to get a raise is for them to threaten to leave, then the company’s compensation system is broken, and they will struggle to retain employees.
Do you think there is a possibility to develop an application for career development? (If yes, let’s meet :))
I know of one, https://www.progressionapp.com/, our design team uses it at Onfido. In general, it is hard to develop a career development tool that would be useful for multiple companies because, by requirement, it would need to be generic. I’m advocating for each company to build a unique career development framework. So, there is a mismatch there.
Is there a single company doing all of this well?
I think that many companies do this well. I only have direct experience with the ones that I have worked at, though.
Thank you, Kevin, for your interesting speech. I don’t have a question but as introvert I wanted to thank you for bringing up this issue.
You are welcome!
What about the people who are not promoted, do they have to work on continuous improvement to reach the goal.? Or are they considered as under achiever.
That will be contingent on the company culture. Some companies have an “up or out” culture that expects people to be continually working towards the next milestone in their career progression, or they will be moved out of the company. Other companies support folks who do not choose to pursue the next level. It’s a decision that must be incorporated into the design of the framework itself.
What to do when someone wants to be a manager but is clearly not going to be a good manager – how do you present other options to them?
You should have tracks that support people who want to remain and progress as individual contributors as well as those who want to move into management. If people can move forward in their careers in either track, then there is no pressure to move into management to advance in their career.
If you have good IC and manager paths and someone wants to move into management even if they aren’t suited for it, then you can have an honest conversation about how you see their skills relative to the role that they want.
You mention candidates having all the criteria for the next role. In some cases is it not impossible to have this without experience in the next role?
It is the responsibility of the manager to help people progress in their careers by allowing them to grow by taking on additional responsibilities when they are ready.
The manager should be giving the employee chances to get the experience needed to progress.
Any tips on creating a meaningful growth path in a small company without much hierarchy, where vertical growth is very limited?
In a smaller company, people can feel meaningful growth and career progression without a formal career path. If you do create a career path, you have to build it so that people can progress even without company growth.
Rather than building a career path, instead, focus on making sure people have opportunities to learn new things and take on new responsibilities and then find ways to celebrate them when they do. You can give people meaningful milestones in their personal development without having to make a formal career path.
This mostly considers a straight or slight forking line of development. What about moving across the business. A dev becoming product etc. How do you roadmap?
You can decide if you want to support this in your career pathing design explicitly. It does not tend to happen that often, so unless you want this to be part of your culture, you can probably treat it as an exceptional case.
If you don’t want to be explicit about how to transfer between functions, you can have people apply to the role they would like to switch to and evaluate their level against the new career path just like any candidate.
How can you make an attractive offer in regards to career development for developers? Can you give some concrete examples?
Make your career path and core parts of your culture public, given documentation on them to candidates. When you interview the candidate, ask them where they want to go with their career and then discuss how your career pathing framework can support their development.
I’m not sure I understand what you mean by concrete examples here. Feel free to clarify in the comments.
Would you recommend including values into the promotion process? If yes, how?
Your company’s values should be part of your career pathing framework and, therefore, part of the promotion process. If your companies values are not part of your career pathing, then you send a message to the organization that the values are not real values.
One way to tie the values into the pathing is to think about how each value would be demonstrated at progressive levels of professional maturity and use that as part of the requirements for each level.
What do you think about creative naming of roles? A lot of companies are renaming roles to make them sound more exclusive etc does this confuse career paths?
At this point, I think software engineering titles are so polluted from title inflation that it doesn’t matter what we call them anymore.
There are two benefits I see from using a more “traditional” title naming scheme. One advantage is that the job posting is easier to find, and when a recruiter contacts a candidate about the role, it may make it easier to decide if the candidate approaches them back. The second benefit is that people may have a title that they are mentally aiming for, “Principal Developer” is more likely to be someone’s career goal than “Master of the coding arts.”
You can always choose to have internal and external titles. One is the title that people inside the company use when talking to each other, and the other is the one that people use when talking to external people. People will do that naturally when speaking to outside people anyway, rather than try to explain some unique title naming scheme.
How do you make sure we don’t bias towards the people we have now, while ignoring the people who may join in the next year with different motivations?
If you have experience working in companies at different stages, you can try to anticipate what the next set of joiners may want, but it is probably a futile exercise. Focus on the company you are today. Make sure that you revisit and revise your career pathing framework with a reasonable frequency so that if it doesn’t perfectly align with the motivations of a new joiner, then they know they will have an input into the next iteration.
My last company decided to create a career path when I asked and for me that was a red flag. It means they were not thinking about employee progression. Agree?
Without more context, it is tough for me to make a sound judgment. In my talk, I make the point that you can create an employee progression framework too early. The company needs to move from the phase of “will the company exist a year from now?” to “what will my role be if I choose to stay another three years?” If you are asking the second question at a seed-stage startup, you are probably in the wrong place.
Many companies treat tech career pathing by intertwining seniority and leadership. What about people who want to grow and not lead? There can only be one CTO?
I am going to rephrase your question in the way that I believe you intended to ask. Please correct me in the comments if I got it wrong. I think you meant “intertwining seniority and MANAGEMENT.” I expect senior individual contributors to be technical leaders just as much as I expect senior people managers to be leaders.
In my talk, I make the point that companies should not intertwine these two things. There should be an individual contributor track and a management track, and as much as reasonable, the responsibilities and pay should be comparable, right up to the C-level.
As you say, there can be only one CTO. However, there can also be a Chief Architect, who may not be part of the executive team, but who has a level of responsibility and leadership equivalent to an SVP or similarly senior role.
When you have built a well functioning engineering team and things have stabilized, how do you keep career progression open?
I have been privileged to work in some very stable, well-functioning engineering teams, including people with multiple decades of tenure.
The biggest challenge to keeping career progression open is an unhealthy retention rate (too high), especially if organizational growth is near flat. I have seen this create a problem in an organization where up-and-coming talent leaves because they have a cap on their growth since there are no senior roles available. This describes one company I left after several years of very positive career growth. I knew that I had reached a level where I would be stuck, potentially for many years, because there were no roles likely to open above me for some time, and the company kept headcount flat year over year.
The best thing you can do in that situation is to help your up-and-coming talent find new roles outside the organization, and then keep in touch with them, so that when a senior role does open up, you can hopefully get them to return.
How do you know an idea is worth stealing/it makes sense in your organisation?
Has the idea worked in an organization your size? Do you see it scaling up or down to your dimensions of an organization? Does it align with your culture? Would you need to do some culture change work to implement it? Is there a way you can try it in a limited way to test it in your organization? And of course, does it sound like a good idea to you? An idea that you can champion?
Do you think it’s good to have career path development for start up with 60 people?
How old is the company? What is the average tenure of an employee? If both of those are low numbers, then probably not. It is maybe too early for you to focus on career path development. Unless managers are getting questions about career development commonly from employees, that should be your first warning sign that it is time to think about career pathing.
Doesn’t waiting for the timing to build a career pathing seem reactive? Ideally an organization should be proactive i.e think ahead of your employees.
Yes, it is reactive. I think about it this way: One of the most attractive elements of a startup is the flexibility and freedom that come from being a small, growing company. Trying to impose structure and process too early can feel “corporate” too soon to those employees. It is also hard to establish what responsibilities levels should have as the team is still forming. You are likely to build a framework that matches current employees (who don’t want structure) instead of the one that will work for the employees who join later and are expecting more structure.
If your company is already established and is of a certain size and attracting employees who want more structure, then you have probably waited too long.
Do you think a good career development framework kan help with developer retention?
A useful career development framework primarily exists to support retention. Beyond establishing career paths for employees, it also helps benchmark salaries for new hires. A career development framework ensures that pay is fair relative to the market (which is also essential for retention).
If you read the slides and have questions not answered above, please ask them in the comments.