Engineering teams that have created a safe environment to fail

You can’t innovate if you’re afraid to fail.

Our success rests on our ability to keep innovation alive at every level of the company. We know that the fastest way to suppress innovation is to make it unsafe to fail, which is why we’ve worked hard to ensure that everyone at Mode is able to take risks. We’ll try something out even if we’re not 100% sure it will succeed because we know that as long as we did the best we could to de-risk it, no one will shame us if it doesn’t work out. In everything we do, we have a “Shared Responsibility, Shared Risk” mentality.

Making it safe to fail requires a blameless post-mortem process, and for that, we have After Action Reviews (AARs). During an AAR, we start with the assumption that everyone did the best they could with the information they had at the time. Participants focus on how to understand and improve the systems and processes that led to the failure, rather than who did what wrong. We use these meetings as opportunities to reinforce that Mode is a safe place to fail, build stronger safeguards against future failures, and bring the team closer together.

“The cost of failure is education.” – Devin Carraway, Google SRE Handbook

We’ve learned the most from our failures. We’ve had companies, products, and projects all fail and they’ve been some of our most impactful learning experiences. Making mistakes has allowed us to become more thoughtful and intentional about our decisions, including carefully thinking about roles before we post them. We’ve found when a project fails, it’s not any one person’s fault, but rather mismanaged expectations or a misalignment/misappropriation of resources. Only in an environment where you fail enough to learn – but not so much that you feel ineffective – will you be successful.

We want Indent to be a warm place, where you can feel safe, do your best work, and feel good about what you do. This means being able to take risks and learning how you can iterate and improve moving forward.

Trade-offs: We lean toward failing early and often to learn what we need to make the next iteration successful, rather than risk going too deep on a shaky foundation that can lead to a significant opportunity cost. There are certain problems that require a correct answer on the first try and we try to distinguish if it’s a “one-way door” or a decision we can approach iteratively.

We believe in getting in the boxing ring and learning by doing.

In order to innovate and move quickly, Sibi is willing to take risks. Knowing it takes a few misses to land a winning swing, we are more than happy to hop into the metaphorical boxing ring. To us, the best way to learn is by recognizing that no mistake is the fault of any one person. Rather, we view each misstep as a learning opportunity to make ourselves better. For example, when a team member recently broke production, we gathered together, shared a few cupcakes and made the experience fun while introducing an opportunity to learn. In general, there are very few safeguards because we celebrate learning from our mistakes. While there’s a pull request in order to push to master, we view it as more of a communication tool. It’s primarily a way to radiate community and intent, not to judge if you’re writing flawless code.

We take pride in having a safe, trusting environment.

The Zapier team is highly collaborative. Engineers work closely with product and design to come up with a product plan, then work together as an eng team to spec it out and build prototypes. We encourage people to speak up and share their ideas, and recognize that we’ll all make mistakes along the way. Any time something goes wrong, we all jump in to help solve the problem and never place the blame on any one individual. In fact, one of our senior engineers, Grant, openly shared his experience: “A change I made brought down Zapier for about 30 minutes. 🤦‍♂️ Someone opened up an incident and I jumped right on it to get a temporary fix in place. At no point did anyone blame me for what happened. We all stayed focused on arriving at a solution. Everyone brought empathy and patience to the incident, which only helped us get to a resolution more quickly and reflect [blamelessly] during the post-mortem. It made me really happy to work at Zapier.”

Mistakes are inevitable and we view them as learning opportunities.

Startups are inherently risky ventures and if we don’t take enough risks we won’t succeed. In order to support a culture of risk taking, we strive to create a safe environment to fail. For example, when we ship bugs, or accidentally cause outages we use those as opportunities to improve our process as a team rather than focus on the mistakes of any one person.

Postmortems are standard and we always use blameless language. Creating a place where people feel safe to take risks, make mistakes, and learn along the way is what allows us to iterate quickly and build the best product. In fact, one of our engineers actually set a personal goal “to make at least N mistakes this year,” which only underscores how comfortable team members are to take risks, reflect, and learn. At the end of the day, we take our work seriously, but we’re lighthearted folks.

We deploy daily and aren’t afraid to make mistakes.

In order to innovate quickly, we believe in taking measured risks. Not everything will go flawlessly, and that’s okay. Since deployments happen on a daily basis, the longest that anybody is really dealing with the aftermath of a potentially dangerous line of code is 24 hours. Team members are always willing to jump in and help if something doesn’t go as planned and we never place blame on any one person.

When failures do happen, we’re able to speak honestly about what went wrong, and what we can do better next time in our bi-weekly engineering retrospectives. That said, we also have certain guardrails in place, including a solid review and QA process to ensure we’re shipping high-quality code.

Failure and mistakes are healthy as long as we're making new ones.

If we're not failing at all, we’re not taking enough risks or pushing ourselves beyond our comfort zones. Being eventually correct is much better than being consistently incorrect, and we focus on getting to the right answer through reflection and discussion, instead of trying to be right the first time. We know we’ll make mistakes along the way, but we believe each failure should be examined dispassionately, objectively, and without blame. Instead, we treat each misstep as an opportunity to improve. An environment of mutual trust and respect empowers us to debate ideas effectively and create the best outcomes for our customers and our team.

Listen to our VP of Engineering, Uma Chingunde, share her strategies on how to cultivate a blameless culture here.

Failing is a great way to learn, and high-profile bugs often teach more than they cost.

We ship to production often, sometimes multiple times a day. Quickly iterating helps us learn and if nothing ever breaks, we’re under-indexing on speed. However, we always view missteps as learning opportunities and never place the blame on any one person. For instance, when we experienced a production outage or a recent time zone bug, it only motivated analysis and improvement of our systems.

Different features require different levels of certainty around success – both technically and from a product perspective. Ascertaining these levels and executing appropriately is critical to the success of the business, and if we avoid all failure on a micro level, we believe we’ll fail on a macro level. At the end of the day, while perfect code doesn’t exist, we each take complete responsibility in producing excellence. We review code thoroughly, write tests and design architecture collaboratively, and practice blameless postmortems so we can learn from failures and celebrate wins together.

You’ll never be criticized for making a mistake at LDT.

It’s completely okay to get it wrong. In fact, we have a strong bias toward taking risks and making mistakes versus playing it safe. As a result, one of our core company values is, “We are an elite force. It’s always we, never me.” We are all open about making mistakes across all levels, including our CEO. Our most senior developers admit to hacks and come forward about being wrong when they are. We’re honest in our PRs and the feedback we give, and also recognize risks and knowingly accept them. We have dedicated time to build test harnesses to catch bugs and alert us (i.e. alarms, emails) because we encourage speed over perfection. It makes us more comfortable to take risks, too, because we are reassured that mission critical regressions will be caught early.

To us, feeling safe to fail goes hand in hand with open feedback. We frequently reference the radical candor graph when we are giving or asking for feedback, and make sure everyone has context around what they’re building. We document our in-person meetings, whether they be dev team retrospectives or standups, and share them in our shared Slack office. We also have a Facebook Portal in both of our offices, which is on at all times. It provides an additional wormhole from one office to another. And despite our time differences, we schedule weekly All Hands meetings and weekly #devs meetings when our timezones overlap so that everyone can attend. Ideas are shared, concerns are voiced, and we happily address successes and failures from the week.

When something goes wrong, we figure out how to fix it next time – not assign blame.

Creating a safe space where we can take risks, innovate quickly, and learn from our mistakes is key to helping us build the best product possible. In order to do that successfully, it means fostering an environment where we’re working together as a team first and foremost. Mistakes are inevitable, but we always view them as learning opportunities and avoid prescribing blame to any one person. Instead, we write post-mortems, figure out solutions, and document them for next time.

The goal is to foster a space where our engineers feel safe speaking up, asking for help, or admitting mistakes – this is how we grow individually and as a team. Having mutual trust and respect for one another allows us to debate ideas effectively and figure out how we can improve things moving forward.

There is perhaps over-enthusiasm on our team when reviewing PRs, improving code, and doing retros.

Our Reliability Team provides one of the infrastructures at New Relic that is incredibly on point: Do not Repeat Incidents (DRI). The retros for DRIs are completely blame-free and when managers write up summaries, we won’t even include names. There’s a lot of acknowledgment that anyone can make a mistake and we never focus on who did it. Instead, we try to identify what’s missing in our process or infrastructure and what we need to solve in order to alleviate those gaps.

Specifically on our Data OS team, there is a tremendous amount of support reviewing PRs and improving our code. There’s actually quite a bit of silliness and joy! Feedback is never given from a punitive place and always comes from a place of wanting to support you and your career.

In order to innovate, we’re not afraid to take calculated risks.

Teams are encouraged to take calculated risks, knowing we’ll make some mistakes along the way. We view this as the best way to inspire trust and innovation within each pod. For example, as part of a code change to support a new API version for one of our services, an operational oversight occurred by one of our newer engineers, causing some customer impacts. Having recently joined the team, she was worried about what had happened. However, her leadership team assured her it was totally okay and helped her reframe things as a learning opportunity. The focus will always be on the solution, rather than how we arrived at a certain problem.

We run a blameless culture and our post-mortem process reflects that. Our team operates under the assumption that everyone did the best they could with the knowledge they had at the time. By seeking to understand first and then learn from our failures we can work toward our common goal of improving our systems and processes.

Errors give us valuable insight to improve user experience.

Effective error handling is important both for the developer and the user. We try to make sure both internal and external error messages make sense from a user’s point of view (human-friendly output please!). And in the case of internal errors, they should always be actionable, communicating what was expected versus what was received. To track runtime errors we use Rollbar, which fires Slack notifications once an error exceeds a pre-set threshold. You can’t fix what you can’t see, so it’s crucial to have visibility to the exceptions that are occurring in order to maintain a great user experience.

Failure is expected at Indeed.

Yes, some of the stuff we do will fail, but we believe if we're not failing sometimes, we're not being bold or risky enough. As our CEO, Chris Hyams says, “The greatest soil for personal growth is made of the things that you mess up on badly, but are willing to look at honestly.”

This is a key principle of Indeed Incubator, an internal program that gives Indeedians an opportunity to pitch their ideas and get funded like a venture-capital investment, in various amounts from Angel to Series C. We provide guardrails, not speed bumps to solve real-world problems facing job seekers. This gives us the freedom to test ideas. Even if they fail, we take what we learned and apply it to another project.

Key products that have come from or are currently testing by Indeed Incubator include:

• Text2Apply
• Virtual Interview Platform
• Virtual Evaluations
• Job seeker communities
• Employer video messaging

For more successes check out:

• Hiring Events
• Certifications
• Indeed for Chrome

Indeed Incubator partners with Indeed University, which launched in 2015 with the mission to empower new hires out of college to prototype ideas, some of which have become Indeed products.

Innovation requires taking risks.

Across the board, we work hard to ensure we create a safe space to try new things. Even if we’re not 100% sure something will succeed, we take calculated risks and learn from our mistakes along the way. While we don’t have structured postmortems at this point, we do have regular roundtable discussions to reflect on takeaways. For example, we recently launched our first version of a new platform to users and then gathered as a frontend team to take stock of what went well and what went less well. Then, as a broader software team, we compiled a list of things we intentionally put off from the previous quarter in order to get the release out and prioritize our work.

We’re all in the same boat, working toward a common goal, so there’s never any blame placed on an individual. Instead, we strive to create a culture where it’s safe to call out if you didn’t do something well, think there could be an improvement, or simply need to ask for help. Ultimately, both our successes and failures are shared.

Failures lead to learning and we focus on the cause, not the individual.

We believe it’s crucial to learn from incidents and take actions to prevent them from recurring, which is why we’ve formalized this process with blameless post mortems. Our primary goal in conducting a post mortem is to ensure the incident is documented, that all contributing root causes are well understood, and that effective preventative actions are put in place to reduce the likelihood and impact of a recurrence.

For a post mortem to truly be blameless, we make sure it focuses on identifying an incident’s contributing causes without placing blame on an individual or team for bad or inappropriate behavior. At Wealthfront, a blamelessly-written post mortem assumes everyone involved in an incident had good intentions and did the right thing with the information they had. This means engineers whose actions contributed to an incident can give a detailed account of what happened without fear of punishment or retribution.

Newfront’s vision is to create the greatest brokerage in the world for our brokers and customers, and the technology required to get there isn’t built neatly.

Our industry is incredibly complicated and opaque. To break through, we need to innovate and make mistakes.

We want developers who bring an inherent curiosity to everything they do and a desire to try new things. We value enthusiasm and smart engineering more than expertise in a particular stack. Some of our most talented engineers have mastered programming languages while on the job.

Following major projects, we conduct thorough post-mortems where everyone is encouraged to contribute. You’ll never hear blame in these meetings. We take shared responsibility for mistakes and a collaborative approach to fixing them. If you try something and fail, we’ll share learnings together and choose a different strategy next time.

We approach product and development from a learning angle.

It’s a horizontal organization at Monograph. We’re very open about how no technology or process is sacred and we’re all figuring it out together as we go. Operating under the guiding mantra of “blame systems, not people,” we reinforce the blameless culture in our feedback and communication. For instance, we’ve actually had to rebuild Monograph two times now because of early architecture choices, and each time was a decision we made together. We hope you’ll experiment and work openly with us on both Monograph and whatever side project(s) you have.