Engineering teams that have created a safe environment to fail

‘Stay hungry, keep improving’ is a motto we live by.

While mistakes are inevitable, we have a thoughtful, humble culture that always puts the focus on what we can learn from failures to make us better. Innovation requires risk taking, and if something doesn’t go as planned, we view it as a system failure, not the fault of any one person. Even if you break production or make a mistake that costs $30K in extra AWS costs (true story, it’s happened), we won’t point fingers. Instead, we’ll look at why something happened and how we can better design cost controls, for instance, to learn from that lesson.

We practice blameless postmortems and find that building stronger safeguards against future failures helps bring the team closer. Ultimately, it’s this safe environment to fail that ensures folks aren’t afraid to openly speak up and share a mistake in real time. It’s safe to say we’ll always rally behind each other to jump in, help solve any issue, and learn from it together.

You can expect to ship code on your first day.

While we strive to write high-quality code, we recognize mistakes and edge cases that cause errors are inevitable. We have a bias for speed over perfection; the sooner we release a feature, the faster we can gain more data and iterate to provide our users with the best experience possible. We’re not afraid to break production (it happens!). In fact, it’s only by doing so that we can learn how to build better safeguards against future failures. Ultimately, this leads to a cluster immune system, which helps us deploy even faster and more securely (see Continuous Delivery below).

If something goes wrong, we never place the blame on any one person. Rather, we’re happy to jump in, help fix the problem at hand, and learn from it during blameless retros. For instance, we once misnamed an environment variable and accidentally ran a stub client in production. To avoid this class of mistake from happening in the future, we added a runtime check to prevent a server with this issue from ever serving real traffic. While this is a small example, by repeating the blameless retro process over 90+ times, we've built an environment where it's actually safe to make mistakes because there's a robust safety net.

“The cost of failure is education.” – Devin Carraway, Google SRE Handbook

We’ve learned the most from our failures. We’ve had companies, products, and projects all fail and they’ve been some of our most impactful learning experiences. Making mistakes has allowed us to become more thoughtful and intentional about our decisions, including carefully thinking about roles before we post them. We’ve found when a project fails, it’s not any one person’s fault, but rather mismanaged expectations or a misalignment/misappropriation of resources. Only in an environment where you fail enough to learn – but not so much that you feel ineffective – will you be successful.

We want Indent to be a warm place, where you can feel safe, do your best work, and feel good about what you do. This means being able to take risks and learning how you can iterate and improve moving forward.

Trade-offs: We lean toward failing early and often to learn what we need to make the next iteration successful, rather than risk going too deep on a shaky foundation that can lead to a significant opportunity cost. There are certain problems that require a correct answer on the first try and we try to distinguish if it’s a “one-way door” or a decision we can approach iteratively.

We believe in getting in the boxing ring and learning by doing.

In order to innovate and move quickly, Sibi is willing to take risks. Knowing it takes a few misses to land a winning swing, we are more than happy to hop into the metaphorical boxing ring. To us, the best way to learn is by recognizing that no mistake is the fault of any one person. Rather, we view each misstep as a learning opportunity to make ourselves better. For example, when a team member recently broke production, we gathered together, shared a few cupcakes and made the experience fun while introducing an opportunity to learn. In general, there are very few safeguards because we celebrate learning from our mistakes. While there’s a pull request in order to push to master, we view it as more of a communication tool. It’s primarily a way to radiate community and intent, not to judge if you’re writing flawless code.

We take pride in having a safe, trusting environment.

The Zapier team is highly collaborative. Engineers work closely with product and design to come up with a product plan, then work together as an eng team to spec it out and build prototypes. We encourage people to speak up and share their ideas, and recognize that we’ll all make mistakes along the way. Any time something goes wrong, we all jump in to help solve the problem and never place the blame on any one individual. In fact, one of our senior engineers, Grant, openly shared his experience: “A change I made brought down Zapier for about 30 minutes. 🤦‍♂️ Someone opened up an incident and I jumped right on it to get a temporary fix in place. At no point did anyone blame me for what happened. We all stayed focused on arriving at a solution. Everyone brought empathy and patience to the incident, which only helped us get to a resolution more quickly and reflect [blamelessly] during the post-mortem. It made me really happy to work at Zapier.”

Failure and mistakes are healthy as long as we're making new ones.

If we're not failing at all, we’re not taking enough risks or pushing ourselves beyond our comfort zones. Being eventually correct is much better than being consistently incorrect, and we focus on getting to the right answer through reflection and discussion, instead of trying to be right the first time. We know we’ll make mistakes along the way, but we believe each failure should be examined dispassionately, objectively, and without blame. Instead, we treat each misstep as an opportunity to improve. An environment of mutual trust and respect empowers us to debate ideas effectively and create the best outcomes for our customers and our team.

Listen to our VP of Engineering, Uma Chingunde, share her strategies on how to cultivate a blameless culture here.

We deploy daily and aren’t afraid to make mistakes.

In order to innovate quickly, we believe in taking measured risks. Not everything will go flawlessly, and that’s okay. Since deployments happen on a daily basis, the longest that anybody is really dealing with the aftermath of a potentially dangerous line of code is 24 hours. Team members are always willing to jump in and help if something doesn’t go as planned and we never place blame on any one person.

When failures do happen, we’re able to speak honestly about what went wrong, and what we can do better next time in our bi-weekly engineering retrospectives. That said, we also have certain guardrails in place, including a solid review and QA process to ensure we’re shipping high-quality code.

You’ll never be criticized for making a mistake at LDT.

It’s completely okay to get it wrong. In fact, we have a strong bias toward taking risks and making mistakes versus playing it safe. As a result, one of our core company values is, “We are an elite force. It’s always we, never me.” We are all open about making mistakes across all levels, including our CEO. Our most senior developers admit to hacks and come forward about being wrong when they are. We’re honest in our PRs and the feedback we give, and also recognize risks and knowingly accept them. We have dedicated time to build test harnesses to catch bugs and alert us (i.e. alarms, emails) because we encourage speed over perfection. It makes us more comfortable to take risks, too, because we are reassured that mission critical regressions will be caught early.

To us, feeling safe to fail goes hand in hand with open feedback. We frequently reference the radical candor graph when we are giving or asking for feedback, and make sure everyone has context around what they’re building. We document our in-person meetings, whether they be dev team retrospectives or standups, and share them in our shared Slack office. We also have a Facebook Portal in both of our offices, which is on at all times. It provides an additional wormhole from one office to another. And despite our time differences, we schedule weekly All Hands meetings and weekly #devs meetings when our timezones overlap so that everyone can attend. Ideas are shared, concerns are voiced, and we happily address successes and failures from the week.

Failing is a great way to learn, and high-profile bugs often teach more than they cost.

We ship to production often, sometimes multiple times a day. Quickly iterating helps us learn and if nothing ever breaks, we’re under-indexing on speed. However, we always view missteps as learning opportunities and never place the blame on any one person. For instance, when we experienced a production outage or a recent time zone bug, it only motivated analysis and improvement of our systems.

Different features require different levels of certainty around success – both technically and from a product perspective. Ascertaining these levels and executing appropriately is critical to the success of the business, and if we avoid all failure on a micro level, we believe we’ll fail on a macro level. At the end of the day, while perfect code doesn’t exist, we each take complete responsibility in producing excellence. We review code thoroughly, write tests and design architecture collaboratively, and practice blameless postmortems so we can learn from failures and celebrate wins together.

Innovation requires taking risks.

Across the board, we work hard to ensure we create a safe space to try new things. Even if we’re not 100% sure something will succeed, we take calculated risks and learn from our mistakes along the way. While we don’t have structured postmortems at this point, we do have regular roundtable discussions to reflect on takeaways. For example, we recently launched our first version of a new platform to users and then gathered as a frontend team to take stock of what went well and what went less well. Then, as a broader software team, we compiled a list of things we intentionally put off from the previous quarter in order to get the release out and prioritize our work.

We’re all in the same boat, working toward a common goal, so there’s never any blame placed on an individual. Instead, we strive to create a culture where it’s safe to call out if you didn’t do something well, think there could be an improvement, or simply need to ask for help. Ultimately, both our successes and failures are shared.

Errors give us valuable insight to improve user experience.

Effective error handling is important both for the developer and the user. We try to make sure both internal and external error messages make sense from a user’s point of view (human-friendly output please!). And in the case of internal errors, they should always be actionable, communicating what was expected versus what was received. To track runtime errors we use Rollbar, which fires Slack notifications once an error exceeds a pre-set threshold. You can’t fix what you can’t see, so it’s crucial to have visibility to the exceptions that are occurring in order to maintain a great user experience.

We’ve also increasingly leaned into using tests to make sure our front ends pass muster; we use React Testing Library for testing individual React components and Cypress for testing application flows (and legacy non-React code). We also leverage compilers like TypeScript to reduce both runtime errors and the number of unit tests we need to write.

Two of our company values are “always learning” and “responsible.” We maintain a strong desire to know more in order to better ourselves and Course Hero, never letting a fear of failure get in the way of learning and progress. We each take smart, strategic risks and are accountable for our successes and failures. We take pride in making progress and enabling those around us to do the same. While mistakes are inevitable, we view them as learning opportunities. When something goes wrong it may sting in the moment, but it ultimately yields valuable insight to improve our customer’s experience and save us from future stress.

Newfront’s vision is to create the greatest brokerage in the world for our brokers and customers, and the technology required to get there isn’t built neatly.

Our industry is incredibly complicated and opaque. To break through, we need to innovate and make mistakes.

We want developers who bring an inherent curiosity to everything they do and a desire to try new things. We value enthusiasm and smart engineering more than expertise in a particular stack. Some of our most talented engineers have mastered programming languages while on the job.

Following major projects, we conduct thorough post-mortems where everyone is encouraged to contribute. You’ll never hear blame in these meetings. We take shared responsibility for mistakes and a collaborative approach to fixing them. If you try something and fail, we’ll share learnings together and choose a different strategy next time.

We approach product and development from a learning angle.

It’s a horizontal organization at Monograph. We’re very open about how no technology or process is sacred and we’re all figuring it out together as we go. Operating under the guiding mantra of “blame systems, not people,” we reinforce the blameless culture in our feedback and communication. For instance, we’ve actually had to rebuild Monograph two times now because of early architecture choices, and each time was a decision we made together. We hope you’ll experiment and work openly with us on both Monograph and whatever side project(s) you have.