To create a culture of experimentation, you first need to create a culture of safety.

We encourage everyone at Custora to try and test new things, and we want everyone to also feel comfortable with the failures that often come with experimenting. In fact, our actual product emphasizes the importance of experimentation. Our app allows users to test different types of emails based on their customer demographics and compare how those emails perform. A/B testing is a core feature and a value prop of Custora, so much so that we provide it to our customers.

We’re open about experimentation even on an organizational level. Inspired by Spotify’s squad system, we tried implementing squads at Custora, too. After some time though, we realized it was too process-heavy and we were too small of a team for squads to be effective. Even though it was a bit of a failed experiment, it was a useful exercise for everyone, and we all happily reverted to our old way of doing things, operating in teams.

Even though engineers always work on a team, individuals are never restricted to working on any one particular thing. While our workloads are shaped by business goals and product plans, engineers have agency to choose what they work on each product development cycle. Having a say in what you work on keeps everyone excited and challenged, and it also cultivates more creativity in how we solve problems. Just like our customers, we encourage one another to experiment and run tests, learning from both our failures and successes.

If we’re not extending ourselves to where we might fail, we’re not challenging enough.

The default for most startups is failure, so in order to fight against those odds, we need to push ourselves in ways that will feel uncomfortable at times. We learn a lot more from failing than we do immediate success (where you often pat yourself on the back and not truly understand why you succeeded). With this in mind, we run a lot of experiments and A/B tests with the expectation that more than 50% will fail. As Thomas Edison said, “To have a great idea, have a lot of them.”

When we asked Yi Sun (one of our engineers) how he’d describe our team, he said, “Success on the engineering team means not being afraid of failure. The team leadership truly supports an environment to fail. That might sound crazy, but if you are aiming to never fail, you are not getting out of your comfort zone. When you have confidence to try new things no matter if it fails, you are on a path to innovation.”

It's nice to have the fail-fast-then-fix philosophy, but only if it's safe-to-fail as well.

We talk about our fail-fast-then-fix philosophy a lot at Peerspace. Not all releases go according to plan, so we want everyone to have the latitude to triage, fix, and deploy appropriate fixes as necessary. Within your first week at Peerspace, you’ll notice that everyone is comfortable with failures, and more importantly, we’re comfortable taking ownership for any mistakes or shortcomings.

Our engineering system was built to tolerate failures so that we can free up our mental energy and focus on taking risks, trying new things, and delivering value to the company. We’re proud that everyone feels comfortable experimenting because we’ve already put guardrails and systems in place to protect us from irreversible failures.

We also feel comfortable taking risks as a company. We sometimes face problems that no other companies have tackled before and in these cases it may take many iterations to get it right. More often than not, these are the same scenarios where we learn most about our users and customers. Failure leads to learning, and we’re better for it.

Be Brave is one of our company-wide values.

In order to push boundaries and truly experiment, we place a heavy focus on testing. These enable us to take chances without worrying about irrevocable failures. For example, we have multiple test labs featuring equipment in which we’ve invested millions of dollars for our firmware/product side. We run these testbeds almost 24 hours a day for basic feature testing, end-to-end RF testing, competitive performance tests, scale testing of our VPN features, and more. Software engineers use these testbeds as a key part of feature development and it enables them to innovate with greater freedom.

However, while we’re always looking for opportunities to be better, challenge the norm, and improve our products, we believe it’s important to collectively learn from failures and understand what went wrong, so the same mistake isn’t made twice. To that end, we foster an environment where open discussion and collaboration are the norm, and blame and finger pointing aren’t part of the conversation. If there is a customer-facing outage or a serious bug, we have a blameless postmortem. Instead of sharing it publicly, we’ll have a follow-up meeting so that helpful lessons are learned across the organization to guide us in the future. This cultural consistency can be attributed to a reporting structure where all teams ladder up to a tight-knit engineering leadership team comprised of our CTO Bret Hull and two senior directors, Dan Aguayo, who looks after the Firmware Engineering teams, and Jack Horner, who looks after the Cloud Engineering teams.

We currently have ~200 engineers in our software engineering department. We’re organized into 17 teams which range between 1 and 30 people in size. We spend our energy on creative problem solving, making sure we improve the next time, and having support along the way. Meraki is the type of place where people are always up for lending a helping hand, whether you’re new to the company, or a seasoned Merakian.

When we asked one of our software engineers, Phil Dayboch, about Meraki’s culture of safety, he said “I worked on the Meraki phone from launch day until the product’s cancellation. At every turn of the journey, senior management made it clear that no one was to blame for challenges and that they were always willing to provide additional resources needed to get the job done. After the product’s cancellation, I was provided with the opportunity to join any team I was passionate about. It was a positive experience when all of the other teams actively pursued me and my teammates to join them.”

Summer interns soaking up the sun on our roof deck overlooking the San Francisco Bay Bridge.

We believe it’s important to be able to try things - whether it’s choosing a new tech stack or updating our workflows.

We think this is necessary to learning and improving as a team. Inevitably, some of the things we try aren't going to be successful. When something goes wrong or we fall short of our goals, we don't look to figure out who is to blame -- we figure out where the gaps are in our processes (or our expectations!). We go into retrospectives with the understanding that people did the best they could with the conditions they had. A failed experiment isn't viewed as a personal failure due to our culture of support and accomplishment.

Failure = Information

We jump on the opportunity to learn from our mistakes. In a world where not all problems are solved, there are bound to be answers that don’t quite hit the mark. Our customers depend on our software to be reliable, so we have many processes in place to prevent regressions. We understand that everyone on the team is human (except you SlackBot), so we’re bound to make mistakes. When we do make errors, we compile information about the instance, document it, and focus on creating preventative measures to make us more reliable.

Here’s an example to illustrate how we handle failures. Our website’s signup form was recently hit with a malicious script to batch signup a bunch of accounts. The form didn’t include our Google Recaptcha snippet and thus, opened us up to an attack. The original programmer of the signup form was not targeted or blamed for this. Rather, the team came together to add the Google Recaptcha code. As a team, we put together a process to prevent sign ups like this from occurring in the future. Our team has each other’s back and “figuring out who pushed the bug” wasn’t on our agendas. Ultimately, we were able to create a stronger sign up experience in the process.

We hold Blameless Post-mortems for when there are production-facing incidents.

For example, a major increase in writes to our main database caused degraded performance for our entire suite of products. This was isolated to be an issue with fetching data from one of our APIs stuck in a loop. The post-mortem focused on the technical problem, and the root causes, including recommendations for process improvements. One of the attributes our team holds dear is that of Team > Self, where any incident is the team’s fault, not the individual who may have pushed or deployed some code.

We’re constantly experimenting in new ideas and inventing new technologies on an ongoing basis to meet the needs of large enterprises.

Sometimes, these experiments do not yield the results we desire—or simply don’t work— but we encourage experimentation and failing quickly because learning from these is what enables future success.

In our experience, failure is rarely a direct function of the individual contributor but rather a process inefficiency that needs to be remediated. To diagnose the issue, we practice blameless retrospectives to understand what happened, how to improve, and communicate our findings to the team. We look to continually improve our output.

For example, we will push out early software under feature toggles to test and observe customer usage with production traffic. If the experiment and associated a/b tests prove to be successful, we’ll expand the rollout accordingly. On the other hand, if the experiment fails we put the idea on the shelf and move on.

Work to Learn is a key behavior we pride ourselves on here at BetterUp, and we understand that failure is often a necessary step along the path to learning.

When we fail, we focus on what we can take away from the experience to apply moving forward. One way we implement this behavior is through a practice we refer to as “learning memos.” Employees are encouraged to take time to reflect on what they’ve learned through various trials and tribulations, and write their thoughts out in a way that can be shared with the rest of the team.

When we introduce a critical bug into a production environment, we follow-up the incident with a post-mortem that focuses both on what went wrong and how to prevent the incident from being repeated in the future. We are strong believers in approaching this process with empathy and acting as humans, not robots during the post-mortem analysis.

We are also acutely aware that psychological safety is key to both innovation and creating a positive work environment where people feel empowered to speak up with ideas, issues, or concerns. Our managers across the organization welcome and invite feedback of all kinds.

Failing is a path to success.

We take every failure as an opportunity to learn and be better. We know we may not fix the problem right away, but it’s about testing and trialing, and learning from it to make sure the next time we tackle it, it’ll be better than the last. We released the first beta of our API in a few weeks when it wasn’t really rock solid yet. Our first big user actually broke it, as high-availability was not even there. However, it allowed us to get feedback and to focus on the highest impact items. Since then, we have done tons of iterations to arrive at the product we have today and we continue to iterate on it every week. Our biggest failures are often times are best learning opportunities, which is why we spend time to thoroughly document post mortems (DNS DDos incident, 8-min indexing downtime). We share these with the public too, in hopes that everyone can benefit from the lessons we’ve learned the hard way.

When people “fail” we pick them up and dust them off.

Carbon Black is a constant work in progress and the best way we navigate this is by living our values. One of the biggest ways we live our values is by doing a retro when we fail to figure out how we can do better in the future – and not who caused the issue. By shifting the language to “doing better going forward” rather than “which throat do I choke?” we start from a place of safety. Then we take the recommended steps to improve and add those steps to our future work.

A great example of this is when we had a migration issue on CB Protection a few years ago. A person wrote a script that had a mistake which ended up modifying data. Once we recognized that the issue came from a poor migration step, we went right to remediation and lessons learned for the team. This meant three weeks of planning for how to address the root cause and how to get our customers back to health. We spent zero time calling out the person who wrote the script. We, instead, focused on how to tactically fix the software issue and help our customers.

Only after the two above issues were met did we have a personal retro with the person. We discussed what they learned from the process and what, if anything, they’d change. (In this case, the person said they’d add several tests to smoke tests. And that’s what we did!)

We also do post-mortems, and we always do a “lessons learned/retro” for major failures. The example given above is one such example. We don’t always share the results of the post-mortem externally; however, we do make sure our customers know what the problem was and how we’ll fix it. For example, in the scenario above, we talked to every customer impacted and worked with them to remediate the data change.

We strive to push ourselves to try new things and build a culture of learning from failures instead of assigning blame.

Things are going to go wrong or not work –– it is how we react to them that define how we move forward. We’ve built several services and products that worked for a while but needed to be torn down and rebuilt later. Our needs change, the industry changes, and over time, our customers expect more, too. As a result, we need to be able to adapt and respond quickly. Our data is used to power the growing on-demand economy, and beyond, and we need to be able to deliver what our customers need.

We have a strong practice of internal and external post-mortems when an issue occurs and take ownership when things go wrong. For example, when switching to new infrastructure, a command was accidentally run in production that was designed for development… this resulted in a database being dropped! 😱 Our whole team rallied and we had it restored within minutes along with pull requests out to prevent it from happening again. All the while, we experienced zero downtime! 😎

We honor curiosity and experimentation as a cornerstone of intelligence.

We actively encourage the offering and exploration of new ideas, even if their value is uncertain to the broader team. If a next step forward on an idea can’t be agreed upon, we encourage parties to agree on an experiment instead. An experiment is something that is safe to try and has a clear hypothesis of what outcomes are expected. From its execution, we’ll all learn something of value, and should it fail it won’t cause the organization unavoidable or serious harm.

Failing is a necessary part of growing.

At Handshake, we recognize a growth mindset is only possible when individuals are given a safe environment to fail. Many Handshake managers consider failing a necessary part of growing as an engineer here, which is why we embrace failures as learning opportunities. In fact, many of our employees favorite the #learning-from-losses Slack channel, where employees show humility with their recent failures and how they have learned from them.

As discussed previously (see EQ > IQ), we practice blameless postmortems mistakes. Rather than pointing fingers, we focus on how to fix the issue at hand and prevent it from occurring in the future. Postmortems are then shared across the entire engineering organization so everyone can learn and improve collectively. For instance, one of our engineers wrote a feature to run a promotion within the Handshake platform, but unfortunately, the data from the promotion was getting corrupted due to a misunderstanding of how Rails destroys records. “There were no finger-pointing or blaming of why this happened,” the engineer said, “only a calm focus on how can we prevent the next incident. In fact, one of the team members even researched more of why this problem happened in the first place and gave a tech talk to fellow engineers on how to avoid these record destroy pitfalls.”

We recognize that these mistakes don't make us bad engineers but actually they are lessons for all of us to be better and help others avoid making the same mistakes.

We’re pushing the status quo which means failures are inevitable and something we fully welcome.

A fundamental ideology we hold is to always be curious. Curiosity sometimes lead us down the right path, such as our membership offering. Sometimes it leads us down the wrong one. Take “joystick” for example - this was a navigation overhaul that, while aesthetically pleasing and pushing the boundaries of UX and design, ultimately ended up leading to a negative end user experience. Regardless, we look to bring on individuals who are curious and hold a growth mindset above all.

You could say our Growth team fails regularly, but it’s never seen as a “failure,” rather it’s an important learning. “Shipping winning experiments is great, but every experiment teaches us something even when it fails. We learn something about the user psychology and can better determine what to do the next time around,” says Henry Ly, an iOS engineer on our Growth team.

We challenge ourselves, we challenge each other, and we challenge what we do as a company without fear of failure or repercussion, because that is the only way we grow and innovate.

This is one of our core values. We highly value continuous learning, both at the level of the individual and as a team navigating the ever-changing dating space. We constantly reflect on and find opportunities to do better, which inevitably comes with making mistakes. As one example, we send “nudges” to our users when there’s a lull in their chat conversation. We thought maybe if our nudges were sassy and sarcastic, they might increase chat participation by giving both parties something to laugh about or comment on. Unfortunately, people were writing in that our nudges were too abrasive and over the top. We thought nothing of it and switched back to the old messages. Failure? Not at all. We know more now than we did before.

At G Adventures we practice blameless postmortems.

Failure isn’t the fault of an individual developer. It’s the fault of the systems and practices that failed to prevent it. We learn from one another and continuously strive to learn from our mistakes and the mistakes of others. Senior developers and management are transparent about our past failures (dropping production databases for example!) and encourage new staff to try new things and experiment. We chronicle these failures through a postmortem repository and support documentation to and make each error an opportunity to learn.

We approach product and development from a learning angle.

It’s a horizontal organization at Monograph. We’re very open about how all of this is new to us and we’re all figuring it out together as we go. It’s part of the process. There is no blame whatsoever and with how much we communicate with one another, it’s never one person’s fault. We’ve actually had to rebuild Monograph two times now, and each time was a decision we made together. We hope you’ll experiment and work openly with us on both Monograph and whatever side project(s) you have.