Engineering teams in data-driven companies

We’re revolutionizing the way rideshare and delivery drivers use their data.

We are upending the way that data flows in the gig economy. Companies like Uber and DoorDash vacuum up massive amounts of data from their workers and users. Then they use that data to optimize their operations, manage their workforce, and shape the public story about their companies, resulting in an unfair playing field. But what would happen if the people generating that data in the first place – the rideshare and delivery drivers themselves – had the power to capture their own data? What if they had the technological tools to make use of that data? These are the questions we’re answering at Driver’s Seat.

New engineers and data scientists who join the team will have the opportunity to work with large, complex datasets. These include high frequency GPS and telemetry data, workforce data (time, earnings, and activities), and mobility data (e.g. transportation mode, trip purpose). Drivers can choose to upload their data automatically or manually. We then use a suite of analytics tools, including PostgreSQL and R to provide concrete insights for users. For example, how much are rideshare and delivery drivers really earning? When and where is the most productive time to work? Is tipping higher on rides or food deliveries?

We discover valuable patterns in our data and can talk directly to drivers to understand the motivations and behaviors behind them. In turn, this also helps city agencies, who can leverage the data to make better decisions about transportation planning. We’re accessing massive amounts of interesting data and are only at the tip of the iceberg.

We have access to a vast amount of data that lets us measure the impact of our work on real-world user behavior.

At CircleCI, we are in a unique position to be able to review large amounts of data on how technology delivery teams act in the wild. Our cloud continuous integration and continuous delivery platform processes over 1.6 million job runs per day for more than 40,000 organizations and over 150,000 projects. To that end, we’re constantly looking at data to inform best practices and make improvements.

Every engineering team has a product manager and data scientist/analyst. Engineers work closely with data scientists to determine what data we should be capturing, how easily we can measure the things we’re interested in, and how to measure the impact our work has on user behavior and usage patterns.

We track OKRs at the company, department, and team level. For example, at the team level, we pay attention to delivery data (i.e velocity, cycle time, PR review time), SLOs and SLIs, escalation data, and PR review time. For production incidents, we track mean time to diagnosis and resolution.

We also look at market data, usage data for our product (like adoption rates), and growth metrics using A/B testing. Since insights is part of our product, we’re starting to build out a data intelligence/AI Ops team. Some of our analysis dashboards include Looker, DataDog, Amplitude, Segment, Honeycomb, and more.

We track the percent of time teams spend on different kinds of work: new features, customer support, and technical investment/maintenance. When we see teams that have a consistently high percentage of support and maintenance work, we might do any of the following:

• Review our documentation to ensure customers understand how a capability works
• Identify an opportunity for a new feature to address the need customers are trying to satisfy
• Split the team to reduce its area of responsibility
• Focus on reliability or scalability improvements until support requests and maintenance needs are reduced
• Hire more people to make the team bigger

We’re obsessed with using data to inform every decision.

Anything is open for debate, provided the argument is backed by data. To that end, we rely heavily on Segment, Snowflake, Dbt, and Mode for Business Intelligence. We use a combination of LaunchDarkly and Amplitude for A/B testing and Full Story to replay user sessions and uncover UX insights.

Individually, engineers own a set of metrics that they are responsible for analyzing, reporting, and developing a product roadmap against. We also believe in self-serve analytics. Regardless of job title, everyone in the company analyzes data.

“If we can measure it, then we can improve it.”

This is a mantra leadership repeats often and one that permeates throughout our culture. We take a hypothesis-driven approach and look to the data to inform decision making. That means we test and measure as much as we can, whether it’s conducting hundreds of A/B tests, constantly evaluating what search terms job seekers are using, or building and improving a large-scale recommendation system powered by machine learning. Engineers can expect to work closely with our data scientists on projects such as Search Ranking, Employer Recommendations, Budget Optimization and Salary Extraction, and Estimation.

If you’re a data nerd like we are, New Relic is your paradise.

There are a lot of companies that drive initiatives based on executives’ feelings, but they’re not necessarily grounded in actionable data. At New Relic, simply feeling that something is wrong won’t cut it for us – we aim to make decisions rooted in data. We pride ourselves on collecting data on everything from sprint velocity to data quality in order to make decisions regarding process and system improvements. DOS strives for monitoring-driven development, where monitoring is the first step of a project instead of the last. This is key in establishing a baseline for the current state and for demonstrating the utility of new features and system optimizations. That being said, we try to strike the right balance between making decisions based on math/data and remaining empathetic.

We aren’t just data-driven, we are data-obsessed.

When patients sign up, we receive their consent to retrieve and aggregate their medical records. Our records platform is built around human-in-the-loop machine learning to take the hundreds of pages of scans and faxes of medical records per patient, digitize them, then extract and organize key medical concepts. For our patients and research partners, we deliver structured, queryable data to help patients identify key information in their records and help research partners find deeper insights that were previously only unlocked by manual, painstaking analysis of records. In each of these three key pillars, data is a first class product that we are obsessed with. Like most companies, we use data to power our product, technical decisions, strategy, and roadmaps to ensure that we are delivering value to our customer; but where we truly differ is that data is the product that we acquire (with consent), refine, and deliver. Our data is used to accelerate research and treatment programs. We believe in the transformational power of data – for patients, researchers, and our team.

We rely heavily on data-driven decision making.

We’re building a wide range of passwordless, frictionless solutions on one platform, with a focus on both security and conversion. By using both data and qualitative user research, we can create the best products for our customers. We’ve invested in telemetry of services from the beginning and have several dashboards in the office that show traction toward goals. Analytics also help inform our site map and information architecture. For instance, we used analytics and research to revamp our onboarding experience.

One of our core engineering values is “Evidence is our North Star.”

At the highest level, the nature of our domain is clinical. Our product is evidence based and backed by industry-leading research and randomized controlled trials. Approximately 10 million people have access to Sleepio and Daylight – our digital therapeutics for insomnia and anxiety. Data informs everything we do as we continue to iterate and improve these offerings for our customers. We build our OKRs based on the data we collect and are constantly looking for ways to improve our DevOps metrics.

Data improves the insurance experience for our brokers and clients.

By running a full-stack brokerage, we sit in a privileged position to analyze data. We collect structured information about risk from clients (e.g. property data, business metrics, etc.) and risk pricing data from insurance markets. This industry is particularly opaque and we have a rare opportunity to provide transparency and build the first ever multi-sided marketplace for risk.

There are lots of thorny technical problems to solve at Newfront, from workflow systems to data modeling, to document generation. We always rely on the data we collect from brokers and clients to inform new projects and test initiatives before release.

Several engineers on our team have backgrounds in data analysis and mathematics, and we encourage the entire team to leverage their expertise when shipping products.

As an analytics platform, data drives everything we do!

Our core philosophy is to use data to help us learn where to evolve our product. We use Amplitude to instrument everything (>1 trillion data points per month) we do and all of our teams focus on understanding user behavior and using it to influence our roadmaps. In short, we use our own product in order to build a better product. We rely on Amplitude to research what we should be working on, identify which customers to reach out to, validate hypotheses, and determine leading and lagging indicators as we build new features. When we find shortcomings in our own system, we invest in building it. In fact, we never use other tools to do the job. When we needed a scalable data-ingestion platform, we built it in-house.

Sometimes, what we build for our internal use ends up getting rolled out. For example, Period-over-Period Analysis was a feature we built for ourselves to compare this week’s data to last week’s and give us a look at percentage change over time. We eventually rolled this out as a free feature for all of our customers to use and it continues to be one of the most popular.

Instacart is a data-driven organization.

We build and test often, and use as much data as we can gather to inform new features for our customers and shoppers. Our data science team in partnership with our product team develops hundreds of experiments per quarter to improve the customer, shopper, retailer, and advertiser experiences across our product portfolio.

If you’re interested in learning more, we’d love to hear from you!