How Uber Increased Deployment Automation From 7% to 65%
Explore how the ride-hailing giant streamlined its code delivery process and transformed its microservices ecosystem
Ever felt like your deployment process was a chaotic traffic jam, with code changes honking impatiently while manual approvals play traffic cop? That's precisely where Uber found itself not too long ago.
Uber juggled 4,500 microservices across three monorepos, with a whopping 5,600 commits zooming in weekly. And only 7% of their services used continuous deployment.
So, how did Uber turn this deployment gridlock into a smooth-flowing highway of continuous delivery?
Let's find out!
Uber's messy deployment system
In 2022, uber managed 4,500 microservices across three monorepos, with 5,600 weekly commits. Despite this scale, only 7% of services used continuous deployment, while 34% relied on manual processes. Talk about a manual mess!
With over 100 unique pipeline templates, it offered flexibility but sacrificed consistency. Lack of standardized testing and monitoring left quality hanging in the balance.
To top it off, Uber juggled two separate deployment systems. This confused developers and made implementing best practices challenging. As Uber grew, this fragmented approach became a roadblock, demanding a complete deployment detour.
So they introduced Up CD
Facing deployment chaos, Uber's team set out to create a new system. "Up CD" was designed with five key goals:
Standardize and automate deployments, bringing order to the Wild West of code releases.
Put safety first by integrating observability and testing because nobody likes a surprise outage.
Make CD the default, embedding best practices into everyday workflows.
Craft a UI/UX that eases the navigation in monorepos.
Seamlessly integrate with Uber's Up platform, creating a unified development experience.
The architecture of the new CD system
But how did Uber tackle the monorepo monster? They got smart with data structures.
Up CD efficiently handles 1000+ daily commits by tracking only service-specific changes. This targeted approach reduces the scope for individual services, eliminating unnecessary rebuilds.
The system creates simple, direct links between services and relevant commits, making tracing changes and their impacts easy.
Developers can now focus on what matters – building great features – instead of wrestling with deployment complexities.
Up CD's smart workflow
Up CD doesn't just manage commits; it orchestrates them.
The system uses Kafka to process commit events, ensuring no change goes unnoticed. Each commit undergoes an automated impact analysis, pinpointing affected services.
This smart approach means only relevant services get updated, saving time and resources.
Deployments follow standardized stages but with a twist. Configurable gates act as quality checkpoints, including:
Soak time to catch sneaky bugs
Deployment windows to avoid peak-hour surprises
Operational status checks for system health
Alert monitoring to catch issues before users do
Up CD's user experience is also optimized. Developers get a service-centric view of commit history, focusing on what matters to them. A collapsed view highlights only relevant commits, cutting through the noise.
The visualization gives a quick status check across environments, while detailed event tracking per commit helps pinpoint issues fast.
Integration is key to Up CD's success. It works well with existing systems, creating a unified deployment experience. It's compatible with manual operations, too, because sometimes you need that human touch.
The incremental adoption path means teams can ease into the new system, making the transition smooth and painless.
The results is in the numbers
Fast forward to March 2024, and Uber's deployment has transformed dramatically.
Uber's service count grew from 4,500 to 5,000, and weekly commits nearly doubled, jumping from 5,600 to 11,000. Weekly deployments skyrocketed from 7,000 to a whopping 50,000.
That's not a typo – Uber increased its deployment frequency by over 600%.
CD-orchestrated deployments surged from 66% to 95%, showing a clear shift towards automation. The number of fully automated services leaped from 7% to 65%. That's a lot of developers sleeping better at night.
Incidents per 1,000 code changes dropped by 50%. So Uber didn't just speed up – they got safer, too.
With great power come great responsibility
As with any major system overhaul, Up CD brought its own set of challenges.
The increased speed of deployments amplified the risk from shared code changes. A single update could now affect multiple services simultaneously.
To tackle this, Uber implemented a tiered deployment strategy. High-risk changes are first rolled out to less critical services, acting as canaries in the coal mine. This staggered approach allows Uber to catch potential issues before they impact core services.
Managing parallel deployments became crucial as deployment frequency increased.
Uber developed an automated halt mechanism that pauses deployments if it detects issues, preventing a domino effect of failures.
This system works hand-in-hand with an author notification system, alerting developers immediately if their changes cause problems.
Cross-service monitoring arose as a critical challenge in this interconnected environment.
Uber's solution? A comprehensive success rate monitoring system that tracks deployments across all services. This bird's-eye view allows Uber to quickly identify and address any anomalies, maintaining the health of the entire ecosystem.
For companies looking to implement similar strategies, consider:
Analyzing your current deployment patterns and pain points
Investing in tools that can intelligently determine which services are affected by each commit
Implementing robust monitoring and rollback mechanisms
Developing a gradual rollout strategy for high-risk changes
Continuously gathering feedback and iterating on your CD process
Learn more about it here.
Here are some of the insightful editions you may have missed: