The Journey from Manual to Automated: A Pragmatic Approach

In the fast-paced world of startups and growing tech companies, there’s often pressure to automate from the beginning. However, I’ve found that the most sustainable path follows nature’s own progression: crawl => walk => run. Let me share a real-world story of how we transformed our operations in the Online Storage team at a 300-engineer company.

Starting from Ground Zero

When I founded the Online Storage team, we faced a common dilemma: balance the need for speed with building sustainable processes. Instead of jumping straight into full automation, we took a methodical approach that paid dividends in the long run.

The Evolution of Our Process:

1. Crawl: Document Everything First, we focused on making our processes repeatable. Our humble beginning? Google Docs. We created detailed runbooks that captured not just the “what” but the “why” behind each step. These living documents became our foundation, complete with annotations about how to adapt procedures for different scenarios.

2. Walk: Identify High-Impact Opportunities As our operations matured, patterns emerged. We began analyzing our runbooks to identify which processes were consuming the most time and being executed most frequently. This data-driven approach helped us prioritize which manual processes to automate first.

3. Run: Dynamic Runbooks and Automation The game-changer came when we evolved our most-used runbooks into dynamic, parameterized versions. We integrated them with Terraform modules and built tooling that could automatically plan required inputs. This wasn’t just automation – it was intelligent automation that could adapt to different scenarios.

Real Impact: Database Migration Success

This approach proved invaluable during our complex database migrations from Aurora MySQL to TiDB. Our dynamic runbooks enabled our small team to execute these transitions with precision and confidence, despite each service having unique requirements and 200 steps.

Sharing with the Community

I believe in the power of open source and sharing knowledge. That’s why I designed our runbook tooling on my own time and made it available to everyone: dynamic runbooks. I excited to see how other teams adapt and improve upon this pattern.

The Key Takeaway

The path to automation doesn’t have to be an all-or-nothing approach. By starting with solid documentation, identifying high-value automation targets, and building flexible tools, teams can create sustainable processes that evolve with their needs.

I’m passionate about seeing this pattern adopted more widely in our industry. What’s your team’s approach to automation? How do you balance immediate needs with long-term sustainability?