I had the task of deploying three production databases with minimal downtime. Here’s the takeaways.
Moving Redis with persistent data
Redis needed to move off a couple providers and into another provider. This needed to happen inside a 30 min maintenance window for one application (which performs critial writes) but some novelty loss of other low value writes was an acceptable tradeoff for having 0 downtime of other services.
One db was easily imported using DB host’s Import tool. Another db was not able to use that mechanism and was transfered by redis-transfer. I enjoyed extending the tool to make it work well for this purpose.
Simplest of all, it was a matter of generating Heroku backup, downloading that link and importing it into other db.
#!/usr/bin/env bash # References https://devcenter.heroku.com/articles/heroku-postgres-import-export # # Requires heroku commandline tool. # The following ENV are required # HEROKU_API_KEY= # The following envs are required for the destination DB and are automatically # used by PG. # PGPASSWORD= # PGUSER= # PGHOST= # PGPORT= # Set this for simpler scripting # PGDATABASE= # Install heroku toolkit https://zph.xargs.io/heroku-toolkit-install.sh | bash # sudo apt-get install postgresql OUTPUT_FILE="latest.dump" APP_NAME=$HEROKU_APP heroku=$HEROKU_BIN $heroku pg:backups -a $APP_NAME capture && \ curl -o $OUTPUT_FILE `$heroku pg:backups -a $APP_NAME public-url` && \ pg_restore --verbose --clean --no-acl --no-owner -d $PGDATABASE $OUTPUT_FILE
The Day Of
I ran through all the steps, outlined them, then setup working scripts for each portion of process. Those were then setup as commands in a command station type
Each Tmux tab was a phase of the process:
maintenance_mode:on, redis_migrations, maintenance_mode:off, pg_migrations, logging
Inside each tab it had the commands I would need to one, one per section of the window:
|-----------------|------------------| | redis1_migration| redis_migration2 | |-----------------|------------------| | point to new r1 | point to new r2 | |-----------------|------------------|
Performing the Migration
- Notified stakeholders in advance
- Prepared steps, conducted trials against staging
- Setup migration scripts
- Walk through checklist in 15 min before time
- Set one heroku app to maintenance mode
- Import 2 redis dbs
- Verify result
- Run script to point to those new endpoints
- Maintenance mode off
- PG migrate
- Verify results
- Run script to point to new endpoints
Glad redis-transfer was available to help with a recalcitrant server. And I’m glad to be preparing postgres for more active duty in our stack.
My takeaway from accomplishing this migration was that careful planning leads to quick and uneventful maintenance windows. Also, I’d rather migrate pg than redis.
And have a migration buddy :). Makes it far more enjoyable and extra hands in case things go wrong.