I recently modified our failover protocol at work for MongoDB in a way that reduces the interruption from 14 seconds down to 3.5 seconds by altering election configurations ahead of controlled failovers. This was tested on a 3.4 cluster but should hold true up until modern versions. Starting in 4.0.2 it’s less valuable for controlled failovers but still useful as a tuning setting for uncontrolled failovers.

How it works

The premise is to make the shard call a new election as fast as possible by reducing electionTimeoutMillis and heartbeatIntervalMillis.

Procedure:

// on the primary
cfg = rs.conf()
cfg.settings["electionTimeoutMillis"] = 1000
cfg.settings["heartbeatIntervalMillis"] = 100
rs.reconfig(cfg)

// wait 60 seconds for propagation
rs.stepDown()

// wait for 60 seconds for election to settle
// connect to primary

cfg = rs.conf()
cfg.settings["electionTimeoutMillis"] = 10000
cfg.settings["heartbeatIntervalMillis"] = 1000
rs.reconfig(cfg)

This is valuable to tune also if you’re on high quality, low latency networks. You’re missing faster failovers in non-controlled circumstances every time mongo insists on waiting 10 seconds before allowing an election, even when receiving failing heartbeats.

PS - While browsing the docs I found this ^_^ which is non-intuitive since I would expect no writes to one shard but no impact to other shards. Presumably it’s a typo and cluster means replicaset.

During the election process, the cluster cannot accept write operations until it elects the new primary.

Ruby’s bundler inline installs to --system destination and does not respect BUNDLE_PATH (as I would expect).

Errors will be about permission errors and server requesting root access for the bundler install.

Digging around in github issues, this is desired behavior: https://github.com/rubygems/bundler/pull/7154

Solution:

export GEM_HOME=vendor/bundle

It’s 2022 and the macroeconomic environment is severely correcting from the easy days of cheap capital.

In this environment it’s wise on a personal and business level to consider expenditures and ensure they’re providing value. I’ve been through this with personal subscriptions and with family financial planning.

Today I spent a day off work helping a friend and colleague from a former startup in optimizing his company’s tech infrastructure spend. His business is a web application with a valuable service but low traffic levels ( < 10 RPS ) and running on Heroku.

Within the first 30 minutes of him screensharing and describing the business behavior of the app, I was able to recommend $200/mo (25%) savings on his plan.

With another 2 hours, I had a recommendation for how to save another $450, which is a great savings for an indie lifestyle business. That will save him $7,800 per year and reduce his Heroku bill from ~$800 down to $150 per month.

In debugging that, I discovered how he can also trim 20 seconds (ie 75%) off of a critical feature which will translate to increased conversion.

It was a fun problem to solve and fits well into my interest in either increasing business through technology or increasing efficiency and profit through technology. I’ve done similar projects on database and server spend which yields amazing results at scale (7 figures per year), on backend and frontend performance in small and medium size companies, and take pleasure in using my expertise at the intersection of technology and business.

I’ll test the waters to see if there’s demand for consulting to help companies outsource these efforts and reduce their AWS / Heroku / NewRelic / Datadog / etc bills.

The timing is right for this adventure.

I’ve upgraded Hugo, ditched webpack for esbuild and removed most javascript from blog 🍋. The best part is that it now does syntax highlighting on the server side and builds in < 500ms 🐎.

While reading docs, I learned that Hugo allows for custom type blocks such as the mermaid diagramming in a code block below that’s rendered into an image. Unsure if this will have practical use in blog, but it’s a problem in search for an answer ;).

Make your OSX setup reproducible by exporting settings using defaults cli tool:

$ defaults find com.apple.driver.AppleBluetoothMultitouch.trackpad

Found 24 keys in domain 'com.apple.driver.AppleBluetoothMultitouch.trackpad': {
    Clicking = 1;
    DragLock = 0;
    Dragging = 0;
    TrackpadCornerSecondaryClick = 2;
    TrackpadFiveFingerPinchGesture = 2;
    TrackpadFourFingerHorizSwipeGesture = 2;
    TrackpadFourFingerPinchGesture = 2;
    TrackpadFourFingerVertSwipeGesture = 2;
    TrackpadHandResting = 1;
    TrackpadHorizScroll = 1;
    TrackpadMomentumScroll = 1;
    TrackpadPinch = 1;
    TrackpadRightClick = 1;
    TrackpadRotate = 1;
    TrackpadScroll = 1;
    TrackpadThreeFingerDrag = 1;
    TrackpadThreeFingerHorizSwipeGesture = 1;
    TrackpadThreeFingerTapGesture = 0;
    TrackpadThreeFingerVertSwipeGesture = 0;
    TrackpadTwoFingerDoubleTapGesture = 1;
    TrackpadTwoFingerFromRightEdgeSwipeGesture = 3;
    USBMouseStopsTrackpad = 0;
    UserPreferences = 1;
    version = 5;
}

Then convert them into items in the format seen here: https://github.com/zph/zph/blob/master/home/.osx to create your own custom configuration to apply during next system setup.

Use defaults domains to find all the key domains.

I must not fear outages.

Fear is the mind-killer.

Fear is the little-death that brings total obliteration.

I will face my outages.

I will permit it to pass over me and through me.

And when it has gone past I will turn the scorecard to see its path.

Where the outages have gone there will be 5x 9s. Only availability will remain.

:face_palm:

I spent one hour tonight writing a new blog post about storage and then my 9 month idle blog

1. Refused to build from md to html
2. Couldn’t compile node-gyp with dependency on EOL’d python2
3. Debugging alpine build depdencencies for node-gyp
4. Had webpack v1 issues on M1 apple silicon
5. I refused to upgrade and stay on webpack
6. Which led to me switching to esbuild for js
7. Then I needed a custom plugin for esbuild sass
8. Finally I learned how to execute node in yarn’s package context

Entropy rules everything around me and this is after setting up CI, pinning versions, and using a static site generator for my blog.

I had to sleep on the issue and debug it with fresh eyes… it took 2.5 hrs of debugging and that speed of tool rot makes me consider reducing blog to even more basic form for improved longevity.

One of my teams at work is the online storage team (love it!), so I’m focusing efforts on how we improve the high availability of these systems and improve the system behavior during incidents or degradation. ie all the MTs: MTTR, MTBF and MTTD.

I’ve been spending a lot of time considering reliability improvements and turning those into a series of architecture and storage design principles to follow.

For these systems we’ve learned that our greatest leverage is in MTBF, which is predicated on operating these systems in the messy real world of cloud compuuting with the expectation of hardware failures (gp2 for 99.8 to 99.9% SLA/yr/volume).

What’s a recipe for improving current system behaviors?

• Partition critical and non-critical workloads
• Use read replicas as heavily as consistency requirements allow
• Choose the right cost/reliability threshold for your workloads (gp3 vs io2)
• Remove cross-shard queries from your critical path
• Run storage systems with sufficient headroom that you don’t have to performance tune in a panic
• Ensure 1-2 years of architectural runway on systems in case you need to shift to a new storage platform (ie eat your veggies first)
• Horizontally shard at application layer by promoting out hot or large tables to dedicated cluster (ie for aurora mysql)

With enough business success and a data intensive features, you’ll hit an inflection point you must either adopt a new online storage technology OR invest heavily in a sharding layer on top of your existing storage (vitess on mysql). Based on evidence of adoption at other companies, adopting a complex sharding solution is more expensive and less featureful than adopting a full storage platform including those features.

Ideal online storage system characteristics

Legend: ✅=yes, ☑️=partial, ⭕=no

Legend

(Chart filled in using my significant experience with MongoDB (<= 3.x) and Aurora Myql (5.x) but knowledge of TiDB, ScyllaDB and FoundationDB comes from architecture, documentation, code, and articles)

Predictions for the next 10 years of online storage

1. Tech companies will require higher availability so we’ll see a shift towards multi-writer systems with robust horizontal scalability that are inexpensive to operate on modest commodity hardware.
2. Storage systems will converge on a foundational layer of consistent distributed K/V storage with different abstraction layers on top (TiDB/Tidis/TiKV, TigrisDB, FoundationDB) to simplify operations with a robust variety of features.

What happens to your digital assets, legal assets, crypto, photos, emails, laptops, servers, and bereaved when you suffer from a fatal accident and exit non-zero?

If you’re the one left behind, what will make the process less layers of trauma?

(Based on a recent tragedy I’m close to… RIP mah friend </3)

First off, keep your passwords in a password manager and preserve the emergency recovery kit with one or more parties. If paranoid, split the content among N parties where it requires N-2 in order to recreate the full unlock info. If you have an attorney or a safety deposit box, consider keeping a copy there. Like physical security and trusting your local hardware, if you can’t trust your lawyer or deposit box you have bigger problems that need solving.

Keep a running list of your accounts, stocks, investments, company equity, stock options, RSUs, beneficiaries, etc. Either share the link to this spreadsheet ahead of time with loved ones or keep record of it in your password manager. Perhaps use a subtle tag like exit-127 in your password manager on all logins that should be investigated in the case of your death.

Use OTP instead of SMS and store it somewhere that could be recovered if you’re hit by a bus.

Store your laptop login in your password manager, with clear naming, so your bereaved can get in to figure out your accounts.

Make sure you and sig-other are both listed on accounts and cross referenced as beneficiaries, esp if unmarried.

Setup a Medical Power of Attorney and a Will or Trust… talk to an estate lawyer.

Share instructions about your hardware wallet. Make sure the ones you’ll leave behind know how to unlock it and have a trusted friend that can assist for redundancy.

Expect beneficiaries to need to engage with your most recent place of employment to wrap up loose ends, like returning company property (laptop, notebooks).

As the bereaved, pace yourself. Tackle the immediate needs and deal with grief, you can cancel Netflix and Spotify later. Speaking of which, grab credit card statements, bank statements, Mint login or YNAB and figure out weeks later what needs to be canceled. Get help from others for all the things you can outsource…. it’s going to be additional hardship working through the paperwork and logistics while thinking about your beloved.

So long for now.

G’luck and don’t exit non-zero.

[Facts adjusted to protect the innocent]

A friend approached me to talk about what to look for when hiring their startup’s first backend engineer. The company is ~8 people, with a frontend/3d CTO and 2 strong engineers on the frontend. There is currently no backend tech or staffing, so this is hiring from the ground up.

First off, it’s hard to hire well for a skillset you lack, so get advice from people who have that expertise.

For personality, they need to be scrappy and able to do more with less. This is entirely at odds with the expectations that exist for staff in larger corporations. You want more “indie hacker” style than deep expertise on LSM Trees.

They need to be personable and able to work well with various business units, since it’s all hands on deck to survive the first couple years of a startup. They might need to be front line support for Enterprise customers and will require the patience and tact to retain that customer despite less stable technical systems.

They need to understand the build-versus-buy debate and strongly prefer to buy when staffing is short. They will do things with Zapier, Airtable, etc that you find appalling… and they’ll use that saved time to invest in mission critical systems getting the product features or reliability they need.

They know how to build reliable systems, but save their efforts for the systems that need it. Sometimes a scholastic grade of C+ is the right benchmark for Tier 2 systems and they know how to make the tradeoff.

They’re biased towards boring technologies and realize how few innovation tokens a young startup can truly spend. At this stage of my own development, this means they’re using tech like Golang, gRPC, Aurora MySQL, and hosted solutions. They realize every hour needs to be delivering business value and reinventing wheels with self-hosting or new flashy tech is a failure mode.

They see technology as a means to an end of delivering product features and “wow” for customers.

They’ll need to be the DevOps and Developer Efficiency team, on top of their main backend architect and coder role. They need the skillset and willingness to design IAM permissioning, wrangle sensible and secure defaults into AWS/GCP, and ensure developers have a minimal amount of tooling and support to be effective.

They’re the ones who will setup your Continuous Integrations, Continuous Deployment, deployments, rollbacks, operational manuals, paging system, alerting, metrics, commit hooks, linting, unit testing, end to end testing, staging environments, etc.

They’re designing a storage layer that will likely survive for 5+ years… so while they’re not planning on ROFL scale, they have a general idea of how the current system would evolve into ROFL scale. (TLDR Please use Aurora MySQL… if that can’t handle your ROFL scale… consider DynamoDB or MongoDB can rofl-scale with coaxing and mild care).

They’ll setup your data warehousing and long term archival (blob storage). They don’t need to be an expert here but should have seen it done before. If they can use a hosted solution for warehousing, that’s best. Otherwise, they know enough to choose among various hosted/self-hosted solutions (snowflake, spanner, redshift, clickhouse).

They’ll work with your analysts and eventually will set them up with a dedicated Business Intelligence tool. When I sourced this in a prior company we settled on Periscope and it treated us well. They’ll make sure you run it off a read replica so you don’t endanger production.

They’ll do the rest of your backend hiring, they should either have the skill and experience in hiring, firing, and leadership. They don’t have to be a people manager but do need to be willing to act in whatever way is best for the organization.

They need to be scrappy: ie able and willing to tackle any problem. They should also be willing and able to work on the frontend if that’s the most critical task of the moment. If they can’t solve a problem, they know to find someone who can for advice or admit they’re stuck and brainstorm solutions.

They need to be product minded and think of how their expertise can unlock that product vision. They’ve either considered founding their own startup, worked in a small one or done entrepreneurial work before.

They need to understand the value of dollars, in your startup. It’s different than in larger corporations with greater runway. If someone isn’t productive, it could result in the full startup failing and putting everyone out of work… better to coach that person and fix it or send them packing.

They need to be presentable among a wide audience: sales staff, investors, techcrunch, other engineers.