20 Jan 2017

Announcing MoreSQL (Realtime Mongo -> PG streaming)

I’m proud to announce that MoreSQL is live and production ready :)!

We’ve been using it in production for a few months to stream production data mutations from Mongo to PostgreSQL. The latency is normally subsecond and has scaled well with a small footprint in the two production use cases.

Background

It’s written in Golang and conceived of based on a Ruby project called mosql (built by the wonderful folks over at Stripe). After using that in production our telemetry revealed that it was lagging behind during mutation intensive periods. We were further stymied from upgrading MongoDB due to version incompatibilities in MoSQL.

Implementing the project from the ground up in Golang allowed for better concurrency, lower latency, and a small memory footprint (often 20-50MB under load).

License

Moresql is released under a permissive license and is open source software. You’re welcome to set it up in production environments and submit pull requests to improve the project.

Consulting

If you’d like a turnkey solution implemented for your business in order to have realtime data from Mongo sent to Postgres for reporting, analytics, query performance or as a mongo to postgres migration strategy, send me a note. As the author of Moresql and with running this solution in production on different systems, I’m ready to solve your problem!

Update

We used this project in production from 2017-2019 for our primary data storage and using MoreSQL was a critical component of moving off of MongoDB.