Last week one of the engineering juniors that I mentor ran into a strange environmental issue.
When he ran npm run karma
it would run for ~8 minutes and then suddenly spit out an out of memory error. He tried debugging it for awhile himself and then reached out to me to assist.
We ran through the normal set of troubleshooting steps:
- Verify NPM and Node are on versions appropriately matched to production. (They were newer so we re-installed the ones used in prod)
rm -rf node_modules/
followed bynpm install
. (This semi-frequently resolves issues when old dependencies are not cleared out)
And when we tried running the offending command again, we suffered the error once more.
Which was when I reached into my bag of tricks and thought back on articles by @b0rk and @brendangregg. I remembered tutorials about using Dtrace to track down system calls from particular process identifiers. And I remembered a similar tool called DTruss that allows for attaching to PID and observing the system calls. For more info on DTruss, go check it out here: http://www.brendangregg.com/DTrace/dtruss or by vim $(which dtruss)
.
So I explained the barebones that I knew about how DTruss operates and we fired up dtruss npm run karma
.
We had time to talk a bit about system calls and the meaning of the readout. After 2 minutes we noticed that the log continued to fly by but the same folder was being accessed. Over and over and over. We had a recursive dependency due to an out-dated library that was stored inside the project tree.
Thanks to DTruss, we realized the issue, wiped out the offending folder and tried again with success!
PS - While writing this article I learned that Brendan Gregg wrote DTruss. Many thanks both for DTruss and for writing articles about how to use these tools! I also owe a thanks to Julia Evans who exposed me to these tools through her blogging and Zines :).