On Being On Call

December 07, 2016

Join my mailing list…

There was a conversation on Twitter about developers being on call, started (I think) from this tweet (in reaction to a series of tweets that, thusfar, starts with this great blog post by Alice Goldfuss):

Put your developers on-call. You’ll be surprised what stops breaking.— Aaron אהרן (@as_w)

I 100% agree with this (for a particular interpretation of “put”—see below).

What You Learn On Call

I’ve observed, both in myself and others, that being on call leads to honing skills that are otherwise difficult to gain and are useful in writing software.

As I said in Production is All That Matters:

How your software runs in production is all that matters. The most amazing abstractions, cleanest code, or beautiful algorithms are meaningless if your code doesn’t run well on production.

Where I’ve grown as a developer in relation to being on call is in seeing how my code works in production and what happens when it breaks down. When code I wrote experiences a problem, I get very valuable feedback about my code. Was it really as simple as I thought when I was writing it? Do I regret omitting comments? How happy am I to see a Git commit message of “updated”? Did that failure case that “should” never happen actually happen? Did a think through the user experience as well I thought I had?

These are hard questions to answer, but being directly responsible for your code in production helps answer them.

Putting developers on call also fosters a DevOps culture that avoids an us-vs-them mentality with the technical operations team. In fact, a great reason to use a service like Heroku is that you don’t (initially) need an ops team, and developers have to be on call. Heroku is a really powerful tool for a developer to learn what it’s like to be on call, because the scope of your responsibility is greatly reduced as compared to a more classic hosting setup.

But, being on call should not be assumed to be part of your job.

“On Call” Must be an Explicit Requirement

If you are hiring developers, mention the on-call requirements. If you are looking for a job, ask about it. You need a sense of the frequency of when you’ll be on call, and a baseline for how often you’ll be woken up while you are on call (this is also a great question to reveal the actual quality of the software you’ll be working on).

The reason for this is that not everyone can actually be on call. Not everyone actually wants to be on call either, and that’s OK. You aren’t a bad developer for not wanting to be on call.

But, if your job requires it, then it requires it. That’s why it’s better to explicit up front.

It also makes it hard to suddenly put developers into an on-call rotation when the expectation has been set that this is not a job requirement. While it’s perfectly normal for a company’s needs to change as the company changes, you have to be cognizant of the people who now have a new job requirement being thrust upon them.

If you decided that your Ruby shop is now going to write 100% Go, you’ll need to accommodate the team during that transition. Adding an on-call requirement is an even more drastic change. You have to prepare yourself for people that are unable (or unwilling) to take on this new requirement. Whatever gain you feel you may get from adding an on-call rotation must be weighed against the downsides.

Conclusion

If you have the opportunity to be on call, and have the bandwidth in your life to do it, I highly recommend it. You might hate it, but I’ll bet you’ll learn things about software development, and you won’t always be in a place in your life to do it.