Performing a resilient operation on bulk data can be challenging, especially if the operation relies on a third party. You can safely do this by fanning out the work to idempotent background jobs that operate on only one piece of data at a time. Those jobs can retry independently as needed, making the entire operation more easy to manage. This post will show an example of how that works and why you might want to use this pattern.
If you want to know how to sustainably manage background jobs in your Rails app, you should be using Sidekiq, and you should by my book, Ruby on Rails Background Jobs with Sidekiq. $9.99, ebook only, a scant 70 pages.
Since 2011, I’ve been responsible for apps that had a lot of background jobs, and I’ve been on-call for those apps. Believe me when I tell you, you don’t want to learn how to write idempotent jobs, create self-healing retriable processes, or manage queues that way. It takes too long and its too stressful. Just read the book I wish I had back then.
To effectively use any abstraction, you must—eventually—have a solid understanding of what it’s abstracting. I’m calling this the Katz Conjecture because Yehuda Katz made this statement in a conference talk (that I cannot find and hope actually existed). The conjecture has two implications: any abstraction that doesn’t sufficiently simplify what it’s abstracting makes everything worse, and any abstraction you don’t understand is a risk to your project.
This week, I’m on Robby Russel’s “Maintainable” podcast, talking about Sustainable Rails, testing, Angular JS, expertise and agile aphorisms. This is the intro and a reference to the link, as well as a quote:
I’ve been working on an in-browser interactive fiction/game for the last few years that I have finally decided is complete: Starlight Dawn. It’s a tile-based RPG in the vein of the old Ultima games. It was made with Ruby and JavaScript. You can play it right now, and your location is saved in your browser’s local storage. There’s no sign up or other requirement. Just start playing! You can also follow on Mastodon.
iOS has a feature on any website called “Save to Home Screen”, which creates an icon that then goes to that site. If you craft your website a certain way, this will launch the site in an app-like mode that is often referred to as a “Progressive Web App” or PWA. iOS does not support many features of PWAs that are available on Android or ChromeOS, but it supports some. This post will document exactly what you need to do, because it’s not documented very well.
Collin Donnell and Joel Drapper were kind enough to have me on their podcast, Rooftop Ruby. We talk about Background Jobs, the terribleness of SOLID, Service Layers in Rails and much more!
My first Rails job involved a lot of background jobs, and the year and a half I worked at LivingSocial was, in many ways, a crucible of managing background jobs. Today, the book I wish I had been given at the time is now in Beta: “Ruby on Rails Background Jobs with Sidekiq”.
The book is tightly focused at under 70 pages. It’s a pragmatic guide to using Sidekiq background jobs to make your app manageable without notification fatigue. Putting code in a background job is almost trivial—making it fault tolerant, idempotent, and self-healing is the hard part, and it requires care when designing your app’s features.
This book is all about that.
This is a bit of a long post describe the custom toolchain I created—and philosophy behind it—for the books that I’ve self-published. It was created to solve a problem that plagues technical books: keep them updated as libraries and programming environments change. My current solution completely automates all code samples, command-line invocations, screenshots, and many of the diagrams.
Jason Swett asked on Twitter if anyone has an app with 2000+ tests that does not have a severe flaky test problem. I have two such apps, and I want to share the lengths I’ve gone to to make the tests not flaky.
In a nutshell, you have to build and design for testability at all levels, plus ensure your tests are clear about what is the cause of a failure.