Our own app · 2015
A health score for your Ruby project's gems.
Prospector watched every gem in a Ruby project, graded its health and told you which updates were safe to take, without ever seeing a line of its code. I built all of it myself, and launched it in September 2015.
No longer runninggemprospector.com
The brief
Keeping a Ruby project's gems up to date was manual, easy to put off and important, both for security and for keeping up with the frameworks underneath. I wanted it automated, so I built a service that watched every gem in a project and said which could be safely updated and which would take real work.
The Challenge
I didn't want Prospector to need access to anyone's repository. It had to learn everything it needed at runtime, from the application's own bundle, and report it over an API, so teams could use it without handing over their source code, which helped with compliance too.
What I Built
An open source Ruby gem that reported a project's gems and Ruby version on every deploy, an API that took them in, and a Rails web app that graded each project, split its updates into safe and major ones, and kept a snapshot of every deploy. Teams paid for it, people added it through Heroku, and every one of Lotus's clients was recommended to use it.
Why I built it
At the time I’d noticed how much manual work it was to stay up to date on a project’s dependencies. They matter a lot, because an old gem can carry a vulnerability or a security issue, and because the frameworks underneath keep changing whether you keep up or not. Doing it all by hand was painful, so of course, as a developer, I wanted to automate it, and I could see there was a space for something that did.
I worked in Ruby on Rails all the time, so I made it for Ruby. It worked with Bundler rather than with Rails itself, which meant it could watch any Ruby project that had a Gemfile, from a Rails app to a Sinatra service or a RubyMotion app. I made the first commit in April 2015, and Prospector went public in September, as Lotus’s first product of its own.
That was before Dependabot, which started in 2017 and was bought by GitHub in 2019.
The client
One of the things I’m proudest of is that Prospector never needed your source code. Instead of adding something to a GitHub repository and asking for full access to it, I wanted it to work at runtime. You added a small open source gem to your project, and when your app was deployed, it looked at the application’s own bundle and reported back over an API: the name and version of every gem it had loaded, dependencies and all, and the version of Ruby it was running on. Prospector never got access to your actual code, and that helped with compliance too.
In a Rails app it took care of itself. When the app booted in a real web server, rather than in a console or a Rake task, the gem queued its report in the background with ActiveJob or Sidekiq, whichever the app already used. Anywhere else, one Rake task sent it, so a release script could run it on every deploy, and a RubyMotion app reported itself every time it was built for a device.
The gem went from 0.1 in June 2015 to 1.0 in March 2017, and it’s still on RubyGems, with 30k downloads to date.
The updates
Every gem Prospector saw was checked against RubyGems, where Prospector fetched each one’s latest version as soon as a project first reported it, and checked it again every day after that. A gem was current if you had the latest release, and expired if you didn’t.
A plain list of expired gems isn’t much help on a real project, where there can be dozens, so Prospector split them. A major update was one whose first version number had gone up, the kind that most likely needs real work to take. A safe update was one that semantic versioning says shouldn’t break anything, worked out the same way Bundler’s own pessimistic operator does. For the safe ones, it wrote the bundle update command for you, ready to paste into a new branch, run your tests against and ship.
The score
A list of outdated gems is useful, but I wanted something a team could glance at and watch move over time, so every project got a health score out of 100 and a letter grade. The sum is simple: start at 100 and take something off for every gem with an update waiting, more for a major update than for a safe one, and never go below zero. A is 93 and up, then B from 86, C from 78, D from 70, and anything under that is an F.
Getting the weights right took three tries. The first version took 10 points off for every major update, which was too harsh, so seven weeks later I brought it down to 2. In January 2016 I softened it again, to 1 point for a major update and half a point for a safe one, which moved one real project from a 2 to a 51. Try each of them here.
C80100 − 6 − 14 = 80
For Storefront, the demo project in the screens above, with 6 major updates and 28 safe ones, the August 2015 weights give it a 12, an F; September 2015’s give it a 60, still an F; and January 2016’s give it an 80, a C.
The grades
Over time
Every report was kept as a snapshot, with the gems you had, the score they earned and the Ruby version you were on, so you could look back at a project on any day and see how it had changed. From December 2015, a project’s history showed its score and its gem count as charts, over a timeline of every import.
The news came to you, too. Every Monday, on the plans that included it, an email went out with every project’s score and grade, and from March 2016 a project could post each new score into a Slack channel, in its grade’s colour. Permissions were built in from the start: an account could belong to several organizations, each with its own projects, users and teams, and a team saw only the projects it was given. You could sign in with GitHub, too.
Heroku
Prospector itself ran on Heroku, and at the end of 2015 I built it into a Heroku add-on as well. Running heroku addons:create prospector against an app created an organization, a project and its keys in Prospector, and handed them back to Heroku as config vars, so once the gem was in the Gemfile, the app reported itself the next time it booted. Signing in from Heroku’s dashboard took you straight into your project, and removing the add-on took everything with it.
People did add Prospector through Heroku, alongside the teams that paid for it directly, and every one of Lotus’s clients was recommended to use it too. It launched on Product Hunt as well, in August 2017, although I don’t think it brought much traffic.
The craft
A report fanned out into one background job for each gem, so a big project’s import ran in parallel on Sidekiq. Then it had to know when they had all finished, so the score was worked out once, and posted to Slack once, and never twice from two jobs racing each other. Batches, a short poll and a database lock made sure of it.
If a report arrived while the last one was still being processed, the API accepted it and quietly let it go, so the gem never retried and nobody’s deploy was held up by Prospector.
Prospector tracked its own code from August 2015, and in January 2016 one commit to it reads “Updated all minor gems as directed from Prospector site”.
I built every part of it: the Rails app and its API, the Ruby gem, the Heroku add-on, billing and trials, and the marketing site, over 950 commits across the three.
What happened
Prospector had some initial traction, but it seemed difficult to market, and I didn’t have as much customer uptake as I’d have liked to keep investing in my own product while running a very busy consultancy. So within about two years of the launch I stopped the feature development, and I just kind of let it die.
It had also become fairly clear that what people really wanted was for it to open the pull request in their own repository, and not just give them a dashboard and alerts about what was available. Sadly, I just didn’t put enough time into exploring what was possible there and following it through, or I think I could have ended up with a product that was acquired.
Then Dependabot came on the scene, and by the time it was clear that it was platform agnostic, and not tied to just Ruby, I realized there was no way I would win the race in the market. The last commit I made to Prospector, in June 2018, was a security update to Rails’s asset pipeline.
