Have you tested your tests lately?

February 14, 2014 Molly Trombley-McCann

Summary: If you are using the rspec_api_documentation gem to test and document your API, you can use this code to test that you have a spec for each of your routes. Check out the gist here: http://gist.github.com/kayline/8868438

If you’re building an app with a public API, accurate and clear documentation is a blessing for both you and your third-party developers. The rspec_api_documentation gem is a nice solution that allows you to test your API and generate your docs in one fell swoop. You know your docs are up to date, because the tests that generate them are green. But what if an endpoint has no API test? Can we write a test to tell us when another test is missing?

As it turns out, yes! This post will explain how to create an Rspec test that fails if any of your routes do not have matching API documentation tests.

A Test Coverage Test

The test file itself lives in the same folder as your API specs. This is important. You must run your API specs at the same time that you run this test, or it won’t work properly. We put this code in spec/api/route_spec.rb.

require 'spec_helper'
require 'support/route_check'

  it "tests all routes" do
    route_check =
        routes: Rails.application.routes.routes,
        world: RSpec::world)

    route_check.missing_docs.should == Set.new

The test is implemented using a RouteCheck class, which takes in a list of routes and an Rspec test world. Where do these inputs come from?

Rails provides an API for getting all defined routes within an application, although the call is a little odd. Rails.application.routes returns a non-enumerable RouteSet object, on which we then call .routes again to get an array of routes. (We call Rails.application.reload_routes! first to ensure that all our defined routes have been loaded into the application.)

Getting the test world is fairly straightforward: just call Rspec::world. The trick here is that the world is loaded when the test is run, and its contents depend on which tests are being run. If you run a test in isolation, the world will only contain that single test. More to the point, if you are not running your API specs then Rspec::world won’t contain them and RouteCheck won’t be able to find them. That’s why this test lives in the API specs folder – it only works if the API specs are also running.

Test-Driving the Test Coverage Test

In order to implement our API coverage test, we need to define a RouteCheck class with the necessary behavior. In the spirit of recursive TDD, we wrote a series of Rspec tests to define how RouteCheck should behave. The following lives in spec/lib/route_check_spec.rb.

describe RouteCheck do
    { ActionDispatch::Routing::RouteSet.new }
    { route_builder(
      application_routes) }
    { route_builder(
    { route_builder(
      application_routes) }

  describe '#filtered_routes' do
    let(:routes) { RouteCheck.new(routes: application_routes.routes) }
    let(:filtered_routes) { routes.filtered_routes }

    it 'includes all non-filtered routes' do
      filtered_routes.should == [route]

   describe '#api_specs' do
     subject(:route_check) { RouteCheck.new(world: world) }
     let(:configuration) { RSpec::Core::Configuration.new }
     let(:world) { RSpec::Core::World.new(configuration) }

     describe 'returns routes for tests found in the world' do
       context "when there are no examples" do
         it "is empty" do
           route_check.api_specs.should be_empty

       context "when there are non-API examples" do
           { RSpec::Core::ExampleGroup.describe("regular group") }
           { resource_group.context(
             { :api_doc_dsl => :endpoint,
               :method => "METHOD",
               :route => "/path/to/api" }
            ) }
         let(:resource_group) do
            describe("resource group",{:api_doc_dsl=>:resource}) do
              context "something different"

        before do

        it "returns only API test groups" do
          route_check.api_specs.should == [action_context]

  describe '#missing_docs' do
     { RouteCheck.new(routes: application_routes.routes, world: world) }

      { RSpec::Core::Configuration.new }
      { RSpec::Core::World.new(configuration) }
      { ActionDispatch::Routing::RouteWrapper.new(route) }
      { ::Route.new(
        wrapped_route.path[//[^( ]+/]) }

    it 'detects routes for which no api test exists' do
      route_check.missing_docs.should == [formatted_route].to_set

    it 'does not return routes for which an api spec exists' do
      group = RSpec::Core::ExampleGroup.
        describe("resource group", {:api_doc_dsl => :resource}) do
           {:api_doc_dsl => :endpoint,
            :method => :get,
            :route => '/projects/lebowski'})

      route_check.missing_docs.should be_empty

  def route_builder(method, path, action, route_set)
    scope = {:path_names=>{:new=>"new", :edit=>"edit"}}
    path = path
    name = path.split("/").last
    options = {:via => method, :to => action, :anchor => true, :as => name}
    mapping = ActionDispatch::Routing::Mapper::Mapping.
      new(route_set, scope, path, options)
    app, conditions, requirements, defaults, as, anchor = mapping.to_route

    route_set.add_route(app, conditions, requirements, defaults, as, anchor)

We wrote tests for three methods on the RouteCheck class. The first, filtered_routes, should return the list of all routes for which we want matching API specs. This should not include internal Rails routes or routes without an HTTP method, and should allow us to filter out other routes we may not want to test (for example, routes generated by the Devise gem).

The second method, api_specs, should return a list of all currently defined rspec_api_documentation tests. It should not include non-API tests.

Finally, the missing_docs method should use the outputs from filtered_routes and api_specs to return all of the chosen routes that do not have matching tests.

The most complex part of these tests is creating new routes and tests in the test world without defining them in the app. We started by creating an empty RouteSet object and an empty Rspec world. In order to add routes to the RouteSet we created a route_builder method, which handles the creation of necessary intermediary Rails objects.

To add a new example group to the Rspec world we call RSpec::Core::ExampleGroup.describe() and pass in the test description and an options hash. Inside the example group we define a context, which is an actual test. In order to indicate that we want a certain test to be considered an API test, we add the api_doc_dsl key to the metadata of both the example group and the test. We did not find a way to organically add an API documentation test to the Rspec world, other than hard-setting the metadata key.


Now that we have some solid tests, we are ready to implement the RouteCheck class. We put this class in spec/support/route_check.rb.

class RouteCheck
  attr_reader :routes, :world

  def initialize(routes: nil, world: nil)
    @routes = routes
    @world = world

  def filtered_routes
    collect_routes do |route|
      next if route.internal?
      next if route.verb.blank?
      next if route.controller =~ /^devise.*/
      next if route.controller =~ /^docs.*/
      next if route.controller == "users"
      next if route.controller == "authentication_jig"
      next if route.controller == "invitations"
      next if route.controller == "admin"
      next if route.controller == "index"

  def api_specs
        reject{ |g| g.metadata.fetch(:api_doc_dsl, :not_found) == :not_found }.
        reject{ |g| g.metadata.fetch(:method, :not_found) == :not_found }

  def missing_docs
    existing_routes = Set.new(matchable_routes(filtered_routes))
    existing_route_specs = Set.new(matchable_specs(api_specs))

    existing_routes - existing_route_specs

  def matchable_routes(routes)
    routes.collect do |r|
      ::Route.new(r.verb, r.path[//[^( ]+/])

  def matchable_specs(specs)
    specs.map do |spec|
      ::Route.new(spec.metadata[:method], spec.metadata[:route])

  def collect_routes
    routes.collect do |route|
      route = yield ActionDispatch::Routing::RouteWrapper.new(route)

class ::Route < Struct.new(:method, :path)
  def eql? other
    self.hash == other.hash
  def == other
    method.to_s.downcase == other.method.to_s.downcase and
      path.downcase == other.path.downcase
  def hash
    method.to_s.downcase.hash + path.downcase.hash

The implementation of the filtered_routes method uses a private method collect_routes to wrap each route in a RouteWrapper object (a Rails class that provides simple accessor methods for route properties), and collect the results in a new array. The filtered_routes method itself passes a block to collect_routes that tells it to ignore any routes that meet certain conditions. The first two conditions filter out routes that are
a) internal to Rails
b) have no method (‘GET’, ‘POST’, ‘CREATE’, ‘DELETE’)
You will probably always want to include these two conditions in your filter. These rest of the filter conditions are project-dependent. We wrote this class for a project that was using the Devise gem for user management. Because Devise is an externally-maintained library with its own set of tests, we did not write API specs for Devise endpoints. We therefore tell filtered_routes to ignore any routes pointing to a Devise controller (/devise, users, or invitations). We also ignore a few routes that are not part of our public API. Which routes to filter is up to you, but keep in mind that the purpose of this code is to keep you honest about your API test coverage.

On to spec parsing! The RouteCheck class takes in an Rspec world and uses the api_specs method to extract each rspec_api_documentation test. Rspec has a pretty friendly API, so the implementation is relatively simple. We extract all the example groups and their descendants from the world, which gives us an array of rspec test objects. Each of these objects has a metadata hash, and the rspec_api_documentation gem helpfully adds an :api_doc_dsl key to that hash for all api_spec tests. By filtering our array of tests for those that have this key, we end up with an array of rspec_api_documentation test objects. However, this includes both actual tests and context blocks, so we filter again for objects that have a :method key in their metadata. Voila! We now have a test object for every API spec, and this object knows its method and path (via its metadata).

Now that we have an array of routes and an array of tests, we can call the missing_docs method to find the difference between the two. This method formats both the routes and the tests as ::Route objects with a custom equality function. It then subtracts the set of all tested routes from the set of all filtered routes. Any routes that have not been tested will be contained in the diff.

There you have it. Use this code on your next project and never push an untested API endpoint again! You can find the code here:


About the Author


Cloud Foundry Serving Around the Globe
Cloud Foundry Serving Around the Globe

The Church of Jesus Christ of Latter-day Saints (LDS Church) is a really large organization with even larg...

Development workflows with Grunt
Development workflows with Grunt

After working in one technology stack for a while, it is easy to become reliant on a toolchain you know and...