Wojtek Mach

On Ruby & Rails

Integration Testing on Different Levels

| Comments

Last time I wrote about sharing examples in Minitest. This time I want to show an idea I had for a long time about reusing the same test to verify system’s behavior on different levels.

Let’s say we’re building a simple signup application. We may end up with a test like this:

(Check out full code here: https://github.com/wojtekmach/signups)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class SignupWebTest < ActionDispatch::IntegrationTest
  def test_success
    visit "/"
    fill_in "Email", with: "example@gmail.com"
    click_button "Sign up"

    assert page.has_content? "Thanks!"
  end

  def test_failure
    visit "/"
    fill_in "Email", with: "invalid"
    click_button "Sign up"

    assert signup.has_content? "Email" "is invalid"
  end
end

Now, let’s say we also want to have an API. Often times we are testing the same two scenarios as above, usually with the same test data:

1
2
3
4
5
6
7
8
9
10
11
12
class SignupAPITest < ActionDispatch::IntegrationTest
  def test_success
    post '/signup', signup: {email: 'example@gmail.com'}
    assert last_response.succes?
  end

  def test_failure
    post '/signup', signup: {email: 'invalid'}
    refute last_response.succes?
    assert_equal Hash['email' => ['is invalid']], JSON(last_response.body)['errors']
  end
end

Finally, we also have the lower level test that’s using the application logic directly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class SignupTest < Minitest::Test
  def test_success
    signup = Signup.new(email: 'example@gmail.com')
    signup.submit
    assert signup.valid?
    # assert email was sent etc.
  end

  def test_failure
    signup = Signup.new(email: 'invalid')
    refute signup.valid?
    assert_equal Hash[email: ['is invalid']], signup.errors.messages
  end
end

We can extract the common part from all tests into helper methods like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
class SignupWebTest < ActionDispatch::IntegrationTest
  def test_success
    signup(email: 'example@gmail.com')
    assert page.has_content? "Thanks!"
  end

  def test_failure
    signup(email: 'invalid')
    assert signup.has_content? "Email" "is invalid"
  end

  private

  def signup(attributes)
    visit "/"
    fill_in "Email", with: attributes[:email]
    click_button "Sign up"
  end
end

class SignupAPITest < ActionDispatch::IntegrationTest
  def test_success
    signup(email: 'example@gmail.com')
    assert last_response.succes?
  end

  def test_failure
    signup(email: 'invalid')
    assert_equal Hash['email' => ['is invalid']], JSON(last_response.body)['errors']
  end

  private

  def signup(attributes)
    post '/signup', signup: attributes
  end
end

As I am writing this, without thinking about it, I was just gonna work on cleaning up the 3rd test but, which is kind of the point of this post, there isn’t anything to clean up there. There’s no duplication that’s worth extracting out or some test/production API quirks worth hiding. Since we fully control the application code we can design it however we want.

This brings us back to the title of this post about reusing the same test on different levels. What I want to do is to design an interface that will behave like the Signup class, but under the hood will either call the application logic directly or use Web UI or API. The test must be written in such a way it’s easy to inject dependencies. Here’s one approach; I write it as a module that will be later included into concrete test cases.

1
2
3
4
5
6
7
8
9
10
11
12
module SignupTests
  def test_success
    signup = @app.signup(email: 'example@gmail.com').submit
    assert signup.valid?
  end

  def test_failure
    signup = @app.signup(email: 'invalid').submit
    assert !signup.valid?
    assert_equal Hash[email: ['is invalid']], signup.error_messages
  end
end

What’s an @app? It’s an object that knows how to construct object that can play a role of a Signup. Object that can play role of @app need only to implement #signup message. For Signup role they need #submit, #valid? and #error_messages. Here are possible implementations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
class WebClient
  def signup(attributes)
    Signup.new(attributes)
  end

  class Signup
    include Capybara::DSL

    def initialize(attributes)
      @email = attributes[:email]
    end

    def submit
      visit '/'
      fill_in 'Email', with: @email
    end

    def valid?
      page.has_content? "Thanks!"
    end

    # ...
  end
end

class APIClient
  def initialize(base_uri)
    @base_uri = base_uri
  end

  def signup(attributes)
    Signup.new(self, attributes)
  end

  class Signup
    def initialize(client, attributes)
      @client, @attributes = client, attributes
    end

    def submit
      RestClient.post(@client.base_uri + "/signup", signup: @attributes)
      # ...
    end

    # ...
  end
end

class App
  def signup(attributes)
    Signup.new(attributes)
  end
end

Now we can write the remaining concrete test cases:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
class SignupAppTest < Minitest::Test
  include SignupTests

  def setup
    @app = App.new
  end
end

class SignupAPITest < Minitest::Test
  include SignupTests

  def setup
    WebMock.stub_request(:any, /signup.test/).to_rack(Rails.application.routes)
    @app = APIClient.new('http://signup.test')
  end
end

class SignupWebTest < Minitest::Test
  include SignupTests

  def setup
    Capybara.app = Rails.application
    @app = WebClient.new
  end
end

There’s a few nice benefits about this design.

For one thing this setup is highly configurable. We can easily switch certain levels on and off. What’s more, we can take this configuration further and for the UI & API tests point them to live servers (e.g. staging.example.com) instead of local servers on development machine. This has added benefit that we can find more errors this way, like for example asset pipeline & general deployment issues, DNS etc. Granted, this works extremely well for a simple application as such that’s basically stateless but it should still be doable for more complex cases.

This test design also forced us to write mostly production (albeit not used by the production app) code and just a little bit of simple test code. A nice side effect of this is I think you’d generally keep this code more organized if it’s not a part of the test suite. More importantly though as a way of testing the app we built client libraries to access API (See http://robots.thoughtbot.com/how-to-test-sinatra-based-web-services) and the Web UI. If you’re lucky enough to have a dedicated QA team they may appreciate that they can drive the app using quite convenient interface yet still be able to access raw features of capybara etc.

Finally, there’s one more thing maybe worth mentioning. If we have 2 instances of the app running on app1.example.com and app2.example.com it’s entirely possible to configure app1’s controllers to use APIClient (instead of simply App) pointed to app2.example.com without a single change in the application code. Again, probably not that useful but I think it’s pretty cool :–)

Sharing Examples in Minitest

| Comments

Last time I wrote about enforcing Liskov principle via tests. It was pretty simple to do in Minitest using just class inheritance. Sometimes, however, we can’t inherit test methods because the framework forces us to inherit from a test case class like:

  • ActiveSupport::TestCase
  • ActionDispatch::IntegrationTest

etc. In these cases we need to find some other way to share behavior and with Minitest’s design the answer is pretty simple – modules.

Example

Let’s write a simple data store library inspired by Moneta

1
2
3
4
5
6
7
8
9
10
11
12
13
class DataStore
  def initialize(adapter)
    @adapter = adapter
  end

  def get(key)
    @adapter.get(key)
  end

  def set(key, value)
    @adapter.set(key, value)
  end
end

Now let’s write an adapter:

1
2
3
4
5
6
7
8
9
10
11
12
13
class DataStore::InMemoryAdapter
  def initialize
    @hash = {}
  end

  def get(key)
    @hash[key]
  end

  def set(key, value)
    @hash[key] = value
  end
end

Let’s write a test for this. Knowing we will later reuse test methods, we start with a module:

1
2
3
4
5
6
7
8
9
10
module DataStore::AdapterTest
  def test_get_not_found
    assert_equal nil, @adapter.get(:invalid)
  end

  def test_set
    @adapter.set(:foo, 42)
    assert_equal 42, @adapter.get(:foo)
  end
end

Now the actual DataStore::InMemoryAdapter test (note, I’m using Minitest::Test which comes from minitest 5):

1
2
3
4
5
6
7
class DataStore::InMemoryAdapterTest < Minitest::Test
  include DataStore::AdapterTest

  def setup
    @adapter = DataStore::InMemoryAdapter.new
  end
end

Running this we see that two examples have been “inherited” from the shared module:

1
2
3
4
5
6
7
8
9
10
~% ruby shared.rb
Run options: --seed 18221

# Running:

..

Finished in 0.001126s, 1776.1989 runs/s, 1776.1989 assertions/s.

2 runs, 2 assertions, 0 failures, 0 errors, 0 skips

With this foundation it’s pretty easy to add new adapters and we don’t really have to write new tests. Including shared module in the test is enough to have confidence that an adapter is conforming to an interface.

Let’s say we package the data store as a gem. We can ship the AdapterTest as an integral part of the gem distribution and let the users write their own application specific adapters. Just as Rails ships with ActionDsipatch::IntegrationTest.

minitest/spec

It’s actually pretty easy to use shared modules with minitest/spec. It’s simple because minitest/spec is really just a DSL on top of minitest/test (minitest/unit). A describe block creates a new Minitest::Test class, an it block defines a new test_ method. With this in mind we can start with our custom DSL like this:

1
2
3
4
5
6
7
8
9
10
module DataStore::AdapterSpec
  it "returns nil for an invalid key" do
    @adapter.get(:invalid).must_equal nil
  end

  it "can set a value" do
    @adapter.set(:foo, 42)
    @adapter.get(:foo).must_equal 42
  end
end

And the spec:

1
2
3
4
5
6
7
describe DataStore::InMemoryAdapter do
  include DataStore::AdapterSpec

  before do
    @adapter = DataStore::InMemoryAdapter.new
  end
end

Running this will result in error like:

1
<module:AdapterSpec>: undefined method 'it' for DataStore::AdapterSpec:Module (NoMethodError)

Let’s fix this; we basically have to implement Module#it for it to work:

1
2
3
4
5
class Module
  def it(description, &block)
    define_method "test_#{description}", &block
  end
end

Tests should be passing now.

I mentioned before that minitest/spec is just a DSL. In fact, there’s literally a Minitest::Spec::DSL module that Minitest::Spec is including. The DSL module is so good in fact that it can be included both in classes and in other modules:

1
2
3
class Module
  include Minitest::Spec::DSL
end

and it just works! We now can do stuff like:

1
2
3
4
5
6
7
8
9
10
11
12
module SomeTest
  before { "..." }
  after { "..." }

  let(:foo) { "..." }

  it "returns this" do
  end

  it "returns that" do
  end
end

etc.

The way Minitest::Spec::DSL is implemented is actually pretty simple. It doesn’t do anything special; it just defines a bunch of methods like setup, teardown, foo, test_returns_this etc. It means that after the “DSL” phase we end up with just a ruby module that we can include (or not), and nothing is evaluated until the module is included somewhere.

Conclusion

Minitest’s simple design allows us to extend it with standard tools we use in day to day ruby programming. We can use the same exact constructs like classes, modules, inheritance & mixins for both the production & test code. As a consequence of this design writing minitest extensions is imho pretty easy too!

Liskov Principle & MiniTest

| Comments

What is Liskov Principle?

In layman’s terms Liskov Substitution Principle says that if class Foo inherits from class Bar, then you should be able to use (substitute) derived class in every place that the base class is used. For a better definition and further references check out The Liskov Substitution Principle by Uncle Bob.

Testing LSP with MiniTest

MiniTest has a really simple design. A test case is a class and an example is a method of that class. After requiring minitest/autorun every subclass of MiniTest::Unit::TestCase is instantiated and test methods are executed one by one.

One very nice result of this design, which is kind of obvious when you think about it, is that you can not only inherit helper methods (eg. you subclass ActionController::TestCase to have get, post etc) but you may as well inherit whole examples! This is a perfect way to test LSP because, again, you should be able to substitute base class with a derived class.

Example

Let’s re-implement Ruby’s built-in Set class. I’ll write a test first:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
require 'minitest/autorun'

class SetTest < MiniTest::Unit::TestCase
  def setup
    @set = Set.new
  end

  def test_size
    assert_equal 0, @set.size
    @set.add 42
    assert_equal 1, @set.size
  end

  def test_include?
    refute @set.include? 42
    @set.add 42
    assert @set.include? 42
  end

  def test_add
    @set.add 13
    @set.add 13
    assert_equal 1, @set.size
  end

  def test_to_a
    @set.add 1
    @set.add 4
    @set.add 2

    ary = @set.to_a

    assert_equal 3, ary.size
    assert ary.include? 1
    assert ary.include? 2
    assert ary.include? 4
  end
end

Note I didn’t write the exact result of Set#to_a because a cannonical set is unordered. A Ruby 1.9 built-in Set is actually ordered, it simply preserves the order of insertion.

A basic implementation is very easy using Hash like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class Set
  include Enumerable

  def initialize
    @hash = {}
  end

  def size
    @hash.size
  end

  def add(obj)
    @hash[obj] = true
  end

  def include?(obj)
    @hash.include? obj
  end

  def each(&block)
    @hash.keys.each(&block)
  end
end

Let’s run it:

1
2
3
4
5
6
7
8
9
10
~% ruby set.rb
Run options: --seed 59316

# Running tests:

....

Finished tests in 0.000589s, 6791.1715 tests/s, 15280.1358 assertions/s.

4 tests, 9 assertions, 0 failures, 0 errors, 0 skips

Now, let’s write a SortedSet that will keep values sorted. Again let’s write a test and run it first:

1
2
class SortedSetTest < SetTest
end
1
2
3
4
5
6
7
8
9
10
~% ruby set.rb
Run options: --seed 54235

# Running tests:

........

Finished tests in 0.000944s, 8474.5763 tests/s, 19067.7966 assertions/s.

8 tests, 18 assertions, 0 failures, 0 errors, 0 skips

We now have exactly twice assertions because all test methods have been inherited. Let’s now build a simple SortedSet class and adjust the test, so that we actually use the derived class:

1
2
3
4
5
6
7
8
9
10
11
class SortedSetTest < SetTest
  def setup
    @set = SortedSet.new
  end
end

class SortedSet < Set
  def each(&block)
    @hash.keys.sort.each(&block)
  end
end

Sure enough all tests passes and we’re now certain that a Set object can be substituted with a SortedSet object.

Let’s also test the unique behaviour of the SortedSet. We won’t just define test_to_a method, because we would overwrite assertions from the base test. We’ll pick a different name instead:

1
2
3
4
5
6
7
8
9
10
11
12
13
class SortedSetTest < SetTest
  def setup
    @set = SortedSet.new
  end

  def test_to_a_sorted
    @set.add 1
    @set.add 4
    @set.add 2

    assert_equal [1, 2, 4], @set.to_a
  end
end

Now, we could stop it right here, but you propably noticed some duplication between test_to_a and test_to_a_sorted. Again, because we’re using just classes and methods, we can actually write:

1
2
3
4
5
6
7
8
9
10
class SortedSetTest < SetTest
  def setup
    @set = SortedSet.new
  end

  def test_to_a
    super
    assert_equal [1, 2, 4], @set.to_a
  end
end

I’m not sure if it’s that useful and you should use it, but you must agree it’s pretty neat!