Regaining trust in your test suite with Docker.

Docker, and containerised services in general, have brought a lot to the world of software development. While not everybody is (or ever should be) using docker as their deployment method, it's usefulness far exceeds being just a platform to run services on in production.

Testing PHP integrations.

For Flysystem, an open source PHP package to deal with filesystems, I needed a way to test FTP (barf) interactions. FTP servers are notoriously bad at abiding by the spec.

In the past I've used some of PHP's dirty little secrets to mock the ftp_* functions. This works, but it's not elegant. In the stubbed version I used responses from actual FTP servers, so I knew I could rely on those tests. But because Flysystem is an open source project this became a problem over time. This might sound weird, but getting help with FTP related things actually made things worse. Bugs were fixed, response fixtures were added. Over time I no longer knew if the mocked responses were any good. I didn't know for sure if I had broken anything because I no longer fully trust my test suite.

And that's when things need to change. The whole point of having a test suite is that it gives you confidence. You don't want to know it works, you want to proof it. So when you find yourself loosing trust in your test suite, find a way to restore it.

For Flysystem's FTP(d) adapter an integration test, using an actual FTP server, brought back the level of confidence it needed.

Running Docker containers on Travis CI

In order to run docker services on travis some specific configurations are required. In your .travis.yml you'll need to following settings:

sudo: required
services:
  - docker

This will ensure you're running on VM's that can run docker commands. Note that this setup also requires a little more setup time at Travis' end, your test process will slow down. It's a trade-off, having a fast test suite is nice, but having confidence in your test suite is far more important.

Now that we're all setup we can use docker to run an FTP server. In the before_script section of our .travis.yml we can setup our docker services:

before_script:
  - docker run -d --name ftpd_server -p 21:21 -p 30000-30009:30000-30009 -e "PUBLICHOST=localhost" stilliard/pure-ftpd:hardened
  - php wait_for_ftp_service.php
  - docker exec -it ftpd_server sh -c "(echo test; echo test) | pure-pw useradd bob -f /etc/pure-ftpd/passwd/pureftpd.passwd -m -u ftpuser -d /home/ftpusers/bob"

The first line starts a FTP container with some ports exposed. The third line creates a test user (bob) with a password (test). The middle statement is a waiter script.

The wait_for_ftp_service.php script checks if the container is ready for use. There are more "docker-ish" ways to do this, but this does the job. For these kind of scripts I like to write the dirtiest PHP I can:

<?php

$tries = 0;
start:
$tries++;
$success = @ftp_connect("localhost", 21, 2);

if ($success) {
    fwrite(STDOUT, "Connected successfully.\n");
    exit(0);
}

if ($tries > 10) {
    fwrite(STDOUT, "Failed to connect.\n");
    exit(1);
}

sleep(1);
fwrite(STDOUT, "Waiting for a connection...\n");
goto start;

In this script we try to establish an FTP connection. It's so dirty, I ❤️ it! It's got a goto statement, an @ warning suppressor, and even a sleep!

And then what?

Well, then you can just use the services as you normally would. You'd actually use them. No mocking, no stubbing, no nothing like that. For Flysystem the test cases are little scenario's. Connect with the FTP server at localhost:21 and use it. Writing files, reading them back, and then deleting them. Listing the contents of a directory you've just created. Those kind of things.

You can use this same approach for many other things. The most important thing is that we have proof our code does what it's supposed to do. Is searching your content important for your business? Spin up an ElasticSearch container, fill it with documents, and test your PHP code that uses it. You can do the same for many other databases too!

Other uses for Docker

Using Docker in your test suite is just one other use-case. You can use Docker in your CI/CD pipeline to build your deployable artefacts even if you don't run docker in production. You use docker to run one-off script in a language you don't have installed on your machine.

Aim for trust

The most important thing to take away from this; it's important to look for things that provide the proof you need. The more your business relies on something the higher level of confidence you'll want to aim for. If something is difficult, do it often. If something is critical, do it often. If you need to be better at something, do it often and you'll be better at it.