Over the last month a lot of my free time has been spent working on a new project I'm really excited about. It combines two of my programming passions; asynchronous PHP with Amp and unit/integration testing. The project is a testing framework built from the ground up with first-class asynchronous support called AsyncUnit. It was designed to solve a specific set of problems encountered writing integration tests with an Amp powered application. Though there's still work to do the project has reached a minimum viability and ready for initial feedback! If you're interested in writing unit or integration tests with Amp please take a look!

The Problem - Why this exists

Before any code is shown, should address why a new testing framework is needed. Starting this project was not on a whim or a lark, but to solve a specific problem encountered writing integration tests. I have written thousands of unit and integration tests in my career as a programmer, both professionally and in my personal code. Generally I strive for 100% code coverage in my own projects and want to test all aspects of my library or application… including interactions with the database. Testing database code can be tricky and hard to do right. Asynchronous applications crank that challenge up a notch.

If you're writing tests with Amp and PHPUnit you should be aware of the awesome amphp/phpunit-util library. It provides an AsyncTestCase that eases a lot of the pain experienced with writing most types of asynchronous unit tests. I helped introduce the initial implementation and since then the maintainers have made some pretty great improvements to it. It is my de facto goto for writing asynchronous unit tests. However, the design of the wrapper, indeed the design of PHPUnit itself, was causing significant problems when I reached a certain number of database tests.

Each test in an AsyncTestCase has its own Loop that runs exclusive to that test. An Amp database connection requires that a Loop be running to keep it active. Each database test requires a separate connection to the database. Very quickly I reached a point where the number of database connections and disconnections overwhelmed Postgres, it was simply locking up and eventually crashing. I could have increased the connection limit on Postgres but that's a band-aid and just ignores the underlying problem. First thing I did was to dig through PHPUnit to see if this could be resolved with the existing wrapper. I almost immediately ran into an expected problem where some value from PHPUnit needs to be a Promise and turning it into a Promise would be a tremendous amount of work. I thought about attempting to refactor a mature, established codebase to do something it wasn't meant to do and determined it was not the best course of action. At that point I knew I'd have to drastically reduce the amount of database tests or write a framework with async support built-in from the ground up. Hence, AsyncUnit was born!

The Solution - An async testing framework

When I decided to implement a testing framework I started thinking about the functionality I use most often writing unit and integration tests. I considered the pains experienced writing my database tests and set out to design a testing framework that handles asynchronous I/O out-of-the-box. I decided that the framework needed to check off the following feature set for a minimum viable product.

• All tests need to run on one Loop so heavy-initialization resources, like database connections, can stay active.
• Take advantage of new features available in PHP8 like Attributes and Fiber support coming in 8.1.
• Tests can return a Generator, Amp\Promise, or Amp\Coroutine and have those values resolved completely.
• Have an assertion API that supports synchronous and asynchronous assertions.
• Support "normal use-case" functionality like data providers, expecting exceptions, and test setup and tear down.
• A CLI application to easily execute written tests.
• A process for solving the too-many database connections described earlier.

There's plenty of other features AsyncUnit aims to support; like code coverage, better diff output, and supporting a variety of mocking libraries. However, everything necessary for the MVP has been implemented, and the additional features have a solid path forward.

Writing AsyncUnit Tests

Let's finally look at some code! The demonstrations assume the following directory structure. It also assumes that a PSR-4 Composer autoloader has configured the namespace Acme\Examples\BlogIntroduction to load from the tests directory. All console commands are expected to be executed with the app_root as the working directory and all Composer dependencies installed.

app_root
  |_ tests
    |_ ExampleOne
    |_ ExampleTwo
    |_ ExampleThree
    |_ ExampleFour
  |_ composer.json

Below is a very simple TestCase that shows the minimum implementation required to see a running test… that will fail.

<?php declare(strict_types=1);

namespace Acme\Examples\BlogIntroduction\ExampleOne;

use Cspray\Labrador\AsyncUnit\TestCase;
use Cspray\Labrador\AsyncUnit\Attribute\Test;

class MyTestCase extends TestCase {

    #[Test]
    public function checkStringEquals() : void {
    }

}

vendor/bin/asyncunit run tests/ExampleOne

The terminal output indicates that this test failed because no assertions were made. Each method annotated with the #[Test] Attribute MUST make at least 1 assertion… anything less than that is considered a failure.

Writing an assertion

The failure in the previous example can be fixed by asserting something in the test. Let's confirm that two strings are equal to one another. For now, the test will use the synchronous API; a test showing use of the asynchronous API will be detailed further below. This version of the test will also fail, but it will be one step closer to green.

<?php

namespace Acme\Examples\BlogIntroduction\ExampleTwo;

use Cspray\Labrador\AsyncUnit\TestCase;
use Cspray\Labrador\AsyncUnit\Attribute\Test;

class MyTestCase extends TestCase {

    #[Test]
    public function checkStringEquals() : void {
        $this->assert()->stringEquals('AsyncUnit', 'PHPUnit');
    }

}

vendor/bin/asyncunit run tests/ExampleTwo

The terminal output displays the test failed because the assertion we made did not pass. It shows the two strings that are not equal to one another and specifies which file and line the assertion occurred. Let's make this test pass and see what a successful test looks like.

Writing a passing test

It is probably pretty apparent how to fix this test… compare two strings that are equal to one another!

<?php

namespace Acme\Examples\BlogIntroduction\ExampleThree;

use Cspray\Labrador\AsyncUnit\TestCase;
use Cspray\Labrador\AsyncUnit\Attribute\Test;

class MyTestCase extends TestCase {

    #[Test]
    public function checkStringEquals() : void {
        $this->assert()->stringEquals('AsyncUnit', 'AsyncUnit');
    }

}

vendor/bin/asyncunit run tests/ExampleThree

Now the terminal output displays what every TDD programmer wants to see… the green dot and ok! More tests can be added to this TestCase, or new test cases, until all the code under test is sufficiently covered.

Writing an asynchronous assertion

While the synchronous API is expected to be used frequently and demonstrates how to write tests there isn't anything particularly special there. Everything demonstrated can easily be done with PHPUnit. In the next demonstration a new test will be added to the existing TestCase that makes an assertion on the result of an asynchronous function. The function will be inlined in the test for this example but in normal use cases would be in its own file.

<?php

namespace Acme\Examples\BlogIntroduction\ExampleFour;

use Cspray\Labrador\AsyncUnit\TestCase;
use Cspray\Labrador\AsyncUnit\Attribute\Test;
use Amp\Promise;
use Amp\Delayed;
use Generator;
use function Amp\call;

function getAsyncString() : Promise {
    return call(function() {
        yield new Delayed(100); // emulating some interaction with I/O
        return 'AsyncUnit';
    });
}

class MyTestCase extends TestCase {

    #[Test]
    public function checkStringEquals() : void {
        $this->assert()->stringEquals('AsyncUnit', 'AsyncUnit');
    }

    #[Test]
    public function checkAsyncIo() : Generator {
        yield $this->asyncAssert()->stringEquals('AsyncUnit', getAsyncString());
    }

}

vendor/bin/asyncunit run tests/ExampleFour

Now the new test, checkAsyncIo, is more interesting! The asyncAssert()->stringEquals() call returns a Amp\Promise that resolves when the assertion is complete. Note that the 2nd argument passed to stringEquals() is also a Promise. The asyncAssert() API expects a type of Generator, Amp\Promise, or Amp\Coroutine for the actual value being checked. The value compared to the expected value will be whatever is resolved. This allows to easily check asynchronous calls without a bunch of boilerplate in the test itself.

AsyncUnit Resources

There's a lot more to the framework, even in its MVP state, than can be discussed in a single blog post. We didn't even get to set up and tear down, data providers, expecting exceptions, handling async timeouts, or how to solve the initial problem that caused the framework to be created in the first place! Needless to say there's a lot to document when it comes to a testing framework. Fortunately, the great folks over at GitBook have agreed to sponsor documentation hosting for AsyncUnit! If you haven't checked out GitBook you absolutely should! I found it easy to get started, intuitive to use for somebody familiar with Git, and the entire UI/UX feels really solid from a developer-writing-documentation perspective.

Here's all the important links for AsyncUnit!

• Source
• Documentation
• Bugs
• Discussion

What's Next?

For a project that I've implemented by myself in my free time I feel like it is in really solid shape. Numerous tests make sure that the framework and CLI tool operate as expected. The documentation site includes a Roadmap that details what features are actively being worked on as well as what features are coming up next. The project is currently still in 0.x development. If you're not familiar with semver that means the API could still change in breaking ways from version to version; there's a CHANGELOG where I detail which pieces have changed in each version and indicates which of those changes are considered breaking. At this point I'm considering the foundation in a pretty solid place and don't expect substantial breaking changes… just the addition of new features!

If you're interested in the project please check out the documentation and source code. Use the framework to write and run some asynchronous tests. Let me know what you think about the project! If you like what you see give the repo a star so I know there's interest. It helps keep me motivated during those long nights. :) Happy asynchronous testing!