Timing tests
------------

The "ts" utility from "moreutils" is quite handy:

  $ CH_TEST_SCOPE=quick make test | ts -i '%M:%.S'

Note: a skipped test isn't free; I see ~0.15 seconds to do a skip.


How to write a test image
-------------------------

The Charliecloud test suite can build images by three methods:

1. From a Dockerfile, using ch-docker-build.
2. By pulling a Docker image, with "docker pull".
3. By running a custom script.

To create an image that will be built, unpacked, and basic tests run within,
create a file in test/ called {Dockerfile,Docker_Pull,Build}.foo. This will
create an image tagged "foo".

To create an image with its own tests, documentation, etc., create a directory
in examples/* containing a file called {Dockerfile,Docker_Pull,Build}.
Optionally, include a file test.bats containing Bats tests for the image.

Each of these files must specify its scope for building and running, which
must be greater than or equal to the scope of custom examples-phase tests in
test.bats. Exactly one of the following strings must be in each file:

  ch-test-scope: quick
  ch-test-scope: standard
  ch-test-scope: full

Other stuff on the line (e.g., comment syntax) is ignored.

Additional subdirectories can be symlinked into examples and will be
integrated into the test suite. This allows you to create a site-specific test
suite.

You cannot have multiple images with the same tag.

Dockerfile:

  * It's a Dockerfile.

Docker_Pull:

  * First line states the address to pull from Docker Hub.
  * Second line is a scope expression as described above.
  * Examples (these refer to the same image as of this writing):
      alpine:3.6
      alpine@sha256:f006ecbb824d87947d0b51ab8488634bf69fe4094959d935c0c103f4820a417d

Build:

  * Script or program that builds the image.
  * Arguments:
    * $1: Absolute path to directory containing Build.
    * $2: Absolute path and name of gzipped tarball output.
    * $3: Absolute path to appropriate temporary directory.
  * The script must not write anything in $PWD.
  * Temporary directory can be used for whatever and need not be cleaned up.
    (It will be deleted by the test harness.)
  * The first entry in $PATH will be the Charliecloud under test (i.e., bare
    ch-* commands will be the right ones).
  * The tarball must not contain leading directory components; top-level
    filesystem directories such as bin and usr must be at the root of the
    tarball with no leading path ("./" is acceptable).
  * Any scripting language is permitted. To be included in the Charliecloud
    source code, a language already within the prerequisites is required.
  * Exit codes:
    *    0: Image tarball successfully created.
    *   65: One or more prerequisites were not met.
    * else: An error occurred.
