Towards automating tests for examples in docs

LogoTM_verticalbigForgeRock documentation includes many (but never enough) examples. When reading about editing configuration files, you expect to see excerpts of the configuration files. When reading a developer guide, you expect to see code samples. When following a tutorial that involves the command line, you expect to see command line examples for each step. Most of us figure things out a lot more quickly given both a good explanation and also a working example.

Trouble is, examples can go stale and break when the software changes. Unless you have a test harness, this sort of breakage happens silently. If the doc source contains only example input and output, it can also take time to set the software up in order to reproduce the conditions for the example. And yet readers hardly want to search for the relevant part of the example in a mass of scaffolding code and configuration.

Some of that work can be done behind the scenes by quality engineers. They can set up a context that allows them to test examples in the documentation, and indeed some of the quality engineers at ForgeRock like Jean-Charles and Laurent are already starting to solve the problem that way. It would be better for them and for everyone else, however, if it were a lot easier to prepare the context, and not something separate from the examples.

So it would help both to include only the salient excerpts in the doc but also to link to all the material needed to set up the software to try or to test the examples. (Of course everything must be versioned alongside the software. If OpenAM changes between versions 11 and 12, then the examples must change as well.)

With XInclude and JCite support, this is already technically possible for XML and Java samples. We use JCite in the OpenDJ LDAP SDK Developer’s Guide to quote from code samples tested as part of the build. But throughout the docs we have samples that involve neither XML nor Java code.

For all the other samples, we are adding two things to the next major release of the forgerock-doc-maven-plugin: copying arbitrary resources into the documentation, and quoting from any text-based file.

In the next major version, copying arbitrary resources will involve setting <copyResourceFiles> to true. If the resources are not under src/main/docbkx/resources, you will set this by using the <resourcesDirectory> setting. There are some notes on this in the draft README for the nightly build version of the plugin. This way docs can include long example scripts and other files, without including the entire text in docs.

Quoting from any text based file will depend on a new plugin called xcite-maven-plugin. The basic idea is as follows. In the file you want to cite, you either add nothing (if you want to quote the entire file) or you add markers (if you want to quote only part of the file). The markers are just strings. So they could be in comments, but they could also be part of the text. Then in the file where you want the quote to appear, you add a citation, à la JCite. For example: [file.txt:start marker:end marker]

Suppose file.txt looks like this:

# start marker
This is a great quote & stuff.
# end marker

Then in your book.xml file, you might have the citation:

<para>[file.txt:start marker:end marker]</para>

After you run the plugin, the quote ends up in your book.xml file:

<para>This is a great quote &amp; stuff.</para>

The examples above are overly simplified. But you can imagine how this might work to include excerpts of a long shell script that involves some setup and configuration, then a number of example commands.

More to come…


Leave a comment

Filed under Docs, Tools

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s