Use a spec to generate spec input

I’ve been enjoying Aruba a lot recently. Full integration command line testing is a great tool to have when working with legacy code.

Productivity drag

One thing that drives my productivity down when using aruba: massaging spec output! I have to run the program with certain input, copy output from the terminal, paste that into my spec, and massage it so the file looks decent.

If there are lots of changes or different paths through the program, this massaging gets tedious fast. The spec file gets filled with program output text. The chances of fat fingering the spec output increases the more output text there is.

Is there a better way? Yes, more specs!

Write a spec to save program output to a file, then use the output as input to other specs.

This way, run the spec to generate the output, every time after, use the generated output as a spec, whenever there’s a change in the output, re-run the output generation spec.

Increase productivity by not copy, paste, massaging output text. No more fat fingering spec output! Best of all, spec files only have specs, no output!

Demo

Let’s get started with a demo how I generated spec output by hand and then how to write a spec to generate the output, and configure RSpec to only run those specs when a flag is set.

Test ls /

Yes, it’s a bit contrived, but it is one of the easiest examples I can think of: Test the output of ls on the root file path of a new GNU/Linux install.

This is what a passing spec would be with Aruba:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
it 'gets output from ls by hand' do
  setup_aruba
  run('ls /')
  output =
    "bin\n"\
    "boot\n"\
    "dev\n"\
    "etc\n"\
    "home\n"\
    "initrd.img\n"\
    "lib\n"\
    "lib64\n"\
    "lost+found\n"\
    "media\n"\
    "mnt\n"\
    "opt\n"\
    "proc\n"\
    "root\n"\
    "run\n"\
    "sbin\n"\
    "srv\n"\
    "sys\n"\
    "tmp\n"\
    "usr\n"\
    "var\n"\
    "vmlinuz\n"
  stop_all_commands
  expect(last_command_started.output).to eq(output)
end

The spec itself is trivial, less than five lines that do work, but there is twenty lines of just output from ls. (note: This is how I would format it normally in my specs as well. It can be shorter if more things were fitted on a line, but this is just an example. ls /proc is really ugly and would definitely take over twenty lines!)

Use spec to generate output:

Make an output directory

I like to keep output files generated by specs in a directory, since there can be many files and I don’t want to litter the spec folder with spec output.

u@system: ~/spec_output_generator/$ mkdir spec/output

Generate output

1
2
3
4
5
6
it 'records program output', record: true do
  setup_aruba
  run('ls /')  stop_all_commands
  expect(last_command_started.output).to_not be_nil  ls_output = last_command_started.output
  File.open('spec/output/ls_output.txt', 'wb') { |f| f.write(ls_output) }
end

This basically runs the command in question and writes its output to file, specifically spec/output/ls_output.txt.

Cleaner spec

Now, by using the output generated, the first spec which tested the output can now become:

1
2
3
4
5
6
7
it 'gets output from ls from file' do
  setup_aruba
  run('ls /')
  output = File.read('spec/output/ls_output.txt')
  stop_all_commands
  expect(last_command_started.output).to eq(output)
end

Now that is a tight spec, less than ten lines.

The output is read from file, the one created in the test generation (spec/output/ls_output.txt) and compared to. There’s no need to touch the file at all. This is the best part. With much longer output, there’s no need to massage the output every time the code changes.

Caveat: full suite

There is one problem with this setup: every time the full suite of specs are run, the output is created and read in the same run (or even in a different order.)

If you are working on a legacy code project and want to generate a golden master record of the original code before touching it, regenerating the output on a full suite test run is not ideal since you want to keep the original output, not each new output.

Configure RSpec to ignore record: true specs

In the spec generator, there is this line:

it 'records program output', record: true do

The record: true is a tag in RSpec. Tags allow which specs are run under certain conditions. By default, all specs are run, with or without a tag. To exclude specs with a certain tag from running on default, just add this line to the spec_helper.rb file under RSpec.configure:

config.filter_run_excluding record: true

and to ONLY run all specs with the record: true tag, use command:

u@system: ~/spec_output_generator/$ rspec --tag record

and to run all specs without the tag use:

u@system: ~/spec_output_generator/$ rspec

Code

All code used above can be found at:

https://github.com/a-leung/spec_output_generator

Overall

Using a spec to generate output to a file is a very efficient way of handling long specs and most importantly, keeps you productive programming instead of massaging output.