Of all the aspects of testing, preparing test data is probably the most time consuming and tedious. It is however, one the most important parts. If your test data isn’t correctly prepared and doesn’t tie in with your test plan then you are immediately at a disadvantage. Valuable time can be wasted examining code for bugs, when in fact it is the data that is at fault.

Often the tendency is to rely on client supplied test data however it is usually the case that their test data will not reflect all the possible variations of a document.  It is vital therefore, that before exposing the application to the client for testing with their data, that you are as confident as possible that, if given the correct data, the application will behave as expected. The best way to achieve this is with a comprehensive test plan and the data to match.

1. Document Your Test Cases!

Often the tendency is to do lots of testing during the development, and then only document five or six test cases. This means you are losing valuable test cases. Every test case run should be documented with expected results and actual results. Having these available means that when changes come along in the future you have a ready to run test suite which can be used for regression testing.

2. Use Control Files

Use control files (or their equivalent if you’re not using Dialogue) to assist with your testing. If you create a control file for each test case it means you can easily re-run tests without the need to rename files or worse still change the file properties within Dialogue. A sample control file could look something like this:-

  -packagefile=C:\Dialogue\pubs\application.pub
  -filemap=input.xml,C:\Dialogue\Input\testCase1.xml
  -outputfile=C:\Dialogue\Output\testCase1.pdf
  -messagefile= C:\Dialogue\Output\testCase1_messages.txt
  -reportfile=C:\Dialogue\OutputTestCase1_Report.txt

This means that anytime you need to re-run the test you can simply select the control file from the “Run Engine” dialogue. In addition if you have a full set of test cases that you need to run, you can create a windows batch file such as :-

  Engine.exe –controlfile=testCase1.opt
  Engine.exe –controlfile=testCase2.opt
  Engine.exe –controlfile=testCase3.opt
  Engine.exe –controlfile=testCase4.opt
  Engine.exe –controlfile=testCase5.opt
  Engine.exe –controlfile=testCase6.opt

This will then give you the ability to re-run a complete set of tests by simply running the batch file.

3. Design With Testing in Mind

Remember to consider your testing requirements as part of your design process.  Using a debug rule to include paragraph identifiers in the body of the documents can make the unit testing process much easier as you no longer have to compare big blocks of often very similar text to identify which particular paragraph has been included in the document.  Use of these references in your test plans also makes documenting the test cases much less tedious

4. Automate Data Preparation

Manual data preparation, has relatively low set-up costs however it is time consuming, especially when large volumes of test data are required. It can also be expensive to maintain if changes to the data format are introduced. This can lead to valuable test files becoming out of date with the result that either these files are rebuilt by hand or, as is more often the case, tests being skipped for future changes.

On the other hand automated test file preparation can be expensive to set up initially, however if the file is likely to be rerun on a regular basis for regression testing and the there is a large amount of data in use then these costs can often be recouped many times

The vast majority of the time, data will be manually put together using tools such as Text Pad, XML Spy and Excel. This is fine for small scale, rarely repeated testing. However the more often test data needs to be generated and the more often tests need to be re-run, the more value there is in seeking a way to automate the process.  The methods available will vary greatly as it depends on the tools you have available at your site.  Tools such as the Perl XML::API module make writing csv to xml converters very easy, and make it straightforward to increase the complexity of the test cases as the documents increase in complexity over time.

5. Print File Comparison

Another useful tool in the battle against bugs is the print file comparison tool.  There are a number of these around now (my own personal favourite is StreamDiff from CDP) and using these combined with a comprehensive regression test bed can be a huge time saver once you get into managing business as usual changes.

Conclusions

This is the last of the testing series (for now), and I hope you have found it useful.  There’s no doubt about it, testing is always a balance of time/effort against reward, and time is a very scarce commodity.  If you don’t take anything else away from this series remember that the earlier in the process you consider and start planning your testing the less time overall the project will take.  I’d love to hear what you think so feel free to comment or drop me a mail.