Monday, June 27, 2011

Creating Plugins for Wave Test Manager

A customer recently contacted me describing a problem of how to get notified when a script fails in Wave Test Manager.  In this case their alert system did not work with the default options built in to Wave Test Manager (email, XMPP, etc) and thus they wanted have another way to know when one or more scripts had failed.    Fortunately Wave Test Manager 1.1 offers a very simple way to do this kind of integration based on its Plugin system.   In this post I’ll walk through a very simple example plugin that first saves the details about failures in a file, and then to extend it I’ll show how to make it store failure details in a database.   To do this I'll be assuming a very small amount of familiarity with the concepts of programming languages.  However what we need to do is so simple that if you have any basic experience with any kind of scripting language such as VBA, Javascript, Python or Java then you should be fine.

Wave Test Manager plugins are implemented in a simple scripting language called Groovy.  Although it’s a strange name it is widely known, most especially because it is nearly entirely based on Java, but with a greatly simplified syntax to allow much easier use in scripting environments.   If you don’t know Groovy, don’t worry – the use you need to make of it is quite minimal to achieve a lot and there are plenty of examples that are easy to find for how to do a huge array of tasks.

Step 1 – Locate or Create the Plugins Directory
The first step in creating a Wave Test Manager plugin is to locate your Wave Test Manager data directory.   This depends a bit on which version of Windows you are using, but a sure fire way is to use the shortcut in the Start Menu for the Scripts folder.  This is created by the WTM installer and is handily in exactly the directory we are looking for.  So open this folder using the Start Menu and then navigate one level up in Windows Explorer.   Here you need to create a directory called “plugins”.  Here is how it looks in my test computer:

image

Step 2 – Create a Folder for your Plugin
The folder for your plugin goes inside the plugins folder that you just created above.  You can name it whatever you like – ideally make it match the name or function of your plugin.  Here I’ve called mine “failurelog”:

image


Step 3 – Create a Test Plugin Script
Here you need to use a text editor such as Notepad.   For now the script we make will be extremely simple – we’re going to do nothing more than display a message when Wave Test Manager starts to see if your plugin is working.   Here is what you need to put in your text file:

import javax.swing.JOptionPane
JOptionPane.showMessageDialog(null,
"If you are seeing this then your plugin is working!", 
"Plugin Test", JOptionPane.INFORMATION_MESSAGE);      


This script displays a message box when Wave Test Manager starts up.  Save it as the file “plugin.groovy” inside your plugin folder (in my case, inside the “failurelog” directory).  Then if you restart Wave Test Manager you should see something like this:

image

If this doesn’t work, check your steps above before proceeding!   Make sure you saved your file as “plugin.groovy” and not as “plugin.groovy.txt” (what Notepad will do by default).

Step 4 – Create the Real Script

Here is our first real script below:

   1:  //
   2:  // A WTM plugin that records a log of script failures in a file
   3:  //
   4:  events.connect(scriptRunnerService,"testPlanComplete") { testPlan->
   5:   
   6:     def f = new File("failures.log")
   7:     if(!f.exists())
   8:       f.text = "" 
   9:   
  10:    def date = new Date()
  11:    for(s in testPlan.scripts) {
  12:      if(s.result == "Failures") {  
  13:        f.text += "${date} : script ${s.testPlanScript.scriptFile.name} failed\n"  
  14:      }
  15:    }
  16:  }

Here you can see on line 4 the plugin connecting to events belonging to the scriptRunnerService inside Wave Test Manager to receive notifications when a Test Plan is complete.   On line 6, 7 and 8 it creates a file to store script failures in, initializing it with blank text if it does not already exist (line 8).    Then on line 11 it iterates through the scripts in the Test Plan that completed and checks each one to see if it failed (line 12).  When a script has failed, it appends a line to the the failure log file showing details about the failure.

Step 5 – Add Rows to a Database

Just to cap off the example, we make an extension to the above script to make it record all script failures in a database.   Thanks to Java’s built in database interfaces (JDBC) the script we need works for any database, as long as you add the correct libraries (jar files) and connect string in the code below.  For this simple example we’ll use MySQL.

  1. Firstly, since we are connecting to MySQL you will have to add the MySQL database driver jar file.  Normally you could do that by making a “lib” directory inside your plugin’s directory, but due to the special way in which JDBC drivers are initialized in this case you need to place it in a different directory that already exists.  Look for the webapps/bbq/WEB-INF/lib directory which should already exist and  then download the jar file from the MySQL web site and add it there.
  2. To use this example you’ll need to create a MySQL database and add a failure log table to it.  The failurelog table can be created by executing the following statement:
    create table failure ( id int(9) unsigned auto_increment,  script   varchar(255),  date timestamp, primary key (id) );
  3. Now we make three changes to our script as shown below:  import the Groovy SQL module at the top (line 4), add a connect statement to connect to the database (line 8), and replace the line that appends to the file with a line that instead, inserts into the database (line 16).

   1:  //
   2:  // A WTM plugin that records a log of script failures in a database
   3:  //
   4:  import groovy.sql.Sql
   5:   
   6:  events.connect(scriptRunnerService, "testPlanComplete") { testPlan ->
   7:   
   8:    def sql = Sql.newInstance(
   9:        "jdbc:mysql://localhost:3306/failurelog",  // database
  10:        "root",  // user name
  11:        "",  // password
  12:        "com.mysql.jdbc.Driver")
  13:    def date = new Date()
  14:    for(s in testPlan.scripts) {
  15:      if(s.result == "Failures") {  
  16:        sql.execute("insert into failure values (NULL, ${s.testPlanScript.scriptFile.name}, ${date})")
  17:      }
  18:    }
  19:    sql.close()
  20:  }

Conclusion

As you can see above, Wave Test Manager plugins make it really easy to integrate WTM and Badboy into your organization’s ecosystem.   With just a few lines of code you can do some very powerful things.   As always I’m interested to hear what uses Wave Test Manager users are finding for plugins and, of course, what challenges you are having and what we can do to make plugins better and easier to develop.   And of course, if you think you’ve created something that can be useful for others, drop us a line at support@badboy.com.au and we would love to post it on our Plugins Page.

Wednesday, June 15, 2011

Documenting the Kindness of Strangers!

One of the wonderful things about being a small business making a widely used product like Badboy is the passion and loyalty of users who love the product so much that they put in their own time to contribute to it.   Whether it’s on the forum, blog posts or – in this case – not just one but two separate users who donated versions of the Badboy Documentation in Microsoft Word format for others to use.   In the end I had to choose which version to post on the web site – even though both were extremely professional and high quality and show an amazing level of care and attention to detail.   It is thanks to these two people that you can now find the Badboy Help downloadable in either PDF or Word format.  
I know from the frequent requests how useful it is to have the documentation in a format that is both printable and readable off line.   Thank you to the amazing Badboy users who made this possible!

Monday, March 28, 2011

Documenting Badboy Scripts

Today I thought I would write about something that many people would rather avoid, but which can be essential in larger projects – making documentation that explains what your test scripts are doing and how and why they work.

Why Write Documentation?

For someone starting out or working on a relatively simple project, making documentation can seem like a bit of a nicety rather than a “must have”.   However, as anyone who engages in a non-trivial amount of quality assurance work knows, your library of automated test scripts soon represents a significant value that needs to be preserved and understood by more people than just yourself, or even your team.   Sometimes it even needs broader exposure to people who understand automated testing tools.

This problem was brought home to me recently very sharply when I engaged in a project to build FDA regulated software for managing medical images and health records.   All of a sudden the problem of correctly documenting QA assets goes from being a nicety to a necessity mandated by law.   In this kind of situation it is not good enough that tests exist – they have to be “seen” to exist, traced to requirements and even be able to be understood by external third parties such as an auditor.  During this project several lesser known features of Badboy proved extremely useful for helping to keep these problems under control and, to my surprise, they turned out to be useful beyond just the purpose of automated testing and as a general library of information about how to navigate and use the web application.

Adding Documentation to Your Scripts

Every item in a Badboy script automatically gets a field for documentation text that can be used to write notes about what it does, how it works, which requirements it is testing or what bugs are related to it.  This text is completely free form and supports simple formatting such as bold, underline, italics and bullet lists.   It also supports adding links to external content so that you can easily open up any web based resource that might be related such as a wiki, project management software or a bug tracker straight from Badboy.
There are several shortcuts for adding documentation that make it a bit easier.   The simplest is just to select the item in the Script Tree and press Ctrl+Shift+D.  This will pop you straight into the Documentation section of the item ready to start typing.  From there you can type to your heart’s content:

image
To create links in your text, select the text and then press Ctrl+K and a small window will pop up to let you enter the URL that you want to point the link at.

Using Documentation

Once you’ve added some documentation, Badboy uses that documentation in several ways.  The most obvious is that it shows the documentation in the Summary pane inside Badboy so that it’s always there for quick reference as you scan your script:

image
A much more interesting use of the documentation comes, however, when you Export it using Badboy’s Export HTML Documentation feature which is found in the “Save As” menu.

image
Upon exporting like this we get a nicely formatted report showing all of our Steps with excerpts from the documentation included for reference:
image

This is a nice simple HTML document that you can save on your hard drive, email to your boss, or – perish the thought – even print out and put in a filing cabinet!

An intriguing about this kind of export is that not only is it nice documentation about the test, but (assuming you documented it well), it contains most of the essential information for a manual tester to reproduce the test.   This means that you can grab any test you like, export documentation, print it out, and you have in your hand something that a human being can process to perform the test.    This raises a rather interesting possibility of reversing the usual process of writing test specifications – why write a test plan, then implement it with Badboy, when you could do it the other way?  Record your test, add documentation and export it and there you have your test specification!

As counter-intuitive as it seems, there are lots of reasons to like having human readable and manually executable versions of your automated test scripts.    The simple fact is, as useful as automated tests are, they are never as good as a real person carefully looking at the page and assessing it.    Of course you want to offload as much of your manual testing into automated means as you can – but when you need to look deeper – whether it is because things are failing and you need to know why, or whether it is because you just want to do a “final” pass check, having a way to export your test sequence in a human readable form is extremely powerful.

In Conclusion …

These documentation features of Badboy are, in my opinion, greatly under utilized.   The world would be a better place if more people made more documentation of their test scripts.   I would love to hear via the Badboy Feedback Page or comments how usable people find the documentation features of Badboy to be and what we could do to make them even easier to work with.

Sunday, March 20, 2011

Mobile Access to Wave Test Manager

If you have ever tried to use Wave Test Manager from a smart phone such as an iPhone or Android device you will probably have noticed that while it’s functional it is far from optimized for the mobile experience.  Probably it comes up looking something like this:
image
Fortunately there is some good news for smart phone junkies who are also Wave Test Manager users – the next version of Wave Test Manager (1.1) will ship with an optional, mobile optimized interface that you can use to view and manage your tests.  The mobile optimized version of this page is far more finger friendly and usable:

image

Tapping on on one of the Test Plans will bring you a nice graphical view of the Test Plan showing how it has been running and more options:

image

Not only is this interface much easier to use from a mobile device, it is also much faster due to being built to take advantage of HTML5 Offline Caching abilities of phone browsers1.    This even lets you browse your tests if you don’t have a network connection (although it won’t get the latest status from the server unless you’re connected, obviously).   The pages are also designed to work well when you save shortcuts to them to the home screen – in which case they will get nice shortcut icons, use the whole screen and behave similarly to native apps in other ways.

These interfaces are still under development but you can start using an early version which will be included in the next beta for Wave Test manager 1.1 quite soon.   For now the focus is iPhone and Android compatibility, but if you use different devices, tell us at the Feedback Page so we know where the demand is for support on other platforms.

1 Unfortunately the latest iOS update broke some capabilities of web apps on iOS devices such as the iPhone and iPad.   Hopefully Apple will fix these problems soon.

Monday, March 14, 2011

Using Badboy and Ant Together

As more and more people use Badboy and integrate it into their development and testing workflows a question that has been becoming more and more frequent has been “how do I build Badboy into my automated build scripts so that it can run as part of my normal build process?”   This question becomes even more important when you attempt into integrate Badboy into a continuous integration environment such as Hudson (or Jenkins, as it is now called).

The good news is that with the latest versions of Badboy (as of this writing, Badboy 2.1.1 and up) using Badboy with one build tool -  Ant – just got a lot easier.    In fact, even if you use a different build tool, new features in Badboy 2.1.1 will help you better integrate Badboy into your workflows because there are new command line options for Badboy that significantly streamline the process.   However this post will focus on support for Ant as this extremely common build tool now has direct support via some very handy Ant plugin tasks that make it extremely simple to integrate Badboy.

Getting Started

To get started with using the Badboy Ant plugin you first need to download the jar file containing the plugin tasks and place them in your Ant “lib” directory.  For example, if you have Ant installed at “C:\programs\apache-ant-1.8.2”, then place the jar file in “C:\programs\apache-ant-1.8.2\lib”.   There are other ways to do it but this is the simplest and most tried and true method.  
Once you have the jar file installed you have to do just one more thing to your build file before you can use the Badboy Ant tasks – just add two taskdefs at the top of your build file (right after the project tag) like so:
<taskdef name="runscript" classname="com.badboy.ant.RunScriptTask"/>
<taskdef name="runTestPlan" classname="com.badboy.ant.WTMTask"/>

Once you’ve done this you’re ready to use the tags just like any other inside your build file.

Using the RunScript Tag


Once you’ve declared the tasks putting the tags to use is easy.  Running a simple script using Badboy on the same computer as the build script is running is, in fact, just a single tag:
<runscript script="test.bb"/>

This will fire up Badboy and cause it to run the script from beginning to end without stopping.  All the niceties such as auto-dismissing popups, Javascript errors or other problems that might cause the script to halt midstream are taken care of for you.   In addition, if the script has errors or assertion failures it will cause the build to fail and abort – just as if you had introduced a compile error or a normal unit test failure!

If you prefer different behavior you can stop the build from failing when a script fails by simply adding a failOnError attribute and setting it to false:

<runscript script="test.bb" failOnError="false"/>

If you want to pass variables to your script then you can easily add them as child elements:

<runscript script="test.bb">
  <var name="foo">bar</var>
  <var name="cat">dog</var>
</runscript>

Using the Wave Test Manager Tag

The RunScript tag is simple and easy to use, but it has one glaring disadvantage: it runs the script on the same computer as that where the build script itself is running. This may not be a problem in some situations but there are a range of scenarios where it just doesn’t work at all – perhaps you like to do useful work while your build is running and don’t like a giant Badboy window popping up in the middle of the build, or perhaps you need to run multiple builds concurrently and having many Badboy instances popping up together causes problems – or perhaps you want to support developers who aren’t using Windows and can’t run Badboy at all. All these issues are easily dealt with using Wave Test Manager and the runTestPlan task. Before you can do this, of course, you need to install Wave Test Manager somewhere – anywhere will do, including on the same computer as the build is running on. For the purpose of this post, I’ll assume that you have the server running on a server called “wave.acme.com” – but it could be on any computer, the build script doesn’t care. Assuming you’ve got a live Wave Test Manager server, running your script is once again, just a single tag:

<runtestplan name="Example" server="wave.acme.com:8030"/>

There are some differences to how this task works to the previous one that need to be noted. Firstly, with Wave Test Manager, you don’t run script files directly but rather Test Plans which are essentially just a list of scripts with preconfigured settings for how they should run. So instead of specifying a script file name now you are specifying a Test Plan name that matches the name one of your Test Plans in Wave Test Manager.

Another important difference is that instead of pausing while the test plan executes, the build will, by default, continue on in parallel while the test executes, and the build script will not fail if the script has errors or assertion failures. The reasoning here is that a script scheduled to run in Wave Test Manager is simply placed in queue and there is not a guarantee that it will begin executing straight away – that depends on the availability of a Badboy Agent to run the script. Wave Test Manager also supports its own notification mechanisms to tell you about script failures so you don’t necessarily need a build failure to tell you about it.

However if you prefer your build script to wait while the script executes and fail when it fails, you can add attributes to make that behaviour occur:
<runtestplan name="Example" server="wave.acme.com:8030" waitFor="true" failOnError="true"/>

Now your build will patiently wait until the script finishes executing and it will fail the build if any of the scripts in the Test Plan has errors or assertion failures.

Summary

Hopefully these Ant tasks will make integrating Badboy into automated builds much easier and help Badboy users streamline execution of their test scripts. Of course, there are many other build tools out there and many more features that could be added to these tags as well, so let us know what you want at the Badboy Feedback Page and we’ll endeavour to oblige!