Wednesday, 16 September 2009

Introduction to QTP with Flex

Over the last couple of years Flex has brought the Flash platform in to a new era of RIAs. Flex is now being used for a range of projects of all scales, from simple animated websites to complex systems providing real time data, which have never previously been seen in the browser. As larger applications are developed with Flex, the challenges of testing large systems which are traditionally only associated with desktop applications arise. Automated testing claims to provide cost, resource and time reduction to the testing teams of large projects. These are my thoughts and findings on the matter from the perspective of a Flex developer on a large Flex project.

QTP

Quick Test Pro (QTP) is an automated test tool used for testing all sorts of applications written in a plethora of languages, from html web pages to .NET applications. The product is produced by HP ( and formally Mercury ) and provides integration with the test and defect tracking tool Quality Centre. The Flex SDK is bundled with the add-ins required to build a QTP testable Flex application. Two steps are needed to automate a Flex application, the installation of the QTP Flex plugin and compilation of your Flex application with the QTP/Automation SWCs, both of which are provided with the Flex SDK. In theory, from this point on you should be able to record and run scripts for testing a Flex application with QTP. The test scripts are in VBScript and rely heavily on the concept of the QTP object repository.
QTP builds up it’s own view of the hierarchy of components in your application, known as the object repository. This is build at the time the scripts are recorded. The VBScripts then run against this object repository to locate objects in your Flex application and dispatch events on them to mimic user interaction with the controls. The QTP SWCs load in XML files which describe the components they can interact with, this includes the properties and functions of the components. A standard TEAFlex.xml file describes the standard Flex components and a TEAFlexCustom.xml file is used to describe your custom components.

How it All Works

QTP can only connect to applications running in Internet Explorer (watch out for this if you cannot get QTP to connect to your application) as it relies on using ActiveX to communicate with Flash Player. Flex attaches a delegate object to every component so that automated tools can communicate with the objects. A delegate will be automatically attached to every object. The use of delegates is a way of removing automation code from production code so that automation code is not build into non-automation builds of an application, hence keeping the SWC/SWF size down. These delegates add event listeners to your objects so that they can inform the automation tool of the action that was performed so that it can be recorded. The standard automation delegates are provided in automation.swc, but you are free to add your own for custom components or if you wish to deal with a situation that the standard implementations do not support.
When you replay an automation script it will find the object in question, by traversing the hierarchy of objects in your application (which it gets from its object repository). It will then recreate the event which was recorded and use the delegate to dispatch the event on the object to mimic the dispatching of the event as if the user had interacted with the component.

Pros

The advantages of automating the testing of large systems is quite clear. Testers can spend less time regression testing every time there is a release candidate ( or even on a nightly build basis) and can spend more time building up thorough test suites for new functionality. Regression testing is a tedious task and despite a developer saying “it’s a small change and won’t affect any other part of the application”, should always be carried out! The overhead of this on a large application can be very high, sometimes a matter of days, to a week. The knock on effect of this is making release cycles longer and increasing the time to apply urgent patches. The theory is that the effort put into writing manual test cases can instead be put into scripting automated test cases and from that point on, minimal effort is required to repeatedly run these tests. This does, of course, require testers with these skills, and programming scripts in VBScript is not necessarily a skill that most testers have.

Cons

In my experience, the cost of maintaining these scripts is high. QTP/Flex rely heavily on the layout of the application to locate objects and on their IDs if a specific automation name is not applied. The problem with this is that if a developer changes the ID of an object of moves it in the view hierarchy the scripts will break. Even changing the type of a container will cause a script which uses the child of that container to fail as it cannot locate the required child. On a reasonable stable code base this may not cause a huge problem but on a project at the peak of development these changes can break huge amounts of scripts. For example, changing the type of a container directly under the main application from a Canvas to a VBox will most likely cause every script to break. The only way around this is to ensure that scripts are written in a way to avoid this, for example using variables to hold high level components, so that their type/automationName only needs changing in one place, at the variable declaration. Note that the Object Repository will also need changing, but this is only a change in one place. These sort of changes will lead to false negatives, tests failing not because the application’s functionality has regressed but because the test script is no longer in line with the application.

The speed of execution of these scripts is not particularly fast, I would expect a standard length test including application startup to take between 20-60 seconds. Ideally the application should be reloaded between every test so that any side effects of the first test do not affect the next test. For a large test suite I have found that automated testing is not as quick as people would expect, but it is still quicker than a human performing the test, although somewhat less intelligent and more brittle.

Automated testing is good at testing certain types of tests, such as those involving data entry. Testing for regressions in the appearance of the application however cannot be performed, this still requires the watchful eye of a tester. An example test may be that once an online store order has been placed using a Flex UI, confirm the order has been correctly entered into a database and is now in another order fulfilment system. Once again, this is something that QTP is not going to be able to help you with, you need a tester to access these down-stream systems to check that everything has worked.

One particular feature that I have found to be missing from the Flex integration with QTP, is the ability to “sync” on objects. When interacting with most platforms QTP can wait until an object is available before trying to interact with it, for example when you are waiting for a popup to appear, this is know as “syncing” on an object. The lack of this with Flex means that you have to hard code waits of set time periods, which can obviously fail based on the indeterminate time that some operations may take. The alternative is to write a slightly more complex function that continuously checks for an object’s presence and only interacts with it when it is present. Both methods add bulk to your scripts and add complexity for testers writing the scripts who shouldn’t be too concerned with this problem.

4 comments:

  1. First of all. Thanks very much for your useful post.

    I just came across your blog and wanted to drop you a note telling you how impressed I was with the information you have posted here.

    Please let me introduce you some info related to this post and I hope that it is useful for software testing community.

    There is a good Software Testing resource site, Have alook

    http://SoftwareTestingNet.com

    simi

    ReplyDelete
  2. I need more information on how to make script faster for Flex application

    ReplyDelete
  3. Hi Nirja,
    Hopefully my latest post "Debugging Flex/QTP Issues" might help you with this. It describes how to improve the performance by selectively adding more levels into the automation visible hierarchy of containers.
    Ben

    ReplyDelete
  4. Hi,
    Thanks for your great post.

    I have installed HP QTP 10 and Flex 4 plugin and its working fine except for one object.

    I am using Apple Accordion menu to display menu list. And on mouse over of menu header it is going to expand menu items. But in QTP it is not capturing menu header object.

    Menu header is used as headerItemRenderer in Accordion.

    I have created delegate for header but its giving the same result :(

    Will you please help me to solve this problem?

    Thanks in advance..

    Regards,
    Pradip Jadhav

    ReplyDelete