www.google.com used to be a simple page

For some time WebTest users started to report that even simple examples from the demo tests available when creating a new project failed due to JS errors like this:

As this test used to work, this means that Google's start page isn't so simple any more that it used to be. It was predictable that this page would change one day but it is quite embarrassing as we use this kind of tests to show how easy and powerful WebTest is even for sites using a lot of JS stuff.

www.google.com has really a JS error!

HtmlUnit's JS support is quite good (and improved continuously), but of course not (yet ;-) ) as good as the one of "real" browsers, therefore I've started to search for the cause of the problem in HtmlUnit, looking for missing/incorrect implementation of some JS functionalities. At the end I've found that the problem occurred only when setting directly the value attribute of the field rather than calling the type(...) method on it because keypress handlers were being used. This means that WebTest should probably use type(...) rather than setting the field's value directly.

But this means as well that Google's home page relies on the keypress handlers and that the JavaScript code doesn't work correctly if these handlers are not called like in following scenario:

1. - go to http://www.google.com/ncr (to be sure to have the English version of the page)
2. - right mouse click in the search field to paste some value to search for
3. - click the "I'm Feeling Lucky" button

The results are quite similar to the JS error obtained by WebTest, no matter if you use Firefox or Internet Explorer (perhaps does it work correctly with Chrome ;-) ):

Rather painful for Google than for WebTest!

OK, this error is nothing dramatical but for an Internet giant like Google that has plenty of talented engineers (and ..., and..., and .... You perhaps know better than I what the web means for Google), I find quite painful to have such an error on its main page!

The error scenario is not particularly complicated and such a case should surely be covered by automated tests. Perhaps is it a question of inadequate tools? Google seems to use intensively Selenium ;-) .

Concerning WebTest, I'll start the discussion on the mailing list to decide what is the best way to follow: setting directly the field value, typing it, or having both possibilities.

[slideshare id=569950&doc=webtest1-1219754182328161-8&w=425]

There is no time limitation (according to Michael) therefore I plan to show different new demos of existing features as well as of future ones that should trouble / motivate / question the attendees.

There will be a Lightning Talk on Selenium by Simon Tiffert before my presentation. I think that this will be quite interesting as well because Simon has far more experience with Selenium than nearly all persons blogging about this tool.

1) The Groovy compiler can be used to generate 'Java Bytecode', which can be integrated with any other java module/project.

2) Most of the Java code is also a valid syntax for Groovy.

3) Groovy is dynamically typed, whereas Java is static and strongly typed. Dynamically typed languages have lesser code, but more execution time.

4) Groovy also provides special XML processing through the classes XmlParser, XmlSlurper, DOMCategory.

Apart from these, Groovy has features like 'Dynamic Typing', 'Closures' and 'builders'. Also, it's pretty useful for testing as it supports unit-testing and mocking out-of-the-box.
After doing a bit of reading and coding some basic scripts, I tried to code a Groovy solution for the Java code that I wrote for solving the "Google's Billboard Puzzle" (from my other blog). It's for calculating "Euler's Number (e)" , which is pretty basic and simple code. Here's the code:

BigDecimal e = new BigDecimal(1.0G)
BigDecimal temp = new BigDecimal(1.0G)

for(i in 1..100) {
temp *= i
e += new BigDecimal("1").divide(temp, new java.math.MathContext(10000))
}

println("e : " + e)
println("length of e calculated : " + (e+"").length())

As you can see, the 'Groovy' script also has 'Java' code in it (and is valid). Since, Groovy is a dynamically typed language, it took more time than the Java code took, to produce the solution.

Here's one more script, which outputs the 'hashcode' of a string:

def str = "abc"
int hashcode = 1;
int len = str.length()

for(i in 0..(len-1)) {
hashcode = 31*hashcode + str.charAt(i)
}

println("hashcode : " + hashcode)

The above code is a naive approach to calculate the 'hashcode' and is not a cryptographically secure function. Its based on the formula "s[0] * 31^{n − 1} + s[1] * 31^{n − 2} + ... + s[n − 1]", which is actually 'Rabin-Karp Algorithm (RK)'. One might be wondering why there in '31' in that Math expression. Its because Java's int is '32' bit, in length.

Groovy can also be used to test web applications, webservices, et al. Groovy can be used to test webservices in the following way:

* Act as a normal webservice client
* Use 'WebTest'
* Use 'SoapUI'

Ok..I decided to code a webservice client and googled for 'Free Publicly Available Webservices' and found a couple of websites with publicly available webservices. I picked the CDYNE''s 'Profanity Filter' webservice (which filters words that are considered profanity) and provided the WSDL url to the Groovy script. Here's the script (minimalistic):

import groovyx.net.ws.WSClient

def proxy = new WSClient("http://ws.cdyne.com/ProfanityWS/Profanity.asmx?wsdl", this.class.classLoader)

def result = proxy.SimpleProfanityFilter("You crappy bugga!")
println(result)

For the above script to run on 'Groovy Console', you have to copy the 'groovyws' jar file into Groovy's 'lib' directory.

But it was not that easy to realize my task due to the lack of good documentation and finding the correct xpath expression.

The following code snippet grabs the text value of a text input field with the id surname and saves it in the ant property xyz.

def testSomething() {
   webtest('Some description') {

        // saving initial value
        storeXPath(xpath: "id('surname')/@value", property: "xyz")

	// restore initial value
	setInputField(name: "surname", value: "#{xyz}")

    }
}

Pretty simple in the end, but it took a while till I came to this. Maybe it helps someone.

The Ultimate Personality Test - Tickle Personality Tests > Results

http://web.tickle.com/personality/result.jsp

Skärmurklipp gjort: 2008-05-08; 22:35

Helt fel, säger jag! Jag har ju inte ens besökt Stockholm i 18 år, inte heller Sverige för den delen. Då jag bodde i Stockholm i början av 1980-talet, var det närmaste till Södertälje jag någonsin hamnade i Midsommarkransen, i länet dock Västra Flemingsberg.

Så det så!

via Birger Schlaug

This is particularly true for functional tests of web applications. WebTest is known to be very fast compared to other functional test tools, nevertheless when your test suite grows or simply when you want to quickly receive feedback after a commit, you often feel that tests are too slow. A new experimental feature of WebTest allows to specify the number of threads that should be used for the tests what can bring enormous speed improvements without modification of the tests.

Simply configure the number of threads

Rather than just calling
[sourcecode language='java']
ant[/sourcecode]
you just specify the number of worker threads
[sourcecode language='java']
ant -Dwt.parallel.nbWorkers=20[/sourcecode]
The optimal number of worker threads will depend from the usage.

How it works: enqueue rather than execute

The idea is quite simple and the implementation totally non-intrusive. In fact this is a shame that nobody has had the idea previously (including you! ;-)). This can be simply explained with a few lines of code.

WebTest is based on Ant what means that the task mapped to <webtest> contains something like this:
[sourcecode language='java']
class WebTestTask extends Task
{
void execute()
{
// do the WebTest
}
}[/sourcecode]
For the parallel execution we map <webtest> to a new class that looks like this:
[sourcecode language='java']
class WebTestTaskParallel extends WebTestTask
{
void execute()
{
workQueue.add this
}

void executeReally()
{
super.execute()
}
}[/sourcecode]
Once this is done, Ant can do its job normally. When it calls the execute() method of the task, the instance adds itself to a queue and the execute() method returns allowing Ant to continue its normal execution. A set of worker threads look at the queue, calling executeReally() on the WebTestTask instances to really run the tests. That's nearly everything, the rest is just synchronization glue code.

Improvements from 0% to 50% and more

I've seen many cases where using a few threads allows to gain over 50% execution speed. In fact this will depend from the capacity of the application under test and how it can handle the increased charge. With a server - not necessary fast - that smoothly handles the charge generated by numerous worker threads, I can imagine tests suites that execute approximatively in the time of the slowest test. On the other side I've already seen test suites that take more time to execute with many worker threads when the AUT runs on the same physical machine than the tests themselves. It's not really surprising: if the tests need more "power" to execute, then there is less for the application.

Limitations

With a recent WebTest build, you just have to specify the number of worker threads to start using this new feature. You many need to refactor your tests to include processing made after a </webtest> within the test otherwise this will be done before the tests itself gets executed and may be the cause of problems. Finally current implementation of this feature doesn't contain any facility to manage dependencies between tests.

Towards functional load testing?

Tests with large numbers of worker threads have shown that WebTest scales quite well: both memory and CPU usage stay correct. This opens a new perspective for WebTest: the usage as load testing tool. In fact it should probably be bound to something like JMeter but I see great advantages in WebTest usage compared to low level http processing as usually done by load testing tools, particularly in test maintainability, AJAX testing or for the correctness of the simulated scenario.

Update (Feb 6): a former client told me that they just started to use a slightly modified version of this feature I prepared them for some time (allowing to manage some dependencies between tests). The results are even better as expected: over 75% saved time! With 8 working threads the tests complete in less than 25% of the original time.

In the discussion in Grails-user mailing list following my previous post "WebTest vs Selenium", Marc Palmer and James Page requested the creation of a kind of meta DSL in Groovy for automated functional tests of web applications.

The idea was to provide a DSL allowing to write functional tests in a tool agnostic way and to run them with WebTest as often as needed because it is fast and for instance once each night using Selenium because it uses a real browser but is quite slow.

I'm still not fully convinced on the utility of such a feature because WebTest is so good ;-) but I think that there is no need for a new DSL: it already exists! The AntBuilder allows WebTest to have a really nice syntax in Groovy and this could be simply reused for other "target tools" thanks to Ant.

Ant theory

When you write something like this (the examples in this post use Groovy AntBuilder but the same would apply for "pure" Ant in XML format):

ant.webtest(name: "a simple test)
{
  invoke "http://webtest.canoo.com"
  verifyTitle "WebTest website"
  clickLink "Manual"
  verifyXPath(xpath: "count(id('navigation-top')//li)", text: "5",
      description: "check the number of top menu elements")
  ...
}

this looks like WebTest but this is not WebTest as long as you haven't configured Ant with for instance something like

ant.project.addTaskDefinition("webtest", com.canoo.webtest.ant.WebTestTask)
ant.project.addTaskDefinition("invoke", com.canoo.webtest.steps.request.Invoke)
...

This has 2 consequences:
- it is possible to configure other tasks than WebTest's ones for "WebTest steps"
- it is possible to inspect the "parsed" tasks tree

This means that it wouldn't be too difficult to run a "WebTest" test with an other automated test tool like for instance Selenium (as long as the tests don't use any WebTest feature for which no Selenium equivalent exists). Let's see how this could be done.

First solution: redefine "WebTest steps"

The first approach consists in redefining the "WebTest steps" to provide an alternative implementation for each WebTest step before executing the test. This could look like following for verifyTitle:

import org.apache.tools.ant.*class SeleniumVerifyTitle extends Task
{
  String text
  void execute()
  {
    if (text != selenium.title)
      throw new BuildException("Wrong title: expected $text, got ${selenium.title}")
  }
  def getSelenium()
  {
    project.references.'selenium' // assuming that test start placed it there
  }
}
ant.project.taskDefinitions["verifyTitle"] = SeleniumVerifyTitle

verifyTitle is quite simple, for other steps it would be more tricky or even impossible (like the pdf or email steps) to write a Selenium equivalent.

Generate script from Ant tree

The second approach consist in the generation of a script from the Ant structure. This has the "advantage" that the
generated script doesn't necessarily have to be run on the Java Virtual Machine.

class WebTest2SeleniumRubyConverter extends WebTest
{
  File targetFile // additional attribute to Ant task: the file to write in
  private converters = [
    'invoke': { "open \"${it.attributeMap.url}\"" },
    'verifyTitle': { "assert_equal \"${it.attributeMap.text}\", @selenium.get_title" },
    'clickLink': { "@selenium.click \"link=${it.attributeMap.label}\"" },
    'verifyXPath': { "# ?? I haven't found an example" },
    'pdfVerifyTitle': { "# skipped because not supported: pdfVerifyTitle ${it.attributeMap.title}" },
    ...
  ]
  def execute()
  {
    def rubySteps = runtimeConfigurableWrapper.children.collect { converters[it.elementTag](it) }
    def scriptTemplate = """
require 'test/unit'
require 'selenium'
class ExampleTest < Test::Unit::TestCase
include SeleniumHelper

def setup
@selenium = Selenium::SeleniumDriver.new("localhost", 4444, "*firefox", "http://localhost", 10000);
@selenium.start
end

def teardown
@selenium.stop
end

def test_something
<% steps.each { %>
$it
<% }%>
end

end"""

    def engine = new groovy.text.SimpleTemplateEngine()
    def template = engine.createTemplate(scriptTemplate)
    targetFile.withWriter {
      it << template.make([steps: rubySteps])
    }
  }
}
ant.project.taskDefinitions["webTest"] = WebTest2SeleniumRubyConverter

NB: I don't have any experience in Ruby and have written the example code above only by looking at the samples on Selenium's website.

Naturally this is only an example and it is a bit more complicated to do it correctly (handle different combinations of task attributes, special characters, Ant macros, ...) but surely not so much.

Conclusion: not very complicated but does it makes sense?

These two examples shows that it wouldn't be complicated to "convert" simple WebTest tests to allow them to be executed with an other tool like Selenium (but other tools like for instance WebDriver could be a target too) as long as they use the subset of WebTest features that the target tool accepts. Of course WebTest features like its particularly rich reporting wouldn't be available either.

Personally I'm not really convinced of the utility of such a "generic DSL" due to WebTest's excellent quality and unless a new client (you? ;-)) really wants this feature I don't plan to work on it but I'm ready to provide technical assistance if someone wants to do the job.

I know that some of you are familiar with these two things and some of you are not. For this past month I have been using these plugins and these are also my favorites. They are really useful in terms of time and efficiency.

Let me explain these things to you.

XPath Checker is a Firefox Extension for testing XPath expressions interactively written by Brian Slesinsky. It is also like a regular expressions for web pages. XPath make it easy to extract content from deeply nested markup in a web page or an XML document. This would be useful in unit-testing code that generates XML or HTML.

Here are examples of xpath expressions:

Get the title of a page:
//title/text()

List all the images on a page:
//img

List the images that are inside a link:
//a//img

List the images that have alt tags:
//img[@alt]

List the images that don't have alt tags:
//img[not(@alt)]

Show all the alt tags:
//img/@alt

Show the href for every link:
//a/@href

Get an element with a particular CSS id:
//*[@id='mainContent']

REGULAR EXPRESSION tester is simply a tester for regular expressions on the fly. It is a plugin for your Eclipse especially you are dealing different patterns.

These two are small but powerful tools in programming. These are also my favorites and I wish you will find it useful and helpful than doing your own parsers in a weeks or in a months.

Features

• Test and search for regular expression
• Matches are colorized, for an easy visual clue
• Support for pattern flags (e.g. Pattern.DOTALL)
• LiveEval evaluates your regular expression while you are typing it, gives feedback on possible errors and shows any matches automatically
• LiveEval is supported for changes of the regular expression, the search text and the pattern flags
• 4 distinct match modes:
  • Find a sequence of characters
  • Match a complete text
  • Split text
  • Replace every occurence of the regex with a different string
    Replacing supports back references ($1,$2,...)
• LiveEval for match mode changes
• Context sensitive "Regular Expression Assist"
• Selective evaluation of expressions
• Bracket Matching
• Generation of string literals based on the regexp, e.g. "\(x\)" becomes "\\(x\\)"
• De-escape patterns in your code, e.g. \\(x\\) becomes \(x\)
• Improved "Clear Menu", choose which parts of the view you would like to get cleared every time you press the clear button
• Easy movement through matches: Choose "Previous Match" or "Next Match" and cycle through all matches found.
• Polished and accessible user interface, everything is reachable via keyboard

Content: (duration ~2'30'')
I- Do you know Google Calculator?
II- Do you know the dataDriven Ant task?
III- Testing Google Calculator with data driven WebTest

It should be interesting even for experienced WebTest users: the dataDriven task is so simple and powerful, and Google Calculator is a funny toy.

What does it bring? In fact nearly nothing.

It's just a small information displayed during test execution. It becomes funny when you start a few dozen tests in parallel because you nearly don't have time to follow each single test that get started or finished: it goes too fast. But this will be the subject of a future post once this feature is integrated in WebTest builds.

Marc Guillemot postou uma comparação entre os dois, e o seu resultado foi vitória do WebTest.

Particularmente, quando fiz meus testes, achei o WebTest muito lento/chato para fazer os testes quando comparado ao Selenium IDE.

The maintainability of automated tests depends primarily on the skills of the test authors, but different tools have different features that impact their efficiency. This blog post compares the features of two open source automated web testing tools: Canoo WebTest and Selenium.

A short introduction to the contenders:

Canoo WebTest is a free open source tool that has existed since 2001. It is written in pure Java and contains a set of Ant tasks that drive a simulated, faceless browser (originally HttpUnit, but for the last few years HtmlUnit).

Selenium is a free open source tool as well, created in 2004. It uses injected JavaScript to work within real browsers. Different components exists under the name Selenium: Selenium Core, Selenium RC, Selenium IDE (!), ... In this blog post I will only consider Selenium RC used with Selenium IDE on Firefox or Selenium HTA on Internet Explorer due to the limitations of the other possibilities.

Features comparison

To be clear, as a WebTest (and HtmlUnit) committer I'm undoubtedly biased. On the other hand, I have experience with huge functional test suites being developed and maintained over periods of years. Trying to be objective, I may overcompensate in the other direction and give Selenium too much credit. Of course I will diligently fix errors I may have in my Selenium understanding. But please read this post until the end before starting with criticisms ;-)

• Browser fidelity: WebTest 0 - 1 Selenium

This is probably the most overestimated characteristic of a web testing tool. Automated tests don't make manual testing useless because automated tests can't cover everything (at least for affordable costs). You still have to walk through your application (just think of everything you've checked just reading this: page layout, font size, font colors, ...).
The consequence is that an automated web testing tool's purpose is not to ensure that an application works "well" as it is not possible, but to detect most of the failures that could happen. This is a huge difference because it means that tests don't have necessary to run in a "real" browser.
Nevertheless the browser's real behaviour has to be approximated as closely as possible. HtmlUnit's JavaScript support has made impressive progress but it still doesn't (and will never) behave exactly like a normal browser.
Even though Selenium modifies the normal JavaScript execution of an application, it uses the browser itself and therefore is nearer to the standard behaviour of the browser.

• Reports: WebTest 1 - 0 Selenium

JUnit-like reports are far too limited for web test automation. This is probably something that you first see when you have reached a certain volume of tests. If the tests are successful, you don't need any report at all, but when some tests fail, you need the information to find as quickly as possible what is the failure cause and an error message is often not enough.
With comprehensive reports like those provided by WebTest, you don't have to debug your tests, just to analyse the reports. Furthermore it allows you to understand (and fix) the worst kind of bugs: those that don't occur systematically.

• Speed: WebTest 1 - 0 Selenium

Tests are never fast enough. Selenium is known not to be very fast and even slower on Internet Explorer (just read the mailing list) and seems to suffer from memory leaks. On the other side, WebTest is quite fast (see for instance this thread in Selenium Dev mailing list for a non representative test where Selenium took ~ 10 seconds and WebTest < 2s).
It's not surprising due to Selenium's architecture (3 tiers involved) and all the rendering that happens in browser. Even if HtmlUnit's HTML handling algorithms are not as good as the real browsers, WebTest has simply less to do and everything happens in the JVM.

• Integration in development process: WebTest 1 - 0 Selenium

WebTest is "just" Ant which means that it can directly be called from CruiseControl for instance or from each developer's workstation.
On the other side for Selenium you need a real browser with an own profile, a proxy - possibly on another computer if you want to test with IE and run the tests from a non Windows system.

• Scalability: WebTest 1 - 0 Selenium

For a large application (or set of applications) with a good functional test coverage your test suite(s) will rapidly grow and scalability may become an issue. WebTest scales far better than Selenium mostly because it's faster and because you can simply run many tests suites in parallel (just think of the hardware requirements and browser limitations to do that with Selenium).

• Capture JS errors: WebTest 1 - 0 Selenium

This is what surprises me most about experienced developers working with Selenium: they find it acceptable to ignore JS errors. Would you accept compilation errors in your program as long as your unit test pass? Surely not! But in fact this is exactly what you do with Selenium as it doesn't detect the javascript errors contained in your application (unless they directly impact the specific tests causing them to fail).

• Testing AJAX: WebTest 1 - 1 Selenium

Contrary to popular belief, you don't need to run your test as JavaScript inside a browser to test AJAX functionality. HtmlUnit and thus WebTest is just as well up to the task. It can even be considered superior as it allows better control over how to schedule the in-page requests making the unpredictable browser behavior predictable (see for instance my previous post).

• Beginner friendly: WebTest 0 - 1 Selenium

Beginners (as well as managers ;-)) better understand test automation of web applications, when they see what happens.

• Documentation: WebTest 1 - 0 Selenium

Extensive and up-to-date documentation is very important. A quick look at both web sites will show you that WebTest manual is clearly the winner. It should even be a negative point for Selenium as advice on what makes test suites maintainable is completely missing.

• Predictable behaviour: WebTest 1 - 0 Selenium

Should be a minimal requirement for a test tool, but if you look regularly at Selenium mailing lists or at different posts (like this one), this is not yet fully the case for Selenium.

• XPath support: WebTest 1 - 0 Selenium

WebTest currently uses Jaxen as XPath engine which means that XPath 1.0 is covered as well as some XPath 2 functions (do you know that starts-with is in XPath 1 spec but ends-with first appears in XPath 2?).
Additionally you can customize it to define your own XPath functions.
Selenium uses native XPath support when it's available (like in Firefox) and evaluates XPath expressions using JavaScript libraries otherwise (like in IE). This JS library is slow and many XPath expressions aren't interpreted correctly. Even in Firefox, the support is limited to XPath 1.0.

• Support for badly formatted HTML code: WebTest 0 - 1 Selenium

Browsers are able to cope with really badly formed HTML code and so does Selenium as a consequence. WebTest's parser (NekoHTML) is able to handle some malformations but not that much. Even though it is quite questionable to see it as a feature when your goal is to write your web application as well as possible, sometimes testers do not have access to the development resource to correct the source and just want to test functionality so I'll give this point to Selenium.

• Extensibility: WebTest 1 - 0 Selenium

Selenium accepts custom extensions but first this is cumbersome because the extensions have to be deployed in the target browser(s) and second interactions are limited as extension code executes in the browser and not in your test program.
In WebTest you have full control over the "browser" from within your tests which you can use to simply write global extensions as well as project or test specific ones.

• Data driven tests: WebTest 1 - 0 Selenium

No discussion, the dataDriven Ant task used with WebTest is simple and powerful!

• Multi-language support: WebTest 0 - 1 Selenium

Selenium RC has bindings in different languages (Java, Ruby, PHP, ...) whereas WebTest is bound to Ant which means XML or for instance Groovy with its nice AntBuilder (in fact any of the over 200 languages for the JVM could probably be used). I think that Selenium is missing a real specification language (please don't talk about Selenese!) like Ant is in this case for WebTest, but I need to give some points to Selenium...

• Internationalisation support: WebTest 1 - 0 Selenium

Using WebTest, you just need to put your language specific strings in property files and use Ant's built in property task to load the right resources before executing your tests.

Update 05.11.07:
Dan Fabulich correctly indicates that Selenium RC tests written in the same language than the AUT can directly use the application's i18n resource bundles. Therefore WebTest can't be seen as winner outside of the Java world.

• Support for non HTML content: WebTest 1 - 0 Selenium

HTML is only one of the content types used by a web application and it's a common need to have mixed content within the same applications with for instance a PDF file containing the invoice after the checkout. Selenium is limited to HTML content (+XML and text). On the other side WebTest has built in support to work with PDF documents and Excel files as well as Applets and Emails.

Conclusion:

Automated functional tests of web application should become as natural as unit tests. Some tests are better than no tests, no matter which tool is used. Selenium does a good job to introduce newcomers and has many advantages (besides the price) over its commercial model QTP. Nevertheless at least when the size of your suite grows, you should pay attention to your efficiency if you want to last and "the ROI on WebTest is many orders of magnitude higher than any other tool I've used" (Lisa Crispin, author of Testing Extreme Programming).

When comparing Selenium and WebTest, 3 categories of web application can be considered. First the applications that are supported both by WebTest and Selenium. This contains most of the applications. Then the ones that use browser features (mostly javascript) that are not yet supported by HtmlUnit. Due to HtmlUnit's awesome progress in JavaScript support, the size of this category continuously shrinks. The last category concerns applications that use for instance PDF documents or applets and for which Selenium has no support. In all cases where WebTest can be used it is far more efficient than Selenium to ensure the quality of web applications because it provides more feedback and takes less time (both to execute and to analyse results).

When I started writing this post I didn't expect that Selenium would get such a bad score. Comments are welcome to show which advantages of Selenium over WebTest I've missed!

Run tests in parallel

Selenium Grid's aims to allow running tests on multiple computer because:

• "Selenium remote control is quite slow at driving the browser".
• " You can only run a limited number of concurrent tests on the same remote control before seriously impacting its stability. Practically speaking, launching more than 6 browsers on the same Selenium Remote Control is not advisable. The limitations are even more drastic for Internet Explorer."

In other words: slow and not scalable. If I correctly understand the documentation, Selenium Grid will allow to run approximatively 6 x N tests in parallel, where N is the number of computers in the grid. Not really impressive.

An HtmlUnit user reported that he performed load testing with HtmUnit using 350 threads in the same VM and one WebClient instance per thread. For this purpose he allocated 512M of his 1Gig laptop to the VM and it worked very well. WebTest surely adds some overhead to HtmlUnit "pure" but, due to the indications of this user, I could imagine that it would be possible to run 500 threads of WebTest tests on one computer within a single VM. This would save just... 80 computers compared to Selenium Grid!

Parallelisation often limited

Selenium Grid's documentation says: "Selenium Grid cuts down on the time required to run a Selenium test suite to a fraction of the time that a single instance of Selenium instance would take to run". Sadly this isn't that easy. Not all tests can be run in parallel: as soon as you test the functionalities of registered users, you're often limited to a serial test execution (or a just a few parallel executions) when your application doesn't accept numerous simultaneous login of the same user and you don't have so much users available with the right settings for the tests. In this case only pure speed can help.

Grid computing is an exciting large area, perhaps is Selenium Grid only a side product of some more significant activity...

Content: (duration ~4 minutes)

1. Download WebTest from http://webtest.canoo.com
2. Check download and environment
3. Create a new WebTest project
4. Start the demo tests
5. Browse automatically opened reports
6. Look at the sources
7. Look at the documentation

This is quite simple and probably not so interesting for experimented WebTest users (or maybe a bit as the features of the utility webtest.xml are quite new).

My colleague Tomi Schütz and I are working on other screencasts to show WebTest features. Stay tuned, the next one should be quite impressive.

The idea is quite simple:

• start with the initial values F0 = 0 and F1 = 1
• invoke Google to compute the sum of these values
• extract the result from the html page, this is F2
• invoke Google to compute F1 + F2
• ...

Expressed as a WebTest this looks like:

<webtest name="Fibonacci">


<invoke url="http://www.google.com/ncr" description="Go to Google (in English)"/>
<verifyTitle text="Google" />
<setInputField name="q" value="WebTest" />
<clickButton label="Google Search" description="dummy search to cope only with one kind of submit button in the repeat"/>
<storeProperty name="previous" value="0" description="F0"/>
<storeProperty name="current" value="1" description="F1"/>
<repeat startCount="2" endCount="10">

<setInputField name="q" value="#{previous} + #{current}" />
<clickButton label="Search" />
<storeProperty name="previous" value="#{current}"/>
<storeXPath xpath="substring-after(//tr[.//img[@src='/images/calc_img.gif']]/td[3]//b/text(), '= ')" property="current" description="Store Fibonacci #{count}"/> <!-- ugly XPath but it's Google's fault!-->

</repeat>
<verifyProperty name="current" value="55" description="10th Fibonacci number"/>

</webtest>
(sorry for the formatting, WordPress.com is not the best blog hoster for code examples)

Running this script you get results like this: WebTest Fibonacci - Test Results

Ok, it was just for fun because, even if WebTest and Google are very fast, computing the 10th number of the Fibonacci sequence involved getting 11 html response pages and took 2496 ms here!

webtest is declarative and XML configuration file driven. If you like configuration management more than programming, you might like this format. All you need is edit the test definition and hit the run command, no compilation necessary. If you have well formed HTML and working JavaScript, then you should not have too much trouble running this. My initial simple test case works once I commented out the non-working and irrelevant JavaScript reference in the HTML. I tried to take another mini-step forward by attempting to "configure" a click to select one of the link items in a JavaScript widget. Webtest has 3 ways to click on a link, by ID, label, or xPath. The widget's link item doesn't have an ID, so I can't use that. I tried label and that didn't work. Then I tried xPath and that didn't work for me either. So, that second mini-step didn't go very far for me. Any one know an open source xPath validation tool or web site?

jWebUnit is one of the many unit testing frameworks based on jUnit/HtmlUnit. The interesting aspect of this framework is that it supports both HtmlUnit and Selenium test cases. jWebUnit uses similar programmatic model as jUnit, so you can stay comfortable within your favorite IDE. I was able to create a new project in NetBeans and leverage it's jUnit feature, by creating an unit test and rewrite that as a jWebUnit test case, while using NetBeans to drive the test execution. jWebUnit, or the underlying HtmlUnit engine appears more picky about JavaScript than Webtest, I had to start tweaking YUI libraries in an attempt to load a page, which shouldn't be something that I supposed to be doing anyway. Perhaps YUI engineers use a different unit testing framework. The test page actually loads without a major issue in Firefox.

Both frameworks do share one thing in common, both do not use actual browser instances to drive the testing. And that explains why they don't work as well as some of testing frameworks that built within or on the real browser instance because of behavior differences when processing JavaScript. At the same time, both may have one advantage over browser based testing framework, which is the ability for non-interactive testing through ant and is very useful for continuous integration servers such as CruiseControl. Again, I've only looked a few testing frameworks and let me know if you know a browser based unit testing framework that works with ant. Thanks.

Last week I had again a case showing the power of XPath. Colleagues had to write a WebTest script that should (among other steps) change the value of a select like this one:

<select name="mySelect">
<option value=""></option>
<option value="1">English</option>
<option value="2">French</option>
<option value="3">German</option>
</select>

(this blog doesn't allow to "display" this select field it as it would normally appear, but you have surely enough HTML knowledge to understand it)

No assumption could be made on the current value of the select, the test had just to select an other valid option (meaning not the first one).

First approach
The first approach could be to read the index of the selected option, increment it, perform a modulo operation and avoid the first option. More pragmatic would be to take the second one (here "English") if it is not already selected and the 3rd one (here "French") if "English" was selected.
These solutions work but require some logic that can only be expressed in a script.

More elegant solution with xpath
In fact you can simply formulate the test specification with an XPath like this one:

//select[@name='mySelect']/option[text() and . != ../option[@selected]]

This XPath just selects all the options of select field mySelect that have a "content" (i.e. not the first, empty one) and that are not currently selected.

------
NB: WebTest's setSelectField step takes automatically the first option when the xpath evaluation returns more than one result. For other test tools, you may need to select explicitly the first option like this (//select[@name='mySelect']/option[text() and . != ../option[@selected]])[1]

A lot of tools for automated testing of web app are available and have different features that may appear appealing in a first time. WebTest shines in what I call "professional test automation" because it allows to simply write test scripts that are reliable, fast, easy to run, cheap to maintain, helpful to quickly find errors and it scales fine and is easy to extend. Lisa Crispin, author of Testing Extreme Programming, described this perfectly: "the ROI on WebTest is many orders of magnitude higher than any other tool I've used".

This new motto is only the first step. Keep tuned!

We've already presented a WebTest tutorial at the previous edition of this conference last year in Frankfurt. This was really nice and I think that our tutorial was the most visited one. I hope that this year's edition will be as good as the last one.

I've discussed recently with a colleague having worked 2 years with QTP and now working since nearly one year with WebTest. I was interested to hear from someone with real QTP experience how the comparison would sound.

He misses 2 QTP features in WebTest:

• the possibility to combine tests of web applications with test in other Window applications (like a test starting with a host connection and going on in a browser)
• the possibility to "see" the responses during the test execution which helps beginners

but considers WebTest as really superior to build reliable, easy to maintain tests of web applications mostly because:

• no time lost for synchronisation configuration during the creation of tests
• native support for XPath
• easy to integrate in continuous integration
• reports provide comprehensive on failed tests
• easily extendible

This mainly confirms my first appreciation: QTP is a tool for people that don't test!

Perhaps should we go away from Open Source and change WebTest's license to sell it, let's say, just half the price of QTP ;-)

The error being located in a js file with "prototype" in its name, I thought that HtmlUnit still had problems with the prototype library although I was convinced that these had been fixed.

Exactly the same with Firefox
On the next day I was motivated to look at it more attentively and was motivated to get it fixed for the presentation we would have on the day after. Indeed the JAX website could be a good example to show some of WebTest features as all attendees should know it. Therefore I started to examine HtmlUnit's execution of the javascript with a debugger. Unfortunately I couldn't find anything and first after a while I've had a look at the JavaScript console of my Firefox:

WebTest was correct!
This was the explaination: WebTest had no problem and just reacted like Firefox providing precise information concerning the js error (including the line number and file name)!
The faulty line was
for (var i = 0; i < element.childNodes.length; i++) {
and both Firefox and WebTest gave correct message (a bit different) concerning the problem.

It's possible to tell WebTest to ignore JS errors but it's not recommanded as it doesn't help producing high quality Web Sites and http://www.jax.de is an example of WebSite that didn't get tested correctly (or with a tool that doesn't help discovering such failures).

Why a jump from 2.1 to 2.5?

This was mainly motivated by the very important changes performed in WebTest internals which have only small visible effects (smaller memory consumption, seamless integration of macros or external target calls within <steps>...</steps>).

What's special in this release?

Besides the internal changes, the main changes are probably:

• the upgrade to htmlunit 1.11 allowing a far better javascript support
• the new selectWebClient step allowing to drive more than one web client within a single test (particularly useful when you have a scenario involving users with different roles that should interact)
• the xpath attribute for the repeat step
• the new ready to import webtest.xml build file
• and of course diverse improvements in the reports

What's next?

Difficult to say as changes often come when the need arise. Many AJAX related improvements will probably come like a way to synchronize some asynchronous calls to make them testable without sleep or waitFor... Look at the mailing list to stay up to date.