Automatic Functional Testing for Mac and Linux?

testYourGUIS asks: "I work with a company that tests consumer and business software for various venues. One of the programs we rely on heavily is called Rational Visual Test, which is a VB-like compiled programming language that works directly with Windows API to simulate mouse clicks, text entry, menu pulldowns, and so on. We use them to stress-test various applications, collect the results, and then send the vendors a spreadsheet showing what we found. Recently, pressure has started to produce the same results on Mac OS X and possibly Linux applications. I was wondering if anyone knew of such software that works on Mac OS X, and how to go about doing this for Linux?"

"For example, a vendor has a database interface, and wants to know how many times we can access a series of records, what the results show (are the right, blank, time out?), and how long it took for the search to take place during various times of the day, and so on. So we emulate one of their data entry people by logging onto the system, putting in some search parameter from a script, and then record what the text (or error message) was when it came back. Yes, we could connect directly to the SQL database without their front end, but it's the combination of their front end and the database we usually test. We have found many examples where were can connect raw, for instance, but the front end displays a timeout. We also test things like 'What happens if the user enters a colon or asterisk?' or 'What if the text entered in the box is over 256k?' Then we tell the vendor the results, and they hopefully fix them based on the data we gave them."

Comments

AppleScript

[krog]

you can do this with AppleScript, quite possibly the most underrated language of its kind.

Re:AppleScript

[chris613]

"underrated"? Applescript is cumbersome, painful, and WAY too unreliable to be used for verification testing. I've had serious trouble just getting applescript to run and then close certain stock applications. It's a broken mess, and I wouldn't recommend it to anyone, especially not for testing.

Re:AppleScript

[scrod]

Actually no, that wouldn't do him any good. He would still have to implement the scripting commands in his program. He wants something to simulate the use of the actual interface, not call the underlying functions.

Re:AppleScript

[DLWormwood]

This assumes that the application being tested is fully scriptable. This requires a factored application design and/or an object model to convert AppleScript commands into their corresponding events. Historically, Mac OS development tools don't automatically create apps with such a structure. And as another poster noted, the means to have AS work with non-scriptable apps requires the use of unreliable "osaxen" that make timing and reliablity metrics meaningless.

Re:AppleScript

[guttentag]

How about QuicKeys? When I couldn't get AppleScript to translate spoken commands into certain keystrokes and mouse clicks in my mapping program, I found that QuicKeys was able to do the job. (I ended up using AppleScript for the Speech Recognition input and QK to manipulate the interface)

Someone mentioned server load testing earlier... if that's what you're looking for, remember that Mac OS X is basically a pretty window manager on top of a flavor of FreeBSD -- you can have cron automate shell scripts and perl.

testing tools

[Anonymous Coward]

try OpenSTA. Its an open source automated testing application, similar to Rational's product.

www.opensta.org

There is a commercial company that provides support at www.opensta.com

Re:testing tools

[leandrod]

For all I could gather OpenSTA is still MSW32-only, with plans to porting to other platforms; and it can't yet do tests other than http yet.

IANATester.

Re:testing tools

[Anonymous Coward]

Trust me, I use OpenSTA at work and its not what I would call 'stable' yet.

Also, it is designed to give perforance stats on web-based apps, not checking for failures like the person asked for (although I guess you could look at the number of 404 errors and then track them through the audit log for each test).

Also another major drawback is that it seems to hook into several components of NT/2000/XP to run the tests so it won't work on any other OS's, although you can test whatever type of web server and OS you want.

Well thats my 2p on the subject...


* This Christmas give your child a gift they wont forget, Mental Blocks! *

Hmm.

[dasmegabyte]

Automatically uses the API to interact with the machine? Yeah, they've got this for Mac...it's called AppleScript and it's been around for roughly 20 years.

Don't be scared by the name "script"...there are visual tools for ascript and the language itself is so simple my mom could do it.

XRunner

[micantos]

Check this out for some pretty reasonable X-windows testing. Heard quite a few positive things about it.

Re:XRunner

[Anonymous Coward]

We currently use Winrunner, The win32 equivalent of XRunner by Mercury Interactive. We've had lots if problems with this product, It's very buggy for what my standard of a automated test suite should be (hmmm a Automated test suite that actually has ben TESTED?). Constant restarts, variables changing values and so on are day to day recurrences. The support from MI is extremely poor, generally taking days to weeks to get any issue resolved. XRunner might be a bit more stable, but given its high price tag and the poor quality of this product I'd definitely advise anyone to stay away from this.

Re:In case it gets slashdotted

[yorgo]

Check out http://www.testingfaqs.org for a short list of possible contenders (GUI test drivers).

We're working on one...

[matticus]

Actually, the company I am currently employed by is in the process of writing a system/gui test environment for this very purpose. One caveat-it's only for Java, but when we complete it (very soon), we will release it to the public. our address is www.atomicobject.com.
The name of the project will be HASTE, look for it in a month or two.

Re:We're working on one...

[spicysquid]

you can get system.out stream and write out the character sequences to that byte if it is a terminal/telnet program. For swing programs, you can just fire off actionevents to take care of the items.

XTest and Expect ?

[too_bad]

Isnt this what XTest and
Expect allows us to do on Linux
(and a vareity of other operating systems) ?

Re:XTest and Expect ?

[Vee Schade]

XTest must be compiled into X as an extension and appears only to allow "fake key presses" which will probably not be very useful in this context. Expect allows for "automating interactive processes" (and is VERY good at it - IMHO), where such processes are CLI-driven (e.g. telnet, ftp, etc.), and so is also not useful in this context.

Re:XTest and Expect ?

[cduffy]

Actually, android lets you script fake keypresses, mouse events and such from a TCL interpreter (which expect happens to be). However, it doesn't let you see and check widget contents as text in the same way that many of the Windows-based test systems do. The commercial products for X suffer from similar failings when not using explicitly supported widget sets; as such, automated testing is just one of those things that isn't so well supported under X Windows. Hopefully Berlin (or some other successor) will help in this.

(FWIW, my job very recently involved buying or writing an automated test scaffold for graphical apps on Linux that would work across architectures and windowing systems -- it turned out that the latter was necessary, as the available commercial products just weren't/aren't cross-platform capable, and android has some serious faults which I'll expound on if asked. The product partially exists, having been put on hold for other internal development... perhaps when it's finished we'll release it under an open license).

Check out Test Quest Pro

[Jay]

It's a PC you can use to drive keyboard / mouse input into another system. It comes with a video capture card to read the output and make decisions about whether the test was successful or not.

It's fully scriptable using a c-ish syntax, and comes with a nice IDE and runtime environment to get the work done. It seems like it may be a nice match for what you want to do.

Here is more info.

F-Script

[Tomy]

For Mac OS X have a look at F-Script. You can inspect a running program and call any method in that program and any method on any object in the Objective-C runtime. This should allow you to generate events or enter text in text fields, etc.

Of course this will only work for programs developed using Cocoa. Can't help you with Carbon apps. You might be able to write something using the Apple Event manager.

Last time I had to do this for Linux (and Gtk+) I had to roll my own.

Applescript, QuicKeys for Mac OS X and OSA

[plsuh]

AppleScript is great, but the app that you are testing must be scriptable. Many are today, but some are not. Furthermore, in this situation you want to test the GUI, not just the underlying object model which is what AppleScript talks to.

However, AppleScript is not the only scripting environment for the Mac. Underlying all of the native scripting languages on the Mac is something called the Open Scripting Architecture. This allows any OSA component (i.e. scripting language) to talk to any other OSA component.

One OSA component is QuicKeys, a great product that actually does simulate mouse clicks, keystrokes, etc., and is completely scriptable. It integrates completely with AppleScript and via the power of the OSA, with shell scripts and Perl as well.

On top of this, Apple provides a powerful GUI building tool for AppleScript called AppleScript Studio, which is free. Plus, AppleScripts can talk across a network to do RPC. Imagine, you can have an AppleScript Studio front end that drives scripts on a set of machines over the network (so that you can assess performance under load). The individual scripts rely on AppleScript to drive the logic while using QuicKeys sequences to drive the GUI.

The only issue here is how easily you can assess results. There's no general way to check what the GUI is showing other than a pair of Mark I mod 0 eyeballs, since grabbing data out of the app being tested (even simulating a copy-paste) only talks to the underlying data structures and does not address GUI bugs. A possible way is to arrange the windows in a known manner, turn off the menu bar clock, take a screen shot, and use a graphical comparison tool to XOR the bitmap with a known good screen shot to highlight differences, but this won't work if you're working with variable or randomly generated data.

Disclaimer: I work for Apple, but these are my opinions only and do not represent any sort of official endorsement.

--Paul

Re:QARun, QADirector?

[KaiserSoze]

Holy piss, madd scientist, I can't believe you recommended that ;)

QADirector is not a testing language in itself. It is a harness from which to run the script generated by script writers. It used to be a Unix app, but when the switch was made to Wintel there were a few missteps. If you are looking for a test harness that is comprehensive (for Windows anyways) in its command-line options, look into QAD, otherwise I would go with Test Director by Mercury or maybe Segue's new product (I believe its called SilkPlan or something similar).

QARun on the other hand is a pain in the ass. If you are looking for automation, don't use it. It's designed for straight-up "record and playback" style testing. If you wish to deploy these tests to client machines (something which some, but not all people want to do), do not use QARun. You can look into XRunner (it is the predecessor of WinRunner) by Mercury if it is still being sold/supported for *nix testing. I honestly don't know about Mac, our product doesn't run on it.

If you want a serious evaluation document, I highly suggest heading over to http://www.qaforums.com and head to the Automation section. Post a query about the Mac and you'll probably receive a response from one of the moderators with a white paper that was written comparing all of the available tools.

DISCLAIMER: My group uses QARun, QAD, Segue Silktest, and WinRunner, so as not to be accused of bias.

Re:Automatic Functional Testing for Mac and Linux?

[Anonymous Coward]

no you won't

The problem is that tools like VisualTest and WinRunner can find specific objects (buttons, dropdown boxes etc... and treat them as objects. When you use such a tool with an X Server, all the tool can see is a mouse click at a specific location. This makes your automated playback screen size and window location dependent. This is not a reliable way to handle this.

WinRunner

[jishak]

My company uses WinRunner for Windows GUI testing. I know that Mercury Interactive which makes WinRunner and LoadRunner another GUI Testing utility creates Unix version of their software in addition to the Windows versions. I don't know about Mac though.
Mercury Interactive
Hope that helps
Joe

Pounder and Abbot

[Anonymous Coward]

Can be used to create these sorts of scripts:

http://sourceforge.net/projects/pounder
http:// sourceforge.net/projects/abbot