Automatic Functional Testing for Mac and Linux?

testYourGUIS asks: "I work with a company that tests consumer and business software for various venues. One of the programs we rely on heavily is called Rational Visual Test, which is a VB-like compiled programming language that works directly with Windows API to simulate mouse clicks, text entry, menu pulldowns, and so on. We use them to stress-test various applications, collect the results, and then send the vendors a spreadsheet showing what we found. Recently, pressure has started to produce the same results on Mac OS X and possibly Linux applications. I was wondering if anyone knew of such software that works on Mac OS X, and how to go about doing this for Linux?"

"For example, a vendor has a database interface, and wants to know how many times we can access a series of records, what the results show (are the right, blank, time out?), and how long it took for the search to take place during various times of the day, and so on. So we emulate one of their data entry people by logging onto the system, putting in some search parameter from a script, and then record what the text (or error message) was when it came back. Yes, we could connect directly to the SQL database without their front end, but it's the combination of their front end and the database we usually test. We have found many examples where were can connect raw, for instance, but the front end displays a timeout. We also test things like 'What happens if the user enters a colon or asterisk?' or 'What if the text entered in the box is over 256k?' Then we tell the vendor the results, and they hopefully fix them based on the data we gave them."

Comments

testing tools

[Anonymous Coward]

try OpenSTA. Its an open source automated testing application, similar to Rational's product.

www.opensta.org

There is a commercial company that provides support at www.opensta.com

Hmm.

[dasmegabyte]

Automatically uses the API to interact with the machine? Yeah, they've got this for Mac...it's called AppleScript and it's been around for roughly 20 years.

Don't be scared by the name "script"...there are visual tools for ascript and the language itself is so simple my mom could do it.

XRunner

[micantos]

Check this out for some pretty reasonable X-windows testing. Heard quite a few positive things about it.

Java... sort of

[d3xt3r]

Are these applications you need to demo like this, Java Swing apps? If so, you can use built-in javax APIs to do this. You could simulate user actions this way, mouse movements, etc as a "demo" mode for the user.

Obviously this would work on Linux, OS X, and Windows as well.

Hope this helps.

Re:In case it gets slashdotted

[yorgo]

Check out http://www.testingfaqs.org for a short list of possible contenders (GUI test drivers).

Legos!

[djtack]

I've got it... you need Lego Mindstorms. Just build some robotics to push the mouse around, whack random keys, etc. Best of all, such a system would be fully platform-independant. I bet Microsoft is already doing this, only they call it "genetic programming".

XTest and Expect ?

[too_bad]

Isnt this what XTest and
Expect allows us to do on Linux
(and a vareity of other operating systems) ?

Re:XTest and Expect ?

[cduffy]

Actually, android lets you script fake keypresses, mouse events and such from a TCL interpreter (which expect happens to be). However, it doesn't let you see and check widget contents as text in the same way that many of the Windows-based test systems do. The commercial products for X suffer from similar failings when not using explicitly supported widget sets; as such, automated testing is just one of those things that isn't so well supported under X Windows. Hopefully Berlin (or some other successor) will help in this.

(FWIW, my job very recently involved buying or writing an automated test scaffold for graphical apps on Linux that would work across architectures and windowing systems -- it turned out that the latter was necessary, as the available commercial products just weren't/aren't cross-platform capable, and android has some serious faults which I'll expound on if asked. The product partially exists, having been put on hold for other internal development... perhaps when it's finished we'll release it under an open license).

I've got two words for you...

[pi_rules]

Trained Monkeys.

Applescript, QuicKeys for Mac OS X and OSA

[plsuh]

AppleScript is great, but the app that you are testing must be scriptable. Many are today, but some are not. Furthermore, in this situation you want to test the GUI, not just the underlying object model which is what AppleScript talks to.

However, AppleScript is not the only scripting environment for the Mac. Underlying all of the native scripting languages on the Mac is something called the Open Scripting Architecture. This allows any OSA component (i.e. scripting language) to talk to any other OSA component.

One OSA component is QuicKeys, a great product that actually does simulate mouse clicks, keystrokes, etc., and is completely scriptable. It integrates completely with AppleScript and via the power of the OSA, with shell scripts and Perl as well.

On top of this, Apple provides a powerful GUI building tool for AppleScript called AppleScript Studio, which is free. Plus, AppleScripts can talk across a network to do RPC. Imagine, you can have an AppleScript Studio front end that drives scripts on a set of machines over the network (so that you can assess performance under load). The individual scripts rely on AppleScript to drive the logic while using QuicKeys sequences to drive the GUI.

The only issue here is how easily you can assess results. There's no general way to check what the GUI is showing other than a pair of Mark I mod 0 eyeballs, since grabbing data out of the app being tested (even simulating a copy-paste) only talks to the underlying data structures and does not address GUI bugs. A possible way is to arrange the windows in a known manner, turn off the menu bar clock, take a screen shot, and use a graphical comparison tool to XOR the bitmap with a known good screen shot to highlight differences, but this won't work if you're working with variable or randomly generated data.

Disclaimer: I work for Apple, but these are my opinions only and do not represent any sort of official endorsement.

--Paul