Slashdot Mirror
Automatic Functional Testing for Mac and Linux?
"For example, a vendor has a database interface, and wants to know how many times we can access a series of records, what the results show (are the right, blank, time out?), and how long it took for the search to take place during various times of the day, and so on. So we emulate one of their data entry people by logging onto the system, putting in some search parameter from a script, and then record what the text (or error message) was when it came back. Yes, we could connect directly to the SQL database without their front end, but it's the combination of their front end and the database we usually test. We have found many examples where were can connect raw, for instance, but the front end displays a timeout. We also test things like 'What happens if the user enters a colon or asterisk?' or 'What if the text entered in the box is over 256k?' Then we tell the vendor the results, and they hopefully fix them based on the data we gave them."
you can do this with AppleScript, quite possibly the most underrated language of its kind.
Cretin - a powerful and flexible CD reencoder
try OpenSTA. Its an open source automated testing application, similar to Rational's product.
www.opensta.org
There is a commercial company that provides support at www.opensta.com
umm... interns?
Automatically uses the API to interact with the machine? Yeah, they've got this for Mac...it's called AppleScript and it's been around for roughly 20 years.
Don't be scared by the name "script"...there are visual tools for ascript and the language itself is so simple my mom could do it.
Hey freaks: now you're ju
Check this out for some pretty reasonable X-windows testing. Heard quite a few positive things about it.
Just issue random click and random key stroke ....
I've worked with Rational Visual Test(or Irrational Test as we often refer to it.) With both you can call commands from different programs. I have't tried doing similar tasks in PERL, but mostly because I have not immersed myself in PERL.
Obviously this would work on Linux, OS X, and Windows as well.
Hope this helps.
Check out http://www.testingfaqs.org for a short list of possible contenders (GUI test drivers).
I've got it... you need Lego Mindstorms. Just build some robotics to push the mouse around, whack random keys, etc. Best of all, such a system would be fully platform-independant. I bet Microsoft is already doing this, only they call it "genetic programming".
Actually, the company I am currently employed by is in the process of writing a system/gui test environment for this very purpose. One caveat-it's only for Java, but when we complete it (very soon), we will release it to the public. our address is www.atomicobject.com.
The name of the project will be HASTE, look for it in a month or two.
Expect allows us to do on Linux
(and a vareity of other operating systems) ?
DO NOT PANIC
You might be able to use Rational Visual Test with a windows based X-server to control Linux (or any Unix) apps.
They can stress all the bugs out a program in no time :)
One way (the simpler way) is just write a program using the xtest extension to send keyboard/mouse events to said applications...
:), This way you can do REAL test without some of the speed overhead that the video cards induce :)
But another way (and more affective way) is to use XGGI and force it to use your own data streams for you input and then you don't even have to really display you output to a video card, you can just keep it in memory
- r0gu3
Remember when Rational made that?
Opened up a whole world beyond 16 bit DOS apps.
Behold the power of Watcom C!
Here we lock the accountants in a room with one bottle of Orangade and a bag of Fritto's, and tell them the only way out is by finding the hidden "key" in the app. Then we video tape them, and if the results are not good for the client, we can always send the tape to Fox for "when stupid white-collars go bad"
Don't off topic me too quickly. It is slightly relevant. Remember back in 1999 and 2000 when companies would pay you to surf the web with on of their ad windows at the bottom? Well a friend of mine and I wrote a little Visual Basic program that would randomly move the mouse to some point on the screen and click. It had a random time movement from anywhere from 20-80 seconds and we restricted where it would move so that it would not hit the start menu and accidentally log you out! I worked great and each of us made a few hundred bucks over a few months using it on a few different computers with different accounts. My friend finally got caught on one of them so we decided to quit, but it sure was an easy way to make some cash.
I think there's more to that story. A company hired to test program written for Mac OSX and Linux, not just Windows. Anyone else catch what I'm getting at?
The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
Why don't these people just do a search on freshmeat and google like the rest of us? You'll undoubtable just hear from the other slashdotters who just got down googling and freshmeating (he he he) for their results.
Okay there's an offchance that another slashdotter might be working on an esotric project like he is and happens to have a viable solution. Highly unlikely. Or perhpas slashdot just wants to bring this to our attention. Don't know why though? Or mabey IT'S SHAMLESS ADVERTISEMENT PLUGGING yet again =P
Go ahead and give me your worst repliers.
Since you're already a Rational customer, I would check into Rational preVue . Many years ago it had another name - and was a product from a company that Rational bought...
"Software is the difference between hardware and reality"
you could always port it to OS X ... :-)
I was on a UNIX project almost 10 years ago that used MetaCard to do this kind of thing. Haven't used it myself and I think it is limited to simulating a user running a GUI (and for creating demos, etc.), but it is cross-platform...
It's a PC you can use to drive keyboard / mouse input into another system. It comes with a video capture card to read the output and make decisions about whether the test was successful or not.
It's fully scriptable using a c-ish syntax, and comes with a nice IDE and runtime environment to get the work done. It seems like it may be a nice match for what you want to do.
Here is more info.
You think emacs is evil?! You've never used VM's XEDIT have you?!! That's evil, baby!
we call it "Beta" testing...
Trained Monkeys.
We use a product called Test Quest.
http://www.testquest.com
It is hardware based, so it may be less dependent on OS.
I believe it was $50,000 but not positive.
We are just starting to use it and have found it to work quite well and also work over our KVM but a clean signal is needed over KVM's.
Let a cat in the room and pretend you are doing some serious work.
Here's a website that talks about the language: http://expect.nist.gov/. I bet there's other sites so check google. O'Reiley also has a book about it. I still think I'd use Applescript under MacOS, but under Linux I'd give expect a try. I played with it a little under Solaris and it looked pretty useful. I was able to automate the setup for US Robotics modem racks and Cabletron ethernet hubs using expect. Never did get that stuff totally debugged and my script would go crazy, but that's not the language's fault.
If you're already using winblows, to test X apps, just VNC to the application you want, and then run the stress test on the VNC session. All mouse clicks and keypresses will be performed on the remote computer. That way you can use the same program across platforms, instead of figuring out a seperate application for each OS you test.
You can also reverse this process if you'd rather use your linux box to test winblows.
Everyone is entitled to their own opinion. It's just that yours is stupid.
Depends on your budget. Can you afford a thousand monkeys?
I object to that article, and to the next reply.
Yeah, right. That paraphases exactly why we've used VHS for 20+ years instead of Beta. NOT! It is arguable that the Apple was a superior computing platform to the IBM PC of its day. The IBM-PC had the full corporate power and marketing strength of, well, IBM behind it while the Apple had the comparatively limited resources of two guys named Steve.
For the uninformed... Back in the days when the VCR was just being released on the world there were two competing formats, JVC's VHS format (which we all know) and Sony's Beta (a.k.a BetaMax). The latter was far superior but, alas, was first out marketed (a bit of history), then out litigated (Read case story). To date the majority of video production houses still use Beta because of it's superior image and audio quality.
My office has been taken over by iPod people.
You could probably roll your own rather quickly with the CGRemoteOperation API: /System/Library/Frameworks/ApplicationServices.fra mework/Frameworks/CoreGraphics.framework/Headers/C GRemoteOperation.h
The Future: Some assembly required; batteries not included.
For Mac OS X have a look at F-Script. You can inspect a running program and call any method in that program and any method on any object in the Objective-C runtime. This should allow you to generate events or enter text in text fields, etc.
Of course this will only work for programs developed using Cocoa. Can't help you with Carbon apps. You might be able to write something using the Apple Event manager.
Last time I had to do this for Linux (and Gtk+) I had to roll my own.
Google
I'm assuming you're talking about java.awt.Robot. This, with BeanShell would allow test scripts for GUIs to be built.. A more sophisticated system would be able to capture AWT events, and store them in a file for later playback...
A while back I found an app called xmacro. Just a simple app to send mouse clicks and such. If you can't find a package to do what you want, you might be able to roll your own from it.
Xmacro
Slashdot is like Playboy: I read it for the articles
it's called AppleScript and it's been around for roughly 20 years.
Let's see here... 1993-2002 is roughly 20 years?
Has CowboyNeal been teaching you "new math"?
I just posted a set of tools for automating X at my website that I recently created to make a bot that could play Yahoo! Spelldown. There are 2 parts to the tools, finding things on screen using a "visual grep" program that finds images inside of images, and xte, an app built on the XTest extension that allows scripting of moving the mouse, clicking, dragging, keypresses, etc. Together you can use them to make scripts that can identify buttons on screen or letter or words or numbers, whatever, and interact with programs. Still a pre-1.0 version, but it does everything listed above so far.
BTW, don't use xse, which is a wrapper for XSendEvent, if you roll your own. There are so many places where it just doesn't work, or doesn't work right, because the XSendEvent function just doesn't work reliably. I lost a lot of sleep over that before I gave up and just wrote xte.
Free Online Woodworking Resources Directory
i think compuware's QARun or QADirector work on *nix
MARIJUANA, SHROOMS, X: ONLINE?! - E
your dick is that small?
for mouse events, you can try QuicKeys X from CE Software for Mac OS X, anyway. It can enter text, too, and launch applescripts.
1) They're made by Sony 2) They can use the same size cassette shell 3) They have Beta in their name
They are all professional formats and are not the same as BetaMax, BetaMax II and EDBeta of old....
-- oldthinkers unbellyfeel ingsoc
Finally, I've discovered a use for this strange box on my desk.
I design user interfaces for a free network management application,
Over the years I've asked Rational salespeople if they plan on porting some of their more popular debugging tools to Linux, such as Purify, Pure Coverage, Quantify, etc. The earliest responses to the question were, "What's Linux?" In later years, the response has simply been "No". I do not understand why, but it has been a major sticking point for me. More and more, people are developing software for Linux, and it's hard to understand why companies like Rational won't embrace it.
I'll tell you, it's a major hassle to develop a product for Linux while being forced to debug it on Solaris because that's the only place our debug tools will run. The first company to make decent Linux tools that are similar to Rational's will have my business. And Rational won't.
AppleScript is great, but the app that you are testing must be scriptable. Many are today, but some are not. Furthermore, in this situation you want to test the GUI, not just the underlying object model which is what AppleScript talks to.
However, AppleScript is not the only scripting environment for the Mac. Underlying all of the native scripting languages on the Mac is something called the Open Scripting Architecture. This allows any OSA component (i.e. scripting language) to talk to any other OSA component.
One OSA component is QuicKeys, a great product that actually does simulate mouse clicks, keystrokes, etc., and is completely scriptable. It integrates completely with AppleScript and via the power of the OSA, with shell scripts and Perl as well.
On top of this, Apple provides a powerful GUI building tool for AppleScript called AppleScript Studio, which is free. Plus, AppleScripts can talk across a network to do RPC. Imagine, you can have an AppleScript Studio front end that drives scripts on a set of machines over the network (so that you can assess performance under load). The individual scripts rely on AppleScript to drive the logic while using QuicKeys sequences to drive the GUI.
The only issue here is how easily you can assess results. There's no general way to check what the GUI is showing other than a pair of Mark I mod 0 eyeballs, since grabbing data out of the app being tested (even simulating a copy-paste) only talks to the underlying data structures and does not address GUI bugs. A possible way is to arrange the windows in a known manner, turn off the menu bar clock, take a screen shot, and use a graphical comparison tool to XOR the bitmap with a known good screen shot to highlight differences, but this won't work if you're working with variable or randomly generated data.
Disclaimer: I work for Apple, but these are my opinions only and do not represent any sort of official endorsement.
--Paul
If you want a cross-platform automated testing tool that is:
1. Localizable
2. Non-Intrusive (tests the environment on which the user runs).
Then check out TestQuest .
While a bit pricey - consider it a capital expense - this system is fantastic. It was originally developed for testing medical devices and systems, where the FDA requires that what you test is what you ship exactly .
AFAIK, you can use a VNC X server with a custom "automated" VNC viewer to test any and all X apps.
-JAB
Although not a software solution, we use Test Quest to automate our reboot and application execution tests. Basically the hardware in an external controller machine (a second PC) simulates keyboard and mouse actions, along with a bunch of discrete lines that can be set high and low for things like reboot, power switch, and floppy drive. The system can also read back discrete lines for things like "is the system powered on?", and can take a screen capture to compare bitmaps, or do OCR to read results. It's all controlled by windows DLLs that can be written by anybody with a basic understanding of C, or pseudo-automatically written with the TestQuest recorder system.
They may have other newer systems, or a software solution for now, but for what we do it works great. We only test on PC hardware, but they have a variety of modules and interfaces for touchpads, VGA, svideo, Mac and Sparc hardware as well.
Here's a site which will give you opinions and feedback on a wide variety of tools.
QA Forums.
Scroll on down until you get to the Testing Tools forum.
Cthulhu Barata Nikto
Uh, your a bit of history link -- the urbanlegends.com -- says that VHS was superior. What gives? They were exact equals in quality, but VHS could record for longer, which was a big deal back then....
Java has a Robot class that does this. It's pretty good, the only downside being that you have to specify locations by the pixel.
Check out java.awt.Robot.
~Kevin
A lot of open source projects could benifit from automated testing. The wine project has spent nine years working on translating the windows API to X and reproducing the windows GUI.
Many people think wine is a vital tool for greater use of linux. They rely on people downloading new versions and testing windows programs using whatever versions of X, system libraries and windows DLL's they happen to have and then posting bug reports in a newsgroup. There are still lots of quirks with relativly simple windows programs not redrawing or displaying dialogs correctly.
I believe that wine would progress much faster if the wine developers could make a change then click test and later, with no more effort get a report saying "3 tests which failed previously now pass, 1 test which passed on the last version now fails".
I will be very impressed if somone comes up with a free automated testing tool that allows tests to be set up easily and is powerfull enough to provide human readable results "the bitmap is drawn 1 pixel to the left", "text as expected but wrong font" "dialog appeared behind other window" etc.
DejaGnu is a TCL/Expect testing harness provided as free software by the Gnu Project. Not sure if it has tk hooks (which would allow you to test X-based programs) so you might want to check. If you're trying to test programs on the postscript display of a Mac instead of X, well, I don't know if it would work. Also, check to be sure there is a recent version of TCL/TK and Expect for your machine before you pursue this too far.
There is also LTP, the Linux Test Project, which may be of use to you as well. (Do a Google search for "linux test" and you'll find it quick.)
If you're not a TCL programmer, you're probably not going to like DejaGnu very much. If this is the case, good luck with the GUI tool search; you can expect to spend some serious money for decent test tools. (The pun is simply there. If you look at it too long or pay too much attention to it, it may or may not vanish. Just chalk it up to some crazy cat and don't worry about it.)
Best wishes,
Everything in the Universe sucks: It's the law!
look- it is quite simple to do this with applescript. there exist various OSAX which will simulate key strokes, mouse movement, and even menu selections in ANY application- scriptable or not.
just to name a few:
keystrokes and mouse movement/ clicking: Sändi's Additions OSAX
menu events: Menu Events 1.3.1
i have used this to automate various data entry jobs were the only way to get data into the system was through a painful web-interface. while simulating the keyboard and mouse was not the ideal way to put the data into the system, it was a hell of a lot faster/easier/ and more accurate than doing it by hand.
in terms of claims that AS is less than robust-- obviously you are not writting the code correctly- as i have a number of AS driven CGI's running on a 25mhz machine (thats 25mhz!) that perform quite well- even under high loads.
before you bash something that you all obviously have little knowledge of why not do a little research into it first.
dylan
Android runs on top of these and can be downloaded free for Linux and Unix
See the July 2001 issue of "Dr. Dobb's Journal"
for an overview of Android and how to download.
Plays back keyboard and mouse events for most UNIX implementations. (maybe even Mac OS X)
It is a little bit roll-your-own
Expect to spend some time writing perl to support your android scripts.
Letter To Iran
...aren't really necessary for functional testing, which is why most of these functional testing tools don't [more like can't] run in real time. IMO, these relatively pricey functional tools from the big venders could hardly be called reliable. I suspect that the tiny niche marketplace for such tools ensures that things will probably stay that way.
OTOH, Real life timing is really important for scalability testing, i.e. the domain of load testing tools, which mimic front end client behaviour in real time via back-end automation. These are definitely far more reliable, but also exceedingly more expensive.
ARexx used to do this on my Amiga back in '89. I remember that to launch applications using tooltypes defined in the icons (and to avoid launching via command line tooltypes in the startup-sequence), I ran an ARexx script that double-clicked on my drive, double-clicked on the Applications folder, and launched the apps I wanted running after I logged in. The script would then place focus in a new CLI window. All by scripting mouse events.
*Sniffle, wipes tear from eye* Ah, the good old days.
...and 4: they all use separate y/c video tracks, providing (think svideo vs composite) providing far superior video quality.
The professional formats also use stereo audio tracks and such, but are VERY similar, and some are backwards compatible hardware wise to the original Betamax, IRRC.
Sony licensed itself out of the market with a closed format.
This is an Apple developer thing... VU or Virtual User... It has hooks into the OS.... It is more apprpriate than AppleScript and other normal user scripting tools...
My company uses WinRunner for Windows GUI testing. I know that Mercury Interactive which makes WinRunner and LoadRunner another GUI Testing utility creates Unix version of their software in addition to the Windows versions. I don't know about Mac though.
Mercury Interactive
Hope that helps
Joe
Rational suite, including tests was a piece of crap when we worked with it. We switched to XP and everyone started having great time coding, due to allowance of concurrent programming...
If you want to get job done right, I would recommend eXtreeme Programming, over most other ideologies.
2c,
p.
There is a product called KDRunner ( http://www.klaralvdalens-datakonsult.se/Public/pro ducts/kdrunner.html ) available for automating GUI testing of Qt applications. Qt runs on Windows, UNIX and MAC.
I'm not completely sure about this, but I think you can use ATK (accessability toolkit) for sending events and getting widget contents to GTK apps. That works only onn GTK2, though, so perhaps it's not helpful to you, but at least I think the functionality is there.
AppleScript interfaces directly with the program at a lower level than the GUI -- it's essentially an alternate method of controlling the app, plugging into the same API as the GUI widgets do. This tests the app and everything below that, but it's not exactly the same as actually testing through the GUI itself, since you're bypassing it.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Tim Janik came out with a utility called Gerd a while back that lets you script GTK+ applications. It appears to have been abandoned at version 0.0.3, however.
Just install a VNC server on each of your Mac or Linux SUTs. Your test host can connect to each box using a VNC client.
This way you don't have have your developers learn a new environment, and one testing host can test multiple systems.
There are a couple of apps called rfbmacro and rfbproxy, which use the protocol RFB used by VNC.
You'll have to search around for a link (a brief search on Google doesn't reveal much).
From what I remember, these apps are *nix based, but someone on the VNC mailing list once said they got them to work on Win32 via Cygwin.
Basically you record a VNC session, and then can play it back ad nauseam. There are VNC servers/clients for a good many platforms, including OS X, Windows, and *nix.
This may be off topic in that you specified Mac OS X. If so, sorry! Anyhoo, Apple offered for the Classic Mac OS a developer tool called "Virtual User," which as I understand it does what you're asking for, and for all I know might just work on a Carbon application. It's available atg _- _Debugging/Virtual_User_Tools/
ftp://ftp.apple.com/developer/Tool_Chest/Testin
Hope that helps!
--Kimota
Who moderates the meta-moderators?
Can be used to create these sorts of scripts:
/ sourceforge.net/projects/abbot
http://sourceforge.net/projects/pounder
http:/
BTW, can you explain the bit where Mojo typed "Pray for Mojo"? It didn't make enough sense to me for it to be worth having the pet shop owner go inside, get a keyboard, and wait for the monkey to type that, so I figured it must be a reference to something I don't know about.
one hundred twenty
is just enough characters
to write a haiku
at-poke can do this for gtk+-2.0 (maybe also for other libs using atk, not sure)
at-poke README
at-poke in GNOME CVS
Use slashdotters. They can't bring anything to its knees.
No this is what apple means to :P
"Think Different"
--toq
Brings back memories..
I was once hacking pascal apps for the old macintosh toolkit API.. and noticed the app event queue had a method that you could use to insert synthetic events. The documentation on it (PushEvent(?) or something) said it was intended for automated testing. So, what I did was just fill event records with random numbers and left it pounding the queue overnight. In the morning, if the app had crashed, the last event was recorded in a file and you could check it out.
There shouldn't be much of a problem doing this under any X environment, as Xevents of any kind are not hard to generate.
Ordo Militum Unix.
A sci-fi writer had an interesting opininion a while ago;
The computer systems of today are just too complex to be kept working without the Internet. What is the Internet? Slashdot, sourceForge, newsgroups, e-mail, irc? Groupmind!
Prepare to be assimilated. Perhaps you already have.
I'll assume you're not a tester, because that is an awful way to test an application. Any automated script the realise on X/Y coordinates is going to break. Just don't do it!
From today's Freshmeat:
http://hoopajoo.net/projects/xautomation.html
Might be worth looking into...
Didn't you even read your own link.
Beta died a lingering painful death due to inferiorty in that it used a 1-hour playback time when VHS was two hours. You could not record 1 whole movie on a Beta tape, whereas most movies could be recorded on the 2-hour VHS formet.
Pricing was probably an issue (Beta was usually a little more expensive), as well as luck and yes, even marketing. The lawsuit you mention was too little, too late to have killed Beta,
Technical quality of the playback was better on the Beta -- big deal, read your own link -- the average consumer could not even tell the difference even if their TV was capable of the difference.
With respect to Apple Mac vs. IBM PC & clones. Price & Applications were why business continued to by PC's. Mac was so hard to program (new and ugly paradigm you know), that it was a year before any real apps came out, and the cost of developing in-house apps was innordinately high.
A Lamborghini Countach is quite superior in many respects to the Toyota Avalon sitting in my driveway, but it is not a better car for me. Price, Insurance, convenience, capacity, etc. make the Toyota are far better product for me.
PC outsells Mac
Toyota outsells Lamborghini
VHS outsells Beta
All inferior in some respects, All better for the consumer in ways that mattered to them. I've got a PC, a Toyota, and a VHS recorder. I am not a stupid consumer buying stuff because of fancy commercial brainwash me into buying second-rate junk. I bought stuff them makes sense for me, for my needs. I assume most buyers of Macs, Lamborghinis or Betamax can make the same statement.
Here's an open-sourced, VNC-based user-event recording and playback tool:
t heatre.html
http://www.cs.sfu.ca/~walenste/personal/Research/
I am currently bet testing a Mac OSX program that uses TCP/IP and VNC to test software on any platform with a VNC server. The company is called Redstone Software - www.redstonesoftware.com.
Do not taunt Happy Fun Ball
I know this will get moderated down, but that's mostly because /. is group-think hell and because its redundant, but I have to get this out...
Basically, if you're responsible for stress testing software on an OS, you should at LEAST have a cursory understanding of the tools available on that OS... meaning, if you don't, how do you expect to really stress test these applications? If you didn't know applescript could do this, then you're stupid and you shouldn't be stress testing applications for anyone on any platform.
That said... moderators, mod me down.
"I'm getting moderated down for thinking differently than the rest of the k1dd13z..."
-C
"This above all, to thine own self be true"
Funny you should ask. The company I work for is developing software that I think does just what you asked for. It's a Mac OS X application (codenamed Eggplant) that can test software running on Mac OS X or Linux (or Windows or *nix for that matter).
Eggplant uses a TCP/IP connection to remotely control the system being tested, under script control. Because it interacts from the user's perspective, by generating mouse and keyboard events and watching what happens on the screen, Eggplant can test anything that has an interface, no matter what language or toolkit it was created with. I think it's pretty cool. And it's the only project I've ever worked on that had a purple vegetable as an app icon!
Check Redstone Software if you want to join the beta program (happening now). Version 1 should ship soon, but the marketing guys will probably come up with some suitably boring name by then...
crude, but functional, for wrapping installers and other interfaces