Slashdot Mirror
Automatic Functional Testing for Mac and Linux?
"For example, a vendor has a database interface, and wants to know how many times we can access a series of records, what the results show (are the right, blank, time out?), and how long it took for the search to take place during various times of the day, and so on. So we emulate one of their data entry people by logging onto the system, putting in some search parameter from a script, and then record what the text (or error message) was when it came back. Yes, we could connect directly to the SQL database without their front end, but it's the combination of their front end and the database we usually test. We have found many examples where were can connect raw, for instance, but the front end displays a timeout. We also test things like 'What happens if the user enters a colon or asterisk?' or 'What if the text entered in the box is over 256k?' Then we tell the vendor the results, and they hopefully fix them based on the data we gave them."
you can do this with AppleScript, quite possibly the most underrated language of its kind.
Cretin - a powerful and flexible CD reencoder
try OpenSTA. Its an open source automated testing application, similar to Rational's product.
www.opensta.org
There is a commercial company that provides support at www.opensta.com
Automatically uses the API to interact with the machine? Yeah, they've got this for Mac...it's called AppleScript and it's been around for roughly 20 years.
Don't be scared by the name "script"...there are visual tools for ascript and the language itself is so simple my mom could do it.
Hey freaks: now you're ju
Check this out for some pretty reasonable X-windows testing. Heard quite a few positive things about it.
I've worked with Rational Visual Test(or Irrational Test as we often refer to it.) With both you can call commands from different programs. I have't tried doing similar tasks in PERL, but mostly because I have not immersed myself in PERL.
Obviously this would work on Linux, OS X, and Windows as well.
Hope this helps.
Check out http://www.testingfaqs.org for a short list of possible contenders (GUI test drivers).
I've got it... you need Lego Mindstorms. Just build some robotics to push the mouse around, whack random keys, etc. Best of all, such a system would be fully platform-independant. I bet Microsoft is already doing this, only they call it "genetic programming".
Actually, the company I am currently employed by is in the process of writing a system/gui test environment for this very purpose. One caveat-it's only for Java, but when we complete it (very soon), we will release it to the public. our address is www.atomicobject.com.
The name of the project will be HASTE, look for it in a month or two.
Expect allows us to do on Linux
(and a vareity of other operating systems) ?
DO NOT PANIC
You might be able to use Rational Visual Test with a windows based X-server to control Linux (or any Unix) apps.
I think there's more to that story. A company hired to test program written for Mac OSX and Linux, not just Windows. Anyone else catch what I'm getting at?
The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
Why don't these people just do a search on freshmeat and google like the rest of us? You'll undoubtable just hear from the other slashdotters who just got down googling and freshmeating (he he he) for their results.
Okay there's an offchance that another slashdotter might be working on an esotric project like he is and happens to have a viable solution. Highly unlikely. Or perhpas slashdot just wants to bring this to our attention. Don't know why though? Or mabey IT'S SHAMLESS ADVERTISEMENT PLUGGING yet again =P
Go ahead and give me your worst repliers.
Since you're already a Rational customer, I would check into Rational preVue . Many years ago it had another name - and was a product from a company that Rational bought...
"Software is the difference between hardware and reality"
you could always port it to OS X ... :-)
It's a PC you can use to drive keyboard / mouse input into another system. It comes with a video capture card to read the output and make decisions about whether the test was successful or not.
It's fully scriptable using a c-ish syntax, and comes with a nice IDE and runtime environment to get the work done. It seems like it may be a nice match for what you want to do.
Here is more info.
You think emacs is evil?! You've never used VM's XEDIT have you?!! That's evil, baby!
Trained Monkeys.
Let a cat in the room and pretend you are doing some serious work.
Here's a website that talks about the language: http://expect.nist.gov/. I bet there's other sites so check google. O'Reiley also has a book about it. I still think I'd use Applescript under MacOS, but under Linux I'd give expect a try. I played with it a little under Solaris and it looked pretty useful. I was able to automate the setup for US Robotics modem racks and Cabletron ethernet hubs using expect. Never did get that stuff totally debugged and my script would go crazy, but that's not the language's fault.
Depends on your budget. Can you afford a thousand monkeys?
I object to that article, and to the next reply.
heh, i remember that. i also wrote a program to do that. brings back memories of screwing off in college.
Yeah, right. That paraphases exactly why we've used VHS for 20+ years instead of Beta. NOT! It is arguable that the Apple was a superior computing platform to the IBM PC of its day. The IBM-PC had the full corporate power and marketing strength of, well, IBM behind it while the Apple had the comparatively limited resources of two guys named Steve.
For the uninformed... Back in the days when the VCR was just being released on the world there were two competing formats, JVC's VHS format (which we all know) and Sony's Beta (a.k.a BetaMax). The latter was far superior but, alas, was first out marketed (a bit of history), then out litigated (Read case story). To date the majority of video production houses still use Beta because of it's superior image and audio quality.
My office has been taken over by iPod people.
You could probably roll your own rather quickly with the CGRemoteOperation API: /System/Library/Frameworks/ApplicationServices.fra mework/Frameworks/CoreGraphics.framework/Headers/C GRemoteOperation.h
The Future: Some assembly required; batteries not included.
For Mac OS X have a look at F-Script. You can inspect a running program and call any method in that program and any method on any object in the Objective-C runtime. This should allow you to generate events or enter text in text fields, etc.
Of course this will only work for programs developed using Cocoa. Can't help you with Carbon apps. You might be able to write something using the Apple Event manager.
Last time I had to do this for Linux (and Gtk+) I had to roll my own.
Those were the dot.com days... when things started going down the drain, they changed things so you could make $20 when you surf, 2 hours per day for 5 years.. how exciting.
What time is it/will be over there? Check with my iPhone app!
Google
I'm assuming you're talking about java.awt.Robot. This, with BeanShell would allow test scripts for GUIs to be built.. A more sophisticated system would be able to capture AWT events, and store them in a file for later playback...
A while back I found an app called xmacro. Just a simple app to send mouse clicks and such. If you can't find a package to do what you want, you might be able to roll your own from it.
Xmacro
Slashdot is like Playboy: I read it for the articles
it's called AppleScript and it's been around for roughly 20 years.
Let's see here... 1993-2002 is roughly 20 years?
Has CowboyNeal been teaching you "new math"?
I just posted a set of tools for automating X at my website that I recently created to make a bot that could play Yahoo! Spelldown. There are 2 parts to the tools, finding things on screen using a "visual grep" program that finds images inside of images, and xte, an app built on the XTest extension that allows scripting of moving the mouse, clicking, dragging, keypresses, etc. Together you can use them to make scripts that can identify buttons on screen or letter or words or numbers, whatever, and interact with programs. Still a pre-1.0 version, but it does everything listed above so far.
BTW, don't use xse, which is a wrapper for XSendEvent, if you roll your own. There are so many places where it just doesn't work, or doesn't work right, because the XSendEvent function just doesn't work reliably. I lost a lot of sleep over that before I gave up and just wrote xte.
Free Online Woodworking Resources Directory
i think compuware's QARun or QADirector work on *nix
MARIJUANA, SHROOMS, X: ONLINE?! - E
for mouse events, you can try QuicKeys X from CE Software for Mac OS X, anyway. It can enter text, too, and launch applescripts.
1) They're made by Sony 2) They can use the same size cassette shell 3) They have Beta in their name
They are all professional formats and are not the same as BetaMax, BetaMax II and EDBeta of old....
-- oldthinkers unbellyfeel ingsoc
Finally, I've discovered a use for this strange box on my desk.
I design user interfaces for a free network management application,
Over the years I've asked Rational salespeople if they plan on porting some of their more popular debugging tools to Linux, such as Purify, Pure Coverage, Quantify, etc. The earliest responses to the question were, "What's Linux?" In later years, the response has simply been "No". I do not understand why, but it has been a major sticking point for me. More and more, people are developing software for Linux, and it's hard to understand why companies like Rational won't embrace it.
I'll tell you, it's a major hassle to develop a product for Linux while being forced to debug it on Solaris because that's the only place our debug tools will run. The first company to make decent Linux tools that are similar to Rational's will have my business. And Rational won't.
AppleScript is great, but the app that you are testing must be scriptable. Many are today, but some are not. Furthermore, in this situation you want to test the GUI, not just the underlying object model which is what AppleScript talks to.
However, AppleScript is not the only scripting environment for the Mac. Underlying all of the native scripting languages on the Mac is something called the Open Scripting Architecture. This allows any OSA component (i.e. scripting language) to talk to any other OSA component.
One OSA component is QuicKeys, a great product that actually does simulate mouse clicks, keystrokes, etc., and is completely scriptable. It integrates completely with AppleScript and via the power of the OSA, with shell scripts and Perl as well.
On top of this, Apple provides a powerful GUI building tool for AppleScript called AppleScript Studio, which is free. Plus, AppleScripts can talk across a network to do RPC. Imagine, you can have an AppleScript Studio front end that drives scripts on a set of machines over the network (so that you can assess performance under load). The individual scripts rely on AppleScript to drive the logic while using QuicKeys sequences to drive the GUI.
The only issue here is how easily you can assess results. There's no general way to check what the GUI is showing other than a pair of Mark I mod 0 eyeballs, since grabbing data out of the app being tested (even simulating a copy-paste) only talks to the underlying data structures and does not address GUI bugs. A possible way is to arrange the windows in a known manner, turn off the menu bar clock, take a screen shot, and use a graphical comparison tool to XOR the bitmap with a known good screen shot to highlight differences, but this won't work if you're working with variable or randomly generated data.
Disclaimer: I work for Apple, but these are my opinions only and do not represent any sort of official endorsement.
--Paul
If you want a cross-platform automated testing tool that is:
1. Localizable
2. Non-Intrusive (tests the environment on which the user runs).
Then check out TestQuest .
While a bit pricey - consider it a capital expense - this system is fantastic. It was originally developed for testing medical devices and systems, where the FDA requires that what you test is what you ship exactly .
Although not a software solution, we use Test Quest to automate our reboot and application execution tests. Basically the hardware in an external controller machine (a second PC) simulates keyboard and mouse actions, along with a bunch of discrete lines that can be set high and low for things like reboot, power switch, and floppy drive. The system can also read back discrete lines for things like "is the system powered on?", and can take a screen capture to compare bitmaps, or do OCR to read results. It's all controlled by windows DLLs that can be written by anybody with a basic understanding of C, or pseudo-automatically written with the TestQuest recorder system.
They may have other newer systems, or a software solution for now, but for what we do it works great. We only test on PC hardware, but they have a variety of modules and interfaces for touchpads, VGA, svideo, Mac and Sparc hardware as well.
Here's a site which will give you opinions and feedback on a wide variety of tools.
QA Forums.
Scroll on down until you get to the Testing Tools forum.
Cthulhu Barata Nikto
Java has a Robot class that does this. It's pretty good, the only downside being that you have to specify locations by the pixel.
Check out java.awt.Robot.
~Kevin
A lot of open source projects could benifit from automated testing. The wine project has spent nine years working on translating the windows API to X and reproducing the windows GUI.
Many people think wine is a vital tool for greater use of linux. They rely on people downloading new versions and testing windows programs using whatever versions of X, system libraries and windows DLL's they happen to have and then posting bug reports in a newsgroup. There are still lots of quirks with relativly simple windows programs not redrawing or displaying dialogs correctly.
I believe that wine would progress much faster if the wine developers could make a change then click test and later, with no more effort get a report saying "3 tests which failed previously now pass, 1 test which passed on the last version now fails".
I will be very impressed if somone comes up with a free automated testing tool that allows tests to be set up easily and is powerfull enough to provide human readable results "the bitmap is drawn 1 pixel to the left", "text as expected but wrong font" "dialog appeared behind other window" etc.
DejaGnu is a TCL/Expect testing harness provided as free software by the Gnu Project. Not sure if it has tk hooks (which would allow you to test X-based programs) so you might want to check. If you're trying to test programs on the postscript display of a Mac instead of X, well, I don't know if it would work. Also, check to be sure there is a recent version of TCL/TK and Expect for your machine before you pursue this too far.
There is also LTP, the Linux Test Project, which may be of use to you as well. (Do a Google search for "linux test" and you'll find it quick.)
If you're not a TCL programmer, you're probably not going to like DejaGnu very much. If this is the case, good luck with the GUI tool search; you can expect to spend some serious money for decent test tools. (The pun is simply there. If you look at it too long or pay too much attention to it, it may or may not vanish. Just chalk it up to some crazy cat and don't worry about it.)
Best wishes,
Everything in the Universe sucks: It's the law!
Android runs on top of these and can be downloaded free for Linux and Unix
See the July 2001 issue of "Dr. Dobb's Journal"
for an overview of Android and how to download.
Plays back keyboard and mouse events for most UNIX implementations. (maybe even Mac OS X)
It is a little bit roll-your-own
Expect to spend some time writing perl to support your android scripts.
Letter To Iran
...aren't really necessary for functional testing, which is why most of these functional testing tools don't [more like can't] run in real time. IMO, these relatively pricey functional tools from the big venders could hardly be called reliable. I suspect that the tiny niche marketplace for such tools ensures that things will probably stay that way.
OTOH, Real life timing is really important for scalability testing, i.e. the domain of load testing tools, which mimic front end client behaviour in real time via back-end automation. These are definitely far more reliable, but also exceedingly more expensive.
ARexx used to do this on my Amiga back in '89. I remember that to launch applications using tooltypes defined in the icons (and to avoid launching via command line tooltypes in the startup-sequence), I ran an ARexx script that double-clicked on my drive, double-clicked on the Applications folder, and launched the apps I wanted running after I logged in. The script would then place focus in a new CLI window. All by scripting mouse events.
*Sniffle, wipes tear from eye* Ah, the good old days.
My company uses WinRunner for Windows GUI testing. I know that Mercury Interactive which makes WinRunner and LoadRunner another GUI Testing utility creates Unix version of their software in addition to the Windows versions. I don't know about Mac though.
Mercury Interactive
Hope that helps
Joe
Rational suite, including tests was a piece of crap when we worked with it. We switched to XP and everyone started having great time coding, due to allowance of concurrent programming...
If you want to get job done right, I would recommend eXtreeme Programming, over most other ideologies.
2c,
p.
I'm not completely sure about this, but I think you can use ATK (accessability toolkit) for sending events and getting widget contents to GTK apps. That works only onn GTK2, though, so perhaps it's not helpful to you, but at least I think the functionality is there.
AppleScript interfaces directly with the program at a lower level than the GUI -- it's essentially an alternate method of controlling the app, plugging into the same API as the GUI widgets do. This tests the app and everything below that, but it's not exactly the same as actually testing through the GUI itself, since you're bypassing it.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Tim Janik came out with a utility called Gerd a while back that lets you script GTK+ applications. It appears to have been abandoned at version 0.0.3, however.
This may be off topic in that you specified Mac OS X. If so, sorry! Anyhoo, Apple offered for the Classic Mac OS a developer tool called "Virtual User," which as I understand it does what you're asking for, and for all I know might just work on a Carbon application. It's available atg _- _Debugging/Virtual_User_Tools/
ftp://ftp.apple.com/developer/Tool_Chest/Testin
Hope that helps!
--Kimota
Who moderates the meta-moderators?
Can be used to create these sorts of scripts:
/ sourceforge.net/projects/abbot
http://sourceforge.net/projects/pounder
http:/
BTW, can you explain the bit where Mojo typed "Pray for Mojo"? It didn't make enough sense to me for it to be worth having the pet shop owner go inside, get a keyboard, and wait for the monkey to type that, so I figured it must be a reference to something I don't know about.
one hundred twenty
is just enough characters
to write a haiku
Use slashdotters. They can't bring anything to its knees.
No this is what apple means to :P
"Think Different"
--toq
There shouldn't be much of a problem doing this under any X environment, as Xevents of any kind are not hard to generate.
Ordo Militum Unix.
Didn't you even read your own link.
Beta died a lingering painful death due to inferiorty in that it used a 1-hour playback time when VHS was two hours. You could not record 1 whole movie on a Beta tape, whereas most movies could be recorded on the 2-hour VHS formet.
Pricing was probably an issue (Beta was usually a little more expensive), as well as luck and yes, even marketing. The lawsuit you mention was too little, too late to have killed Beta,
Technical quality of the playback was better on the Beta -- big deal, read your own link -- the average consumer could not even tell the difference even if their TV was capable of the difference.
With respect to Apple Mac vs. IBM PC & clones. Price & Applications were why business continued to by PC's. Mac was so hard to program (new and ugly paradigm you know), that it was a year before any real apps came out, and the cost of developing in-house apps was innordinately high.
A Lamborghini Countach is quite superior in many respects to the Toyota Avalon sitting in my driveway, but it is not a better car for me. Price, Insurance, convenience, capacity, etc. make the Toyota are far better product for me.
PC outsells Mac
Toyota outsells Lamborghini
VHS outsells Beta
All inferior in some respects, All better for the consumer in ways that mattered to them. I've got a PC, a Toyota, and a VHS recorder. I am not a stupid consumer buying stuff because of fancy commercial brainwash me into buying second-rate junk. I bought stuff them makes sense for me, for my needs. I assume most buyers of Macs, Lamborghinis or Betamax can make the same statement.
Here's an open-sourced, VNC-based user-event recording and playback tool:
t heatre.html
http://www.cs.sfu.ca/~walenste/personal/Research/
I am currently bet testing a Mac OSX program that uses TCP/IP and VNC to test software on any platform with a VNC server. The company is called Redstone Software - www.redstonesoftware.com.
Do not taunt Happy Fun Ball
I know this will get moderated down, but that's mostly because /. is group-think hell and because its redundant, but I have to get this out...
Basically, if you're responsible for stress testing software on an OS, you should at LEAST have a cursory understanding of the tools available on that OS... meaning, if you don't, how do you expect to really stress test these applications? If you didn't know applescript could do this, then you're stupid and you shouldn't be stress testing applications for anyone on any platform.
That said... moderators, mod me down.
"I'm getting moderated down for thinking differently than the rest of the k1dd13z..."
-C
"This above all, to thine own self be true"
Funny you should ask. The company I work for is developing software that I think does just what you asked for. It's a Mac OS X application (codenamed Eggplant) that can test software running on Mac OS X or Linux (or Windows or *nix for that matter).
Eggplant uses a TCP/IP connection to remotely control the system being tested, under script control. Because it interacts from the user's perspective, by generating mouse and keyboard events and watching what happens on the screen, Eggplant can test anything that has an interface, no matter what language or toolkit it was created with. I think it's pretty cool. And it's the only project I've ever worked on that had a purple vegetable as an app icon!
Check Redstone Software if you want to join the beta program (happening now). Version 1 should ship soon, but the marketing guys will probably come up with some suitably boring name by then...