Slashdot Mirror
Ask Slashdot: Best Test Case Manager Plugin For JIRA?
Bomarc writes: I've been working with software testing ... for a few years now. And there seems to be a serious lack of QA — Test Case Management (TCM) tools. The company that I'm working for needs a good test case manager. Currently JIRA is the tool of choice for other aspects of project management. I'm not asking to jump ship from JIRA, but the Atlassian TCM "Zephyr" has several problems, some of the key ones include: It does not have (any) matrix capabilities, no test case suite capabilities, if you change one test case (including assignments) the system changes all of the runs from that test case, the integration between the defect tracker and the TCM is archaic (at best), the number of actions to pass/fail a step (or test case) are annoying (way to many). Whoever designed it doesn't use it. If you watch the "Introduction" for Zephyr – it is amusing to see how the person performing he demo skips over and fumbles when dealing with the flaws I've mentioned above.
I have used the product "TestLog" which is a well-thought-out product; has test matrix capabilities (and other good features); however it does not have any integration with JIRA. (Hint, hint: Atlassian, this is what you need!).
Is there any company that makes a "plug-in" for JIRA with a similar features to TestLog – test case management that is well thought out, not just an afterthought?
I have used the product "TestLog" which is a well-thought-out product; has test matrix capabilities (and other good features); however it does not have any integration with JIRA. (Hint, hint: Atlassian, this is what you need!).
Is there any company that makes a "plug-in" for JIRA with a similar features to TestLog – test case management that is well thought out, not just an afterthought?
The last few places I've worked that use Jira for have used Enterprise Tester. Seems to do an okay job for the QA people.
http://catchsoftware.com/testi...
Okay, Testlog looks like an awful proprietary Windows-native tool. But if you can find a TCM with your required feature set and a documented API, it shouldn't be too big of a deal to make an own Jira plugin that interacts with that hypothetical TCM.
Maybe Testrail has those features?
Truth arises more readily from error than from confusion. -Francis Bacon
I have been testing TFS and got a parsing error at "(way to many)".
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
How about a test case manager for the crooks at dice? Stealing projucts, beta, and now pop overs? Fuck you
Silence is a state of mime.
We're just getting ready to publish or Test Case Management plugin to the Atlassian marketplace. I can provide trial installation packages to people interested in trying it out. Collaborative editing, tight integration with Jira, great UI. https://www.evernote.com/l/AAL... Comment if interested, I'll set up trial link.
We use TestRail (http://www.gurock.com/testrail/) with much success. It integrates with Jira (or not, up to you) and has a very simple and intuitive interface. I am a former Zephyr user and that experience is, in part, why I ended up with Testrail. I'm surprised it hasn't been mentioned yet.
Good luck.
TK
After a long break in research I've been doing client work again. This client is pretty big, a small European airline company. For some reason they have a lot of trouble getting the Jira suite of products to run stable. Stash is offline complete afternoons. I find this quite bizarre. But is this really the Jira software, or does it have to do with the client's sysadmin team?
8 of 13 people found this answer helpful. Did you?
When I read "jump ship from JIRA, but the Atlassian TCM Zephyr" the only conclusion is that software development has jumped the shark. This is something beyond jargon, totally gibberish.
I understand that TestTrack TCM has been integrated with JIRA at some shops.
Full disclosure: I work on this product.
Try ReqTest. As a developer I mostly use their bug tracker, but their test management seem competent and it has Jira integration.
We use RMsis; it's not the slickest interface in the whole world but it does everything we (medium size engineering company) want to do so far.
The people who support it are very good, and seem to be very amenable to requests for features and support.
"Cattle Prods solve most of life's little problems."
We switched a couple years ago to YouTrack from JIRA and have been quite happy with it. JIRA is a load of shit.
We use Xray (https://marketplace.atlassian.com/plugins/com.xpandit.plugins.xray) and it's very good specially if you use Cucumber
I have no idea WTF JIRA is, but I do know it causes incoherence amongst its advocates.
Troll much?
...I guess that don't like Jira?
We use standalone Zephyr in conjunction with JIRA, Confluence (both hosted OnDemand) and Stash.
What I can't work out is the justification for their pricing.
I pay Atlasssian about $10 per user per month.
Zephyr wants $80 per user per month, others mentioned are $25 per user per month.
Is test management really worth 5-10times as much as I pay for bug and content management ? Or 1000 times as much as I pay for source code and code review ? (Stash is $10 for a year for 10 users)
No.
So we continue to use the Community Edition of Zephyr... I really like Atlasssian's pricing model and they will continue to be my vendor of choice until someone else can match their price/performance combination*
*For more than 10users prices can get high...
...if this Slashvertisement would tell us what the hell JIRA is!
Automated testing only constatates the fact that bugs were introduced in the code at some time. They suffer from the fact that the test case itself may not catch all possible fault conditions of a function.
What most commercial projects lack comparatively to open source is the amount of dedication being given to bug tracking and establishing the chain of accountability. In Google, and later in Microsoft I found tickets being opened on bug trackers by automated QA being closed by project manager who aggressively push for WONTFIX and NOTABUG. Of course, such bugs were popping up again few months, just to be closed again. This did continue up until the real problem comes and these bugs become blockers.
Test cases have little to no place in Agile development. They are the finest example of waste. I know that a lot of places still feel that more test cases equals better quality, that is clearly not the case. Quality is derived from process, which can involve test cases, but again test cases are very inefficient at catching and preventing bugs. I have evaluated multiple solutions and they are all lacking.
While I do advocate some Scenario based testing, which usually are a single line like "Can you place an order", traditional test cases are convoluted and difficult to maintain. Google advocated test tours and ACC (Attribute - Component - Capability) testing that is very efficient, but still can be difficult to maintain. I believe the product owners should maintain the ACC Matrix, but they are too busy dreaming up new ideas. My teams have been using Google Docs for 3-4 years to track scenario based testing, while not perfect it gets the job down, is lightweight, multiple people can be in there, and just gets the job done.
The time would be better spent creating repeatable automation, through unit tests or functional testing solutions like Selenium. I am hoping in the next year or so to move all testing away from manual and just use automation. Get automation plugged into a Continuous Integrated system, so it runs all the time.
Source: Quality Engineering Manager -- Test Early, Test Often and Test What Matters
Half of writing history is hiding the truth.
Didn't you get the memo?
putting the 'B' in LGBTQ+
How do you feel about Behave, Spock, Cucumber, and Thucydides? My experience is that they work well with behavior driven testing. We are testing both APIs and integrating with Selenium using these tool sets.
My major gripe with automated testing is that it is just regression testing. It does not give you new information. Automation and exploratory testing are often at odds with each other. And management wants pretty reports and test case numbers and so carve out little time for exploratory testing.
putting the 'B' in LGBTQ+
We have JIRA inflicted on us at $WORK.
We use Tarantula for managing our test cases.
http://www.tarantula.fi
I'm really sorry, but the best test case management plugin for Jira is Zephyr.
My suggestion is that you actually better more flexibility out of creating custom issue types with fields and for your test cases and depend on issue linkage to handle things like traceability matrixes.
I would suggest adding the following text field types:
Steps To Complete
Expected Outcome
Actual Outcome
Ensure to build whether the test case succeeds or fails into the work flow for the ticket type. You can then create pretty dashboard graphs showing how testing is progressing among other things that you can't do with Zephyr using JIRA.
Zephyr sucks, I do not recommend it over vanilla JIRA customization.
Change is certain; progress is not obligatory.
"I am hoping in the next year or so to move all testing away from manual and just use automation."
Good luck with that, automation has a place but it does not include the replacement of thinking testing performed by informed humans. You must have an incredibly simple system if you think all relevant testing can be executed by a computer.
But they do when certain auditing requirements are involved. I do believe that any step based tests should be automated however and a better approach where auditing requirements are not involved would be to make use of developers doing unit testing and coverage using methodology like test/behaviour driven development.
Scenario testing doesn't deliver reasonable metrics consumption and can leave to numerous auditing issues. This is why I prefer session based testing and take methodologies like scenario testing and include it as part of session testing. There is sense in using certain manual methodologies outside of step-by-step test cases, since they counter issues such as pesticide paradox.
Google Docs is an extremely flexible environment and makes it hard to determine what is really being done. I will say as someone that has used a numerous amount of test management solutions (usually involving large projects with multiple teams) that JIRA certainly does fine for scenario and session based testing. JIRA does have the advantage that using JQL, you can trivially find holes in your data for organisational issues.
I will admit, for step-by-step test cases, JIRA is a bit lacking, but I'd still rather use it over solutions like HP Quality Center which were designed for test cases.
It's disappointing to hear Agile projects not running like this from the get-go.
Change is certain; progress is not obligatory.
You are correct and it has other flaws such as pesticide paradox.
Take a look at session based testing, which provides lots of pretty metrics that are understandable. James Bach wrote a considerable amount of information on why reporting things like test case numbers and relying on how many have been tested etc. do not actually mean much of anything.
If developers were handling unit testing properly, with code coverage, making use of test/behaviour driven development (as in, they are the ones developing the automation as they develop the solution), the testers would be spending time on quality aspects like session based testing rather than step-by-step testing.
This requires a mind set change, where developers can get instant feedback when something does not have the expected out come and fix it before actually doing a release (ensure a failure of a unit test, fails the build and unit tests are always executed when building on the CI env).
Change is certain; progress is not obligatory.
You've clearly not worked with companies where the standard is only step-by-step testing with linking to requirements and no other sort of sort of testing is allotted or even considered to be part of your work. You are either lucky or don't really have much testing experience across multiple firms.
Change is certain; progress is not obligatory.
am hoping in the next year or so to move all testing away from manual and just use automation. Get automation plugged into a Continuous Integrated system, so it runs all the time.
That works as long as you don't have a GUI.
We are forever finding those gosh-darn users will find weird and wonderful ways to use our UI in cases it was clearly not intended for or where the user-space bit calling the kernel-space bit via the commsy-space bit all falls down in one direction or another. Sure we have tested-to-hell drivers with automated tests and that is a god-send, but those UIs?
Someone has to [right|left|double] click those UI buttons and open those window panes and make sure that it all still looks and reports like it did before.
I cannot see how that can ever be automated.
'Won't someone think of the users'!* ;)
* Translation: Won't someone think of the support team dealing with the users via the UI!
Have you considered qTest from QASymphony? - http://bit.ly/1SQpoDI\
They have a great integration with JIRA on both the requirements and defects level, with a bidirectional sync that makes it really easy to keep everything up to date. Sounds like it might be worth you checking out
I used Jira in a few companies after it was pushed down by management because it was cheaper than its competitors. Products like Fogbugz and VersionOne put it to shame in area of testing. There is so much clutter and "flexibility" that you will spend most of the time trying to figure out the right way to use Jira. A typical company will fail same as the previous 3 work environments where I ran into Jira...
The way we've done that issue is having a programmatic interface to provide the "click" inputs to the app. Different projects have either done this as an internal function you can call (e.g. through a debug console while running) or through a second app which sets up a remote connection and issues the commands over the network.
Chaining commands simulates user input, at which point you can validate in a few different ways:
* Check "is X displayed?" and position on-screen programmatically
* Render the screen and do a diff against a "known good" image, through tools like imagemagik
* anything else you want
The end result is similar to Selenium in some high-level respects but can work for GUI-only apps, and for platforms other than PC (e.g. game consoles, windows apps, browsers, etc.) Of course, maintenance is still a major consideration for this type of test, so ROI becomes an important consideration.
Have you checked out qTest?
Hi,
This is essentially one of the reasons I decided to write yet another testing framework (currently just a set of low level APIs). You will be interested in the Use-Case feature, which is hopefully coming out in a few months. It will have a xml file for each Use-Case, which you can include in your test run so you can match up your Use-Cases (a.k.a. Stories) to your tests.
https://www.linkedin.com/pulse/fabricate-tests4j-now-githubcom-scott-morgan
The java code will look something like this;
@UseCaseScope (system=@UseCaseSystem(name="Your System", useCase="Your Use Case")
public class YourTestClass extends UseCaseTrial {
@Test //the name "Basic" is impiled
@UseCaseFlow
public void testSomething() {
}
@Test
@UseCaseFlow (name="exceptionFlowOne")
public void testSomething() {
}
}
Intelligence4j would be a replacement for JIRA, much in the way Tests4j is a replacement for JUnit, and Fabricate is a replacement for Maven.
Intelligence4j will also do much more than JIRA, since it will extract aggregation of test results from one or more CI servers (i.e. Jenkins).
What are your thoughts?
If you like this idea, Adligo could use some sponsorship (donations treated as revenue, hopefully in large sums from companies so they can write off the amounts since there hiring me to improve a free product), email me for specifics at scott@adligo.com.
Cheers,
Scott
I would like to point you at Klaros-Testmanagement which features a close integration with Jira even in the free Community Edition. Working on the implementation of it I am of course biased.