Slashdot Mirror
Ask Slashdot: Best Test Case Manager Plugin For JIRA?
Bomarc writes: I've been working with software testing ... for a few years now. And there seems to be a serious lack of QA — Test Case Management (TCM) tools. The company that I'm working for needs a good test case manager. Currently JIRA is the tool of choice for other aspects of project management. I'm not asking to jump ship from JIRA, but the Atlassian TCM "Zephyr" has several problems, some of the key ones include: It does not have (any) matrix capabilities, no test case suite capabilities, if you change one test case (including assignments) the system changes all of the runs from that test case, the integration between the defect tracker and the TCM is archaic (at best), the number of actions to pass/fail a step (or test case) are annoying (way to many). Whoever designed it doesn't use it. If you watch the "Introduction" for Zephyr – it is amusing to see how the person performing he demo skips over and fumbles when dealing with the flaws I've mentioned above.
I have used the product "TestLog" which is a well-thought-out product; has test matrix capabilities (and other good features); however it does not have any integration with JIRA. (Hint, hint: Atlassian, this is what you need!).
Is there any company that makes a "plug-in" for JIRA with a similar features to TestLog – test case management that is well thought out, not just an afterthought?
I have used the product "TestLog" which is a well-thought-out product; has test matrix capabilities (and other good features); however it does not have any integration with JIRA. (Hint, hint: Atlassian, this is what you need!).
Is there any company that makes a "plug-in" for JIRA with a similar features to TestLog – test case management that is well thought out, not just an afterthought?
The last few places I've worked that use Jira for have used Enterprise Tester. Seems to do an okay job for the QA people.
http://catchsoftware.com/testi...
Okay, Testlog looks like an awful proprietary Windows-native tool. But if you can find a TCM with your required feature set and a documented API, it shouldn't be too big of a deal to make an own Jira plugin that interacts with that hypothetical TCM.
Maybe Testrail has those features?
Truth arises more readily from error than from confusion. -Francis Bacon
I have been testing TFS and got a parsing error at "(way to many)".
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
How about a test case manager for the crooks at dice? Stealing projucts, beta, and now pop overs? Fuck you
Silence is a state of mime.
We're just getting ready to publish or Test Case Management plugin to the Atlassian marketplace. I can provide trial installation packages to people interested in trying it out. Collaborative editing, tight integration with Jira, great UI. https://www.evernote.com/l/AAL... Comment if interested, I'll set up trial link.
We use TestRail (http://www.gurock.com/testrail/) with much success. It integrates with Jira (or not, up to you) and has a very simple and intuitive interface. I am a former Zephyr user and that experience is, in part, why I ended up with Testrail. I'm surprised it hasn't been mentioned yet.
Good luck.
TK
After a long break in research I've been doing client work again. This client is pretty big, a small European airline company. For some reason they have a lot of trouble getting the Jira suite of products to run stable. Stash is offline complete afternoons. I find this quite bizarre. But is this really the Jira software, or does it have to do with the client's sysadmin team?
8 of 13 people found this answer helpful. Did you?
I understand that TestTrack TCM has been integrated with JIRA at some shops.
Full disclosure: I work on this product.
Try ReqTest. As a developer I mostly use their bug tracker, but their test management seem competent and it has Jira integration.
We use RMsis; it's not the slickest interface in the whole world but it does everything we (medium size engineering company) want to do so far.
The people who support it are very good, and seem to be very amenable to requests for features and support.
"Cattle Prods solve most of life's little problems."
We switched a couple years ago to YouTrack from JIRA and have been quite happy with it. JIRA is a load of shit.
We use standalone Zephyr in conjunction with JIRA, Confluence (both hosted OnDemand) and Stash.
What I can't work out is the justification for their pricing.
I pay Atlasssian about $10 per user per month.
Zephyr wants $80 per user per month, others mentioned are $25 per user per month.
Is test management really worth 5-10times as much as I pay for bug and content management ? Or 1000 times as much as I pay for source code and code review ? (Stash is $10 for a year for 10 users)
No.
So we continue to use the Community Edition of Zephyr... I really like Atlasssian's pricing model and they will continue to be my vendor of choice until someone else can match their price/performance combination*
*For more than 10users prices can get high...
...if this Slashvertisement would tell us what the hell JIRA is!
Test cases have little to no place in Agile development. They are the finest example of waste. I know that a lot of places still feel that more test cases equals better quality, that is clearly not the case. Quality is derived from process, which can involve test cases, but again test cases are very inefficient at catching and preventing bugs. I have evaluated multiple solutions and they are all lacking.
While I do advocate some Scenario based testing, which usually are a single line like "Can you place an order", traditional test cases are convoluted and difficult to maintain. Google advocated test tours and ACC (Attribute - Component - Capability) testing that is very efficient, but still can be difficult to maintain. I believe the product owners should maintain the ACC Matrix, but they are too busy dreaming up new ideas. My teams have been using Google Docs for 3-4 years to track scenario based testing, while not perfect it gets the job down, is lightweight, multiple people can be in there, and just gets the job done.
The time would be better spent creating repeatable automation, through unit tests or functional testing solutions like Selenium. I am hoping in the next year or so to move all testing away from manual and just use automation. Get automation plugged into a Continuous Integrated system, so it runs all the time.
Source: Quality Engineering Manager -- Test Early, Test Often and Test What Matters
Half of writing history is hiding the truth.
Didn't you get the memo?
putting the 'B' in LGBTQ+
How do you feel about Behave, Spock, Cucumber, and Thucydides? My experience is that they work well with behavior driven testing. We are testing both APIs and integrating with Selenium using these tool sets.
My major gripe with automated testing is that it is just regression testing. It does not give you new information. Automation and exploratory testing are often at odds with each other. And management wants pretty reports and test case numbers and so carve out little time for exploratory testing.
putting the 'B' in LGBTQ+
"They suffer from the fact that the test case itself may not catch all possible fault conditions of a function."
If we could determine all possible fault states of a function we could write flawless code.
"opened on bug trackers by automated QA being closed by project manager who aggressively push for WONTFIX and NOTABUG"
This is a personnel issue. I've been there and aggressively pushed for fixes. If I find the same bug closed and then it is logged as a new bug I make sure the earlier bug is reopened and the two are linked (We used ALM which allowed links from stories to tasks to test cases and to defects). That way the average defect age spikes which creates question among the managers. I also emailed various development leads, product owners etc. so that the person trying to avoid the bug could not hide.
I won in most cases. It was only when there was a consensus from numerous people that the defect could be deferred that it was set to a state of 'deferred' rather than closed. That way you could keep tabs on how things were going.
putting the 'B' in LGBTQ+
I'm really sorry, but the best test case management plugin for Jira is Zephyr.
My suggestion is that you actually better more flexibility out of creating custom issue types with fields and for your test cases and depend on issue linkage to handle things like traceability matrixes.
I would suggest adding the following text field types:
Steps To Complete
Expected Outcome
Actual Outcome
Ensure to build whether the test case succeeds or fails into the work flow for the ticket type. You can then create pretty dashboard graphs showing how testing is progressing among other things that you can't do with Zephyr using JIRA.
Zephyr sucks, I do not recommend it over vanilla JIRA customization.
Change is certain; progress is not obligatory.
But they do when certain auditing requirements are involved. I do believe that any step based tests should be automated however and a better approach where auditing requirements are not involved would be to make use of developers doing unit testing and coverage using methodology like test/behaviour driven development.
Scenario testing doesn't deliver reasonable metrics consumption and can leave to numerous auditing issues. This is why I prefer session based testing and take methodologies like scenario testing and include it as part of session testing. There is sense in using certain manual methodologies outside of step-by-step test cases, since they counter issues such as pesticide paradox.
Google Docs is an extremely flexible environment and makes it hard to determine what is really being done. I will say as someone that has used a numerous amount of test management solutions (usually involving large projects with multiple teams) that JIRA certainly does fine for scenario and session based testing. JIRA does have the advantage that using JQL, you can trivially find holes in your data for organisational issues.
I will admit, for step-by-step test cases, JIRA is a bit lacking, but I'd still rather use it over solutions like HP Quality Center which were designed for test cases.
It's disappointing to hear Agile projects not running like this from the get-go.
Change is certain; progress is not obligatory.
You are correct and it has other flaws such as pesticide paradox.
Take a look at session based testing, which provides lots of pretty metrics that are understandable. James Bach wrote a considerable amount of information on why reporting things like test case numbers and relying on how many have been tested etc. do not actually mean much of anything.
If developers were handling unit testing properly, with code coverage, making use of test/behaviour driven development (as in, they are the ones developing the automation as they develop the solution), the testers would be spending time on quality aspects like session based testing rather than step-by-step testing.
This requires a mind set change, where developers can get instant feedback when something does not have the expected out come and fix it before actually doing a release (ensure a failure of a unit test, fails the build and unit tests are always executed when building on the CI env).
Change is certain; progress is not obligatory.
You've clearly not worked with companies where the standard is only step-by-step testing with linking to requirements and no other sort of sort of testing is allotted or even considered to be part of your work. You are either lucky or don't really have much testing experience across multiple firms.
Change is certain; progress is not obligatory.
I would like to point you at Klaros-Testmanagement which features a close integration with Jira even in the free Community Edition. Working on the implementation of it I am of course biased.