Slashdot Mirror
Ask Slashdot: To Publish Change Logs Or Not?
Linnerd writes "A software company I work for has decided to no longer publish change logs when updated versions of the software are made available. A change log consists of sections pulled directly from the issue management system that is automatically processed into a spreadsheet. The spreadsheet can be sorted/viewed by many criteria, such as date of the fix, component affected, severity and more. There usually are a fair number of entries (sometimes more than 1000), because each update published contains all the accumulated changes made since some base release in the past and the change log has entries for everything from major bugs to minor improvements to documentation changes and spelling errors fixed. The main reasons for pulling the change logs was the fear of putting the software in a bad light and risking ridicule, especially from the competition. Although I can follow these arguments up to a point, I've personally always been more comfortable with software that had explicit and detailed change logs: Errors and bugs happen, whether they are communicated or not, and I'd rather know what was changed than blindly install some patch without knowing if it's relevant for the issues I'm trying to solve. What is your opinion? Should change logs / errors / bugs be communicated openly? How is this handled in the companies you work for? Can you provide publicly available references on the pros and cons of open and honest communication of changes and bug fixes, especially in commercial environments?"
Your customers are lucky, they get to know that something changed. If you were making 'cloud' software, they wouldn't know anything changed until they logged in one morning and things are broken.
"First they came for the slanderers and i said nothing."
I won't install an update without a change log.
*after* being acquired by dell. If you've worked with that equipment, you know what I'm talking about.
Anybody who has run software on a non-trivial scale knows how important changelogs are.
They give you some idea of what to expect, but more importantly let you know whether a problem you're having now has been fixed in the upgrade. Although developers would like everyone to run the newest version of software, in practice you don't touch production systems without good reason. Fixed pain points, and maybe security (depending on isolation) are valid reasons. "Because it's there" is not.
Elimination is a stupid move. It's a triumph of marketing at the cost of we who must run this shit.
Change logs and proper release notes are very appreciated by administrators and end users. Disclosure of known issues is particularly valued, and it also benefits the vendor because it reduces nuisance calls to technical support. Fixed, pending and wontfix lists are especially appreciated by sysadmins, since they are the ones most immediately impacted by the change - do they install the patch and deploy it immediately, or do they live with the current build until pending issues are fixed, etc.
Plus, it instills trust. A veil of secrecy does not earn trust from your customers, nor does a vague "fixed misc. bugs and implemented misc. performance enhancements" because one is more inclined to not upgrade rather than proceed with it and possibly risk downtime.
Of course I am saying this blindly, since the submitter did not specify what sort of software this is. Is it a server app? A commercial desktop app? Or is it a game or other entertainment software which is not mission-critical, where downtime can cost thousands to tends of thousands per hour?
The Christian Right is Neither (Christian nor right). See: Matthew 23, Matthew 25, Ezekiel 16:48-50
It is one of the major criteria I look for in good software --- open communication from the developers about updates, and changes; active recent update history.
Some of the most successful software companies such as Microsoft and VMware post detailed released notes that show bug fixes and improvements.
Detailed changelogs are even better.
If you are concerned about ridicule from a competitor -- you can probably point out the competitor hides their flaws by not posting detailed change logs.
Another thing you can do that's less recommended -- is put the changelogs behind a click-through agreement. Require site visitors register to review them.
Keep in mind it's not just customers that necessarily want to see documentation, release notes, and changelogs.
When I am considering buying a software product I want, EXPECT, and demand to see on the product vendor's website: (1) Pricing, (2) Release notes, (3) Documentation, (4) Change logs, and (5) A trial version download, before I even talk to a salesperson.
These are signs of a healthy well-marketed product, that will be around for years to come. If any of these are missing -- warning flags, or alarm bells are going off.
If the software doesn't have readily accessible documentation, or notes on bugs --- does anyone use this software in the real world?? Is the documentation crap??; If I buy this product -- is it going to be a complex pile - have spending hours upon hours on the phone with their support, working out bugs at every corner just to get the product up and running with basic functionality?
Will I need to make a phone call, to blow my nose with this software?
This is a boner move by your company. I've seen it happen before, and it's usually a sign of a failing company that fears reality.
Perhaps there was a recent management change? If yes, this is a sign that you've got a cowboy coming in with guns a-blazing and thinks he knows how to run a software shop... GTFO, while the gettin's good.
Different customers are going to feel differently, but in my experience doing releases, a different thing matters more.
The key is to make sure that each release is an improvement over the previous release. If you do that, your customers will be happy, and won't worry much, as you build trust and confidence over time. In your situation, if a particular customer requests the change logs, I would give them, but a general release is not important.
The real problems come when your releases break things. After an especially bad release, your customers might not be willing to upgrade for two or three years (see: Windows Vista). Make sure each release is better than the previous. That is crucial.
"First they came for the slanderers and i said nothing."
Does it have to be all or nothing?
Our software supports the hardware so we don't feel any palpable, fierce competition since we always lag the hardware improvements and those get top billing. Our users are interested in knowing the changes that are meaningful to them.
Since I use our change logs to significantly build our "brochure" touting reasons to upgrade, maybe you all can pare down change logs to the "best of the 1000" to say "20"... a list most can bother to go through.
That was Zen, this is Tao
I have 2 vendors that my company works for that drive me nuts with the lack of change logs. One posts change logs once a year or so, but has a dozen or so versions released annually. A call to tech support usually confirms that the changes don't affect us and aren't security related, but it's a pain to ask every time. The other fixes and changes a bunch of things with every release (and always breaks a few features that have worked for years...), but only documents a few trivial fixes & enhancements each time. I'm evaluating replacements for both vendors' products.
It is certainly useful to the person who filed bug 1234 to know that bug 1234 was closed in release number 5.7.
But... is the rest of the info in your issue tracking system formatted in any way that will be useful to a customer to read? At many of the places I've worked, we get issues with titles like "page formatted wrong, see attached screenshot". That is a pretty useless thing to automatically cut and paste into your spreadsheet. And severity reflects your internal judgement of severity, which may or may not bear any relation to severity-to-customers; something that is stopping other engineers from doing their work today is a Severe Bug and should be marked as such in your tracking system, but if it didn't get into the wild, customers don't need to know it ever existed. Similarly, we often check in code with placeholder text and file an issue to replace the placeholder text with approved final text; why would you ever want to show a customer a changelog that included the issue "replace placeholder text from initial checkin of feature X with final text"?
So: Like anything that is going to be delivered to a customer, this should be run through someone who checks it for relevance, and translates it into customer-facing language. The customer doesn't need to know about issues that never got into the wild, and the customer should have issues that were internally phrased in engineerspeak (because the issue tracking system is designed for use by engineers) rephrased into something useful; don't say "fixed server error 57 on badly formatted JSON", say "Fixed bug that caused web page to hang when user hit submit button" -- the first explains to an engineer what the issue that they're seeing in the logs that needs to be fixed is, the second explains to a customer what the problem they might have seen in the past is that isn't going to be there anymore.
"Open and Honest" communication needs to be clear and relevant communication as well. Don't bury your users in a list of a thousand lines of jargon they don't understand because you think more is better; give them a more compact list of the actual visible to them changes that were made, with the important things highlighted and brought to the top.
If it's an open-source project where you are giving out the source code, give them things like actual file diffs and commit logs too, because maybe some of your users are engineers hacking the source too. But most users are _users_, not engineers.
What reasons are there to NOT put them in?
The main reasons for pulling the change logs was the fear of putting the software in a bad light and risking ridicule, especially from the competition.
This is going to happen no matter what you do. If the competition is going to slander you, they will one way or another.
Another question is: Should I have facebook/twitter for my company, if the competetion can use it to slander us?
The answer is: Even if they try to, if you keep vocal on these feeds and try to solve all the grievences that arise, people will see that you give good support - even to trolls - and the competetion will look like trolls when they try to slander you.
Closing off communication does not enhance people opinions of you - but it does help the competition in making people believe that you are not doing good things.
I'm of the impression that if something was a problem but is fixed, the changelog pointing out the potentially "ridiculous" problem should be more of a badge of pride for resolving it. That said, there's no reason everything has to be included in a changelog, and even fixes that are included can be carefully stated so as to work around anything capable of being ridiculed or revealing additional holes in security or quality.
Since someone is worried about it, entries could be listed as "improved metafile handling" rather than "fixed horribly broken metafile handling". One could even institute a convention that all commits messages indicate the type of change by starting with the either "New Feature:" or "Improvement". Show the complainer a list of 200 new features and 500 improvements and they may see it's value to customers.
Barring that, as a customer I don't really NEED 1,000 entries. I might very well ask "what's new in this version?", though. Mark the 50 most significant ones and pull those for the changelog.
Some OSS bug trackers have a 'relnote' field where people are to enter a release note if the owner feels it's something that ought to be communicated to the user.
As a user, don't bother me with stuff that will never effect me (did you refactor a class? - great, but I don't need to know about it) or I won't need to look up (did you fix a CVE or a spelling mistake?) but do let me know about changes that will affect my workflow, will offer me an improved way of doing something, or should cause me to go revisit results I've generated in the past (i.e. you had a bug).
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
My employer produces a compiler tool chain for its products. Its release notes contain two major things:
1. A list of major customer visible changes.
2. A list of defects fixed
The first represents our internal development efforts. It's written in terms of the actual features, how they affect our users, and how the users ought to use them. They are not written in terms of the series of commits that made the features happen. That would just be pointless.
The second represents the defects fixed in this (and recent) releases, as pulled from the bug tracking system. If a customer filed a bug and we fixed that bug, that bug number and a brief description of the bug are in the release notes. Again, this is not tied to a commit stream that addressed the bug, but rather to defect reports that were closed by the release. Most of these defects come from external customers, but not all.
What's not in there? All the internal churn that got us from point A to point B. We distill it down to what the externally meaningful changes are.
Disclaimer: I am not on the team that produces the tool chain I described. I'm just a happy, internal customer of said tool-chain.
Program Intellivision!
From where I sit there's no question: for compliance reasons we must be able to detail exactly what is being changed when we install updates to software. If we can't, then we have to re-certify the software from scratch before it can be deployed which is a fairly tedious and time-consuming process. Having to do that with any regularity will find us looking for alternatives.
As for a list of bug-fixes making your software look bad, remember this: I know exactly how long our unresolved-issues list is for every single bit of commercial software we use, and exactly how long some of those issues have been sitting there. That your software has bugs that needed fixed too isn't going to come as any great surprise to me. What will impress me is a) actually fixing problems, and b) being clear about what you're fixing or changing which minimizes the amount of work I get saddled with.
If you have enterprise customers, then make the changelog available under NDA. Or just email it to key customers that ask, so that prospective customers never see it.
I have spent 15+ years in IT buying, selling, and using all manner of hardware and software. When a vendor changes something, I want to know. It's the easiest way to quickly see if some weird problem you are seeing can be corrected by installing an update.
While it will take more work, posting EVERY change is completely unnecessary, and oftentimes unhelpful. The publicly released changelog should include human-readable summary's of bugfixes and new features. If a customer asks for more details about a bugfix, feel free to give them the unedited version as you see fit. A changelog should NOT be a git history. Information overload is oftentimes less helpful than nothing at all because someone may spend hours to days sifting through it all only to figure out what they were looking for isn't even there.
Some company like to have put a lot of detail into the update notes even stuff like added this next to useless thing (default off) with a list of why it's next to useless and say it's not recommended to trun on.
And this was part of a bigger update.
Also some of updates due have stuff like fixed crash that some times happens when you do X.
The OP asks "Can you provide PUBLICLY AVAILABLE REFERENCES on the pros and cons of open and honest communication of changes and bug fixes, especially in commercial environments?"
Do any such references exist? I don't know of any.
However, I can attest to having watched one vendor who screwed up multiple times [not a software project] in some pretty major ways, and even though we gnashed our teeeth and wrung our hands, the up-fornt honesty of this vendor to admitting to mistakes (which were ultimately fixed) was actually a big positive in its favor.
Honest communications, no matter how negative they might seem at the time, do wonders in building confidence between a vendor and a customer.
Trust.
It's earned from open communication, then living up to your word. No disclosure, no trust, no investment.
As an employee, my next question would be, if they aren't communicating with their most valuable assets (customers), what aren't they communicating with their next most valuable assets (staff)?
When I've had to support an OS or application I've generally found it helpful for the release notes for an update to contain at least a summary of bug fixes since the last release as a minimum. That is assuming that the comments are detailed enough to be helpful. I don't necessarily need to see them all, but at least the major issues. That can help you decide if you need it to fix issues you are seeing, or if it could effect you. If your software has different collections of bundles of patches, such as a large standard bundle and various individual patches, you can see if you need to go outside of the standard bundle.
One example - A number of years ago there was a Solaris patch that wasn't part of the standard bundle, IIRC, that was needed to fix a problem that occurred only in certain advanced math functions on Ultrasparc III processors. If you were only doing generic database or web work you would be very unlikely to need it. But the applications we ran were very likely to be affected. So, seeing that, I grabbed and applied the patch where it was needed at my first opportunity. Not having a fix list would have prevented me from being able to make that choice and we would have taken a performance hit.
much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
I like to give my users a fairly high level changelog when putting out an update to something I've been working on. Usually its to call attention to new features or important bug fixes. However, I don't go into too much detail, lest I overwhelm or confuse them. Of course I'm also working on something used by average people, not IT admins.
Also, many of the changes tend to be things never directly visible to the user. These things include bugs fixed in core parts of the app the user is only vaguely aware of or updates to data formats and network protocols. Even if changes are visible to the user, sometimes they're simply too minor to call attention to them.
Mostly, I like to give my users a reason to want to upgrade, and to know that I've actually done something in the latest update.
not posting change logs is going to all but disqualify you from consideration from many shops, unless your offering is the only one that can work, or none of your competitors offer change logs. how can anyone manage a deployment to hundreds+ or thousands+ of users without knowing what is going to change.
the alternative is vastly longer deployment testing, since instead of testing what has changed and things related to it, you are essentially doing a ground up test of every single function used every way that is important to the customer.
either that or install and pray.
Snowden and Manning are heroes.
The change log is a product. It needs to be reviewed, readable to the target customers, and compliant to any necessary contractual, legal, or regulatory disclosures with the appropriate disclaimers. It should not reveal any trade secrets, third party confidential information, violate any vendor NDAs, have any unprofessional remarks about the customers, etc.
It sounds like the problem is you're putting out crap change logs using an automated system to copy things from the issue management system. Do you have policies in place to make sure people don't put crap into the issue management system? Are things being reviewed before the change logs are being put out? Is it being vetted by the necessary product/legal/regulatory folks to make sure nothing is in there that is going to bite you?
If a company published a crap product, then it will get bitten. When a company gets bitten, it's instinctive reaction is to stop putting out change logs to stop getting bitten, because that's the easy, lazy, doesn't take more effort answer. Asking "Whether or not change logs are a good idea?" is the wrong question. The right question is more "Okay, we got bitten because we put out crap change logs. How do we stop putting out crap?"
The answer to that question is generally something called 'Hard Work'. If the company isn't willing to put in the effort to make a good change log (appropriate policies to capture the relevant changes, tech writer/tech doc support to clean it up, manager-level review to vet it for compliance, etc.) Then, yes, it may make more business sense to not publish anything rather than to publish garbage. It's not a matter of whether or not change logs are good or bad -- good change logs are good, bad change logs are bad. The question is: How do you generate good change logs?
+1000 to all of the above.
We make a SaaS application. Every major release comes with a change log. Not the raw, unedited and complete change log TFA talks about, but a human readable, edited one of the form
- feature we promised
- change you asked for
- other change you asked for
- new surprise feature
- UI improvement foo
and finally....
- numerous bug fixes
Bugfix releases don't have a separate published change log. Instead, customers who reported a bug are notified directly when it has been fixed.
This way it is useful for the customer. What the customer really wants to know is that they're not paying you for drinking coffee. They don't want information overload. A complete change log would only make sense to people who work with the code.
Two reasons not to publish raw logs like this... (1) Security vulnerabilities in previous versions, unless you guarantee all your customers have updated, and your release notes follow days after a release to give them a chance to avoid being zero-dayed, and (2) Telegraphing new product direction by either the comments themselves, or in such a way that the changes in aggregate allow it to be inferred.
Either of these things will give customers the opportunity to bitch loudly, and even if you have 10,000 customers, 2 of them bitching loudly can lead to weeks of unnecessary meetings.
We encountered the same problem, so a few years ago, we started running two changelogs. One of them is the full changelog, with every ridiculously miniscule change listed. This is made available to customers, but not promoted to them.
The other is the 'enhancements only changelog' - or what we colloquially refer to as 'the readme'. It only contains feature enhancements or significant bug fixes.
We should call this practice stability through oscurity.
My first program:
Hell Segmentation fault
Absolutely. I ran into a kernel bug, which I helped fix. Now when installing a distro kernel I can check the changelog for my name to see if the bug that effects me is fixed in that kernel. I've done the same with firmware updates, like checking a RAID card firmware to see that it had a RAID 5 performance fix that effects me. For those things, a complete changelog is good.
For the main software I work on, the users normally want to know about the top 2 - 4 biggest changes.
Please. if there is a piece of software that I am using, the only reason that I *want* the changelogs are to see if it is worth upgrading and risking other bugs in order to fix the issue I'm stuck with. If there's software that I don't feel a horrendous burden from using it; I'd still like to know what has changed and why I should grab the newest version vs "misc bug fixes".
I worked in a large financial services company in Switzerland. We were one of the most intense users of a particular risk & control application. We understood each corner of this application and with each new release we analysed each change in detail. This was necessary to weigh-up the value of upgrading or not, or timing the upgrade appropriately. Some seemingly insignificant change could be a show-stopper for our users.
Unless the customer has a claim from you that "No Such Bug Exists, It Is Your Fault" and there is a previous or concurrent report in the changelog of a bug doing just that fault, AND that your software is sold under a far more generous and fair EULA than any shrinkwrap software in the world (and most bespoke or "enterprise class" software too) that gives redress for losses that are not limited to the value of the software.
Absent those precise coincident factors, there is no liability.
And if you're going to go "But you can sue for anything", that statement doesn't require changelogs either for it to be "true".
The main reasons for pulling the change logs was the fear of putting the software in a bad light ...
A lawyer got to them and told them they were exposing themselves to liability. Simple.
Changelog is the best way for user to find out about new features and bugfixes. If they don't know about them they don't use new features and don't get rid of their obsolete bug workarounds. If you don't post changelog you make entire program less useful and the work that went into making the update is partly wasted.
This is an ongoing problem for my work. People who refuse to write changelogs often say "read the code, becuase they documentation can be wrong". What I've found is that if their documentation is that wrong, their code is usually even worse. Just reading what the code does will force you to replicate fundamental errors, to preserve unstated API's. In such cases, the change logs or revision history are invaluable, to expose why particular features were altered and when and by _whom_.
If software hs good git logs, or other change control logs, a pointer to those rather than explicit change logs is often enough for my work. But the change logs of packaging systems, such as .deb or .rpm, can provide invaluable information about the changes in the packaging. And the _packaging_ is often not in a public git repository.
I'm a developer support engineer for a company that sells several SDKs - It is absolutely invaluable tor our customers (and ourselves) to be able to see the change logs as they're depending on our product to work in certain ways and could be interacting with dozens of systems/components.
I can't tell you how many times I've found that a claimed bug in our product was actually an issue in Weblogic or Websphere or Tomcat, etc.. that was corrected in a given fix (sadly, its often a case of customers coming to us and saying "this is a bug" and us diggin in only to find that yes it was a bug in that outdated version of the web application server they're using and they should have been doing their homework..
So both our own change logs and those of others are absolutely crucial in troubleshooting problems.
My personal $0.02: saying "here's what was fixed and when is not going to draw ridicule. However, having your software be a "big magic black box" is likely to alienate highly technical customers.
The Digital Sorceress
Uh, reading is fundamental. OP isn't suggesting anything, it's cowardly management that puts covering their own asses over providing a quality product.
Never underestimate the power of stupid people in large groups.
Oh. I thought we were asking Slashdot to publish change-logs.
Never mind.
You are welcome on my lawn.
The issue management system usually includes two fields and one check box. One internal text field that includes information about the issue in great detail aimed for the core developers, and another text fields which includes simplified information about the issue targeted for the end user or management, and then a check box which specifies if this issue should be publicly visible. Usually an issue has both fields filled out and the check box checked, but if an issue is set as private it's usually because it concern a security issue, it's a really minor fix or it's a major embarrassment for the company :)
Comment removed based on user account deletion
When I first started working in IT, back in my early software days (as a tech writer), we did assemble a release report out of the defect tracking system, but then groomed it so that the descriptions were meaningful, and they communicated what the customer would actually see a a change. So many software changes don't leave much of a trace or evidence of a change at all that at some point you need to focus on the things you fixed/enhanced that could be appreciated. Occasionally, when there was a meaningful grouping of individual changes around a particular feature or piece of functionality, we were better served by an umbrella description that spoke generally about a swath of modifications and the area of functionality they affected. Further, we would "sanitize" (ugh, I hate that word) the descriptions so that they did not speak explicitly about protected IP in a way that would permit a casual engineering user to write their own stuff using the description as a template. It was also necessary to moderate the tone of individual defect reports in the summary to ensure that they did not come off as alarmist or use hyperbolic language -- you don't want to send out a report that says "Bezier curve clippings, when extended outside sections of the layout area, cause catastrophic, random, immutable, artifacts to appear in layout prior to full system crash" when it's a problem that (while bad, and fixable) doesn't happen in most situations and is only reproduceable if you use the tool in nonsensical ways. We used to have a couple layers of editorial review by the development management and sometimes legal if there was a sticky topic, but this was more to check work than to artfully craft anything. I had to help write release notes thereafter as a developer and as a manager, and the spirit is the same. If there is a bug in your stuff, your user community has probably already shared it with the world anyway, so acknowledging faults and fixing things is part of the virtuous cycle. You can actually gain user trust by demonstrating that you can successfully identify and promptly fix issues as they come up. Sales can be helped, too, if you had to solve some specific problems (or add functionality) before a customer will make a purchasing decision... they can recognize that their voice is being heard and the product is being maintained in a way that is attentive to their needs. All the same, there's no reason to shoot yourself in the foot in addition to having to fix your problems or make your enhancements.
.. pa-ra-bo-la, pa-ra-bo-la, 2 pi R, 2 pi R, where's your latus rectum, where's your latus rectum, 2 pi R
Some people think we're crazy, but on the game "Card Hunter," we publish all of our check-in messages on Twitter.
Here are some of the benefits:
Obviously, we make an effort not to post things that are going to compromise our security.
Has there been a downside? It hasn't bitten us yet. There is usually no reason to hide what technologies you are using, unless you are using something that is highly vulnerable, or you are making other bad choices. Don't do that. There is no reason to hide that your software has bugs. Everyone knows you have bugs. It's only shameful if you aren't fixing them.
Are we really crazy?
Change-logs should be strictly for internal use! 1) Log entries will contain technical shorthand that is meaningless to novices. 2) Opening these logs up exposes the company to all kinds of liability. 3) Do you think most people want to wade through mostly trivial and redundant entries? The proper forum to introduce meaningful new features to the public is in a blog. It goes without saying that this blog should be actively maintained and have at least one person whose sole job is maintaining it and filtering information from the technical teams, packaging it for public consumption.
'He who has to break a thing to find out what it is, has left the path of wisdom.' -- Gandalf to Saruman