Slashdot Mirror
Ask Slashdot: How Do News Organizations Keep Track of So Much Information?
dryriver writes: Major news organizations from CNN, BBC, ABC to TIME magazine, the New York Times and the Economist publish a tremendous amount of information, especially now that almost everybody runs a 24/7 updated website alongside their TV channel, magazine or newspaper. Question: How do news organizations actually keep track of what must be 1000s of pieces of incoming information that are processed into news stories every day? If they are using software to manage all this info -- which makes a lot of sense -- is it off-the-shelf software that anybody can buy, or do major news organizations typically commission IT/software contractors to build them a custom "Information Management System" or similar? If there is good off-the-shelf software for managing a lot of information, who makes it and what is it called?
There is an industry software that gets used a lot called iNews. There's a reddit thread with comments from people who work at news orgs. Vox Media (The Verge, SBNation, Curbed, Polygon) built its own CMS called Chorus. The NYTimes uses WordPress for some of its blogs. And I assume the Washington Post built their own since, well, Bezos.
Newsrooms depend on well-informed editors and reporters who are often notoriously paper-based. I've worked in six newspaper and magazine newsrooms and it's generally a central CMS/publishing/workflow system, plus a person-by-person armory of solutions (reporter's notebooks, things like Evernote, spreadsheets, etc.) There are systems used in intelligence that could find use, but journalists are kind of sensitive about doing things their own way -- in my experience. The real lifeblood of a newsroom is the channels of incoming info: wires, cable tv, Google News, etc.
Ah, I should add that many news agencies have another layer for information management called a newsroom computing system.
AP, the news-gathering organization, actually sells one of the main ones, called ENPS.
Avid has one a lot of people use, called iNews.
And then there is one a lot of organizations use called Octopus.
MAMs often integrate with ENPS, using a protocol called MOS. This allows you to associate assets in the MAM with placeholders in a rundown put together in the newsroom system, which is used for playout during a live production (off of separate video/gfx payout servers/systems).
I mean, information management in media/news is MANY layers of solution, I simplified it too much above.
Networking
Directory Services
Storage & Storage networking & archive systems
Cloud/hybrid scenarios
MAM/PAM/DAM
Newsroom computing systems
Video servers and video playout automation platforms for production and master control needs
Media transcoding systems
Workflow automation systems including ESB (Enterprise Service Bus)
A big trend is a move from monolithic systems, to SOA (Service-Oriented Architecture). Again, media is a bit behind some other industries in adopting IT. I think because our data is so big, it takes a while for certain elements of IT to be as relevant to us, because our bandwidth/storage needs are so great, relative to many other types of IT challenges. There are formats of 4K uncompressed video that are over a Gigabyte a second. Not many folks outside of Hollywood work in such formats, but still.
I do this for a living, so my answer is somewhat detailed.
Newspapers were using content management systems for this purpose beginning around 1970, before PCs. Previous to that, stories were transmitted electronically, stored on punch tape in a 6-bit format, but edited on paper and re-keyboarded as necessary.
If you wanted to use a story as-is, without editing, you could have a copyboy go find the right punch tape and hand-carry it to the typesetting department.
Computerizing the editing process/approval process allowed written material to be stored, edited on screens, and output directly to electronic typesetters (which were already computerized; a major use of the PDP-8 was automated hyphenation and justification). The story "files" were typically organized in "queues" or "baskets."
The earliest CMS were bespoke, but they quickly became standardized -- "off the shelf" with potentially a great deal of customization, produced by about a dozen companies around the world that often designed and built their own hardware components.
Electronic page layout was pioneered on these systems. One of the first was at the Minneapolis Star and Tribune; the project leader later created founded Aldus, created Pagemaker, and the desktop publishing revolution followed.
As desktop publishing emerged, it displaced bespoke layout systems, and networked PCs displaced proprietary terminals, and SQL databases displaced proprietary storage, but the putting them together into a usable workflow system remained a specialty. In general, the CMS companies got out of the hardware business entirely and focused on software and services.
Photos came later. Keep in mind that the JPEG standard didn't even exist until the 1990s. The first wirephoto storage-and-editing systems were big bespoke monsters that looked like something from a 1950s sci-fi serial, but they were quickly replaced by Mac-based tools, and then the core CMS systems embraced photo management.
Broadcasting trailed all of this in many ways. TV stations actually produce fairly little information in the common sense of the word, and have lighter requirements for handling text, but huge amounts of data in the form of video. When I first worked in TV, video was shot on film, then videotape. As video became digitized and companies like Avid created digital video editors, managing the data became a requirement there as well, and a specialty.
It's now possible to put together a text/image/video workflow system with open source tools. For a single publication, I could do it in a few days with Drupal, and if the Web is the target, it's all pretty straightforward. But the news CMS field is still dominated by specialty vendors.
Print is still a huge driver of revenue, and that means interfacing with advertising workflow and print page layout tools. Adobe InDesign is pretty much the standard there, although I know of one or two systems that have proprietary layout. As a result, a small (and shrinking) number of specialty vendors dominate. They integrate off-the-shelf components, including open source tools and commercial software.
Where I work, writers are using CKEditor, but it's implemented in a proprietary Web-based workflow system that publishes to multiple Drupal sites on the Web and integrates with InDesign for print. Wire service information, agency photos, etc., all come into the CMS.
Because most of the older legacy systems are utterly print-focused, they can be extremely frustrating in a digital world. Some news companies have assemble parallel production systems for the Web, stitching together any number of off-the-shelf components, or writing proprietary code. If you use Django, you should know that it was created at a newspaper company. The Washington Post has created its own system called Arc that it is peddling to other news companies.