Slashdot Mirror
Ask Slashdot: How Do News Organizations Keep Track of So Much Information?
dryriver writes: Major news organizations from CNN, BBC, ABC to TIME magazine, the New York Times and the Economist publish a tremendous amount of information, especially now that almost everybody runs a 24/7 updated website alongside their TV channel, magazine or newspaper. Question: How do news organizations actually keep track of what must be 1000s of pieces of incoming information that are processed into news stories every day? If they are using software to manage all this info -- which makes a lot of sense -- is it off-the-shelf software that anybody can buy, or do major news organizations typically commission IT/software contractors to build them a custom "Information Management System" or similar? If there is good off-the-shelf software for managing a lot of information, who makes it and what is it called?
We're not doing your legwork for you.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
Excel spreadsheets tens of thousands of lines long.
There is an industry software that gets used a lot called iNews. There's a reddit thread with comments from people who work at news orgs. Vox Media (The Verge, SBNation, Curbed, Polygon) built its own CMS called Chorus. The NYTimes uses WordPress for some of its blogs. And I assume the Washington Post built their own since, well, Bezos.
Have you tried contacting and asking such an organization this very question?
Anons need not reply. Questions end with a question mark.
If it's the right, information is not needed. Just shout it loudly and it must be true.
Newsrooms depend on well-informed editors and reporters who are often notoriously paper-based. I've worked in six newspaper and magazine newsrooms and it's generally a central CMS/publishing/workflow system, plus a person-by-person armory of solutions (reporter's notebooks, things like Evernote, spreadsheets, etc.) There are systems used in intelligence that could find use, but journalists are kind of sensitive about doing things their own way -- in my experience. The real lifeblood of a newsroom is the channels of incoming info: wires, cable tv, Google News, etc.
I know this space well. My consulting/integration company works with many, many media companies including the majors on this exact area. AMA? I've been doing this for 13 years, and literally work with many of the largest media companies on the planet.
There are two layers to the answer to this question. The first is storage and networking infrastructure, which is evolving very quickly for many reasons. Object storage, cloud (public/private/hybrid) -- all of these trends are having a massive impact on how the industry does things, but media is 5-10 years behind many other industries in adopting IT to solve particular challenges (our data needs are very, very high). So the move to object and cloud storage, taking advantage of 10GigE much less 40 and 100, seeing where fibre channel goes (SANs are used very extensively), the changing cost environment for all this stuff -- all these things are hitting int he media space big time.
The next layer is the software management layer. We call this "MAM" for Media Asset Management. It's a bit of a catch-all term, and sort of folds up to DAM, or Digital Asset Management, and contains within it PAM, or Production Asset Management. It is sort of a shorthand term that refers to:
Getting your media and other data behind a database
Utilize software automation and integration technologies to orchestrate all sorts of interesting workflows
MAM too is taking more and more advantage of the cloud and hybrid deployments. There are dozens of MAM vendors, with a handful of leaders. For instance Avid has PAM and MAM platforms they brand as "Interplay" (it's two different things). There are dozens of others, and I know many of them quite well. Again, my company does major MAM and workflow deployments for top-tier global M&E companies (among others). If I can answer questions, shoot 'em over.
If you have ever read a news story where you have first hand knowledge of what is being reported, then you should know that most articles get a lot of facts wrong, and sometimes are wildly inaccurate. So the premise of the questions is wrong.
Q: How do news organizations keep track of so much info?
A: They don't.
I don't know what the news organizations use, but governments have some pretty big data sets and they use platforms like ckan and OGPL.
Nullius in verba
They dont, it is very obvious these days that they are not as fantastic as one once thought.
Their editors scour the news agencies, like Associated Press for what they deem "news-worthy". These are standarized gateways, web api for importing purchased articles, which get pushed into local CMS, then manually, or half-automatically laid out. Duplicates are avoided through marking all purchases. If anything newsworthy is announced ahead of time, and the "higher ups" want something exclusive, reporters are send to provide own scoop - but great most of data comes from the agencies.
Generally, a reporter working for a newspaper or media outlet directly is a much more rare sight than a reporter working for a news agency; news aggregated in the agencies and then distributed to news outlets.
Source: worked at a news portal. The token reporter team existed only so that the portal would be still protected by press law, as mere "news aggregation" media can't get that around here.
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
I do this for a living, so my answer is somewhat detailed.
Newspapers were using content management systems for this purpose beginning around 1970, before PCs. Previous to that, stories were transmitted electronically, stored on punch tape in a 6-bit format, but edited on paper and re-keyboarded as necessary.
If you wanted to use a story as-is, without editing, you could have a copyboy go find the right punch tape and hand-carry it to the typesetting department.
Computerizing the editing process/approval process allowed written material to be stored, edited on screens, and output directly to electronic typesetters (which were already computerized; a major use of the PDP-8 was automated hyphenation and justification). The story "files" were typically organized in "queues" or "baskets."
The earliest CMS were bespoke, but they quickly became standardized -- "off the shelf" with potentially a great deal of customization, produced by about a dozen companies around the world that often designed and built their own hardware components.
Electronic page layout was pioneered on these systems. One of the first was at the Minneapolis Star and Tribune; the project leader later created founded Aldus, created Pagemaker, and the desktop publishing revolution followed.
As desktop publishing emerged, it displaced bespoke layout systems, and networked PCs displaced proprietary terminals, and SQL databases displaced proprietary storage, but the putting them together into a usable workflow system remained a specialty. In general, the CMS companies got out of the hardware business entirely and focused on software and services.
Photos came later. Keep in mind that the JPEG standard didn't even exist until the 1990s. The first wirephoto storage-and-editing systems were big bespoke monsters that looked like something from a 1950s sci-fi serial, but they were quickly replaced by Mac-based tools, and then the core CMS systems embraced photo management.
Broadcasting trailed all of this in many ways. TV stations actually produce fairly little information in the common sense of the word, and have lighter requirements for handling text, but huge amounts of data in the form of video. When I first worked in TV, video was shot on film, then videotape. As video became digitized and companies like Avid created digital video editors, managing the data became a requirement there as well, and a specialty.
It's now possible to put together a text/image/video workflow system with open source tools. For a single publication, I could do it in a few days with Drupal, and if the Web is the target, it's all pretty straightforward. But the news CMS field is still dominated by specialty vendors.
Print is still a huge driver of revenue, and that means interfacing with advertising workflow and print page layout tools. Adobe InDesign is pretty much the standard there, although I know of one or two systems that have proprietary layout. As a result, a small (and shrinking) number of specialty vendors dominate. They integrate off-the-shelf components, including open source tools and commercial software.
Where I work, writers are using CKEditor, but it's implemented in a proprietary Web-based workflow system that publishes to multiple Drupal sites on the Web and integrates with InDesign for print. Wire service information, agency photos, etc., all come into the CMS.
Because most of the older legacy systems are utterly print-focused, they can be extremely frustrating in a digital world. Some news companies have assemble parallel production systems for the Web, stitching together any number of off-the-shelf components, or writing proprietary code. If you use Django, you should know that it was created at a newspaper company. The Washington Post has created its own system called Arc that it is peddling to other news companies.
Is this one of those middle-school moments and you are trying to say "NUH UH, YOU ARE!"? Try again. Five minutes on Briebart, Fox News, Twitter, or even Youtube shows how vile and disgusting the right has become with their support of fake news and conspiracy theories. None of you can live in reality it seems and need to try to build your own shared hallucination where the world is out to get you and all the lies, hate and violence you push for is justified.
You spend too much time in an echo chamber with your other alt-right crazies, because you're not making any sense.