The Rise of Machine-Written Journalism

Hugh Pickens writes "Peter Kirwan has an interesting article in Wired UK on the emergence of software that automates the collection, evaluation, and even reporting of news events. Thomson Reuters, the world's largest news agency, has started moving down this path, courtesy of an intriguing product with the nondescript name NewsScope, a machine-readable news service designed for financial institutions that make their money from automated, event-driven trading. The latest iteration of NewsScope 'scans and automatically extracts critical pieces of information' from US corporate press releases, eliminating the 'manual processes' that have traditionally kept so many financial journalists in gainful employment. At Northwestern University, a group of computer science and journalism students have developed a program called Stats Monkey that uses statistical data to generate news reports on baseball games. Stats Monkey identifies the players who change the course of games, alongside specific turning points in the action. The rest of the process involves on-the-fly assembly of templated 'narrative arcs' to describe the action in a format recognizable as a news story. 'No doubt Kurt Cagle, editor of XMLToday.org, was engaging in a bit of provocation when he recently suggested that an intelligent agent might win a Pulitzer Prize by 2030,' writes Kirwin. 'Of course, it won't be the software that takes home the prize: it'll be the programmers who wrote the code in the first place, something that Joseph Pultizer could never have anticipated.'"

Comments

John Henry

[Akido37]

Another "machines will take my job" story. This is as old as technology itself.

As with all other technologies, the future will be vastly different than what we envision.

Re:Niggers

[selven]

If that's a demonstration I feel sorry for the people who think we'll get anywhere by 2030.

nonsense

[symes]

Well-written prose is far from formulaic. While financial institutions and baseball enthusiasts may happily forego a penetrating understanding of a situations meaning and emotions the literate will not.

Re:nonsense

[Anonymous Coward]

Just wait until they even try to automate spin!

enum bias {
          ULTRA_LIBERAL,
          FAIRLY_LIBERAL,
          JUST_LEFT_OF_CENTER,
          JUST_RIGHT_OF_CENTER,
          FAIRLY_CONSERVATIVE,
          ULTRA_CONSERVATIVE,
          RUPERT_MURDOCH

};

At least there's now an automated process for it.

Re:nonsense

[cgenman]

This strikes me as the sort of thing that, without excessive amounts of variation, would get filtered out quickly by the general public. Sure, a machine can write *one* story on a baseball game that is interesting to read. But after the hundredth version of the same story that you've read, the public would stop reading the text entirely and just filter for the important bits. At that point, you might as well just have a table with the interesting stats.

The challenge isn't to write one story. It's to create a machine that can write N stories that remain interesting and fresh, and with less effort and cost than it would take journalists to just write N stories the traditional way.

Re:nonsense

[causality]

Feeling defensive, eh? Thanks for informing everyone you're "literate" and therefore better than those filthy ordinary people. Pick up a newspaper, and tell me how much writing would actually be *improved* with a machine writer, eh? Writing isn't sacred, it's just another occupation like woodchopping or running the cash register at the 7-11.

What you describe is "everyday" writing. This is like the vending machine that reads "insert dollar bills face up" or "team A scored 10 points while the opposing Team B scored 15." It's purely practical, get-the-job-done sort of writing where only the technical correctness is important.

Writing can also be beautiful, powerful, and artistic. A well-written editorial, penned by someone who has a deep understanding of the issues, can and has moved entire communities to change their minds on important issues. A beautifully written book tells the perceptive reader as much about the mind and spirit of the author as it does about the story that is being told. A horror story requires some sense of what is horrible.

The ability to both produce and appreciate good writing is on the decline. Measured scholastically, the grade-level at which the average American reads and writes is significantly lower now than it was say, 50 years ago. A good look at many online forums will also tell you that this skill is not highly valued. You can say that's because only some cultural elite are capable of enjoying it, though the GP made no such claim. You can also say that there is simply less interest in such things.

You can invent and try to substantiate any number of unique explanations for it. The truth of the matter is that in most situations, challenging your readers is now considered highly undesirable. They'll read a competitor's paper written in simpler langauge before they'll grab a dictionary. There was a time when this would have been viewed as laziness and an unwillingness to meet a worthy challenge. It would have been viewed like a wasted opportunity to better yourself that should not have been passed up, just like most people today would not like to pass up a higher-paying job. I'm not saying that previous generations were one homogeneous block who all felt this way, but many more people once did.

Over the last few generations, some kind of cultural shift occurred. People now care much more about avoiding the small-but-significant effort of learning something new than they care about improving important skills. They generally won't do it at all unless it's required by an authority like a boss or a professor. Even then it's in a rote, mechanical sort of way that deprives them of true appreciation. Even then it's reluctant, with a "gun to their head" in the form of losing their jobs or failing school.

Can you comprehend how sad that is?

Re:nonsense

[Ethanol-fueled]

Yep, it's sad. Eloquence has gone out of style. Being a man sucks in this day and age. People think a man is a weirdo or a fag if he reads books by the pool and uses words more than 4 syllables long.

It's an old argument. Anything can be mechanized - art, music, programming, writing, manufacturing, rolling joints - but if you want the good shit then you'd better make sure that there's a human behind it. There will always be a need for handcrafted stuff.

Everytime a nerd calls fractals "art", God kills a kitten.

Censorship

[ground.zero.612]

A great fear of mine is that a machine will decide what I should or should not know about. Another is that a machine like this could be tampered with by any human being to make the same decision.

Big Brother SkyNet is watching you, and telling you all you need to know.

Re:Censorship

[ubrgeek]

Already happened. It's called "Fox News." (Or "MSNBC," depending on one's leanings.)

The reporter is now a touch more obsolete

[onyxruby]

News agencies have already been turned into commodities, they just don't realize it yet. Now the reporter is being sent down that same drain. With original reporting set to become a 'premium' by the news agencies, their market is only shrinking.

Where were the reporters when millions of jobs were outsourced by H1B's or sent overseas? At best most stories were brief, with no follow up, and no outrage at the loss of middle class America. The same thing has happened in Europe and elsewhere as well.

Now the reporter faces the inevitable market forces that they previously ignored, and they expect anyone left to care? The programs will only get better, the markets and stories it applies to will only improve, and for the vast majority of stories the quality will be imperceivable to the average person.

Re:The reporter is now a touch more obsolete

[Anonymous Coward]

There was a time when the collective consciousness of this country embraced technology as a way to free humans from the mundane activities of life. No more work to do, all the menial tasks performed by machines. A utopia of leisure and enlightenment.

This changed over the last hundred years into the dystopia view of the terminator and 1984. Technology advanced but economic and political theory has not. Machines became the enemy because we made them competitors. There will soon be a time when there simply are not enough jobs to be done by humans. People will become obsolete in the work force.

The kick in the ass is that we already have the resources to give every person on the planet a decent life. That surplus of labor and resources will only increase as technology continues to advance.

So why not a utopia? If everyone was equal and happy and content then there is no reason to be rich. The best thing about being rich it getting to shove it in the face of the poor (and the point is for everyone to be poor).

This story is really about the friction between the natural evolution of technology and the need for our masters to make sure we never see the benefits of it..

Re:The reporter is now a touch more obsolete

[Aladrin]

Where were they? They were cherry-picking the easy stories that they knew would get viewers. Stories about H1Bs are hard because those happen 1 at a time and only the collective tale of ALL of them means anything. Who do you interview? What do you actually report?

After the fact, we can point and say 'Wow! That's a lot of jobs lost!' and complain about it. While it's happening, it's nearly invisible.

I agree with your point, though... Reporters are not covering the -real- news. That's why sites like Slashdot exist now. (Not saying ./ is perfect, mind... Just that it is responding to the gap left by 'real' reporters.)

Re:The reporter is now a touch more obsolete

[scamper_22]

There are really 2 good ways to handle this.

1. To place artificial barriers to entry. For example, you could say that any piece of 'news' must be presented by a certified journalist. Just like any prescription must be done by a doctor. Or lawyers have their own provisions. For programmers you could have certain requirements like any piece of software in use must have a certain number of maintainers... Basically turning such jobs into professions with the same level of protection as doctors, lawyers... Oh I'm sure we can come up with some excuse like 'quality' to enforce this :)

2. We embrace the lowering of costs and focus on reducing our cost of living. As good private sector jobs (auto-workers, engineers...) go away, that high-end tax base drops as well. So the payment to the public sector should drop as well. There is no intrinsic reason a teacher should earn more than a waitress. I know I'd rather be a teacher than a fast food server. I've been both :) We can then focus on low property taxes... having a lost cost of living. And we would all be able to have more 'stuff' as things would cost very little. We could simplify the legal and tax system as well as remove the medical monopoly from doctors...

The alternative is the current unstable system. Whereby those in the private unregulated sector keep pushing themselves to efficiency and 0 cost. While those in regulated professions and the public sector keep pushing for more. Basically a form on indentured servitude by those in the public sector and regulated profession upon those in the private sector.

Re:The reporter is now a touch more obsolete

[Aladrin]

It's not racism at all. It's nationalism.

And as far as the common man is concerned, he sees his friends out of work and jobs going overseas. He doesn't understand the complexities involved or that his friends probably can't even DO the job that's being sent there... He just sees a 'lost job'.

But you've gotten away from the point: This is about news reporting and how hard it is to report news other than the usual drivel that they report.

What?!

[Tobor the Eighth Man]

"The latest iteration of NewsScope 'scans and automatically extracts critical pieces of information' from US corporate press releases"

Extracting useful info from press releases? This must be absolutely amazing software.

Re:What?!

[natehoy]

I'm pretty convinced most corporate press releases are machine-generated anyway, so it should be a matter of reverse-compiling them back into plain English and including that as part of a story.

ABC Co. CEO to PressBot: "The market totally screwed us. The building is collapsing because we can't afford maintenance. We have to lay everyone off and we'll be out of business in three months. We can't afford exterminators so weasels are chewing my genitalia into mush."

PressBot's press release: "The company continues to leverage circular market forces to tighten its bottom line, particularly in the area of vertical integration. Resources are plentiful enough, however, that all employees will be allowed to pursue innovative new ideas in an open, creative setting, with plenty of personal time. The CEO is actively involved in the belt-tightening process and has taken steps to ensure that only underutilized corporate assets will be liquidated."

A human or a webbot could probably gather equally-useful information out of it.

Breaking news...

[natehoy]

News flash: Robotic reports indicate that all humans have died.

Oops, sorry, that was a programming error. The robots haven't figured out verb tenses yet.

Update: Ten, nine, eight...

Mod parent up.

[khasim]

This is nothing more than extracting stats and then placing them in pre-generated sentences.

In sports, this is okay. Except when something interesting happens like someone head-butting another player.

Anyone want to place a bet on how long before companies are accused of "gaming" the financial reporting system with their press releases?

Re:Mod parent up.

[PCM2]

This is nothing more than extracting stats and then placing them in pre-generated sentences.

In sports, this is okay.

Countless generations of sports writers and the enthusiasts who read their work would disagree with you.

Re:Mod parent up.

[Archangel Michael]

Anyone want to place a bet on how long before companies are accused of "gaming" the financial reporting system with their press releases?

Inevitable. Most of the Financial world is overstated swings in outlooks, leading to crazy stock price gains and losses.

I have a very simple solution to daily manipulation of financial manipulation. A sliding scale of capital gains taxes, based solely upon how long one owns the financial instrument they are trading.

10 years capital gains tax free.

Or something like that. The problems with our current market are due either directly or indirectly to short term outlooks to income generation, ie next quarters profit/loss statements.

Holding a stock or bond long term is almost insane these days. YET that is the purpose of stocks / bonds, long term financing.

That is my solution.

Re:Mod parent up.

[Tekfactory]

I saw one of the stocks I owned go up when Company A released a press release that Company A signed a deal with company B.

The stock of Company A spiked again 3 days later when Company B released a press release that it had just signed a deal with Company A.

If there are quant systems out there listing to the wire and trading on info like this, the system will surely be gamed. What is worse is that if a human were watching the blips come over the wire would he necessarily catch the problem?

They've been doing crap like this in their accounting for years, Enron charges X to company Y, and Y charges X back to Enron, both of them had 2X extra sales in the quarter, but no money or goods actually change hands, now it extends to journalism.

Re:Mod parent up.

[lgw]

The purpose of stocks is for a company to raise capital by selling shares to the public, in return for some promise of potential future dividends.

The fact that people trade, hold, or speculate on these stocks in the secondary market (all the busy noise that is the "stock market") is nearly irrelevent. A company should have no reason to care about the price of its stock. Sadly, due to double-taxation of dividends, this has gone completely to shit. People who aren't speculating buy stock not for dividends, but to trade it to the next guy at a profit, because this is tax-favorable over dividends.

Companies do all sorts of crazy BS because of the expectation of stockholders to be rewarded not with dividends (a system that you simply can't game for long) but with rising stock prices (a system that is almost entirely gamesmanship).

Individual investors have the absolute right to seek short term gains or long term gains a their preference. The government has no business meddling in that preference. The "problem with the current market" is that it is no longer grounded in the reality of being able to pay dividends, because some previous generation's ideas about social engineering through taxation punishes dividends as the means of earning a profit on one's investment.

Nobody gets it.

[headkase]

Spam is where it's at. Spam is where we are going to see strong artificial intelligence emerge, both defensively and offensively. Spam already represents some of the most cutting-edge algorithms in machine learning today. Think about it. In the undefined when of the future: you will have AI that stops spam. Spam will be AI that attempts to get through your filters. The only spam your AI will let through is spam you are genuinely interested in or that befriends you: it provides something of value. At the base level however it does have its purpose: get you to buy something. This is the motivation of why machine intelligence will emerge in spam first: somebody, somewhere will be making money. Would you like to buy this new computer, it is well built and will enhance the effectiveness of your communication with your network of contacts? Also, if you do I will cover the shipping myself.

Completely automated market crash!

[RyanFenton]

We've completed the circle - various "automated systems" have been blamed for various market failures in recent years, as companies and small traders have used algorithms on computers to "keep up with the speed of the market". Of course, the actual failure was almost always in the design, such as allowing a computer to make blind decisions with large amounts of money faster than you could keep track of.

But here, we have a stronger case for a machine-driven market failure - automated news algorithms. Misunderstanding generated at the speed of the market. I've worked on AI professionally in games, studied it in the contexts of linguistics, nervous system simulation, and such - AI even in its most exaggerated modern state is not going to even know how to figure out how to extract a good quote with human guidance, much less report on a news release. If you thought computer generated music was entertainingly bad - wait until you see some of the awful things produced by automated news misunderstandings... random context switches mixed with "neutral language" bits, it'll be like Fox news switched its agenda to Cthulu-level madness of confusion rather than the usual rage agenda.

And since the market makes its decisions on the basis of news, rumors, and insider trading - and people get the three confused as they hear them, mixing this into the information stream seems a virtual guarantee of another market crash.

That's what I call another serious negative externality for the news business taking the cheaper road to reporting business news.

Ryan Fenton

steakthskynet

[locallyunscene]

I'm trying to figure it out. Is it a typo that wonderfully illustrates the benefit of welcoming automated editors? Is steakthskynet what our meatspace reporters should be called? Or is it simply an insightful tag tragically misspelled?

I for one...

[azgard]

...look forward to our meme-ending overlords.

Gaming more than they do now?

[TiggertheMad]

Anyone want to place a bet on how long before companies are accused of "gaming" the financial reporting system with their press releases?

As opposed to 'gaming' the media with their press releases? Isn't that what a PR person is supposed to to, create press releases that cast the company in a favorable light?

For some definitions of journalism

[SgtChaireBourne]

'scans and automatically extracts critical pieces of information' from US corporate press releases, eliminating the 'manual processes' that have traditionally kept so many financial journalists in gainful employment.

That, I must admit, is an excruciatingly lame definition of 'journalism'.