Will the End of Moore's Law Halt AI Progress? (mindmatters.ai)

johnnyb (Slashdot reader #4,816) writes: Kurzweil's conception of "The Singularity" has been at the forefront of the media conception of artificial intelligence for many years now. But how close is that to reality? Will AI's be able to design ever-more-powerful AIs? Eric Holloway suggests that the power of AI has been fueled by Moore's law more than AI technology itself, and therefore hitting Moore's Wall will bring AI expansion to a fast halt.
Holloway calls that halt "peak AI...the point where a return on the investment in AI improvement is not worthwhile." He argues that humanity will reach that point, "perhaps soon...."

"So, returning to our original question, whether there is a path to Kurzweil's Singularity, we must conclude from our analysis that no such path exists and that unlimited self-improving AI is impossible."

Comments

Transistors and AI

[reanjr]

If you think AI is just more transistors, you probably aren't doing anything interesting in AI research. How many transistors in the human brain? How many regular transistors are necessary to do the work of one quantum transistor? We don't even know how the brain works, and this asshat is asserting that we'll never be able to build a machine that works the same way.

Re:Transistors and AI

[alvinrod]

It's even more foolish since researchers have already been able to build a machine that behaves like very simple creatures. There isn't any reason so think that we can't make something more complex, it's just a matter of being able to map out the wiring and build hardware to mimic the sensory data that the artificial brain needs. However, there's a long way to go. I recall another researcher that was trying to make a robot to fold clothes, but that problem turned out to be much harder than he thought since just getting the robot to be able to recognize individual articles of clothing is challenging.

Re:Transistors and AI

[AHuxley]

Decades of the AI winter https://en.wikipedia.org/wiki/... should have worked on that?
Another few decades of work on the AI and it should be good, like the AI exports said in the 1970's ...

Re:Transistors and AI

[phantomfive]

How many transistors in the human brain?

Of course the answer is zero, because the brain has neurons. But we can have some numbers for comparison. A Graphcore GC2 IPU has 23 billion transistors. In comparison, a brain has:

100 billion neurons.
10 trillion synapses.
300 billion dendrites.

Which of those need to be emulated? A transistor does not do as much as a neuron, and we don't know all the things a neuron does. There is some evidence that the inside of a neuron does some kinds of calculations. So it's much more complicated than just comparing raw numbers. That said, transistors do operate faster than neurons.

Good link for more reading.

Re:Transistors and AI

[Dunbal]

Because unfortunately there is a lack of volunteers willing to give up their living brains to be sliced up for study. Most of what we have learned about the functionality of brains has come from war wounds and tumors. It's one thing to see stained neurons in a microscope or watch patterns on a PET/MRI, and another to experience "if this bit of brain isn't working, then this stops happening, so this must be wired into that, that and that".

Re:Transistors and AI

[qubezz]

The heart has its own pacemaker, it keeps itself beating. The sinoatrial node is in the heart and creates voltages that cause the heart muscles to contract, while reacting to chemical signals such as adrenaline and blood gas levels. The spinal column emits neurotransmitters which only modify the heart's activity level. Many other organs also have the intelligence of their functioning built-in.

Re:Transistors and AI

[Tough Love]

The synaptic weights have not been mapped, instead machine learning algorithms were used. Not dissing the project at all, I think there is a huge amount to learn from it, including helping to determine whether our model of how neurons work is essentially complete or not. I'm betting on not, and that a whole lot more processing goes on in the neuron than what happens at the synapses alone.

Re:Transistors and AI

[careysub]

Also, many neurons have nothing to do with "thinking".

...

That is not happening, because the real bottleneck is our knowledge of how intelligence works.

I generally agree with the thrust of your comment, but your last sentence refutes your first one. We really don't know how intelligence works, or even have a good definition of what it is, still, so your confidence that neurons that maintain homeostasis have nothing to do thinking is misplaced. In fact there is very strong evidence that these background task neurons are also very important in thinking - like in maintaining consciousness. Disruptions in those low level background neurons do not usually interfere with just the autonomic functions but leave consciousness alone, instead consciousness bites the dust first.

Kurzweil is a Shill

[Dayze!Confused]

Kurzweil pretends to know what he's talking about because he can fit a graph with lots of tampering with the data. He fails to see that what he calls exponential growth is nothing more than the beginning of a sigmoid function. A good analysis of Moore's law and computational power shows a sigmoid function, as with many technologies they start off slow, build up quickly, then tapper off.

Re:Kurzweil is a Shill

[Dunbal]

Kind of like your post. You were doing so well until you hit "tapper off" :)

Re:Kurzweil is a Shill

[bcwright]

Moore's Law, strictly speaking, only applies to silicon semiconductor technology, not to computational technology as a whole. As such, it will eventually exhaust its possibilities, and we're not too far from doing that now. You are correct that most technologies do follow a sigmoid function and that few if any can ever be expected to follow an exponential curve.

But Kurzweil was not quite so naive as to claim that silicon semiconductor technology would continue forever. Rather, he claimed that successor technologies would be found to continue the upward ascent. This is probably true as far as it goes, but we have no guarantee that such successor technologies will seamlessly follow the demise of silicon; rather, it's likely that there may be gaps while the new technologies are developed and perfected before progress can renew. We also have no guarantee that they will follow the same growth curve as silicon (ie, doubling every 18-24 months); they might be either faster or slower. Kurzweil's belief that such successor technologies will follow each other seamlessly is based on little more than faith.

We also have no guarantee that continued upward progress is even possible over extended periods of time. Even if we don't reach any physical limits, we're likely to hit limits of power consumption, resource availability, or the simple economic viability of constructing the factories or the chips themselves. Even in the case of silicon, all of this is starting to happen: We have not quite yet reached the physical limits of the medium, but are in danger of running out of either resources (think rare earths), electrical power (think tasks like cryptocurrency mining), or even capital (the ever-increasing costs of building the foundries), any or all of which may turn out to form an effective barrier to reaching the physical end-point of silicon.

For these reasons I do not think that a Kurzweil-style singularity is in our (near) future, although computational progress should continue, possibly with more choppy and unpredictable breakthroughs than has been the case in recent history.

But if Kurzweil is too optimistic, Holloway is much too pessimistic: He completely discounts any successor technologies to silicon. This is already flying in the face of history, since we've had at least three major transitions in computational technology already (from purely mechanical to electro-mechanical to vacuum tubes to semiconductors), and it is foolish to claim that we have already found the best technology possible. Even now there are candidate technologies waiting in the wings, such as quantum computing and various alternatives to silicon that retain traditional computational models.

In other words, the death of Moore's Law (for which read: the progress of silicon technology) marks a transition period, not an endpoint.

Oh my Lord?

[drolli]

Eric Holloway:

* seems to have no qualification in physics/nanotechnology to add anything to the discussion if Moores law will end, and when

* Seem to bagger along with intelligent design folks, with him re-telling the old stories they usually tell about information science and the rest of science

* and seems to write no peer reviewed articles any more (after the paper he wrote unrelated and before his PHD research)

* Did a PHD in a program where the students are identified as "good stewards of God-given talents" (https://www.ecs.baylor.edu/ece/index.php?id=865400)

* Did a PHD program which contains in its description "Engineering is also a value-based discipline that benefits from Christian worldview and faith perspectives; students can also select supportive courses from religion, theology or philosophy. Course selection is broadly specified to provide flexibility and to accommodate a wide-range of student interest." (https://www.ecs.baylor.edu/ece/index.php?id=863609)

* Description of the seminar series of his university where it seems that he presented his PHD: (https://www.ecs.baylor.edu/ece/index.php?id=868860): eBEARS seminars are presented by Baylor ECE faculty, ECE graduate students and transnationally recognized scholars and leaders. The topics lie within the broad area of ECE. In concert with Baylor's Pro Futurus strategic plan to be "a place where the Lordship of Jesus Christ is embraced, studied, and celebrated," some eBEARS seminars focus on the topic of faith and learning.

So praise the Lord for his insights!

Re:Oh my Lord?

[Anonymous Coward]

Interesting opinion, considering many great scientific contributions and advancements throughout the ages have come (and still come) from people who were (or are) very religious. And many of those people attribute their discoveries at least partly to divine inspiration, realizing that despite all the hard work, study, experimentation, and preparation on their part, there is still sometimes the feeling of a sudden flow of knowledge or creativity from an external source, resulting in ideas that they feel they would not have come up with on their own.

Just curious... what are YOUR valuable contributions to science?

Why not yet?

[AndyKron]

Why hasn't some alien already done this and turned the Universe into a massive computer? Oh right, we're living in a simulation...

Now the real work begins...

[BobC]

Much of recent AI progress has come from the awesome amount of cheap computing power available.

That's not going to change! As today's bleeding edge silicon processes evolve, they will get faster and cheaper (both in cost and energy consumption), if not smaller.

Much of AI is inherently parallel: So long as more CPUs and GPUs can be added, larger problems will be solved faster.

We are still in the first two generations of custom hardware for AI. That trend will continue and accelerate as new architectures and algorithms arrive.

I'd say there are at least three full "Moore's Law" generations coming for AI, very likely more. But transistors alone won't be driving it. Fortunately, there are lots of other factors that will.

Re:Now the real work begins...

[raftpeople]

The recent advances have much more to do with Hinton and team's algorithms for training a deep network. That was a major stumbling block that significantly limited the usefulness of neural networks. Even without any increase in computing power, that development created the explosion in success with deep networks.

Right and wrong

[Dan East]

He's right and wrong. He is correct that much of the "advancements" in AI has been because of processing power (and dataset size). Most of what I learned in AI in college a quarter century ago forms the foundation of today's AI (and most of what I learned had been developed decades earlier). The reason we have things like Siri isn't because AI is smarter. It's because processing power is so fast and cheap, and because data storage and ram is so large and cheap, that an absolutely massive data set can be crunched to do speaker agnostic recognition to determine what I said. In fact, Apple can run my voice audio through dozens of speech models (male, female, accents, etc) in parallel to find the best result. So he is right - processing power has enabled AI to become far more useful of late.

However, where he is wrong is in the parallelism and scalability. In my above example, many different nodes (maybe located in entirely different datacenters) are doing that processing to find the best match.

AI doesn't need to exist on one processor, and it doesn't need to execute at any particular speed. If we're talking "turing" type AI, and I were to ask it "How are you feeling today?" and the AI takes 5 hours to reply "I feel the same as I always do.", well it is still just as intelligent as if it were responding in real-time. When we have reached that point in AI intelligence then we can throw more processing power at it in many different ways to allow it to process faster. The point is that the intelligence is not bound by the processing speed. Sure, for Siri to be viable commercially and useful to Joe Blow it needs to be fast, but as far as research and advancing the field of AI, that is independent of the processing speed.

And having said all that, AI has not advanced significantly beyond the full realization and expansion of things like neural nets with massive processing power and data sets to be useful in identifying, say, a tree in a photograph. We could have been doing that in 1980 given the processing power and storage capacity we have now.

Re:Does Moore's law apply to GPUs? ARM?

[michelcolman]

We don't even need quantum computers to speed up AI. All we need is a different architecture that more closely mimicks a brain.

Right now, even the most advanced practical applications of AI are still using serial computations to calculate the propagation of the signals, Even massively parallel GPUs are still serially calculating everything, just doing it in batches instead of one by one.

Meanwhile, some researchers are experimenting with new layouts where the components actually behave like neurons rather than simulating them in a calculation. If we can use modern chip production processes to make those at a large scale, that will be a game changer.

Human neurons take milliseconds to fire, Let that sink in: speeds below 1 kHz. Imagine mimicking those natural neurons with a denser layout of silicon neurons at gigahertz speeds, and comparing that to a classical computer. We'll go way beyond what Moore's law would permit. It would be like saying we're reaching the limits of how much we can improve bicycles, and then replacing them with jet fighters.

Re:AI progress is not bound by computation speed @

[careysub]

The thing about parallel processing is that not everything CAN be neatly decomposed into stateless parallel processes.

However we have really, really good evidence that real, strong AI absolutely CAN be decomposed into stateless parallel processes sufficiently well to allow it to perform in real time at the highest level of competence, with hardware that only has a maximum switching rate of only 1000 HZ, and has a mean firing rate of only 6 HZ. You probably have one of these pieces of evidence about two feet from your keyboard.

Re:Things get small

[drinkypoo]

CPUs stopped getting faster 3-years ago as Intel, AMD, ARM started cramming in more and more cores.

That is complete nonsense. CPU clock rates stopped getting faster, but the CPUs still can retire many more operations per second.