On America's west coast, the world's most valuable companies are trying to make artificial intelligence smarter. Google and Facebook are bragging about experiments using billions of photos and thousands of high-performance processors. But late last year, a project in eastern Tennessee quietly surpassed the scale of any corporate AI lab. And it was run by the US government.
US government supercomputer breaks records
The record-breaking project involved the world's most powerful supercomputer, Summit, at Oak Ridge National Laboratory. This car won the crown last June, returning the title to the United States five years later when China topped the list. As part of a climate research project, a giant computer launched a machine learning experiment that was faster than ever before.
The Summit, covering an area equivalent to two tennis courts, used more than 27,000 powerful GPUs in this project. He used their power to train deep learning algorithms, the very technology that underpins advanced artificial intelligence. In deep learning, algorithms perform exercises at a billion billion operations per second, known in supercomputing circles as an exaflop.
“Deep learning has never achieved this level of performance before,” says Prabhat, research team leader at the National Energy Research Center at Lawrence Berkeley National Laboratory. His team collaborated with researchers at Summit's headquarters, Oak Ridge National Laboratory.
As you might guess, the AI training of the world's most powerful computer focused on one of the world's biggest challenges - climate change. Tech companies train algorithms to recognize faces or road signs; government scientists have trained them to recognize weather patterns like cyclones from climate models that compress centennial forecasts of the Earth's atmosphere into three hours. (It is unclear, however, how much energy the project required and how much carbon was released into the air in this process).
Promotional video:
The Summit experiment has implications for the future of artificial intelligence and climatology. The project demonstrates the scientific potential of adapting deep learning to supercomputers that traditionally simulate physical and chemical processes such as nuclear explosions, black holes, or new materials. It also shows that machine learning can benefit from more computing power - if you can find it - and provide breakthroughs in the future.
“We didn't know it could be done on this scale until we did it,” says Rajat Monga, CTO at Google. He and other Googlers helped the project by adapting the company's open source TensorFlow machine learning software for Summit's gigantic scale.
Much of the work on deep learning scaling has been done in the data centers of Internet companies, where servers work together on problems, separating them because they are relatively disjointed rather than bundled into one giant computer. Supercomputers like Summit have a different architecture, with dedicated high-speed connections linking their thousands of processors into a single system that can work as a whole. Until recently, there has been relatively little work on adapting machine learning to work with this kind of hardware.
Monga says work to adapt TensorFlow to Summit scale will also support Google's efforts to expand its internal artificial intelligence systems. Nvidia engineers also took part in this project, making sure tens of thousands of Nvidia GPUs in this machine are running smoothly.
Finding ways to harness more computing power in deep learning algorithms has been instrumental in the current development of technology. The same technology that Siri uses for voice recognition and Waymo cars for reading road signs became useful in 2012 after scientists adapted it to run on Nvidia GPUs.
In an analysis published last May, scientists at OpenAI, a San Francisco research institute founded by Elon Musk, estimated that the amount of computing power in the largest public machine learning experiments has doubled roughly every 3.43 months since 2012; this would represent an 11-fold increase in a year. This progression helped the Alphabet bot beat the champions in challenging board and video games, and also significantly improved the accuracy of Google's translator.
Google and other companies are currently creating new kinds of AI-enabled chips to continue this trend. Google says that pods with thousands of its AI chips closely spaced - duplicated tensor processors, or TPUs - can provide 100 petaflops of processing power, one tenth of the speed reached by Summit.
Summit's contributions to climate science show how gigantic AI can improve our understanding of future weather conditions. When researchers generate century-old weather predictions, reading the resulting forecast becomes challenging. “Imagine you have a YouTube movie that's been running for 100 years. There is no way to manually find all the cats and dogs in this movie,”says Prabhat. Usually software is used to automate this process, but it is not perfect. The Summit results showed that machine learning can do this much better, which should help predict storms like floods.
According to Michael Pritchard, a professor at the University of California, Irvine, launching deep learning on supercomputers is a relatively new idea that came at a convenient time for climate researchers. The slowdown in the advancement of traditional processors has led engineers to equip supercomputers with an increasing number of graphics chips to improve performance more consistently. “The time has come when you can no longer increase processing power in the usual way,” says Pritchard.
This shift brought traditional modeling to a standstill, and therefore had to adapt. It also opens the door to harnessing the power of deep learning, which naturally lends itself to graphics chips. Perhaps we will get a clearer picture of the future of our climate.
Ilya Khel