Checking in on my 2010 petaFLOP forecast Date: 2023-03-06 Author: John Brennan Source: https://johnbrennan.xyz/essay/petaflop-forecast A look back at technology forecasts made in 2010, checking the progress of computing costs from supercomputers to cloud TPUs. --- "Change is the process by which the future invades our lives, and it is important to look at it closely, not merely from the grand perspectives of history, but also from the vantage point of the living, breathing individuals who experience it." — Alvin Toffler In 2009 and 2010 I drafted a short monograph titled The Future of Analytics. In the appendix I tried to signpost the future by making a series of point estimates about specific technology milestones. For example, I forecast there would be 100 billion embedded Internet devices by 2045. I imagined the first, implantable human-computer interface for long-term memory storage would come in 2040. Another estimate was the first trillion variable analysis would be conducted by 2035. The first petaFLOP computer for under $1,000 would come in 2029. IBM's Roadrunner super computer was the first to sustain more than one quadrillion floating point operations (FLOP) in 2008. It was but one program in a long line of research undertaken by the Department of Energy and DARPA. See Gordon Bell's Amazing Race essay for a history of such programs. Roadrunner's IBM and AMD processors connected together with Infiniband cost $100 million---$126 million in 2021 dollars, which I will now use throughout for comparisons---when it was installed at Los Alamos in 2008. In the intervening years we see a number of super computers achieve lower costs. Cray built the Jaguar super computer at Oakridge National Lab and achieved $82 million per petaFLOP on another AMD architecture. Other super computers surpassed Jaguar's performance. For example, Cray and Oakridge deployed an AMD and Nvidia Tesla architecture in the Titan super computer in 2010, which brought the cost per petaFLOP to ~$6.5 million. IBM, Nvidia, and Mellanox set the next noteworthy target when they installed their Summit super computer ($200 million, 200 petaFLOPs) at Oakridge in 2018, bringing the cost per petaFLOP to approximately $1 million. Certainly the super computer industry in the US, Japan, and abroad has continued to break records in the total number of FLOPs one can execute. From a cost stand point, it is interesting to see how these technologies are leaving the national labs and emerging in enterprise offerings on-premises or via cloud providers. An additional benchmark I could find was Google's tensor processing unit (TPU). While Google is on TPU version 4, I found an analysis showing the TPU v3 achieved a petaFLOP for approximately $6,048 per petaFLOP on Google Cloud. So what are we to do with these increasingly capable computers? Well, so far they have ushered in more confirmations that Wright's Law is alive and well. For example, the cost of sequencing the human genome's 3.2 billion base pairs has gone from around $900 to $30. Source: Our World in Data, National Human Genome Research Institute (2022) Also, this trend has improved our ability to develop and train large language models, propelling us from the equivalent of the Stone Age into the beginning of the Industrial Revolution. When I was researching my dissertation, it took me the better part of a year to search for historical articles, read them, and manually code 11,894 observations into my models. Now feature engineering using the latest data science and data engineering techniques are leading to language models with billions and trillions of parameters. For example, OpenAI progressed from GPT1 to GPT 3.5 (ChatGPT) between 2018 and 2022. GPT is not even the largest of the large language models. In 2022 Google published its progress on Pathways Language Model (PaLM). Their article describes what can be accomplished in about 64 days of processing: "We trained PaLM-540B [540 billion parameters] on 6,144 TPU v4 chips for 1,200 hours and 3,072 TPU v4 chips for 336 hours including some downtime and repeated steps." Select large language models between 2010 and 2022 (Y-axis is log scale). Source: Our World in Data, Sevilla et al. (2022) I'll close by paraphrasing what I said in 2010. Besides nature, the most creative and destructive force in the world is the human brain. It has enabled us to form complex civilizations, turn basic natural resources into astounding technologies, and begin to understand fundamental laws about life. Yet, it fails us with its imperfections. We can be persuaded emotionally to ignore data, and there are concepts we cannot yet understand. In this century we have the potential to connect every member of the planet in communication. We will eventually measure anything, anywhere, at any level of detail, in real time. It is naïve to conclude this era only holds promise and benefit. During our best triumphs and our worst failures we used the best information available, the latest technology, and the best minds. So, as we use these tools to do new work, let us start with our real intelligence---faults and all. --- Canonical: https://johnbrennan.xyz/essay/petaflop-forecast