In the early 1970s, the computing world was a labyrinth of punch cards and room-sized machines. The process of programming involved feeding these cards into massive computers, waiting hours or even days for results to be printed out. This was the state of technology at the time.
Fast forward to today, and we live in an era where interactive computing is seamlessly integrated into our lives. This transformation began with the pioneering work of a team at Xerox PARC, who developed the Alto computer—a device featuring a display, keyboard, and mouse. Despite its initial expense, the Alto’s innovation laid the groundwork for the sophisticated, pocket-sized computers we now take for granted.
AI Mastery: The Next Frontier
Just as the Alto revolutionised personal computing, we are now witnessing a similar upheaval with the rise of Artificial General Intelligence (AGI). AGI refers to a software system’s ability to perform any intellectual task that a human can, without needing specific instructions. This represents a colossal leap from the tools and techniques of early computing.
The rapid advancements in generative AI are nothing short of extraordinary. To understand this, consider the following:
- Moore’s Law and Metcalfe’s Law shaped the evolution of computing and the internet, respectively. Today, an exponential principle is driving the progress of generative AI.
- Scaling Laws in Deep Learning demonstrate that as AI models grow in size and data, their capabilities increase exponentially.
Over the past two years, leading AI models have expanded remarkably. The size of models has surged from 10 billion parameters trained on 100 billion words to 1 trillion parameters trained on over 10 trillion words. This monumental growth is leading to impressive results, but it also underscores the complex journey ahead.
Lessons from Personal Computing
The transition from the Alto to the iPhone wasn’t instantaneous. It involved significant milestones such as:
- Development of Operating Systems: Essential for managing hardware and software resources.
- Creation of Application Ecosystems: Enabling a broad range of functionalities.
- Internet Evolution: Facilitating global connectivity and data exchange.
Similarly, AI’s journey involves overcoming significant challenges. Large Language Models (LLMs) excel in language comprehension but struggle with complex reasoning required for multi-step tasks. Addressing these limitations may necessitate new neural network architectures or innovative training approaches.
The Computational Challenges of AI
At Together AI, we focus on the training and deployment of these sophisticated models. This process involves:
- Bespoke AI Supercomputers: High-performance systems designed by companies like Nvidia. These supercomputers, comprising tens of thousands of processors linked via advanced optical networks, are at the forefront of silicon design.
- Power Consumption and Heat Generation: Training these models consumes vast amounts of power and generates significant heat. A recent Meta study revealed that their 16,000-processor training cluster was operational only 31% of the time due to these constraints.
To address these issues, future innovations will need to:
- Enhance Chip Performance: Optimise for both performance and energy efficiency.
- Transform Data Centres: Fundamental changes in physical infrastructure may be necessary by 2030.
The Geopolitical and Societal Impact of AI
AI is rapidly becoming a crucial geopolitical asset. Its strategic importance is likely to increase, influencing technological leadership on the global stage. As AI continues to evolve, its effects on the workforce and societal structures will also intensify, raising critical discussions and debates.
Shaping the Future with AI
We stand at the precipice of an AI-driven era. The coming years will bring:
- Innovative Digital Products: New tools and services that will enhance and transform user experiences.
- Superintelligent Systems: AI systems that will become deeply integrated into our daily lives.
Just as societies have adapted to previous technological disruptions, AI will reshape our world. The road to genuine AI mastery is complex, but it promises a future where AI is as ubiquitous and transformative as computing has become.
Conclusion
The journey from the Alto to today’s AI models is a testament to human ingenuity and technological progress. As we continue to explore and develop artificial intelligence, we must recognise the lessons of the past and embrace the opportunities of the future. AI holds the potential to revolutionise our world, making it as integral to our lives as personal computing has become.
