How Nvidia built the chip powering the generative AI boom

Featured Sponsor

Store	Link	Sample Product
UK Artful Impressions	Premiere Etsy Store

In 2022, US chipmaker Nvidia released the H100, one of the most powerful processors it had ever built, and one of the most expensive, costing around $40,000 each. The launch seemed ill-timed, even as companies tried to slash spending amid rampant inflation.

Then, in November, ChatGPT was launched.

“We’ve gone from having a pretty tough year last year to an overnight turnaround,” he said Jensen Huang, chief executive officer of Nvidia. OpenAI’s successful chatbot was an “aha moment,” he said she. “It created an immediate demand.”

ChatGPT’s sudden popularity has triggered an arms race among the world’s leading technology companies and startups who are scrambling to get the H100, which Huang describes as “the world’s first computer [chip] designed for generative AI”—artificial intelligence systems capable of rapidly creating human text, images and content.

The value of having the right product at the right time became evident this week. Nvidia announced Wednesday that its sales for the three months ending in July would be $11 billion, more than 50 percent ahead of previous Wall Street estimates, driven by a recovery in data center spending by Big Tech and demand for its AI chips.

Investor response to forecasts added $184 billion to Nvidia’s market cap in a single day on Thursday, taking over what was already the world’s most valuable chip company close to a $1 trillion valuation.

Nvidia is an early winner in the astronomical rise of generative AI, a technology that threatens to reshape industries, produce huge productivity gains, and displace millions of jobs.

This technological leap is set to be accelerated by the H100, which is based on a new Nvidia chip architecture dubbed “Hopper” – named after American programming pioneer Grace Hopper – and has suddenly become the most popular commodity in Silicon Valley.

“This whole thing took off just as we were going into production on Hopper,” Huang said, adding that full-scale production only started a few weeks before ChatGPT’s debut.

Huang’s confidence in continued earnings stems in part from his ability to work with chipmaker TSMC to ramp up H100 production to meet growing demand from cloud service providers like Microsoft, Amazon and Google, internet groups like Meta, and enterprise customers. .

“This is among the scarcest engineering resources on the planet,” said Brannin McBee, chief strategy officer and founder of CoreWeave, an AI-focused cloud infrastructure start-up that was one of the first to receive H100 shipments worldwide. beginning of this year.

Some customers have waited up to six months to get hold of the thousands of H100 chips they want to train their vast data models. AI startups had expressed concern that H100s were in short supply just as demand was taking off.

You are viewing a snapshot of an interactive graph. This is most likely due to the fact that you are offline or JavaScript is disabled in your browser.

Elon Musk, who has bought thousands of Nvidia chips for its new AI start-up X.ai, told a Wall Street Journal event this week that currently GPUs (graphics processing units) “are much harder to get hold of than drugs,” joking on the fact that “it wasn’t quite a high bar in San Francisco”.

“The cost of computing has become astronomical,” Musk added. “The minimum down payment must be $250 million of server hardware [to build generative AI systems].”

The H100 is proving especially popular with Big Tech companies like Microsoft and Amazon, who are building entire data centers focused on AI workloads, and Generative AI start-ups like OpenAI, Anthropic, Stability AI, and Inflection AI because it promises higher performance that can accelerate product launches or reduce training costs over time.

“In terms of access, yes, this is what it feels like to grow a new GPU architecture,” said Ian Buck, head of Nvidia’s hyperscale and high-performance computing business, which has the daunting task of scaling up the offer of H100 to meet the demand. “It’s happening at scale,” he added, with some large customers looking for tens of thousands of GPUs.

The unusually large chip, an “accelerator” designed to work in data centers, has 80 billion transistors, five times the number of processors powering the latest iPhones. While it’s twice as expensive as its predecessor, the A100 released in 2020, early adopters claim the H100 boasts at least three times better performance.

“The H100 solves the scalability problem that has been plaguing [AI] model makers,” said Emad Mostaque, co-founder and chief executive officer of Stability AI, one of the companies behind image-generation service Stable Diffusion. “This is important as it allows us all to train larger models faster while going from a research problem to an engineering problem”.

While the timing of the H100’s launch was ideal, Nvidia’s breakthrough in AI can be traced back nearly two decades to an innovation in software rather than silicon.

Its Cuda software, created in 2006, allows GPUs to be repurposed as accelerators for other types of workloads besides graphics. Then, around 2012, Buck explained, “AI found us.”

Researchers in Canada realized that GPUs were ideal for creating neural networks, a form of artificial intelligence inspired by the way neurons interact in the human brain, which were becoming a new focal point for AI development. “It took us nearly 20 years to get to where we are today,” Buck said.

Nvidia now has more software engineers than hardware engineers to enable it to support the different types of AI frameworks that have emerged in the intervening years and make its chips more efficient at the statistical computation needed to train AI models.

Hopper was the first architecture optimized for “transformers,” the AI ​​approach that underpins OpenAI’s “pre-trained generative transformer” chatbot. Nvidia’s close work with AI researchers allowed it to pinpoint the emergence of the transformer in 2017 and begin fine-tuning its software accordingly.

“Nvidia has probably seen the future before anyone else with its pivot in making GPUs programmable,” said Nathan Benaich, managing partner at Air Street Capital, an investor in AI start-ups. “He spotted an opportunity, bet big, and consistently outperformed his competitors.”

Benaich estimates Nvidia is two years ahead of its rivals, but adds, “Its position is far from watertight on both the hardware and software fronts.”

Stability AI’s Mostaque agrees. “Next-generation chips from Google, Intel and others are catching up [and] even Cuda becomes less of a moat as software becomes standardized.

To some in the AI ​​industry, Wall Street’s enthusiasm this week seems overly optimistic. However, “for now,” said Jay Goldberg, founder of chip consultancy D2D Advisory, “the semi-final AI market looks set to remain a winner-take-all market for Nvidia.”

Additional reporting by Madhumita Murgia