As billion-dollar rivals like OpenAI and Stable Diffusion sprint ahead and make their software available to the general public, companies like Microsoft and Google struggle to incorporate cutting-edge AI into their search engines.
The Nvidia A100, a $10,000 processor that has emerged as one of the most important instruments in the artificial intelligence sector, powers many of these applications.
A technological gold rush has begun with the development of software that can write tracts of text or make images that appear to have been drawn by a person.
As billion-dollar rivals like OpenAI and Stable Diffusion sprint ahead and make their software available to the general public, companies like Microsoft and Google struggle to incorporate cutting-edge AI into their search engines.
An about $10,000 chip that has grown to be one of the most important instruments in the artificial intelligence sector powers several of these applications: NVidia’s A100.
According to Nathan Benaich, an investor who publishes a monthly report monitoring the AI business and includes a partial list of supercomputers using A100s, the A100 has currently become the “workhorse” for artificial intelligence professionals. According to New Street Research, Nvidia controls 95% of the market for graphics processors suitable for machine learning.
The type of machine learning models that power applications like ChatGPT, Bing AI, or Stable Diffusion are perfectly suited for the A100. Its capability to carry out numerous straightforward calculations simultaneously is crucial for developing and utilizing neural network models.
The A100’s technology was first employed to produce complex 3D visuals for video games. Although Nvidia’s A100 is sometimes referred to as a graphics processor or GPU, it is currently set and geared towards machine learning workloads and operates in data centers rather than inside bright gaming Computers.
Hundreds of thousands of Nvidia processors are needed by large corporations or small businesses developing software like chatbots and image generators; they can either buy the chips themselves or obtain access to the computers via a cloud service provider.
For the training of big language models used in artificial intelligence, hundreds of GPUs are needed. To find patterns, the processors must be able to quickly process gigabytes of data. After that, “inference,” or utilizing the model to write text, make predictions, or recognize things in photographs, also requires GPUs like the A100.
This means that a lot of A100s are required for AI businesses. Some business owners in the sector even view the number of A100s they have at their disposal as an indicator of development.
We had 32 A100s a year ago, the CEO of Stability AI, Emad Mostaque, said on Twitter in January. “Children, dream large and stack more GPUs. Brrr.” The business Stability AI, which has a valuation of over $1 billion, worked on the image generator Stable Diffusion that gained prominence last fall.
According to one estimate from the State of AI report, Stability AI now has access to more than 5,400 A100 GPUs. However, cloud providers are not included because their numbers are not made public in the report, which charts and tracks which companies and universities have the largest collection of A100 GPUs.
Nvidia’s riding the A.I. train
The hype cycle surrounding AI could be advantageous for Nvidia. Investors drove the stock up about 14% on Thursday, primarily because the company’s AI chip business—reported as data centers—rose by 11% to more than $3.6 billion in sales during the quarter, showing continued growth—despite overall sales declining 21% during the fiscal fourth-quarter earnings report on Wednesday.
In 2023, Nvidia shares have increased by 65%, outperforming both the S&P 500 and other semiconductor firms.
On a teleconference with analysts on Wednesday, Nvidia CEO Jensen Huang couldn’t stop talking about AI, signaling that the field’s recent boom is central to the business’s strategy.
In the last 60 days, Huang claimed, “the activity around the AI infrastructure that we built, and the activity around inferencing using Hopper and Ampere to influence huge language models, has gone through the roof.” There is little doubt that the past 60 to 90 days have significantly altered any expectations we had for this year at the beginning of the year.
The A100 generation of chips has the codename Ampere according to Nvidia. The new generation, which now includes the H100, has a code name called Hopper.
More computers needed
Machine learning operations can consume the entire computer’s processing capacity, often for hours or days, in contrast to other types of software, like delivering a webpage, which uses processing resources infrequently in bursts for microseconds.
As a result, businesses that develop popular AI products frequently need to increase their GPU capacity to manage peak usage or enhance their models.
These GPUs are expensive. Several data centers employ a solution that contains eight A100 GPUs cooperating in addition to a single A100 on a card that can be inserted into an existing server.
Despite having the required CPUs, the system, Nvidia’s DGX A100, has a suggested price of close to $200,000. Nvidia announced on Wednesday that it would sell cloud access to DGX systems directly, lowering the barrier to entry for experimenters and academics.
It’s simple to understand how the price of A100s can go up.
For instance, according to a New Street Research estimate, the OpenAI-based ChatGPT model used by Bing’s search could need 8 GPUs to answer a query in under a second.
At that pace, Microsoft would require more than 20,000 8-GPU servers only to roll out the model to everyone in Bing, indicating that the functionality might cost $4 billion in infrastructure expenditures.
“At the scale of Bing, if you’re from Microsoft and you want to grow that, that’s maybe $4 billion. According to Antoine Chkaiban, a technology analyst at New Street Research, “If you want to scale at the scale of Google, which serves 8 or 9 billion inquiries per day, you need to spend $80 billion on DGXs.” “The figures we arrived at are enormous. But, they are simply a reflection of the fact that every user who adopts a huge language model needs a giant supercomputer to use it.
According to material online released by Stability AI, the most recent version of Stable Diffusion, an image generator, was trained on 256 A100 GPUs, or 32 computers with 8 A100s each, using 200,000 compute hours.
Stability AI CEO Mostaque said on Twitter that training the model alone cost $600,000 at the going rate, implying in a tweet exchange that the cost was extremely low in comparison to competitors. The price of “inference,” or deploying the model, is not included in that.
According to Nvidia CEO Jen-Hsun Huang, the company’s products are relatively affordable for the amount of computing that these kinds of models require.
Huang added, “We reduced what would have been a $1 billion data center using CPUs to a data center of $100 million. Now, $100 million is practically nothing when split across 100 businesses and stored on the cloud.
Huang claimed that Nvidia’s GPUs make it far more affordable for startups to train models than it would be if they utilized a conventional computer CPU.
Currently, Huang claimed, “You could construct something like a huge language model, like a GPT, for about $10, $20 million.” That is incredibly, really inexpensive.
New competition
There are other manufacturers of GPUs for usage in artificial intelligence besides Nvidia. Along with rival graphics processors from AMD and Intel, major cloud providers like Google and Amazon are also designing and deploying their chips specifically for AI workloads.
The Status of AI computing report states that “AI hardware remains tightly centralized to NVIDIA.” More than 21,000 open-source AI studies cited Nvidia chips as of December.
The majority of researchers included in the State of AI Compute Index used the V100, an Nvidia chip that was released in 2017. However, the A100 grew quickly to become the third-most-popular Nvidia chip in 2022, trailing only a consumer graphics chip priced at or below $1500 that was created for gaming.
The A100 is also unique in that it is one of the select few chips that have export restrictions imposed on it for national defense purposes. Nvidia stated in an SEC filing from last fall that the A100 and the H100 were prohibited from being exported to China, Hong Kong, and Russia due to a license requirement imposed by the US government.
The possibility that the covered products may be utilized in, or diverted to, a “military end use” or “military end user” in China and Russia, the USG said in its filing, will be addressed by the new licensing requirement, according to Nvidia. To comply with American export limitations, Nvidia has claimed that certain of its chips were modified for the Chinese market.
The A100’s replacement might pose the toughest challenge. In chip cycles, the A100’s initial release in 2020 seems like a lifetime ago. The H100, launched in 2022, is already being produced in large quantities; in fact, Nvidia said on Wednesday that H100 chip sales generated more revenue than A100 chip sales in the January quarter.
According to Nvidia, the H100 is the first of its data center GPUs to be optimized for transformers, a technique used by many of the newest and best AI applications. On Wednesday, Nvidia declared its intention to speed up AI training by over a million percent. That might indicate that AI businesses won’t require as many Nvidia chips in the future.
