When you fire up ChatGPT, you're connecting to a big silicon brain that lives somewhere and contains something. So, what exactly is that thing? What is the hardware that runs the chatbot you've fallen in love with? Let's dive into the hardware infrastructure of Chat GPT.
The NVIDIA A100 GPU
At the heart of Chat GPT's infrastructure lies a fundamental building block: the NVIDIA A100 GPU. If you thought the graphics card in your computer was expensive, consider that the A100 costs around ten thousand dollars a pop, roughly equivalent to the price of six RTX 4090s.
Artificial intelligence applications often rely on GPUs because they excel at performing numerous mathematical calculations simultaneously, in parallel. NVIDIA's newer models feature tensor cores, specialized for matrix operations frequently used in AI applications. Although it's called a GPU, the A100 is designed specifically for AI and analytical tasks. In fact, gaming on it isn't a realistic option; it doesn't even have a display output.
You can obtain the A100 in a PCI Express version or an SXM4 form factor. Unlike a conventional graphics card, SXM4 cards lie flat and connect to a large motherboard-like PCB using a pair of sockets with connectors on the underside. The SXM4 version is preferred in data centers because it can handle more electrical power. While the PCI Express version maxes out at 300 watts, the SXM4 version can handle up to 500 watts, resulting in higher performance. An SXM4 A100 boasts an impressive 312 teraflops of FP16 processing power. Additionally, these GPUs are interconnected using high-speed NVLink, enabling GPUs on a single board to function as a single, colossal unit.
Powering the Data Centers
But how many A100s are needed to keep Chat GPT running smoothly for a staggering 100 million users? OpenAI and Microsoft, the forces behind the Chat GPT project, haven't disclosed exact numbers about their hardware, but we can estimate based on processing capacity.
A single Nvidia HGX A100 unit, which typically contains eight A100 GPUs, can run Chat GPT quite effectively. These units are powered by a pair of server CPUs, each equipped with dozens of cores. However, to cater to the demands of a user base this massive, a lot more processing power is required to ensure the chatbot can respond to queries seamlessly for everyone.
While the precise numbers remain undisclosed, it's likely that Chat GPT utilizes approximately 30,000 A100s to meet this demand. To put that into perspective, it's significantly more than the 4,000 to 5,000 GPUs they probably needed to train the language model initially.
Training vs. Running
You might assume that the training process requires more processing power than actually running the model, but the massive amount of input and output that Chat GPT has to handle with 100 million users means that it requires approximately six times more GPUs to run it efficiently.
Given the high cost of these systems, this constitutes a substantial investment on Microsoft's part. While the exact dollar amount hasn't been disclosed, it's known to be in the hundreds of millions of dollars, in addition to several hundred thousand dollars a day just to keep the system running.
But Microsoft isn't stopping there; the company is integrating the newer NVIDIA H100 GPUs into its Azure Cloud AI service. These GPUs significantly outperform the A100 in FP16 performance, and they also introduce FP8 support, which is exceptionally useful for AI due to the complex mathematical calculations involved. This expansion ensures that more people can use Chat GPT and other AI services while enabling Microsoft to train even more advanced and complex large language models.
In conclusion, Chat GPT's hardware backbone is a complex and powerful infrastructure built around NVIDIA A100 GPUs. The investment in this technology by companies like Microsoft is not only substantial but also essential to deliver the seamless and intelligent responses that millions of users rely on every day.
No comments yet