Friday, February 23, 2024
HomeSecurityMeta is building an AI supercomputer with 16,000 GPUs - the fastest...

Meta is building an AI supercomputer with 16,000 GPUs – the fastest ever!

Will it improve a tarnished public image?


A few days ago Meta announced AI Research SuperCluster (RSC). This is a new high-performance computer (HPC) aimed at powering AI technologies from state-of-the-art language models to the metaverse. According to the company, when completed in mid-2022, RSC will be the fastest AI supercomputer, making the company a strong candidate to announce its next AI breakthrough.

The company plans to use RSC to train 1 trillion parameter NLP models, further develop image and speech recognition products, and advances in key areas such as embodied AI and multimodal AI. is doing.


RSC greatly enhances Meta’s hardware capabilities. It currently consists of 760 Nvidia DGX A100s (6,080 GPUs), with plans to scale up to 16,000 A100 GPUs once completed. Incidentally, Nvidia Selene is also DGX A100 SuperPOD (the company’s AI data solution ), boasting 2,240 GPUs (63.4 Pflop/s) and boasting the 6th highest performance in the world.

In addition, Microsoft Azure’s supercomputer , which was jointly built with OpenAI, is equipped with 10,000 V100 GPUs, ranking 10th in the world (30.05 Pflop/s).

Also, Tesla’s current supercluster (which will be replaced by Dojo in the next few years) consists of 720 nodes (5,760 GPUs) of 8 A100 GPUs, making it the fifth largest in the world (although Comparing is difficult because the precision of the floating point format is lower than that of other HPCs).

Meta’s RSC specs are impressive for an AI supercomputer. While it’s no match for the fastest HPC , Nvidia says, “Meta’s RSC is expected to be the largest customer deployment of the NVIDIA DGX A100 system when fully implemented.”

Meta’s previous GPU supercluster featured 22,000 Nvidia V100s, which was enough power for AI R&D so far. RSC now delivers a 20x speedup for computer vision manufacturing tasks and a 3x speedup for large-scale NLP workflows. Once the RSC is complete, these figures will improve by a factor of 2.5.

RSC will use all the user data Meta has amassed over the years to power its AI system. Kevin Lee and Shubho Sengupta (who co-authored the blog post introducing RSC) wrote on their blog: “Unlike previous AI research infrastructures that have only leveraged open source or other public datasets, RSC will allow us to include examples from Meta’s product systems in training our models, allowing our research It allows us to effectively connect to practice.”

The statement means the company will leverage the trillions of data points it collects from users across platforms to create the next generation of AI products and tools.

Jerome Pesenti, vice president of AI at Meta, said in an interview (for The Wall Street Journal) that RSC “will unleash an AI that understands the world around you much better. The goal is to build an AI model that can take in huge amounts of data, recognize things like speech and vision, and then use multiple inputs to think like the human brain and understand situations contextually.” says (*translation note 4).

If this prediction becomes a reality, we will be closer to AGI than ever before. But can Meta safely implement the resulting model? Pesenti criticized GPT-3 for its tendency to produce harmful output after being typed at a prompt . Is Meta’s new AI model qualitatively superior to GPT-3 in terms of toxicity and bias?

The models developed by RSC raise concerns about whether more computing power will solve AI problems, if any, in the data fed into the models. Lee and Sengupta have mentioned several times that RSC helps detect and identify harmful content on Meta’s platform.

But he didn’t say how he plans to do that. Will RSC solve the harm Instagram poses to teenage girls ? Or will it remove the fake news circulating in your Facebook feed ? Will the company try to build an AI that is “much, much, much better at just making sense of the world” when it can’t control simpler algorithms?

The bigger the AI ​​model, the more likely it is to use questionable data, and therefore the greater the risk of harm. Considering what is gained and what is lost is in the hands of those who design, manage, train and implement AI systems.

Let’s hope Meta takes the power of RSC and not only creates the most powerful AI models, but also makes them fair and beneficial for everyone. Let’s hope it benefits minorities, especially those who are repeatedly targeted and discriminated against by AI models. To do that, the company would need to make a 180-degree turn from its current position.

Meta published a blog post introducing RSC on January 24th, 2022, and this article was published on February 5th.
(*Translation Note 2) Embodied AI means AI with physical mechanisms that can interact with the real world, and is a concept included in robotics in a broad sense. Embodied AI does not necessarily have to be humanoid, and can be established with just an arm.
On June 30, 2021, Meta published a blog post about Habitat 2.0 , a simulation environment for developing embodied AI . Using the same environment, for example, it is possible to research AI for housework that runs in the kitchen.
 In a blog post published on December 6, 2021, the company reported that a paper utilizing Habitat 2.0 was accepted at NeurIPS 2021.
(*Translation Note 3) For reference, the graph below summarizes the latest supercomputer ranking announced in November 2021 by the TOP500 project , which creates a ranking of supercomputers.
The blue bar in the graph below is Rpeak, which means the theoretical peak performance, and the red bar is the maximum calculated value Rmax measured based on the benchmark LINPACK adopted in the TOP500 project . Both units are teraflops (TELOPS). As of March 2022, Japan’s supercomputer Fugaku is number one . RSC is expected to probably rank within the top five .

In an interview with Jerome Pesenti published by The Wall Street Journal on January 24, 2022 , Meta uses RSC to develop an AI model that thinks like the human brain, There is an explanation that the model aims to realize a multi-sensorial experience in the Metaverse.

Jerome Pesenti was also quoted as saying:“In the Metaverse, 100% 3D multisensory experiences are possible, and we need to create artificial intelligence agents that relate to us in that environment,” said Jerome Pesenti, vice president of AI at Meta.

(*Translation Note 5) In a paper published in March 2021 by Professor Emily M. Bender et al . The dangers of language models are considered. In the same paper, in response to the view that “the larger the (English) learning data collected from the Internet, the more diverse expressions are collected, the more diverse it is,” it is actually based on extreme ideas such as white supremacy.
It is pointed out that the expressions contained in the text are excessive . And she warns that making language models as close to human-like as possible can lead to harmful models .


Please enter your comment!
Please enter your name here

Recent Posts

Most Popular

Recent Comments