What impact will Meta’s Llama 3 have on the future of AI?

In January 2024, Meta CEO Mark Zuckerberg announced in an Instagram video that Meta AI had recently begun training Llama 3. This latest generation of the LLaMa family of large language models (LLMs) follows the Llama 1 model (originally stylized as “LLaMA”). ) was released in February 2023, and the Rama 2 model was released in July.

Although specific details (such as model size or multimodal capabilities) have yet to be announced, Zuckerberg has indicated that Meta intends to continue to open source its Llama-based models.

Read on to find out what we currently know about Llama 3 and how it could impact the next evolution of generative AI models.

When will Rama 3 be released?

No release date has been announced, but it is worth noting that Llama 1 took three months to train and Llama 2 took approximately six months to train. The next-generation model is expected to be released around July 2024 if it follows a similar schedule.

Nonetheless, there is always the possibility that Meta will allocate additional time for fine-tuning and ensuring proper model alignment. Increasing access to generative AI models could empower more entities than just corporations, startups, and hobbyists. As open source models become more powerful, more care is needed to reduce the risk of malicious actors using them for malicious purposes. In his announcement video, Zuckerberg reiterated Meta’s commitment to “training (models) responsibly and safely.”

Is Llama 3 open source?

While Meta has granted free access to the Llama 1 model on a case-by-case basis to research institutions for non-commercial use cases only, the Llama 2 code and model weights have been released under an open license that allows commercial use by smaller organizations. It has over 700 million monthly active users. There is some debate as to whether Llama 2’s license meets the strict technical definition of “open source,” but it is commonly referred to as such. There’s no evidence that Llama 3 will launch any differently.

In his announcement and subsequent press, Zuckerberg reiterated Meta’s commitment to open licensing and democratizing access to artificial intelligence (AI). “I tend to think that one of the biggest challenges here is that if you build something really valuable, you end up being very focused,” Zuckerberg said in an interview. The Verge (Link is external to ibm.com). “On the other hand, making things more open can address a range of problems that can arise from unequal access to opportunities and value. That’s a big part of the overall open source vision.”

Can Llama 3 achieve artificial general intelligence (AGI)?

Zuckerberg’s announcement video highlighted Meta’s long-term goal of building artificial general intelligence (AGI), a theoretical development stage in AI where models demonstrate overall intelligence equal to or better than human intelligence.

“It has become increasingly clear that the next generation of services requires building full general intelligence,” Zuckerberg said. “Building the best AI assistants, AI for creators, AI for business, etc. will require advances in all areas of AI, from reasoning to planning, coding, memory, and other cognitive abilities.”

This does not necessarily mean that Llama 3 achieves (or attempts to achieve) AGI. yet. However, this means that Meta is deliberately approaching LLM development and other AI research in a way that it believes can create AGI. at last.

Will Llama 3 be multimodal?

The new trend in artificial intelligence is Multi-mode AI: A model that can understand and operate on a variety of data formats (or form). Instead of developing separate models to process text, code, audio, image or video data, new state-of-the-art models such as Google’s Gemini or OpenAI’s GPT-4V and open source participants such as LLaVa (Large Language and Vision Assistant) , Adept, or Qwen-VL—move seamlessly between computer vision and natural language processing (NLP) tasks.

Zuckerberg confirmed that Llama 3, like Llama 2, will include code generation capabilities, but did not explicitly mention any other multi-modal features. However, he did discuss how he envisions AI intersecting with the Metaverse in his Llama 3 announcement video. “Glasses are the ideal form factor for AI to see what it sees and hear what it hears,” Zuckerberg said in reference to Meta’s product. Ray-Ban smart glasses. “So you can get help at any time.”

This appears to mean that Meta’s plans for the upcoming Llama 3 release or the next generation of Llama models will include the integration of visual and audio data along with the text and code data that LLM is already handling.

This can be seen as a natural progression in the pursuit of AGI. “One may question whether general intelligence is similar to human-level intelligence, or like human plus, or whether it is some distant future superintelligence,” he said in an interview. The Verge. “But the important part to me is really the breadth of it: that intelligence has a variety of functions that you need to be able to reason about and intuit.”

How does Rama 3 compare to Rama 2?

Zuckerberg also announced significant investments in education infrastructure. Meta plans to have approximately 350,000 NVIDIA H100 GPUs by the end of 2024, which, including the GPUs it already has, brings Meta’s total available compute resources to “600,000 H100-equivalent compute.” Currently, only Microsoft has comparable computing power.

So it’s reasonable to expect Llama 3 to offer significant performance improvements over the Llama 2 model, even if the Llama 3 model isn’t any bigger than its predecessor. As hypothesized in a March 2022 paper in Deepmind and later demonstrated in Meta’s model (and other open source models, such as those from France-based Mistral), training a small model with more data can lead to training a large model with less data. You can get better performance than doing this. .(iv) Llama 2 was available in variants with the same size as the Llama 1 model, specifically 7 billion, 14 billion, and 70 billion parameters, but pretrained on 40% more data.

The Llama 3 model size has not yet been announced, but it is likely to continue the pattern of increasing performance within the 7 to 70 billion parameter models established in previous generations. Meta’s recent infrastructure investments will certainly enable much more robust pre-training for models at any scale.

Llama 2 is also twice as large as Llama 1. context lengthThis means that Llama 2 can “remember” the context of twice the token value during inference, i.e. during context creation or ongoing exchange with the chatbot. Although it is unclear, it is possible that Llama 3 will provide further progress in this regard.

How does Llama 3 compare to OpenAI’s GPT-4?

The smaller LLaMA and Llama 2 models met or exceeded the performance of the larger 175 billion parameter GPT-3 model on certain benchmarks, but did not match the full capabilities of the GPT-3.5 and GPT-4 models available in ChatGPT.

With its next-generation model, Meta appears intent on bringing cutting-edge performance to the open source world. “Rama 2 may not have been the industry leading model, but it was the best open source model,” he said. The Verge. “With Llama 3 and beyond, our ambition is to create cutting-edge products and ultimately the best model in the industry.”

Getting Ready for Rama 3

The new foundational model provides new opportunities to gain competitive advantage through improved apps, chatbots, workflows, and automation. Staying ahead of new developments is the best way to avoid being left behind. Embracing new tools can help organizations differentiate their offerings and provide the best experience for both customers and employees.

Through its partnership with HuggingFace, IBM watsonx™ supports a variety of industry-leading open source-based models, including Meta’s Llama 2-chat. Our global team of more than 20,000 AI experts can help you identify the tools, techniques, and technologies that best fit your needs so your company can scale efficiently and responsibly.

Learn how IBM can help you prepare to accelerate the advancement of AI. Leverage generative AI with watsonx™