Meta Platforms, the parent company of Facebook, Instagram, and WhatsApp, made headlines at its recent Meta Connect conference by unveiling a range of new AI features for its popular consumer-facing services. However, it was a computer science paper quietly published by Meta researchers on arXiv.org that stole the show. The paper introduced Llama 2 Long, an extended version of Meta's open-source Llama 2 AI model, which outperforms competitors in generating responses to long user prompts.
The research paper highlights how Llama 2 Long has undergone continual pretraining from Llama 2, using longer training sequences and a dataset with upsampled long texts. As a result, this elongated AI model surpasses leading competition, including OpenAI's GPT-3.5 Turbo and Claude 2, in generating responses to high-character count user prompts.
Meta researchers achieved these impressive results by enriching the original Llama 2 model with additional longer text data sources. They kept the architecture of Llama 2 unchanged but modified the positional encoding, known as Rotary Positional Embedding (RoPE), which is crucial for the model to handle longer text inputs. By reducing the rotation angle of RoPE encoding, the researchers ensured that more distant tokens, representing less common or less connected information, were included in the model's knowledge base. This modification allows Llama 2 Long to produce accurate and helpful responses while requiring less computing storage.
To train Llama 2 Long, Meta researchers employed reinforcement learning from human feedback (RLHF). RLHF involves rewarding the AI model for correct answers, with human oversight to ensure accuracy. Synthetic data generated by Llama 2 chat was also used to enhance the model's performance in various tasks, such as coding, math, language understanding, common sense reasoning, and answering user prompts.
The release of the research paper showcasing Llama 2 Long's capabilities has sparked enthusiasm within the open-source AI community, including discussions on Reddit, Twitter, and Hacker News. This development not only validates Meta's commitment to open-source generative AI but also demonstrates that open-source models can compete with closed-source alternatives offered by well-funded startups.
With its continued focus on AI advancements, Meta Platforms is positioning itself as a pioneer in the field of artificial intelligence. By leveraging its research expertise and open-source approach, the company aims to foster innovation and drive the future of AI technology across its suite of consumer-facing services.
In conclusion, Meta Platforms has introduced Llama 2 Long, an extended AI model that outperforms competitors in generating responses to long user prompts. The research paper published by Meta researchers showcases the model's capabilities and highlights the modifications made to the original Llama 2 architecture. This achievement not only validates Meta's open-source approach but also demonstrates the potential of open-source AI models to compete with closed-source alternatives. As Meta continues to push the boundaries of AI technology, it solidifies its position as a leader in the field and paves the way for future advancements in the industry.
