Llama 3 presently options two mannequin weights, with 8B and 70B parameters. (The B is for billions and represents how advanced a mannequin is and the way a lot of its coaching it understands.) It solely provides text-based responses to date, however Meta says these are “a major leap” over the earlier model. Llama 3 confirmed extra range in answering prompts, had fewer false refusals the place it declined to answer questions, and will purpose higher. Meta additionally says Llama 3 understands extra directions and writes higher code than earlier than.
Within the publish, Meta claims each sizes of Llama 3 beat equally sized models like Google’s Gemma and Gemini, Mistral 7B, and Anthropic’s Claude 3 in sure benchmarking checks. Within the MMLU benchmark, which generally measures basic data, Llama 3 8B carried out considerably higher than each Gemma 7B and Mistral 7B, whereas Llama 3 70B barely edged Gemini Pro 1.5.
(It’s maybe notable that Meta’s 2,700-word publish doesn’t point out GPT-4, OpenAI’s flagship mannequin.)
It must also be famous that benchmark testing AI fashions, although useful in understanding simply how highly effective they’re, is imperfect. The datasets used to benchmark fashions have been discovered to be a part of a mannequin’s coaching, that means the mannequin already is aware of the solutions to the questions evaluators will ask it.
Meta says human evaluators additionally marked Llama 3 greater than different fashions, together with OpenAI’s GPT-3.5. Meta says it created a brand new dataset for human evaluators to emulate real-world eventualities the place Llama 3 is perhaps used. This dataset included use {cases} like asking for recommendation, summarization, and artistic writing. The corporate says the workforce that labored on the mannequin didn’t have entry to this new analysis knowledge, and it didn’t affect the mannequin’s efficiency.
“This analysis set incorporates 1,800 prompts that cowl 12 key use {cases}: asking for recommendation, brainstorming, classification, closed query answering, coding, artistic writing, extraction, inhabiting a personality/persona, open query answering, reasoning, rewriting, and summarization,” Meta says in its weblog publish.
Llama 3 is anticipated to get bigger mannequin sizes (which may perceive longer strings of directions and knowledge) and be able to extra multimodal responses like, “Generate a picture” or “Transcribe an audio file.” Meta says these bigger variations, that are over 400B parameters and might ideally be taught extra advanced patterns than the smaller variations of the mannequin, are presently coaching, however preliminary efficiency testing reveals these fashions can reply lots of the questions posed by benchmarking.
Meta didn’t launch a preview of those bigger fashions, although, and didn’t examine them to different large fashions like GPT-4.