Google’s next-gen Gemini 1.5 Pro uses Mixture-of-Experts to match Ultra performance at lower compute, with a breakthrough 1M token context window.
The status indicator reflects the current state of the work: - Abandoned: Work that has been discontinued - Notes: Initial collections of thoughts and references - Draft: Early structured version with a central thesis - In Progress: Well-developed work actively being refined - Finished: Completed work with no planned major changes This helps readers understand the maturity and completeness of the content.
The confidence tag expresses how well-supported the content is, or how likely its overall ideas are right. This uses a scale from "impossible" to "certain", based on the Kesselman List of Estimative Words: 1. "certain" 2. "highly likely" 3. "likely" 4. "possible" 5. "unlikely" 6. "highly unlikely" 7. "remote" 8. "impossible" Even ideas that seem unlikely may be worth exploring if their potential impact is significant enough.
The importance rating distinguishes between trivial topics and those which might change your life. Using a scale from 0-10, content is ranked based on its potential impact on: - the reader - the intended audience - the world at large For example, topics about fundamental research or transformative technologies would rank 9-10, while personal reflections or minor experiments might rank 0-1.
Gemini 1.5 Pro released this week with the promise.
"The model delivers dramatically enhanced performance, with a breakthrough in long-context understanding across modalities."
It was designed to be a mid-size multi modal model that matches the performance of 1.0 Ultra (their largest model) while simultaneously managing to use less compute than the prized heifer. 1.5 uses a transformer, and mixture of experts architecture. MoE allows the model to be split into smaller "expert" narrow llms rather than the traditional monolith neural net. Meaning for any given input, only relevant expert pathways active, leading to more effective training and inference.
The defining feature of 1.5 Pro is still it's context window however.
| Metric | Value |
|---|---|
| Standard context window | 128,000 tokens |
| Max context window (preview) | 1 million tokens |
| Tested in research up to | 10 million tokens |
A context window of 1 million tokens is equivalent to 1 hour of video, 11 hours of audio, >30K lines of code, >700K words.
The defining feature of 1.5 Pro is its context window:
| Metric | Value |
|---|---|
| Standard context window | 128,000 tokens |
| Max context window (preview) | 1 million tokens |
| Tested in research up to | 10 million tokens |
What 1 million tokens can hold:
Sign in with GitHub to comment
Yotam, Kris. (Feb 2024). Gemini 1.5 Pro Announced. krisyotam.com. https://krisyotam.com/news/llms/gemini-1.5-pro-announced
@article{yotam2024gemini-1.5-pro-announced,
title = "Gemini 1.5 Pro Announced",
author = "Yotam, Kris",
journal = "krisyotam.com",
year = "2024",
month = "Feb",
url = "https://krisyotam.com/news/llms/gemini-1.5-pro-announced"
}