Organic LLM is a UI lab and cognition lab for chat: memory, grounded tools, and streaming interfaces that stay inspectable.
Anatomy of a Response
A transparent walkthrough of one assistant turn: how Organic LLM packs context, routes tools, streams tokens, and renders the same structured UI you see in chat — using a recorded trace, not a live model call.
Summarize the latest thinking on retrieval-augmented generation and show me a diagram of how it works.
Input
When someone hits send, the server gets their message, the model they chose, which tools are enabled, and any files they attached.
Model for this turn
Enabled tools
Context load
The server assembles a compact context pack: recent turns plus a rolling summary. Older turns stay in the UI but are omitted from the model window when they would blow the token budget.
Incoming turn
Context Window
- Load thread14 latest messages hydrated from storage
- Rolling conversation summary~420 tokens attached
- Memories merged into pack
- Diagram-friendly explanations~48 tok
- RAG latency vs quality focus~52 tok
Window budget
Reserved for completion: 4,096 tok · Last-N turns in view: 8
Memory search
Mem0-style retrieval runs against persisted memories. High-relevance snippets are injected into the system context. Low-score or stale rows are dropped so the answer stays grounded and small.
Injected into context
- 91% match
User prefers concise technical explanations with diagrams when learning new ML topics.
- 84% match
User is comparing RAG architectures for a product doc and cares about latency vs quality.
Filtered out
- 31% match
User once asked for a plain-text summary of RAG without diagrams; low relevance to this diagram-heavy query.
Score 0.31 below threshold 0.55
- 62% match
Older note: user wanted bullet lists for RAG tradeoffs (latency vs quality) from a prior thread.
Superseded by newer memory on same topic
Tool routing
The model emits parallel tool calls: refresh the public literature with web search, then materialize structure as a Mermaid diagram. Each call is validated before execution.
User asked for latest thinking (web) and a diagram (Mermaid). Memory already loaded; run web_search then make_mermaid_diagram.
Tool calls
- Memory searchcall_mem_001Server38 ms
retrieval augmented generation preferences diagram
- Web searchcall_web_002Server410 ms
retrieval augmented generation survey 2025 2026 agentic RAG
- Mermaidcall_mmd_003Server120 ms
Flowchart: user query through embedding, retrieval, rerank, injection, LLM answer.
Token budget & streaming
Streaming keeps time-to-first-token low while a completion budget protects cost. A short system reminder nudges the model to land cleanly before the buffer runs out.
Token meter
Stream buffer target ~900 chars for smooth UI updates.
Live tail (aiAction stream)
Planning response…
Searching memories for RAG preferences…
Searching the web...
Creating diagram...
Streaming answer…
Graceful finish hint
You are near the end of the allotted completion length. Conclude with at most two short paragraphs and avoid opening new tool calls.
Structured card render
Tool JSON becomes first-class UI: Exa rows as glass cards, memory hits as scored rows, and Mermaid as an acknowledged diagram plus inline rendering in markdown.
## RAG in one pass Modern **retrieval-augmented generation** couples a parametric LM with a non-parametric index. Agentic variants add planning loops, but the core contract stays: *retrieve → compress → condition → generate*. ```mermaid flowchart LR Q1["User query"] --> E1["Embed"] E1 --> R1["Retrieve top K"] R1 --> X1["Rerank and filter"] X1 --> I1["Inject into prompt"] I1 --> L1["LLM"] L1 --> A1["Answer"] ``` The diagram mirrors the tool-generated flowchart so you can compare raw fences with the diagram card.
RAG in one pass
Modern retrieval-augmented generation couples a parametric LM with a non-parametric index. Agentic variants add planning loops, but the core contract stays: retrieve → compress → condition → generate.
The diagram mirrors the tool-generated flowchart so you can compare raw fences with the diagram card.
Optional TTS
For read-aloud, citations and code fences are softened into speakable prose. Playback uses the same glass dock styling as chat, but audio here is a short pre-rendered clip—no ElevenLabs call in the browser.