How Compaction Works in Pi

Discussion (5 Comments)Read Original on HackerNews

kennywinker•33 minutes ago

Compaction has been a pretty painful part of local llm usage. Scrapping the current context and parsing almosy 128k of context then generating something like 5-10k tokens - that can take quite a while when you’re working with 10t/s-45t/s (depending on the model).

I pretty much just start a new session whenever i fill the context.

Gecko4072•about 1 hour ago

Can someone recommend a Hermes alternative that is less token hungry? Pi did not work well for my use case.

pkulak•21 minutes ago

I keep my max context really small for personal assistant agents; they don't need it. Especially since compaction keeps anything important around anyway. I use 60k with Pi.

skinfaxi•41 minutes ago

I found hermes to be really lightweight, though I am on a relatively older version and built a custom plugin to lazily load mcps (that's probably in hermes proper by now). Compared to kilo it seems to consume far fewer tokens.

randomblock1•44 minutes ago

TLDR: It keeps ~20k tokens of recent conversations, then hands the rest of the conversation to another model with a special system & user prompt. This then fills out a template with relevant information.

See: https://github.com/earendil-works/pi/blob/main/packages/codi...

How Compaction Works in Pi

⚡ Community Insights

Discussion (5 Comments)Read Original on HackerNews