I'm always curious what you guys are working on; every git repo I've run a local model on and stick below <100k to increase speed seems effective enough to scope patches and changes.
My current Claude Code session has been going on for like 35 hours and has used up around 400 million tokens, thankfully almost all of those being cached (95-98%) - pretty typical for long form agentic work.
First you spend like 2-3 hours working on a plan, once you have that you just tell the model to go and implement it, do adversarial sub-agent review loops before each commit and also make sure that all tooling and tests pass (including coverage requirements). You do need to poke it in a slightly different direction every few hours, though. Not even any novel work, just some refactoring and SSE notification hardening, bug fixes, alongside environment tuning and getting rid of some bottlenecks (also migrated from Oracle to PostgreSQL but that's mostly done).
That said, Kimi somehow manages to use less context in the main thread than Anthropic's models (even when you use sub-agents and also dynamic workflows in Claude Code), might have something to do with either how the model is tuned or their Kimi Code harness - because even in most of the longer form sessions it doesn't seem to fill up quite as quickly (note: because the kimi vis tool doesn't have a full summary view across all agents, these are the main long running agent stats across some sessions, not sub-agents):
I could see 256k context being sufficient for all sorts of work, even if intermediate progress/plan tracking files and docs might have to be used along the way, in addition to whatever plan support the harness has (for example, if you document something that will be relevant for load testing you might need that in 10 turns but not during the ones before then).
Once you have the plan you don't need to keep the 2hrs of research in the context (which is most of it) you can drop that plan into a file and start fresh for implementation.
That is true, but I feel like sometimes if the conversation contains useful rational it can help to keep it. I think sometimes it is a judgement call, I will sometimes compress the context first.
If I feel like the model and I explored a lot of options I won't want to keep context as it might be confusing.
I think the more you use it the better judge you are of whether you should purge, compress, or just keep the context before executing the plan.
Models work best when they have short instructions and no noise. "Conversation [may] contain" also means "conversation has a lot of noise". It degrades performance and increases cost.
I use handoff skill to ask model to write a prompt for itself.
The applicability of your advice is very model dependent. Some like Claude have very good long context performance, whereas others they fall off much quicker past some threshold.
You're right that it's model dependent, even within Claude models.
I've found Opus 5 far better as a subagent with very limited context window use, which could suggest that it might not have good long context performance unlike its predecessors. (I was one of many tearing my hair out trying to work with Opus 5 for the past week.)
I am not only talking about "long context" performance (context rot), but also about noise that is confusing model about its goal (from correctly extracting operator's intent). I think every model will get confused to a degree, so clearing up irrelevant information from context helps a lot.
> That is true, but I feel like sometimes if the conversation contains useful rational it can help to keep it. I think sometimes it is a judgement call, I will sometimes compress the context first.
Yes, pretty much - if there’s a lot of noise and jumping around and wrong conclusions and corrections, compress and only leave the correct stuff (maybe make some plan file briefly mention what NOT to look at/do). But if it’s all fairly straightforward then can just proceed with the execution.
Most of the time the planning stage ends up short of 200k tokens anyways, it mostly takes hours just cause I’m slow and need to explore the various options - still cheaper than building the wholly wrong thing and having to redo everything.
Compressing the context can also drop important information so it might be better to only do that when you need to / use the plan mechanism/files / do it after completing some large stage of the plan and so on - so a judgement call.
Thanks for sharing - is this a normal feature request you are implementing in this example or is this a project from scratch? Trying to get an idea of how your workflow compares to mine.
Existing project and as usual, a few issues mashed together in one mostly coherent plan. The shorter Kimi sessions I mentioned were singular features, there 256k would be wholly adequate.
A greenfield project would probably allow at least 2x fewer tokens to be used in most of those long tasks, but I was mostly after consistency and bug fixes along the way as needed.
Kimi CLI has a mechanism with checkpoints and the ability for the agent to revert to a checkpoint + a message of how to continue based on what went right/wrong. I don't know if that's the cause of what you've seen, but it's plausible.
No. Prefix caching is just an optimisation on the server side. What Kimi CLI does is insert <system> tags that include a checkpoint marker with an id.
The model is then given a tool that allows the model to decide to roll back to a checkpoint + a message containing any additional useful information.
It's specifically instructed to use that tool[1] in cases like when it has inadvertendly read a large file where most of the content is not relevant to the task, or after a web search where it's found what it's looking for but most of the content isn't needed, or when it's written code that didn't work as expected, or similar.
It basically lets the model backtrack and "forget" irrelevant details at the end of the context but give itself hints on how it should continue from the checkpoint.
Though, interestingly they seem to be abandoning it in their new CLI (kimi-code), unless it's been folded into other functionality. Not sure if they just feel it's not needed any more with their newer models or if it just didn't work as well as they expected.
Prefix caching automatically detects priorly read tokens -> more fained grained checkpoints then. With prefix caching you don't have to ask for a checkpoint, its automatic.
One way or the other, a prefix is saved. Only the additional info how to continue is new.
I had Claude build me a Python-inspired .NET language that treats .NET as a first class citizen, and breaks backwards compatibility where some Python nuances don't really apply to .NET for. I was able to get it to build a sample ASP .NET Web application that ran on Culebral code.
Haven't gone back to it, have been using Claude Code on a private project I'm still architecting.
Yeah, that was the goal. I've wanted something like Python for .NET as someone who does both, but backwards compatibility to Pythons ecosystem is something that will always make it harder than just having something Python-like as much as possible, but runs on .NET the other thing I wound up going for Rusts approach to type hints, method signatures should always have type hints, and when instantiating objects or types for the first time, the value should be inferred from there.
I think the distinction is when the model decides to read code top to bottom vs when the model chooses to parse code indirectly to save on tokens, there's also AST tooling to let the model see project structure.
Try doing a refactoring of some sort or larger new feature using just an agent on a moderately sized codebase, 256k will be compacting every few minutes, and result will be unusable.
I do essentially all of my work with auto compact set to 250k and it's fine. It may be due to the way the project tooling is set up and the use of sub agents?
They are just talking to the model in CC, while staying in a single thread. Doubt they have any actual coding knowledge to compartmentalize different problems in the codebase.
Depends on the programming language I'm using for a given project, and the domain I'm working with. I've been coding as a hobbyist for nearly two decades now (since my teens), professionally for 9 years, and was a TA before that for roughly 3 years at one of the best colleges for this field in the state (at least back then it was) where I taught other students about programming, in some cases I was their primary learning resource.
But yeah, I have no idea about anything about software because you made an assumption off very little to go by.
