Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama (patrickmccanna.net)

Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.

> How do you all keep your context so low? Complex tasks just balloon it in my experience.

By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps

The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases.

To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too).

To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices.

To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time.

When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).

My guess is your sessions are interactive conversations rather than planning a course of action then, separately, executing each piece.

That usually ends up being a poor use of LLMs, and is an unsolved problem with LLMs.

RAG was supposed to be the way out on that, and ended up being mostly abandoned.

A lot of this is managed by the inference engine, and has nothing to do with the model.

Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster).

If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.

Yes, please!

Understood, and I realized you'd posted this to HN yourself, so I felt a bit bad making the comment. I think maybe for me, this might have worked better if the motivation had been one separate post, and the details of the gotchas as a post of its own.

But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.

Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par with anything you'll get from hosted providers. You're getting between 60-80 for FP8 and 150+ tokens per second speed for MXFP4 and pp is 4K+. Lookup vllm radiance There is also a lot of progress in running Qwen 3.8 next flash with dual R9700. Obviously one gets less context and speed is a little bit less, but it's still very acceptable. Better than what you're getting with Llama on Strix Halo, that's for sure.

These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash

Your hardware can do way more than 64k tokens context window, can't it? And with Ollama it's very easy, superficially you just drag the slider.

I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.

So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.