A faster way to answer simple messages before the big model wakes up

[Andrea-Bruno]

We built this inside a copy of Gemini CLI, since Qwen Code was made from it, and the fit is close to a direct copy. We are sharing an idea and a working change. The idea is small but it can make the agent feel much quicker and cheaper to run, especially for people who run models on their own computer.

Right now, every message you type wakes up the big language model, even when the message is simple. A greeting, a question about something already on the screen, or a clear request for one small action all pay the cost of a full model call. On a cloud service this costs money. On a home computer or a small graphics card it feels slow.

Our idea is to put a very small and fast decider in front of the big model. This decider does not write text. It looks at the message for a few tens of milliseconds and answers a few simple questions with a confidence score. Is this just chat. Does it need a tool. Can it be answered from what is already on screen. Is it complex or risky. When the decider is sure, the message takes a short and cheap path. When it is not sure, the message goes to the big model exactly as it does today. So the agent is never worse than before. It is only faster when the decider is confident.

We built this inside a copy of your project and we are sending a pull request so you can see how it fits your code. It is off by default. When it is first turned on it only watches and records what it would have done, without changing anything. If the small model is not there or is slow, the system falls back to the normal path on its own.

The pull request is here: #29482

The full idea, the license, and a note about why we chose not to patent it are here: https://github.com/Andrea-Bruno/harness-superfast

The change is offered under a Creative Commons Attribution 4.0 license. If you take the idea or the code, we ask for one small thing. Please keep a short credit to Andrea Bruno and a link to the page above in your project documentation. That is the only condition. You are free to use it, change it, and ship it, including in a paid product.

We would be glad to hear what you think, and we are happy to adjust the change to fit how your project works.