Checking whether Spark actually helps the model

Adding tools around a small model only helps if the answers get better. The September 10–11 FreeSpark work made that a question we could test, comparing the same requests through Spark and through the bare model.

One early comparison found a useful regression. Asked to sort a supplied list, the bare model preserved the numbers while Spark invented a leading value. The fix moved narrowly defined numeric-list operations into deterministic code. Sorting the user's numbers shouldn't depend on the model reproducing every digit correctly.

The suite then grew to 41 cases, including conversation, code, strict formatting, misleading questions, supplied material and injected instructions in research results. Routing changes kept private-context questions out of web search and treated pasted content as local material. A chart tool only needed to run when a chart was actually requested.

Pass 7 recorded 38 of 41 automatic checks for the 2B Spark route and 39 of 41 for the 8B route, against 32 of 41 for the corresponding bare runs. The blind-graded results were recorded separately. Those scores describe this suite and harness, not a universal intelligence score.

The tests also needed review. A correct poem could fail an overly literal word check, and a mechanical formatting pass could miss an answer that sounded awkward or searched unnecessarily. Keeping the actual answers alongside the scores made those differences visible.

For me, that's the useful outcome of this stage: a way to find where Spark improves a model and where the extra machinery hurts. The next pass used that evidence to reduce repeated model calls, improve responsiveness and keep more of the conversation feeling natural.