SAP B1 blog

This is a continuation of the series on this blog, which deals with Artificial Intelligence involved with SAP Business One. It is an attempt to calmly think through and clarify fundamental AI terms. Each episode takes a concept from current AI research and ends by asking the same question: what of this actually arrives in everyday ERP, and what remains vocabulary that sells well? The last episode was about Verification and reward models. This time it's about a simpler, but surprisingly effective idea: simply asking the same question multiple times, or test-time compute scaling.

Read more: Test-Time Compute Scaling: Warum ein kleineres KI-Modell am Ende gewinnen kann — und was der Lieferantenvergleich in SAP Business One damit zu tun hat

More computing power, better AI: a rule of thumb with a gap

A larger, more expensive AI model delivers better answers. That sounds like a solid rule of thumb that people like to quote over and over again. However, it is not quite that simple. A small, cheap model can beat a large one in a direct comparison. The condition for this is simple: you have to allow it to ask the same question more often, and you have to filter out the useful answers afterwards. Model size is then no longer the decisive lever. It is just one of several.

Research on this is quite clear by now. A small model that is tested multiple times and subsequently filtered can outperform a large model that is consulted only once on complex tasks. The advantage does not lie in the better model. Rather, it is the number of attempts and a usable criterion for selection.

Why this matters for budget decisions

For managing directors and CFOs in medium-sized businesses, this is not an academic footnote. The decision regarding AI functions usually focuses on the model question: which system, which provider, which licence. Yet the real adjustable lever often lies elsewhere. It lies in how many times a system is allowed to check a process before it outputs an answer.

A budget that flows exclusively into a larger model can be worse spent than one that funds more testing loops for a smaller model. Test-time compute scaling is therefore often the better investment.

Hidden test-time compute scaling

This does not only concern the choice of the AI software itself. It also concerns the expectation of existing ERP functions that seemingly achieve something similar. Anyone who understands why repeatedly generating and checking yields more than a single expensive attempt will more easily recognise the same principle where it has long been built into their own system. Not infrequently, precisely this logic is embedded in functions that no one would describe as „artificial intelligence“ because they are hard-wired and do not learn from any decision. That does not make them any less effective.

An example can be found in purchasing in SAP Business One, in a place that at first glance has nothing to do with artificial intelligence. More on that later.

Many estimates, a surprisingly good result

A well-known experiment: a jar is filled with beans, and a large group of people are asked to independently estimate how many there are. Almost every single estimate is off, sometimes significantly. But if you take the average or the most common estimate, the result is usually amazingly close to the real number

.

Precisely this logic is behind repeated sampling, the central mechanism of test-time compute scaling. Instead of asking an AI model once and accepting the answer, it is made to solve the same task many times. Individual attempts may be wide of the mark. Over many attempts, however, it is possible to reliably filter out which answers are correct. That is precisely what defines test-time compute scaling: investing processing time at response time rather than additional training. A study entitled „Large Language Monkeys: Scaling Inference Compute with Repeated Sampling“has systematically demonstrated this effect for language models.

How repeated sampling works technically

The procedure can be described in four steps, regardless of the task at hand:

1. A task is formulated, such as a coding or maths question. The model and input remain unchanged, and no additional training takes place.
2. The model is called not once, but ten, a hundred or a thousand times with the same task, with deliberately high randomness in the phrasing of the answer. This generates many independent candidate answers.
3. Each candidate is checked against a correctness criterion, such as a unit test for program code, a formal proof checker for mathematics or a learned evaluation model for more general tasks.
4. The output is determined from the shortlisted candidates, either by majority vote or by best evaluation score.

The proportion of tasks that are correctly solved at least once among all generated candidates is then called coverage. It grows with each additional sampling round. The increase per round decreases in the process, but remains positive.

Why testing is the real bottleneck

All the effort is only worthwhile if the verification step itself is reliable. Generating is cheap: a model produces thousands of responses in a short time. Verification is the actual bottleneck. This applies especially where no automated test exists and another model or a human has to judge instead. If the verification step is weak, even the greatest number of attempts is of little use.

How much this pays off follows a simple pattern: with each additional round, the proportion of solved tasks increases, albeit with diminishing returns. The effect is nonetheless large enough that a smaller model can outperform one of the top models on complex tasks: if it gets enough attempts.

The supplier enquiry comparison: the same principle in purchasing

In the purchasing module of SAP Business One, there is a feature that replicates precisely this pattern, without anyone calling it Artificial Intelligence. When Supplier enquiry comparison the system compiles all received quotes from multiple suppliers for an item in a table. It then highlights the lowest unit price and the earliest quote date in colour as the best option.

An example makes this tangible. For a specific item, multiple quotes from different suppliers come in. The system places them side by side and automatically highlights which has the lowest unit price and which can deliver the earliest. The buyer no longer has to compare the figures themselves; they can see the best option at a glance.

Numbers make all the difference

Structurally, this is the same basic pattern as in repeated sampling. Multiple independently generated candidates, in this case quotes instead of AI answers, are checked against a fixed criterion. Price and delivery date take on the role of the verifier. A difference nevertheless remains: the colour coding is a decision support tool, not an automatic final selection as with the majority vote of an AI. In the end, a human still decides which quote is actually commissioned.

The difference is also evident in the number of candidates. A buyer rarely obtains more than a handful of quotes for an item, as every additional request costs time and effort for the supplier. A language model does not recognise this limit. It can generate ten or a thousand attempts for the same price.

A related feature, the item availability check, goes in the same direction. If a delivery bottleneck is imminent, the system suggests several potential solutions, such as alternative items or stock in other warehouses, from which the processor chooses.

A thought experiment: learned criteria instead of fixed thresholds

Not yet included in SAP Business One, but entirely conceivable with AI: alongside price and delivery date, a further adaptive criterion comes into play, derived from past supplier evaluations – such as on-time delivery or the complaint rate. Thus, a competitively priced offer from an unreliable supplier would no longer automatically take first place.

The risk in this: historical data also carries past negligence or mere habits into the future. An established supplier might continue to be favoured, even though a new tender is objectively better.

However, this distortion effect can be specifically countered by the system checking and displaying multiple criteria in parallel. Instead of rigid automation, purchasing is provided with a multidimensional basis for decision-making: the tried-and-tested, transparent logic remains visible, while knowledge from the past provides supplementary support.

Who it's worth it for

This article is aimed at managing directors and CFOs in the SAP Business One environment who want to understand how much substance there actually is to „more AI computing power“ before deciding on budgets or licence tiers. Anyone not using SAP Business One can still take away the explanation of repeated sampling; the SAP B1 part simply won't apply. Those looking for a concrete purchasing recommendation or product comparison will unfortunately not find it here. However, for those who want to pull back the curtain a little in the AI theatre, I hope they will find a first glimpse behind the scenes here.