Classical ML and generative AI in one operational loop

A surprising number of teams now ask a large language model to forecast next month's demand, flag unusual transactions or work out a delivery schedule. The model answers fluently, and the answer is usually worse than what a gradient-boosted tree, an isolation forest or a linear-programming solver would have produced in a few milliseconds. Language models are not built to fit a time series, estimate a distribution or search a constrained solution space. They are built to read, write and decide what to do next.

The opposite mistake is just as common. Operations teams have had forecasting and anomaly models for years, and the output often sits unused in a dashboard because nobody has time to work out why the number moved or what to do about it. A forecast that says demand for a product will rise 30% is half a decision. Someone still has to check whether it is believable, explain it to the buyer and raise the purchase order.

The useful design puts each tool where it is strong. Classical machine learning and optimisation produce the numbers. A language model turns those numbers, and the evidence behind them, into an explanation a person can check. An agent carries the decision into the systems where work happens, at an autonomy level the business has agreed. This article sets out how we wire that loop, with a worked example from replenishment, and where it goes wrong.

A division of labour

The split we use is simple to state. If the task has a numeric objective and historical data to fit, it belongs to a model you can measure with a numeric error metric. If the task involves reading unstructured context, explaining a result or choosing between tools, it belongs to the language model. If the task changes a record in another system, it belongs to an agent with scoped permissions and an audit trail.

Our Predict accelerator holds the forecasting and anomaly models, Explain holds the explanation layer, and Monarch Bridge orchestrates the agent steps. The names matter less than the boundaries: each layer has its own inputs, outputs and tests, and a failure in one is visible without reading the others.

Forecasting stays a modelling problem

The strongest public evidence on retail forecasting comes from the M competitions. In the M5 accuracy competition, which used Walmart unit sales, the top-performing entries relied heavily on LightGBM and other tree-based models with careful feature engineering, and they beat the statistical benchmarks by a clear margin. That is still where we start for tabular demand data with known drivers such as price, promotion, calendar and stock availability.

Pretrained time-series foundation models are a newer option. Amazon's Chronos and Google's TimesFM produce zero-shot forecasts without per-series training. We use them as a fast baseline for new products or new sites with little history, and we compare them against a fitted model on the same holdout before trusting them. They are not language models answering in prose; they are numeric models with a transformer architecture, and they are evaluated like any other forecaster.

Whatever the model, a forecast without an interval cannot drive a decision. Reorder quantities depend on how uncertain demand is, not just its expected value. We produce quantile forecasts or wrap point forecasts with conformal prediction, which gives intervals with a stated coverage under mild assumptions; Angelopoulos and Bates give a practical introduction. The interval is also what lets the explanation layer say "this is unusually uncertain" with a basis.

The language model does have a place on the input side. Promotions, supplier notices and sales notes often arrive as free text. A model can extract structured features from them (promotion start and end dates, discount depth, affected SKUs) which then feed the forecaster. That extraction is evaluated on its own labelled set, because a misread date silently corrupts every forecast downstream.

Anomaly detection with a triage layer

Detection is the cheap part. A seasonal decomposition with a threshold on the residual, or an isolation forest over transaction features, will find unusual points in stock movements, payments or shipment times. The expensive part is the human hour spent on each alert working out whether it matters. That is where the language model earns its place.

When an anomaly fires, an agent gathers context through read-only tools: recent transactions for the item, open purchase orders, warehouse adjustments, related tickets, the supplier's recent notices. The model then writes a short triage note that states what is unusual, lists the evidence it found, offers ranked candidate causes with the evidence for each, and recommends a next step. The detector decides what is anomalous; the model never overrides that score. It only shortens the path from alert to judgement.

Imagine a distributor whose warehouse system records a sudden drop in on-hand stock for a fast-moving SKU at one site. The detector flags it at three standard deviations below the expected level. The triage agent finds a cycle-count adjustment posted that morning, no matching sales spike, and a ticket from the site about a mislabelled pallet. Its note ranks "count adjustment error" first, cites the adjustment ID and the ticket, and suggests a recount before any replenishment. A planner reads it in a minute instead of opening four systems.

Optimisation belongs to solvers

Replenishment, slotting, routing and staffing are constrained optimisation problems. A solver such as those in Google OR-Tools will find a feasible plan that respects every constraint you encode, or tell you that none exists. A language model asked the same question will produce a plausible plan that may quietly break a capacity limit or a minimum order quantity. We do not let language models produce plans that are executed without passing through a solver or a deterministic checker.

The model is useful at both ends of the solver. Before, it can help a planner express a what-if ("what if the Rotterdam supplier slips two weeks?") by drafting a change to solver inputs, which the planner confirms and the system validates against a schema. After, it can explain the plan: why the solver split an order across two suppliers, which constraint was binding, what the cost of relaxing it would be. Solvers expose this information through slack values and dual prices; the explanation layer turns them into sentences, and does not make up its own reasons.

The explanation contract

Explanations go wrong when the model is handed a number and asked "why?". It will answer, and the answer will sound right whether or not the data supports it. We prevent this with a contract: the model receives a structured evidence payload, and every claim in its explanation must point to a field in that payload.

{
  "item": "SKU-48213",
  "site": "DC-North",
  "forecast": { "horizon_weeks": 4, "p50": 1840, "p10": 1510, "p90": 2260 },
  "baseline_last_4_weeks": 1390,
  "top_drivers": [
    { "feature": "promo_discount_pct", "value": 20, "contribution": 260 },
    { "feature": "week_of_year", "value": 47, "contribution": 140 },
    { "feature": "competitor_stockout_flag", "value": 1, "contribution": 70 }
  ],
  "solver": { "order_qty": 600, "binding_constraint": "dc_inbound_capacity_wk2" },
  "evidence_ids": ["promo:PR-2291", "note:SN-0714"]
}

The driver contributions come from the forecasting model itself, for example SHAP values for a tree model, not from the language model's guess. The explanation layer may say "the forecast is 32% above the recent baseline, mainly because of the 20% promotion PR-2291 and the usual late-November uplift"; it may not introduce a driver that is not in the payload. A post-generation check compares every number and identifier in the text against the payload and rejects explanations that do not match.

Agents act at an agreed autonomy level

Once a forecast and a plan exist and the explanation has passed its checks, an agent can draft the purchase order in the ERP, propose a stock transfer in the warehouse system, or open a ticket for a recount. What it is allowed to do without a person is a policy decision, set per action type and per risk tier: Assist (draft only), Approve (execute after a named person approves) or Autonomous (execute and report).

In the replenishment example, a sensible starting policy is that the agent drafts every purchase order, a planner approves anything above a value threshold or outside the forecast interval, and orders within the interval and under the threshold for long-standing suppliers move to autonomous only after a period of approvals with few edits. The approval screen shows the explanation, the interval and the solver's binding constraint, so the planner is approving a reasoned proposal, not a number. Every action goes through AgentGate with the agent's own identity and scopes, and every approval is recorded.

Measuring the combined system

Each layer has its own metric, and the combined system needs one more. Forecasters are measured on error and interval coverage over rolling holdouts. Detectors are measured on precision and recall against labelled incidents. Explanations are measured on faithfulness to the payload (automated) and usefulness to planners (sampled human review). Agents are measured on approval rate, edit rate and reversals.

The combined measure is decision quality: did the actions taken lead to better outcomes than the old process? For replenishment that means stock-outs, excess stock and expediting cost, compared against a period or a set of sites still on the previous process. This takes months to read, and it is the only number the operations director will care about. Set it up before launch, because a baseline cannot be reconstructed afterwards.

Where it breaks

The explanation layer can make a bad forecast more persuasive. A well-written paragraph with cited drivers reads as authoritative even when the interval is wide or the model is extrapolating. We surface the interval and a plain "low confidence" flag next to every explanation, and we instruct the model to lead with uncertainty when coverage is poor or history is short. Fluency should never be the thing that gets a number approved.

Feedback loops are the second problem. Once agents act on forecasts, the data the next model trains on reflects those actions: stock that was ordered was available to sell, and stock that was not ordered could not sell. Censored demand (sales capped by stock-outs) gets worse if nobody corrects for it. Treat stock-out periods explicitly in training, and keep a small control group of decisions made the old way.

The third is ownership. Data scientists own the forecaster, platform engineers own the agent, and nobody owns the explanation. When planners stop trusting the explanations, the whole loop degrades back to a dashboard. Assign one owner for the end-to-end decision, with authority over all three layers. We would also not build the full loop first. Starting with the explanation layer on top of an existing forecast, with no agent action, is cheaper and shows within weeks whether planners find it useful.

A decision rule

When a team proposes using a language model for an operational decision, we ask three questions in order. If the answer to any is no, the design changes before any code is written.

1. Is every number in the output produced by a model or solver that is measured on a numeric holdout, with an interval or a feasibility check?
2. Does the language model receive a structured evidence payload, and is every claim it makes checked against that payload automatically?
3. Is the agent's action bounded by an explicit autonomy level per action type, with approvals recorded and outcomes compared against a baseline?

Three yeses means the language model is doing the work it is good at, and the numbers are coming from something you can test.