A forecast must live in the time when the decision was made

A small distributor wants to replenish stock using a demand forecast. The team trains a model on sales, randomly splits rows into training and validation sets, and gets an attractive error score. Performance then worsens in production: the training table already contained information about a future promotion, final inventory, or orders that did not yet exist on the purchase date. The robot peeked at tomorrow's delivery note and became yesterday's champion.

This is not a defect of a particular neural network. In an official scikit-learn example, random splitting of a time series produces an overly optimistic estimate compared with time-based evaluation. The textbook Forecasting: Principles and Practice recommends evaluation with a rolling forecasting origin: at each step the model sees only the past, then predicts a period that has not yet happened. For a small business, that check may matter more than choosing the most sophisticated model.

The practical question is not “How accurately did AI guess sales?” but “Would it have helped us order the right quantity in time?” First define the decision a person will make: next week's purchase order, a staffing schedule, or a production plan. The evaluation horizon must match supplier lead time and the point at which an order can still be changed.

Rebuild data as it was known at the time

For a pilot, choose a limited group of products with recurring demand. You need transaction date and time, SKU, quantity, returns, stock, deliveries, cancellations, prices, and promotions. If the supplier takes two weeks, tomorrow's forecast alone does not answer the purchasing question: you need demand until the next replenishment. A holiday calendar can be a factor known in advance; tomorrow's actual weather or final promotion result cannot.

For each historical decision point, reconstruct a snapshot of available information. It is useful to retain two timestamps: when an event happened and when its record became available to the system. If Monday's sale reached the ERP on Wednesday, a simulated decision made on Tuesday cannot use it. A feature table can enforce `available_at <= cutoff`, not merely `event_date <= cutoff`. This is an editorial architecture recommendation, not a claim that one library provides it out of the box.

Mark out-of-stock days separately. Zero sales when the shelf is empty do not prove zero demand: a model may learn the shortage as normal and order even less. Without confirmed unmet orders, treat these periods as uncertain when evaluating demand and review them with the purchasing owner. Do not “restore” lost demand using an invented number; that would start the pilot with a fabricated result.

Evaluation: several past Mondays instead of a random lottery

Split history into sequential windows. For example, train the first window on data through March and forecast April; train the second through April and forecast May. For weekly purchasing, use a weekly horizon and appropriate step. Transformations, feature selection, and hyperparameter tuning must happen inside each training window. Leave a final holdout period untouched until the approach is selected.

When creating lagged features, check when they became available. Average sales over the last four completed weeks can be valid. An average including the week you are predicting leaks the answer. If records arrive late, you may need a gap between training and validation windows; scikit-learn's `TimeSeriesSplit` supports a `gap` parameter. A gap alone does not fix features that were built using future information: inspect their provenance separately.

Compare at least three options: the buyer's current rule, a simple seasonal baseline, and a more complex model. Hyndman and Athanasopoulos show that “last observation” and “same period last season” are useful benchmark methods. If the new system does not beat such a baseline across several windows, its added complexity has no demonstrated return yet.

Break down error by horizon: a one-day forecast and a two-week forecast are different tasks. Review fast-moving goods, rare items, promotions, and stockout weeks separately. Mean absolute error is easy to understand in units for one product. Percentage MAPE behaves poorly around zero or very small sales, as the textbook notes. Across the range, supplement statistical errors with business outcomes: confirmed unmet orders, excess stock, emergency deliveries, and manual overrides. Do not treat one overall percentage as a quality certificate.

A pilot architecture without a mandatory LLM

An export from point-of-sale and ERP, a daily SKU table, reproducible feature calculation, and a simple scheduler retaining the cutoff date and model version are enough to begin. The service produces a recommended order quantity, explains how it differs from the current rule, and shows uncertainty. A buyer approves the decision; the order sent to ERP is theirs, not an autonomous model command.

Local deployment makes sense when commercial data and prices must stay inside the business or regular calculation volume justifies an in-house server. For a small demand table, start with a simple statistical or tabular model on CPU. A large language model does not make time-series evaluation honest. It could later explain an already calculated forecast in natural language, but numeric values and purchasing authorization should come from verifiable computation and human approval.

The integration should log what data were available, which forecast was issued, who changed it, and which order actually went to the supplier. Otherwise it is impossible to separate a bad forecast from a late delivery or a manual cancellation. Restrict access to prices and supplier terms by role; the test environment must not place orders automatically. If multiple stores share a model, keep their purchasing rights and assortment separate: one model does not imply shared access to commercial terms.

Economics: measure the changed decision, not the model score

Consider an illustrative, not real-company, example for 50 products. Under the old rule, one month ends with 600 excess units and 120 documented unmet orders. After a controlled rollout, comparable conditions show 500 excess units and 100 confirmed unmet orders. If carrying one excess unit costs a hypothetical RUB 8 per month, freeing 100 units saves RUB 800. If 20 genuinely recovered sales contribute RUB 250 each, that adds RUB 5,000. Gross effect is RUB 5,800; after RUB 3,000 monthly support, the modeled net effect is RUB 2,800.

These are neither reported company results nor a project payback forecast. They depend on confirming that those twenty orders would actually have been filled, seasonality, capital cost, write-offs, prices, and employee workload. A lower MAE without a different purchasing decision yields none of those modeled rubles. Extra stock may improve availability while consuming margin. Before paying for infrastructure, specify which decisions will change and what an error costs in each direction.

Next step: two weeks in the buyer's shadow

Take 20–50 products with sufficient history and one real replenishment cycle. In the first week, clean availability timestamps, promotion records, and stockout days; then run several historical windows against the current rule and a seasonal baseline. In the second week, issue recommendations in shadow mode: the buyer sees them but continues placing orders through the current process. Record differences and explanations, especially when the model recommends a sharp jump.

Move to limited production only if the approach uses no future data, remains robust across different windows, and improves the cost of actual decisions rather than just one metric. Begin with manual order approval and a quick return path to the old rule. The AI-generated cover for VnutrII shows a manager stopping a robot from moving a future box into past records. The calculation needs the same barrier.