What was released
On 27 October, Yandex and its School of Data Analysis (YSDA) released MTGBench, an open benchmark for testing how a language model behaves across a longer conversation. It contains seven text games: Wordle, Wheel of Fortune, Word Chain, Situation Puzzles, 20 Questions, 20 Features, and Alias. Each next move depends on earlier turns: a model must incorporate new information, plan its steps, and remain consistent with its own previous answers.
Tasks are prepared separately in Russian and English. Four games use algorithmic checks; three use an AI judge whose performance was checked against human ratings. The authors tested 16 models. A paper describing the work appears in Findings of EMNLP 2026, and the project code and materials are available in its repository.
What the authors found
Step-by-step reasoning did not help equally on every task. The clearest gains appeared in games involving letters and symbols, interpreting clues, and ruling out incorrect options. Improvements were less consistent on knowledge-retrieval and semantic tasks, and varied by model. Results also differed across games, while models generally performed worse in Russian than in English.
For a business, this is not a ready-made model ranking. Rather, MTGBench illustrates why one aggregate score or a short test cannot predict the quality of a conversational assistant: outcomes change with the task, language, context length, and evaluation method.
How to use the takeaway
For a support, sales, or internal knowledge-base pilot, borrow the testing principle—not the games themselves. Create a small set of anonymized, multi-turn scenarios from your workflow and check whether the model can:
- retain facts from earlier messages;
- incorporate clarifications and new constraints;
- ask the next useful question;
- avoid contradictory answers;
- complete the scenario at acceptable latency and cost.
Compare models on the same scenarios, separately in Russian, with the same settings. An important limitation: MTGBench’s puzzles do not replace tests of retrieval from company data, policy compliance, or customer-answer quality. Reproducing the experiment also requires environment setup and checking the terms of third-party lexical resources in the repository.
The practical conclusion is to add a few conversations with changing requirements to your usual accuracy checks. It is a low-cost way to catch context loss before an assistant enters a live workflow.