Research notes

1Introduction

In chess, Deep Blue defeated the world champion in 1997 (Campbell et al., 2002). In Go, AlphaGo defeated a top professional in 2016 (Silver et al., 2016). AlphaZero then surpassed humans in chess, shogi and Go using self-play alone, without human game records (Silver et al., 2018). In these games, the search for optimal play has gone beyond the level of human understanding.

In strategy games such as civilization simulations, however, no AI yet consistently beats human experts. The rules of Civilization IV: Beyond the Sword (2007; hereafter Civ4) are defined entirely by formulas and random draws. Even so, the optimal strategy is unknown.

Civ4 is composed of a small number of simple elements: tiles, unit movement, cities, production, research and combat. Each element is simple, but optimizing all of them jointly makes the number of possible action combinations grow rapidly. Experts handle this through experience and intuition, and there is a culture of competing on skill. The developers also released the source code of the game’s core (the SDK) for modders, so each calculation can be verified.

Earlier studies have used Civilization games (Section 2), but computing resources and development effort were major constraints. We think that with today’s hardware and AI coding assistants (such as Claude Code), a single person can build a verified Civ4 simulator and search for strategies across a large number of games. This project aims to fill that gap.

Multiplayer Civ4 is often played with three or more players. More players, however, bring many more variables: diplomacy, alliances and third-party intervention. We therefore first restrict the problem setting to one-on-one play, and extend the methods established there to multiplayer later.

2Related work

Strategy games have long served as benchmarks (shared tasks for comparing methods) at the frontier of AI research, for three main reasons: clear rules and outcomes that let us measure the quality of an AI’s decisions objectively; a long history of human expert play that allows direct comparison with human intelligence; and the way each new kind of game has let AI take on one more difficulty of real-world decision-making (imperfect information, many players, teamwork, negotiation, long-term planning). Strategy games are a miniature of the real world where human and machine decision-making can be compared on the same footing.

The world’s leading AI research labs appear along this line of work.

• AlphaZero surpassed humans in chess, shogi and Go through self-play alone. This was in games where the whole board is visible and each move is small. Silver, D., Hubert, T., Schrittwieser, J., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140–1144.
• Pluribus beat top professionals at six-player poker, a game with imperfect information and multiple players. Brown, N., & Sandholm, T. (2019). Superhuman AI for multiplayer poker. Science, 365(6456), 885–890.
• AlphaStar reached Grandmaster level in StarCraft II. It did not start from scratch: it began with imitation learning from human game records, and kept human play as a reference throughout reinforcement learning. Vinyals, O., Babuschkin, I., Czarnecki, W. M., et al. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782), 350–354.
• OpenAI Five beat the reigning world champion team at the 5v5 team game Dota 2 in 2019, using large-scale reinforcement learning on thousands of GPUs over months. Berner, C., Brockman, G., Chan, B., et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv, 1912.06680.
• Tencent’s JueWu beat professional players at the 5v5 team game Honor of Kings without restricting the pool of playable characters. Ye, D., Chen, G., Zhang, W., et al. (2020). Towards playing full MOBA games with deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS 2020), 33.
• DeepNash reached a level comparable to human experts at Stratego, a war board game where the identity of the opponent’s pieces is hidden. Perolat, J., De Vylder, B., Hennes, D., et al. (2022). Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 378(6623), 990–996.
• CICERO reached the top 10% of an online league in Diplomacy, a seven-player game of alliances and betrayal, while negotiating with players in natural language. As a multiplayer strategy game with diplomacy, it is the closest to multiplayer Civ. Meta Fundamental AI Research Diplomacy Team (FAIR), Bakhtin, A., Brown, N., et al. (2022). Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067–1074.
• On Civilization II, a method was proposed that uses the text of the game manual to guide action selection in Monte-Carlo search; it won over 65% of games against the built-in AI. Branavan, S. R. K., Silver, D., & Barzilay, R. (2012). Learning to win by reading manuals in a Monte-Carlo framework. Journal of Artificial Intelligence Research, 43, 661–704.
• CivRealm is a research environment built on Freeciv. Reinforcement learning made some progress on restricted sub-tasks (mini-games), but neither reinforcement learning nor large language models have achieved strong results on the full game. Qi, S., Chen, S., Li, Y., et al. (2024). CivRealm: A learning and reasoning odyssey in Civilization for decision-making agents. International Conference on Learning Representations (ICLR 2024).
• For Civ4, Better BTS AI and K-Mod improved the built-in AI with hand-written rules (a rule-based approach). Players generally consider it still short of expert level.

In short, AI has overcome hidden information, many players, teamwork and negotiation one game at a time. Civilization simulations, which combine all of these with a broad scope per decision and many interacting elements, remain little studied. As far as we found, no study has directly addressed competitive Civ4 play.

3How we got here

“The experimenter” below refers to the person running this project, who has played competitive Civ4 BtS for a total of about four years.

3.1Background: how modern machine learning works

Recent successful AI systems (AlphaZero, AlphaStar, large language models) share a common form. A function with a very large number of parameters (numbers called weights), a neural network, is adjusted little by little in the direction that reduces the loss (error), using large amounts of data or self-play results (gradient descent). Individual parameters have no meaning a human can read; meaning emerges afterwards from their combination. In chess and Go, this method learned patterns of positional evaluation, from simple inputs such as piece and stone positions, that humans cannot articulate.

3.2First approach: treat everything as flat actions and optimize for wins alone

We first followed that form: treating the AI’s choices as a flat set of actions (an action space) with no human interpretation, and optimizing all of them jointly through massive self-play, with winning as the only objective function. We compared 6 opening strategies and 43 candidate capital sites, each over more than a thousand self-play games.

The differences were buried in noise from the advantage of the starting position and from how wars after turn 50 played out. Even the capital site that came out as most winning was not where the experimenter would settle; it was an answer fitted to the weaknesses of the AI at the time.

At the root is a chicken-and-egg problem. Identifying which choices contributed to a win from a delayed reward such as the game outcome is inherently hard (the credit assignment problem in reinforcement learning). On top of that, while the AI (the engine) is weak, the outcome itself does not correctly reflect which choices were good. Without knowing which choices were good, the AI cannot get stronger. This is generally considered hard, but modern AI broke through the loop with massive self-play and computing power (parallel computation on GPUs and the like). AlphaZero went from a random initial state to beyond human level through self-play alone.

3.3Structure of Civ4: a combination of elements with built-in meaning

In chess and Go the rules are extremely simple, and the pieces and stones carry no meaning in themselves. Humans understand them by applying abstract frameworks such as opening theory and positional judgment, but those frameworks can drift from what is actually best. AI, through combinations of enormous numbers of parameters, found patterns beyond that drift.

Civ4 is the opposite. The meaning and formula of each element (population, research, production, happiness, health, combat, leader traits) are given by the game’s designers from the start. A player’s skill shows less in finding patterns than in how the elements with fixed meanings are combined, and which takes priority in which situation. It is closer to building a castle from blocks of predetermined shapes.

In a game with this structure, treating everything as flat actions and breaking the chicken-and-egg loop with large-scale compute runs into combinatorial explosion. Each turn’s decisions cover every city and every unit; if every single-tile move of every unit is treated as an action, the action space grows exponentially with the number of elements. Not only on a MacBook Air but even with large-scale computing resources, this direction does not fit the structure of the problem and is hard to call a sound approach.

Why does the large-scale-compute approach, which worked for chess, Go and language, struggle here? Our current hypothesis is that it is not only a matter of computing resources, but of the game’s structure: the size of the action space, and rules given in advance as formulas with built-in meaning. AlphaStar building on human play rather than starting from scratch, and the struggles with the full game in CivRealm, are consistent with this view.

Direction: build the principles experts use (named strategies and axes of judgment) into the AI as prior knowledge, and tune through learning how they are mixed, how they are prioritized, and how they are weighed in each situation.

3.4Adding principles by hand

The experimenter carried out error analysis by reviewing the AI’s games ten turns at a time; the verbal feedback was translated into principles with reasons and applicability conditions, and built into the AI (about 100). The effect was measured in controlled experiments, with the new and old AI playing on the same maps and random seeds.

Most of the measured principles stayed around a 0.5 win rate; few showed a clear improvement. Every situational exception (“settle at pop 1 here, pop 4 there,” “except in cases like this”) had to be written as a formula, and the amount to write had no end in sight. Once, a fix broke a behavior taught earlier. This was the same road as classical rule-based AI (expert systems), and it did not use the strength of modern machine learning: parameters improving automatically through learning.

3.5Where learning worked: capital placement

Choosing the starting capital can be framed as a supervised learning problem with one correct answer per question. We made 132 questions from real starting screens and, using the experimenter’s answers as training data, learned weights for about 30 terrain and resource features.

At one point, adding questions lowered leave-one-out cross-validation accuracy (73% → 59%); the cause was not too little data but how the features were defined. Re-splitting how food is counted to match the experimenter’s reasoning brought it back to 73%, and accuracy now rose with more data. A small neural network on the same questions reached 97% on the training data but dropped to 53% on unseen questions: overfitting, where a model too expressive for the amount of data learns quirks specific to the training set.

With appropriate features (axes of judgment), learning can tune a decision from little data. Extending the same approach to turn 1, turn 10, turn 50 and beyond could reach the whole game.

3.6Simulator (hereafter Industrial Civ4)

Learning needs a large number of games, but the real Civ4 spends heavily on graphics, animation and interface, so it cannot run games in bulk. We therefore wrote a Civ4 in Python that performs only the rule calculations.

• Order of processing: the rules are spread across the game’s core source code (C++) and data (XML), and results drift unless even the order of processing matches. For example, trade routes are recalculated at the end of the turn and take effect from the next turn; research still advances slightly during anarchy.
• Combat: outcomes are random, with detailed rules for odds, withdrawal, collateral damage, healing, fortification and bombardment. We built a reconciliation table and verified each rule against the source code. Army behavior was rebuilt from v1 to v5 based on the experimenter’s feedback.
• Execution speed: reusing repeated calculations cut a game from 14.5 s to 9.2 s (150 turns). An 80-turn game now takes about 3.5 s; on a MacBook Air (Apple M2) with 7 in parallel, about 7,000 games an hour.

Verification: the moves of games the experimenter played in the real Civ4 are fed in, and the state is compared every turn. In a game with barbarians and huts, all 50 turns from T0 to T49 matched exactly. Rules are trusted only once they pass this check (differential testing), to keep the AI from gaining spurious strength by exploiting simulator errors (so-called reward hacking). Out-of-scope elements such as diplomacy and espionage are not implemented.

3.7Current approach: design features, optimize parameters

We split learning into two tasks of different natures.

Feature engineering

Deciding which aspects (features) the AI uses to judge: finding an axis of judgment the AI does not see, and turning it into a form the AI can use. Values are not decided here. Example: adding the decision “watch for barbarians” to the AI, with a parameter for its strength (how many lookouts). Knowing what should be a feature requires mastery of the game’s systems and strategy, so this step depends on human expertise.

Parameter optimization

The machine searches for the parameter values. Currently 25 parameters are searched by playing 6 slightly shifted candidates against the original AI for 40 games each and keeping the winner (a (1+λ) evolution strategy, a kind of gradient-free black-box optimization). In the imitation-learning stage, they are planned to be optimized with gradient methods, as with capital placement.

In the first 9 generations of the parameter search, no candidate beat the original AI (a candidate with a 0.57 win rate dropped to 0.47 in a 60-game validation). Over 40 games, the standard error of the win rate is about ±8%, so small differences are buried in noise.

3.8Data collection: three routes

The three ways to play on the public site, Daisensei MOD, are designed as routes for collecting data for the two tasks in 3.7 (the name on the site is in parentheses).

Human error analysis(Watch and heckle)

Material for feature engineering. Through error analysis, people watching the AI’s games point out “this perspective is missing,” which reveals missing axes of judgment. Seeing the game well and putting the angle of a judgment into words matters more than being a strong player. A single perspective introduces bias toward particular strategies and blind spots.

Imitation learning from game records(Train Daisensei with saves)

Material for parameter optimization. Moves actually played by experienced players are used as per-position training data for imitation learning.

Behavior testing against humans(Show Daisensei who’s boss)

Checks whether the trained AI behaves as intended in real games against humans. It surfaces positions that rarely arise in self-play and weaknesses humans exploit. Since the number of games is small, it serves as a check rather than as material for parameter optimization (in preparation).

4Anticipated criticisms and our position

• Isn’t this just a lack of compute? With massive GPUs, couldn’t a flat action space work after all?We do not rule this out; Section 3.3 is a working hypothesis. Still, the large-scale successes in StarCraft II and Dota 2 also built on human play or added hand-designed structure to the action space and rewards. Our position is that, under limited compute, exploiting the game’s structure is the sounder approach. Verification with large-scale compute would be a valuable point of comparison in the future.
• If human principles are built in, can the AI ever exceed human limits? AlphaZero got strong without human data.Humans design the features, but parameters such as how principles are mixed and prioritized are optimized through self-play, so combinations no human has tried can still be found. We acknowledge the limit that ideas absent from the features themselves cannot be discovered. We first build an AI that executes the experts’ common ground without mistakes, then gradually open the fixed parts to learning.
• Is the simulator really the same as Civ4? Strength in the simulator may not transfer to the real game.Real games are fed in and the state is checked every turn (Section 3.6). Rules are trusted only once they pass this check, and out-of-scope elements such as diplomacy and espionage are not implemented at all, so the AI cannot use them. However, the match has been confirmed only for T0–T49 of one recorded game; agreement in the later part of the 80-turn training games, or in positions not covered by records, is not guaranteed. Extending the games and range checked is treated as a precondition of the research.
• The judgment criteria depend on a single experimenter. Isn’t that subjective?The training data for capital placement and the error analysis behind the principles come from one experimenter. To reduce this bias, the public site collects error analysis and game records from many players (Section 3.8).
• Isn’t one-on-one play different from how Civ4 is normally played?We recognize that most multiplayer games involve three or more players (Section 1). The problem setting is narrowed to reduce variables, and the extension to multiplayer is listed as an open issue (Section 5).
• The collected saves may not come from strong players.Rather than averaging everything, we select games by outcome and record (the winning side, top players and so on) for imitation learning. This policy is also stated on the site for people sending saves.
• Are the win-rate differences statistically meaningful?Given a standard error of about ±8% over 40 games, a candidate that wins is re-validated over 60 more games before adoption (Section 3.7). We recognize that the number of games is not enough to distinguish small differences, and will improve both game counts and validation methods.
• Why not let a large language model like ChatGPT or Claude play?Large language models are trained to predict the continuation of text, not to win games. Even at chess they fall far short of dedicated engines such as Stockfish. A language model can call an external engine and play strongly, but that simply delegates the decisions to the engine, which does not work for Civ4, where no such engine exists yet. A game also requires thousands of decisions, so querying move by move makes large numbers of games infeasible in both speed and cost. In CivRealm, large language models have not achieved strong results on the full game either (Section 2). In this project, language models support development of the engine itself (the simulator and the AI) through Claude Code, not in-game decisions.

5Issues

• Save file parserExtract each turn’s position and the moves played from save files
  PythonFile format analysis
• Evaluation functionPredict the number of turns to victory from a position
  Machine learning
• Features for high-level strategyLet the AI choose between growing its economy and attacking
  Civ4 strategy knowledge
• Strength evaluation scaleA rating-like scale comparable with human experts
  Statistics
• Extension to multiplayerAdd diplomacy (open borders, peace and so on) to Industrial Civ4 and extend to three or more players
  Civ4 rules knowledge

6Source code

This is the AI playing in the spectator seat. It combines modules for economy, combat and barbarian lookouts, and its parameters are optimized by self-play win rate.

# Some of the AI’s parameters (initial values before optimization)
ATTACK_RATIO   = 1.25  # attack when military strength is at least this multiple
LAUNCH         = 8     # main-force units needed before setting out
WATCH_PER_CITY = 0.7   # barbarian lookouts per city
WATCH_MAX      = 5     # maximum lookouts

ComingTo be released once prepared for publication.

References

• Berner, C., Brockman, G., Chan, B., et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv, 1912.06680.
• Branavan, S. R. K., Silver, D., & Barzilay, R. (2012). Learning to win by reading manuals in a Monte-Carlo framework. Journal of Artificial Intelligence Research, 43, 661–704.
• Brown, N., & Sandholm, T. (2019). Superhuman AI for multiplayer poker. Science, 365(6456), 885–890.
• Campbell, M., Hoane, A. J., & Hsu, F. (2002). Deep Blue. Artificial Intelligence, 134(1–2), 57–83.
• Meta Fundamental AI Research Diplomacy Team (FAIR), Bakhtin, A., Brown, N., et al. (2022). Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067–1074.
• Perolat, J., De Vylder, B., Hennes, D., et al. (2022). Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 378(6623), 990–996.
• Qi, S., Chen, S., Li, Y., et al. (2024). CivRealm: A learning and reasoning odyssey in Civilization for decision-making agents. International Conference on Learning Representations (ICLR 2024).
• Silver, D., Huang, A., Maddison, C. J., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489.
• Silver, D., Hubert, T., Schrittwieser, J., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140–1144.
• Vinyals, O., Babuschkin, I., Czarnecki, W. M., et al. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782), 350–354.
• Ye, D., Chen, G., Zhang, W., et al. (2020). Towards playing full MOBA games with deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS 2020), 33.