This week, our team continued to work on our GNN reinforcement learning model for Catan. During this, we also sought to improve Catanatron’s trading interfaces.
Originally, Catanatron only allowed maritime trading, that is, trading at ports or with the resource bank, and not with other players. However, we wanted to implement player trading for two reasons: (1) player trading is a central game mechanic of Catan, and (2) we wanted to challenge ourselves to build a machine learning agent that could utilize more advanced tactics.
Catanatron did have functions for player trading, but bots never attempted to perform player trades. This was because all bots base their actions on their provided list of allowed actions. This list never included player trade actions even if player trading was allowed. Thus, our first attempt at enabling player trading was to create a list of possible player trade actions for our players to make.
Here are examples of player trade actions:
- Offer 1 lumber for 1 brick
- Offer 1 lumber for 2 brick
- Offer 2 lumber for 1 brick
- Offer 2 lumber for 2 brick
- Offer 1 lumber for 1 wool
- Offer 1 lumber for 2 wool
- And so on
Our goal was to enumerate every possible trade with a few limits: You can only offer/request a max of 2 resources at a time, and you can only make up to 3 trade offers in one turn. As you can imagine, this list of possible trades can get very long. As such, random choice players (which choose randomly among their list of allowed actions) would almost always attempt to trade since the trade options took up most of their action space.
Thus, we tried a different approach to trading: First, a player must decide if they want to trade. Only after they decide to trade are they given a list of possible trade options. By separating the actions of “deciding to trade” and “making a trade offer”, our bots won’t have to spend as much time considering their possible actions. This ensures our models will continue to face strong opponents during testing.
Trading is a difficult mechanic to implement in our simulation and teach to our AI models. However, we believe this added challenge can reveal more about the strengths and weaknesses of machine learning.
That’s all for now. See you next week!