Relational Markov Decision Processes

Freecraft Application

 

 

 

VIDEOS

For best visualization of these movies, please select the full screen feature in your AVI player.

These videos demonstrate the results of the policies generated by our algorithm on the two Freecraft domains described in the paper:

 

Tactical domain

Planning world

 

3 footmen and 3 enemies   

AVI video

 

We planned in a model with 3 footmen and 3 enemies. These footmen coordinate their attack and are able to kill the enemies, losing only one footman in the process.

 

Generalization world

4 footmen and 4 enemies   

AVI video

 

 

We generalized our value function to a problem with 4 footmen and 4 enemies. The coordination strategy generalizes very effectively. Note in the video that at first, all footmen attack one enemy. As this enemy weakens, they coordinate, decreasing the number of  footmen on this enemy, until only one footman is left. This footman finishes off the enemy, and moves on to the next one.

 

Generalization world

5 footmen and 5 enemies   

AVI video

 

 

We then generalized our value function to a problem with 5 footmen and 5 enemies. Here, the footmen also demonstrated interesting coordination. For example, in the beginning 4 footmen attack one enemy and 1 attacks another. Our model says that the probability of killing an enemy with more than 4 footmen is the same as the one with 4 footmen. Although we never planned in a problem with 5 footmen, our generalization captures this characteristic of the model. Unfortunately, at this point, we do not take into account the position of the units. In this example, one unit spends a lot of time trying to reach an enemy. This inefficient causes our policy to lose to Freecraft's one in a close battle.

 

 

 

Strategic domain

Planning world

 

2 peasants, 1 barrack, 2 footmen and 1 enemy  

AVI video

 

We planned in a model with 2 peasants (a builder and a regular peasant), 1 barrack, 2 footmen, and an enemy. Note that only after both footmen are built, they start attacking the enemy. Nonetheless, both are killed by the stronger enemy. However, as the enemy becomes weak after this attack, in the next round, only one footman is sent to attack it. 

 

Generalization world

 

9 peasants, 1 barrack, 3 footmen and 1 enemy  

AVI video

 

 

We generalized to a model with 9 peasants (a builder and 8 regular peasants), 1 barrack, 3 footmen, and an enemy. Here our policy waits for the 3 footmen to be built before attacking. They are able to kill the enemy, and only one footman is lost.

 

 

More information

More details on the game and models used are available at:

The tactical domain

The strategic domain

Download a copy of the paper:

Postscript