Designing an AI Agent for Touhou-Style Bullet Hell Games

I added avoidance algorithms and a small Q-learning controller to a Touhou-style simulation I had built earlier.

Why not pure reinforcement learning?

Simple Q-learning cannot handle the full state space. I also tried a Dueling DQN, feeding it nearby bullet positions, enemy positions, and the player's position. The model became too large and training took far too long for this project.

Reducing the state space

The controller only considers three groups of information:

• threats from nearby bullets;
• enemy position and health;
• the player's position.

The game contains bullets with straight and function-based trajectories. Giving the controller each trajectory rule would be unfair, so threat prediction assumes that every bullet continues in a straight line, while the actual game still uses its original motion.

I divide the area around the player into nine cells. Within a fixed radius, such as 500 pixels, the controller checks bullets that may enter each cell. Speed and distance produce an estimated time to impact, which becomes the threat weight. The controller prefers the cell with the lowest threat.

When several choices are safe, it moves toward the most important enemy and keeps attacking until that enemy is defeated. It also receives a penalty near the screen edges, with the preferred region centered horizontally and roughly 70% down the screen. Q-learning only decides between options whose total weights are close.

Corner failures

With dense bullet patterns, the first version could get stuck in a corner because the cells outside the screen appeared to contain no bullets. I fixed this by assigning large boundary penalties that increase sharply with proximity.

Another failure occurred when a safe route existed only a few pixels away. A larger secondary care area now counts bullet density by direction. The controller first avoids immediate collisions, then chooses the less crowded route.