3.1 Computer Vision Model

3.2 Model Building and Hyperparameter Tuning

3.3 Model Pretraining and Reinforcement Learning Training

3.4 Evaluation

In this project, I set out to build an AI system capable of playing Pokémon Platinum, an open-world, story-driven video game, using computer vision and reinforcement learning with human feedback (RLHF)!

The project consists of two main models: an annotation model fine-tuned from YOLOv9s for object detection, and a gameplay model built and trained from scratch with RLHF to make strategic decisions within the game environment.

Technical Stack

To develop the computer vision annotation model, I manually annotated over 150 gameplay images, labeling key objects such as Pokémon, items, NPCs, and locations. Data augmentations, including random rotation, color jittering, and contrast adjustments, along with resizing transformations, were applied to enhance the model's robustness and generalization.

Once the annotation model was trained, it was used to automatically annotate an additional 450 gameplay states. Each state was paired with a set of possible actions (in JSON format)[automated in action_prep.py script] based on nine potential options: A, B, X, Y, Up, Down, Left, Right, and None.

These annotated state-action pairs were then used to pretrain the gameplay model, preparing it for the main reinforcement learning phase.

This project was divided into several key components to develop a robust AI model capable of navigating and interacting within the game environment. Each component played a critical role in building an effective and adaptive gameplay agent:

1. Development and training of a computer vision annotation model to identify key game elements.

2. Construction of a PyTorch-based gameplay model with hyperparameter tuning for optimal performance.

3. Pretraining of the gameplay model on annotated data, followed by reinforcement learning with human feedback to refine its interactions.

4. Evaluation of the model's performance in achieving in-game objectives.

3.1 COMPUTER VISION MODEL

The computer vision model, based on YOLOv9, detects key objects like Pokémon, items, and NPCs to enable situational awareness for the gameplay model.

• Architecture: YOLOv9, fine-tuned on a custom dataset of 150 annotated gameplay images.
• Dataset & Augmentation: Key objects manually labeled, with data augmentation (rotation, color jitter) applied to improve robustness.
• Training: Optimized hyperparameters for high detection accuracy. After training, the model auto-annotated 450 additional images.
• Outcome: Achieved high accuracy, providing reliable annotations for the gameplay model's pretraining phase.

Sample Annotation

Computer vision model building script here!

3.2 Model Building and Hyperparameter Tuning

Gameplay Model Architecture

class PokemonModelLSTM(nn.Module):
    def __init__(self, input_size, conv1_dim, conv2_dim, hidden_size, num_layers, num_actions, use_annotations=False):
        super(PokemonModelLSTM, self).__init__()

        # Store hyperparameters as attributes for easy access
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.num_actions = num_actions
        self.use_annotations = use_annotations
        
        # Convolutional layers to extract features from the game state
        self.conv1 = nn.Conv2d(3, conv1_dim, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(conv1_dim, conv2_dim, kernel_size=3, stride=2, padding=1)
        
        # LSTM to capture temporal dependencies; input_size will be adjusted dynamically in forward pass
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
        
        # Fully connected layer to map the LSTM output to the number of actions
        self.fc = nn.Linear(hidden_size, num_actions)
    
    def forward(self, x, annotations=None):
        # Input x is expected to be of shape (batch_size, seq_len, C, H, W)
        batch_size, seq_len, C, H, W = x.size()
        
        # Reshape input for convolutional layers
        x = x.view(batch_size * seq_len, C, H, W)
        
        # Apply convolutional layers with ReLU activations
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        
        # Flatten the output for the LSTM layer
        x = x.view(batch_size, seq_len, -1)  # Shape: (batch_size, seq_len, conv2_dim * H' * W')
        
        # Dynamically adjust LSTM input size if using annotations
        if self.use_annotations and annotations is not None:
            annotations = annotations.view(batch_size, seq_len, -1)  # Reshape annotations to (batch_size, seq_len, annotation_features)
            x = torch.cat((x, annotations), dim=-1)  # Concatenate along the feature dimension
            lstm_input_size = x.size(-1)  # Set the LSTM input size based on the concatenated features
            
            # Reinitialize LSTM if needed to match new input size
            if lstm_input_size != self.lstm.input_size:
                self.lstm = nn.LSTM(input_size=lstm_input_size, hidden_size=self.hidden_size, num_layers=self.num_layers, batch_first=True)

        # Apply LSTM
        x, _ = self.lstm(x)                  # x: (batch_size, seq_len, hidden_size)
        
        # Fully connected layer: map the LSTM output to the action space
        x = self.fc(x[:, -1, :])             # Use the last time step's output
        
        return x

3.3 Model Pretraining and Reinforcement Learning Training

Prior to reinforcement learning, the mdoel was trained on foundational knowledge through gameplay states/action in order to optmize the learning process.

Reinforcement Learning Training Loop

*Note that this is for exploitation, not exploration.

3.4 EVALUATION

The evaluation phase focused on assessing the model's performance in object detection and gameplay decision-making accuracy.

• Computer Vision Model Metrics:
  • Mean Average Precision (mAP): Used to measure detection accuracy for Pokémon, items, and NPCs.
  • Precision and Recall: Evaluated to ensure a balance between correctly identifying objects and avoiding false positives.
• Gameplay Model Metrics:
  • Accuracy of Actions: Percentage of correct actions taken by the model during gameplay.
  • Reward Score: Cumulative reward achieved per episode to assess decision-making quality in achieving game objectives.
  • Consistency: Frequency of successful task completions (e.g., navigating to specific locations).
• Human Feedback Integration: Analyzed the model’s improvement in decision-making accuracy based on human feedback adjustments.
• Real-Time Performance: Ensured that the model could perform efficiently within the game environment with minimal lag or delays.

Model pretraining script here!

Reinforcement learning scripts here!

The results showcase the model’s effectiveness across object detection and gameplay decision-making, with improvements observed through human feedback integration and real-time performance.

• Computer Vision Model:
  • Mean Average Precision (mAP): Achieved 70.5% for detecting Pokémon, items, and NPCs accurately.

  Show graph

  
  • Precision and Recall: The model achieved a balanced precision and recall of approximately 80%, effectively reducing false positives while accurately identifying in-game objects.

  Show graph

  
• Gameplay Model:
  • Action Accuracy: Attained 82% action accuracy, indicating consistent correct decisions during gameplay.
  • Reward Score: Averaged a cumulative reward score of 160 per episode, demonstrating effective decision-making aligned with game objectives.
  • Task Consistency: Successfully completed navigation tasks 75% of the time, reliably reaching designated locations in the game.
• Human Feedback Integration: Increased action accuracy by 12% post-feedback, enhancing decision quality and responsiveness to in-game challenges.
• Real-Time Performance: Achieved smooth operation with <1-second response time, maintaining real-time gameplay without perceptible lag.

Conclusion

As a personal project, this project laid the foundation for the feasibility of using a combination of computer vision and reinforcement learning to create an AI model that can autonomously navigate and interact within a complex open-world game

With the incorporation of human feedback, the model's performance was refined, making it adaptable to real-time challenges.

Future Improvements

Fine-Tune Reward Structure: : Enhance the reward system in the reinforcement learning model to incentivize more complex in-game strategies and behaviors.

Explore Multi-Agent Scenarios: : Develop the model to handle multi-agent environments or interactions with multiple NPCs for a richer gaming experience.

Enhance Model Memory: : Implement a memory module (e.g., extended LSTM or transformer-based) to track map, location, Pokémon status, and recent actions, enabling strategy based on game history.

3.1 Computer Vision Model

3.2 Model Building and Hyperparameter Tuning

3.3 Model Pretraining and Reinforcement Learning Training

3.4 Evaluation