Skip to content
The Chat Archive
o1-mini-2024-09-12· OpenAI WildChat-4.8M (AI2)

build a python pytorch project, aimed to train an AI like Alphazero to play a game named…

  1. User

    build a python pytorch project, aimed to train an AI like Alphazero to play a game named NoTouch, the rule is: The game is played on a board with n rows and m columns.

    Two players participate, the color of their pieces are same.

    Players take turns placing their pieces on empty spaces on the board.

    Rules for placing pieces:

    Pieces can only be placed on empty spaces. New pieces cannot be placed adjacent to already placed pieces (above, below, left, right). Game ending condition: The game ends when there are no legal moves left.

    Winning condition: The last player who can successfully place a piece wins.

  2. Assistant

    Building an AI similar to AlphaZero for the game NoTouch involves several components, including defining the game logic, creating a neural network architecture with PyTorch, implementing Monte Carlo Tree Search (MCTS), and setting up a training loop for self-play. Below is a comprehensive guide along with the necessary code to get you started.

    Project Structure

    Here's an overview of the project structure:

    nontouch_alpha_zero/
    ├── main.py
    ├── game.py
    ├── neural_network.py
    ├── mcts.py
    ├── self_play.py
    ├── train.py
    ├── utils.py
    └── requirements.txt
    

    1. requirements.txt

    First, ensure you have the necessary libraries installed. Create a requirements.txt file with the following content:

    numpy
    torch
    torchvision
    

    Install the dependencies using:

    pip install -r requirements.txt
    

    2. game.py: Define the NoTouch Game Logic

    This module defines the game state, move generation, and game mechanics.

    # game.py
    import numpy as np
    
    class NoTouchGame:
        def __init__(self, n=5, m=5):
            self.n = n  # Rows
            self.m = m  # Columns
            self.board = np.zeros((n, m), dtype=int)  # 0: empty, 1: player 1, -1: player 2
            self.current_player = 1
            self.last_move = None
    
        def clone(self):
            clone_game = NoTouchGame(self.n, self.m)
            clone_game.board = np.copy(self.board)
            clone_game.current_player = self.current_player
            clone_game.last_move = self.last_move
            return clone_game
    
        def get_legal_moves(self):
            legal_moves = []
            for i in range(self.n):
                for j in range(self.m):
                    if self.board[i, j] != 0:
                        continue
                    if self.is_adjacent(i, j):
                        continue
                    legal_moves.append((i, j))
            return legal_moves
    
        def is_adjacent(self, i, j):
            directions = [(-1,0), (1,0), (0,-1), (0,1)]
            for di, dj in directions:
                ni, nj = i + di, j + dj
                if 0 <= ni < self.n and 0 <= nj < self.m:
                    if self.board[ni, nj] != 0:
                        return True
            return False
    
        def make_move(self, move):
            if move is None:
                return  # Pass or game end
            i, j = move
            if self.board[i, j] != 0 or self.is_adjacent(i, j):
                raise ValueError("Invalid move")
            self.board[i, j] = self.current_player
            self.last_move = move
            self.current_player *= -1
    
        def is_game_over(self):
            return len(self.get_legal_moves()) == 0
    
        def get_winner(self):
            if not self.is_game_over():
                return None
            # The player who made the last move wins
            return -self.current_player
    
        def encode_board(self):
            # Encode the board for the neural network
            # Two planes: one for current player, one for opponent
            current_player_board = (self.board == self.current_player).astype(int)
            opponent_board = (self.board == -self.current_player).astype(int)
            return np.array([current_player_board, opponent_board], dtype=np.float32)
    

    3. neural_network.py: Define the Neural Network with PyTorch

    The neural network takes the current board state as input and outputs move probabilities (policy) and the value of the state.

    # neural_network.py
    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    
    class NoTouchNetwork(nn.Module):
        def __init__(self, board_rows, board_cols):
            super(NoTouchNetwork, self).__init__()
            self.board_rows = board_rows
            self.board_cols = board_cols
            self.conv1 = nn.Conv2d(2, 64, kernel_size=3, padding=1)
            self.conv_blocks = nn.Sequential(
                nn.Conv2d(64, 64, kernel_size=3, padding=1),
                nn.BatchNorm2d(64),
                nn.ReLU(),
                nn.Conv2d(64, 64, kernel_size=3, padding=1),
                nn.BatchNorm2d(64),
                nn.ReLU(),
            )
            # Policy head
            self.policy_conv = nn.Conv2d(64, 2, kernel_size=1)
            self.policy_fc = nn.Linear(2 * board_rows * board_cols, board_rows * board_cols)
            # Value head
            self.value_conv = nn.Conv2d(64, 1, kernel_size=1)
            self.value_fc1 = nn.Linear(board_rows * board_cols, 64)
            self.value_fc2 = nn.Linear(64, 1)
    
        def forward(self, x):
            x = F.relu(self.conv1(x))
            x = self.conv_blocks(x)
            # Policy head
            p = F.relu(self.policy_conv(x))
            p = p.view(-1, 2 * self.board_rows * self.board_cols)
            p = self.policy_fc(p)
            p = F.log_softmax(p, dim=1)
            # Value head
            v = F.relu(self.value_conv(x))
            v = v.view(-1, self.board_rows * self.board_cols)
            v = F.relu(self.value_fc1(v))
            v = torch.tanh(self.value_fc2(v))
            return p, v
    

    4. mcts.py: Implement Monte Carlo Tree Search

    MCTS is used to explore possible moves and select the most promising one based on the neural network's predictions.

    # mcts.py
    import math
    import numpy as np
    import torch
    
    class TreeNode:
        def __init__(self, game_state, parent=None):
            self.game_state = game_state.clone()
            self.parent = parent
            self.children = {}
            self.visit_count = 0
            self.value_sum = 0
            self.prior = 0
    
        def is_leaf(self):
            return len(self.children) == 0
    
        def expand(self, action_priors):
            for action, prior in action_priors:
                if action not in self.children:
                    next_state = self.game_state.clone()
                    next_state.make_move(action)
                    self.children[action] = TreeNode(next_state, parent=self)
    
        def update(self, value):
            self.value_sum += value
            self.visit_count += 1
    
        def value(self):
            if self.visit_count == 0:
                return 0
            return self.value_sum / self.visit_count
    
    class MCTS:
        def __init__(self, network, n_simulations=100, c_puct=1.4):
            self.network = network
            self.n_simulations = n_simulations
            self.c_puct = c_puct
            self.Qsa = {}
            self.Nsa = {}
            self.Ns = {}
            self.Ps = {}
            self.root = None
    
        def search(self, game_state):
            if self.root is None:
                self.root = TreeNode(game_state)
    
            for _ in range(self.n_simulations):
                node = self.root
                state = game_state.clone()
    
                # Selection
                while not node.is_leaf():
                    max_ucb = -float('inf')
                    best_action = None
                    for action, child in node.children.items():
                        ucb = self.calculate_ucb(node, child)
                        if ucb > max_ucb:
                            max_ucb = ucb
                            best_action = action
                    node = node.children[best_action]
                    state.make_move(best_action)
    
                # Evaluation
                if state.is_game_over():
                    winner = state.get_winner()
                    value = 1 if winner == game_state.current_player else -1
                else:
                    board = torch.tensor(state.encode_board()).unsqueeze(0)
                    p, v = self.network(board)
                    p = p.detach().numpy()[0]
                    v = v.item()
                    legal_moves = state.get_legal_moves()
                    action_priors = [(action, np.exp(p[action[0] * state.m + action[1]])) for action in legal_moves]
                    node.expand(action_priors)
                    value = v
    
                # Backpropagation
                while node is not None:
                    node.update(value if node.game_state.current_player == game_state.current_player else -value)
                    node = node.parent
    
        def calculate_ucb(self, parent, child):
            if child.visit_count == 0:
                return float('inf')
            return child.value() + self.c_puct * child.prior * math.sqrt(parent.visit_count) / (1 + child.visit_count)
    
        def get_action_probs(self, game_state, temperature=1):
            self.search(game_state)
            counts = {action: child.visit_count for action, child in self.root.children.items()}
            actions, visits = zip(*counts.items())
            visits = np.array(visits)
            if temperature == 0:
                best_action = actions[np.argmax(visits)]
                probs = np.zeros(len(actions))
                probs[np.argmax(visits)] = 1
                return best_action, probs
            visits = visits ** (1 / temperature)
            probs = visits / np.sum(visits)
            return actions, probs
    

    5. self_play.py: Handle Self-Play for Training Data

    This module uses MCTS to play games against itself and collect training data.

    # self_play.py
    import numpy as np
    from game import NoTouchGame
    from mcts import MCTS
    
    def self_play(network, n_simulations=50, temperature=1.0):
        game = NoTouchGame()
        mcts = MCTS(network, n_simulations=n_simulations)
        states = []
        mcts_probs = []
        current_players = []
    
        while not game.is_game_over():
            action, prob = mcts.get_action_probs(game, temperature)
            states.append(game.encode_board())
            mcts_probs.append(prob)
            current_players.append(game.current_player)
            game.make_move(action)
            mcts = MCTS(network, n_simulations=n_simulations)
    
        winner = game.get_winner()
        target_values = [winner if p == winner else -winner for p in current_players]
    
        return states, mcts_probs, target_values
    

    6. train.py: Define the Training Loop

    This module trains the neural network using the data collected from self-play.

    # train.py
    import torch
    import torch.optim as optim
    from torch.utils.data import DataLoader, TensorDataset
    import numpy as np
    
    def train(network, optimizer, states, mcts_probs, target_values, batch_size=64, epochs=10):
        states = torch.tensor(np.array(states))
        mcts_probs = torch.tensor(np.array(mcts_probs))
        target_values = torch.tensor(np.array(target_values), dtype=torch.float32)
    
        dataset = TensorDataset(states, mcts_probs, target_values)
        loader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
    
        for epoch in range(epochs):
            for batch_states, batch_p, batch_v in loader:
                optimizer.zero_grad()
                p, v = network(batch_states)
                loss_p = -torch.mean(torch.sum(batch_p * p, 1))
                loss_v = torch.mean((batch_v - v.squeeze())**2)
                loss = loss_p + loss_v
                loss.backward()
                optimizer.step()
            print(f"Epoch {epoch+1}/{epochs} - Loss: {loss.item():.4f}")
    

    7. main.py: Integrate All Components

    The main script orchestrates self-play and training.

    # main.py
    import torch
    from neural_network import NoTouchNetwork
    from self_play import self_play
    from train import train
    
    def main():
        # Initialize network
        board_rows, board_cols = 5, 5
        network = NoTouchNetwork(board_rows, board_cols)
        optimizer = torch.optim.Adam(network.parameters(), lr=0.001)
    
        num_iterations = 1000
        for iteration in range(num_iterations):
            print(f"Iteration {iteration+1}")
            # Self-play to generate training data
            states, mcts_probs, target_values = self_play(network, n_simulations=50, temperature=1.0)
            print(f"Self-play generated {len(states)} training examples")
            # Train the network
            train(network, optimizer, states, mcts_probs, target_values, batch_size=64, epochs=10)
            # Optionally, save the network
            torch.save(network.state_dict(), f"nontouch_model_iter_{iteration+1}.pth")
    
    if __name__ == "__main__":
        main()
    

    8. utils.py: Additional Utilities (Optional)

    You might need utility functions for logging, evaluation, or visualization. Here's a basic example:

    # utils.py
    import numpy as np
    
    def print_board(board):
        symbols = {0: '.', 1: 'X', -1: 'O'}
        for row in board:
            print(' '.join([symbols[cell] for cell in row]))
        print()
    

    Notes and Considerations

    1. Efficiency: The above implementation is a simplified version and may not be optimized for performance. For larger boards or more complex games, consider optimizing the MCTS, utilizing GPU acceleration effectively, and potentially using more advanced neural network architectures.

    2. Neural Network Enhancements: The current network architecture is basic. You can improve it by adding more layers, residual connections, or other architectural changes inspired by AlphaZero.

    3. Parallelization: To speed up training, consider parallelizing self-play games and utilizing multiple CPUs or GPUs.

    4. Hyperparameters: The number of MCTS simulations, learning rate, batch size, and other hyperparameters can significantly affect performance. Experiment with different values to find the optimal settings.

    5. Evaluation: Implement periodic evaluation against fixed policies or previous versions of the network to monitor improvement and prevent overfitting.

    6. Saving and Loading Models: Incorporate functionality to save the best-performing models and resume training as needed.

    7. Handling Terminal States: Ensure that the game correctly handles terminal states and that the neural network learns to evaluate them accurately.

    Running the Project

    1. Set Up: Ensure all the above files are created within a single directory (nontouch_alpha_zero).

    2. Install Dependencies:

      pip install -r requirements.txt
      
    3. Execute Training:

      python main.py
      

    This will start the self-play and training loop. The model checkpoints will be saved at each iteration.

    Conclusion

    This implementation provides a foundational framework to build an AlphaZero-like AI for the NoTouch game using PyTorch. Depending on the game's complexity and desired performance, you might need to refine and optimize various components further. Happy coding!