give me a Matlab code for windy grid world problem in reinforcement learning
- User
give me a Matlab code for windy grid world problem in reinforcement learning
Assistant
Certainly! Below is an implementation of the Windy Grid World problem in MATLAB. This is a classic reinforcement learning problem where an agent learns to navigate a grid while dealing with "wind" that pushes it upward. The goal is to find the optimal path from a start state to a goal state.
The algorithm used here is Q-learning, which is a model-free reinforcement learning algorithm. The code includes comments for clarity.
MATLAB Code: Windy Grid World with Q-Learning
% Windy Grid World Problem using Q-Learning % Parameters gridSize = [7, 10]; % Grid dimensions (rows x columns) startState = [4, 1]; % Starting position (row, col) goalState = [4, 8]; % Goal position (row, col) wind = [0, 0, 0, 1, 1, 1, 2, 2, 1, 0]; % Wind strength for each column numActions = 4; % Up, Down, Left, Right actions = [-1, 0; 1, 0; 0, -1; 0, 1]; % Action effects on (row, col) % Hyperparameters alpha = 0.5; % Learning rate gamma = 0.9; % Discount factor epsilon = 0.1; % Exploration rate numEpisodes = 500; % Number of episodes for training % Initialize Q-table Q = zeros(gridSize(1), gridSize(2), numActions); % Helper function to check if a state is within bounds function valid = isValidState(state, gridSize) valid = state(1) >= 1 && state(1) <= gridSize(1) && ... state(2) >= 1 && state(2) <= gridSize(2); end % Training loop for episode = 1:numEpisodes % Initialize state state = startState; while ~isequal(state, goalState) % Epsilon-greedy policy for action selection if rand < epsilon % Explore: Choose random action action = randi(numActions); else % Exploit: Choose best action [~, action] = max(Q(state(1), state(2), :)); end % Apply wind effect nextRow = state(1) + actions(action, 1) - wind(state(2)); nextCol = state(2) + actions(action, 2); % Ensure the next state is within bounds nextRow = max(1, min(nextRow, gridSize(1))); nextCol = max(1, min(nextCol, gridSize(2))); nextState = [nextRow, nextCol]; % Compute reward reward = -1; % Default reward for each step if isequal(nextState, goalState) reward = 0; % Reward for reaching the goal end % Update Q-value Q(state(1), state(2), action) = Q(state(1), state(2), action) + ... alpha * (reward + gamma * max(Q(nextState(1), nextState(2), :)) - ... Q(state(1), state(2), action)); % Move to the next state state = nextState; end end % Testing the learned policy disp('Optimal Path:'); state = startState; path = state; while ~isequal(state, goalState) [~, action] = max(Q(state(1), state(2), :)); % Choose best action nextRow = state(1) + actions(action, 1) - wind(state(2)); nextCol = state(2) + actions(action, 2); % Ensure the next state is within bounds nextRow = max(1, min(nextRow, gridSize(1))); nextCol = max(1, min(nextCol, gridSize(2))); state = [nextRow, nextCol]; path = [path; state]; end disp(path);
Explanation of the Code
-
Grid and Wind Setup:
- The grid is defined as a 7x10 matrix.
- The
windarray specifies how much the agent is pushed upward in each column.
-
Q-Learning Algorithm:
- The Q-table is initialized to zeros.
- For each episode, the agent starts at the
startStateand follows an epsilon-greedy policy to balance exploration and exploitation. - The Q-values are updated using the Bellman equation:
-
Wind Effect:
- After taking an action, the agent is pushed upward by the wind corresponding to its current column.
-
Testing the Policy:
- After training, the agent follows the learned policy to find the optimal path from the start state to the goal state.
Output
The output will display the sequence of states (coordinates) that the agent follows to reach the goal. For example:
Optimal Path: 4 1 3 1 2 1 ... 4 8This represents the trajectory taken by the agent from the start to the goal.
Notes
- You can adjust the hyperparameters (
alpha,gamma,epsilon,numEpisodes) to experiment with different learning behaviors. - The
windarray can be modified to simulate different environments. - If you want to visualize the grid or the path, you can use MATLAB's plotting functions.
Let me know if you need further clarification or enhancements!
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.