Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Given this data structure: class PPOMemory: def init(self, numsteps, numenvs,…

  1. User

    Given this data structure:

    class PPOMemory: def init(self, num_steps, num_envs, state_shape, batch_size): self.states = np.zeros((num_steps, num_envs, *state_shape), dtype=np.float32) self.actions = np.zeros((num_steps, num_envs), dtype=np.int32) self.probs = np.zeros((num_steps, num_envs), dtype=np.float32) self.vals = np.zeros((num_steps, num_envs), dtype=np.float32) self.rewards = np.zeros((num_steps, num_envs), dtype=np.float32) self.dones = np.zeros((num_steps, num_envs), dtype=bool) self.batch_size = batch_size

    I need a new array for each one, which has first dimension num_steps*num_envs and elements are sorted by the num_envs dimension

  2. Assistant

    To rearrange your data so that each element is grouped by the num_envs dimension when flattened, you can swap the first two axes of your arrays and then reshape them. This effectively orders the elements so that all steps for a particular environment are contiguous in the flattened array.

    Here's how you can do it for each array in your PPOMemory class:

    # Swap axes 0 and 1, so shape becomes (num_envs, num_steps, ...)
    # Then reshape to combine the first two dimensions
    states_reordered = self.states.swapaxes(0, 1).reshape(-1, *self.states.shape[2:])
    actions_reordered = self.actions.swapaxes(0, 1).reshape(-1, *self.actions.shape[2:])
    probs_reordered = self.probs.swapaxes(0, 1).reshape(-1, *self.probs.shape[2:])
    vals_reordered = self.vals.swapaxes(0, 1).reshape(-1, *self.vals.shape[2:])
    rewards_reordered = self.rewards.swapaxes(0, 1).reshape(-1, *self.rewards.shape[2:])
    dones_reordered = self.dones.swapaxes(0, 1).reshape(-1, *self.dones.shape[2:])
    

    This code does the following for each array:

    1. Swaps the first two axes (swapaxes(0, 1)) to bring the num_envs dimension to the first axis.
    2. Reshapes the array to combine the first two dimensions (reshape(-1, ...)), which flattens the num_envs and num_steps dimensions into one.

    As a result, the first dimension of each new array is num_steps * num_envs, and the elements are ordered so that all steps for each environment are grouped together.

    Example:

    If you have num_steps = 3 and num_envs = 2, the original ordering in self.states would be:

    [
      [state_0_env_0, state_0_env_1],
      [state_1_env_0, state_1_env_1],
      [state_2_env_0, state_2_env_1]
    ]
    

    After reordering:

    [
      state_0_env_0,
      state_1_env_0,
      state_2_env_0,
      state_0_env_1,
      state_1_env_1,
      state_2_env_1
    ]
    

    This groups all the steps of env_0 together, followed by all the steps of env_1.

    Final Answer:

    Swap the first two axes of each array and then reshape to flatten, so elements are grouped by num_envs:

    reordered_array = original_array.swapaxes(0, 1).reshape(-1, ...)