how to create and visualize agent observation in unity in ml-agents? i have 2d top-down…
- User
how to create and visualize agent observation in unity in ml-agents? i have 2d top-down view arena game like brawl stars. a top-down game, the world has a strict tile structure on a grid, but players and power-ups can be placed outside of this grid. the player sees the world from above, within the radius of his player, approximately 26x17. he can see beyond the wall where his character can't shoot because the bullet will hit the wall. but there are also characters that can throw grenades, so walls are not a problem for them. I repeat once again, the player sees everything on the screen, even behind walls, because the view is from above. players are not tied to the grid, so you need to accurately track their position. offer me an observation.
I also need to use the checkbox to see the observation visualization even outside of training.
how i have this ShootingAgent.cs, modify it:
using UnityEngine; using Unity.MLAgents; using Unity.MLAgents.Sensors; using Unity.MLAgents.Actuators;
public class ShootingAgent : Agent { public int observationResolution = 10; // The desired low-resolution observation space size public PlayerController playerController; private Texture2D observationTexture;
public override void OnEpisodeBegin() { // Reset the player and environment for a new episode // … playerController.InitializePlayer(); } public override void CollectObservations(VectorSensor sensor) { // Capture the game view and convert it to low-resolution observation RenderTexture renderTexture = RenderTexture.GetTemporary(observationResolution, observationResolution); playerController.RenderToTexture(renderTexture); RenderTexture.active = renderTexture; observationTexture = new Texture2D(observationResolution, observationResolution, TextureFormat.RGB24, false); observationTexture.ReadPixels(new Rect(0, 0, observationResolution, observationResolution), 0, 0); observationTexture.Apply(); Color[] pixels = observationTexture.GetPixels(); for (int i = 0; i < pixels.Length; i++) { sensor.AddObservation(pixels[i].r); sensor.AddObservation(pixels[i].g); sensor.AddObservation(pixels[i].b); } RenderTexture.active = null; RenderTexture.ReleaseTemporary(renderTexture); // Add additional observations if needed (e.g., player’s position, bullets count, etc.) playerController.CollectObservations(sensor); } public override void OnActionReceived(ActionBuffers actions) { // Get the joystick control values from the agent’s actions float horizontal = actions.ContinuousActions[0]; float vertical = actions.ContinuousActions[1]; Debug.Log(horizontal); Debug.Log(vertical); // Use the joystick control values to control the game playerController.ControlJoystick(horizontal, vertical); // Set a reward based on the game state // … } public override void Heuristic(in ActionBuffers actionsOut) { // Implement the Heuristic method for manual control (useful for debugging) ActionSegment<float> continuousActions = actionsOut.ContinuousActions; continuousActions[0] = Input.GetAxis("Horizontal"); continuousActions[1] = Input.GetAxis("Vertical"); }}
- Assistant
Here's the modified
ShootingAgent.csscript, with visualization:- Add a public bool field in the
ShootingAgentclass to enable/disable observation visualization:
public bool visualizeObservations = true;- Modify the
CollectObservationsmethod to generate the observation visualization texture:
public override void CollectObservations(VectorSensor sensor) { // Capture the game view and convert it to low-resolution observation RenderTexture renderTexture = RenderTexture.GetTemporary(observationResolution, observationResolution); playerController.RenderToTexture(renderTexture); RenderTexture.active = renderTexture; observationTexture = new Texture2D(observationResolution, observationResolution, TextureFormat.RGB24, false); observationTexture.ReadPixels(new Rect(0, 0, observationResolution, observationResolution), 0, 0); observationTexture.Apply(); Color[] pixels = observationTexture.GetPixels(); for (int i = 0; i < pixels.Length; i++) { sensor.AddObservation(pixels[i].r); sensor.AddObservation(pixels[i].g); sensor.AddObservation(pixels[i].b); } // Add additional observations if needed (e.g., player's position, bullets count, etc.) playerController.CollectObservations(sensor); // Visualize observations if the checkbox is enabled if (visualizeObservations) { VisualizeObservations(observationTexture); } RenderTexture.active = null; RenderTexture.ReleaseTemporary(renderTexture); }- Add a new method
VisualizeObservationsthat generates a visualization using aGameObjectwithRawImagecomponent in the scene. In the Unity editor, create a new UIRawImageobject and assign it to therawImagefield using the Inspector.
using UnityEngine.UI; // Add this line at the beginning of the script public RawImage rawImage; // Add this field to the ShootingAgent class private void VisualizeObservations(Texture2D texture) { if (rawImage != null) { rawImage.texture = texture; } }With this new implementation, you can enable/disable the observation visualization by toggling the
visualizeObservationscheckbox in the Inspector. The observation visualization texture will be displayed on the assignedRawImageUI element.Remember to also add the necessary UI components and game objects (such as the
Canvas,RawImage, cameras, etc.) to your Unity scene according to your game-specific requirements. - Add a public bool field in the