Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Could you write a code in GML that automatically detects enclosed/tight spaces and…

  1. User

    Could you write a code in GML that automatically detects enclosed/tight spaces and paints the surface (not the entire object) darker or lighter in a gradient based on that?

  2. Assistant

    Certainly! Creating an effect that detects enclosed or tight spaces within a sprite and applies a gradient shading based on those detections can add depth and visual interest to your game objects. While this task involves complex image processing, it can be efficiently achieved in GameMaker Studio using a combination of GML (GameMaker Language) and GLSL shaders.

    Below is a comprehensive guide to implementing this effect:

    1. Understanding the Approach
    2. Setting Up Your Project
    3. Creating the Vertex and Fragment Shaders
    4. Implementing the GML Code
    5. Final Touches and Optimization

    1. Understanding the Approach

    Objective: Automatically detect enclosed/tight spaces within a sprite and apply a dynamic gradient shading to the surface based on the proximity to these spaces.

    Methodology:

    • Edge Detection: Identify the edges within the sprite where tight spaces occur. This can be achieved using edge detection algorithms like the Sobel operator.

    • Gradient Shading: Once edges are detected, apply a gradient that darkens or lightens areas based on their distance from these edges, enhancing the perception of depth and enclosure.

    Why Use Shaders? While it's possible to perform pixel-level manipulation in GML, it's computationally intensive and not suitable for real-time applications. Shaders utilize the GPU, allowing for efficient parallel processing of pixels, making them ideal for tasks like edge detection and gradient shading.


    2. Setting Up Your Project

    Before diving into the code, ensure you have the following setup:

    1. Sprite Preparation: Ensure your sprite has sufficient contrast and clear edges for effective edge detection.

    2. Shader Resources: You'll need to create two shaders—a vertex shader and a fragment shader.

    3. Object to Apply Shader: Create or select an object in your game to which you want to apply this effect.


    3. Creating the Vertex and Fragment Shaders

    a. Vertex Shader

    The vertex shader processes each vertex of your sprite. For this task, a basic vertex shader suffices.

    File: vertex_shader.glsl

    // Vertex Shader
    attribute vec3 in_Position;
    attribute vec4 in_Colour;
    attribute vec2 in_TextureCoord;
    
    varying vec2 v_vTexcoord;
    varying vec4 v_vColour;
    
    void main()
    {
        v_vTexcoord = in_TextureCoord;
        v_vColour = in_Colour;
        gl_Position = vec4(in_Position.xy, 0.0, 1.0);
    }
    

    b. Fragment Shader

    The fragment shader handles the pixel-level operations, including edge detection and gradient shading.

    File: fragment_shader.glsl

    // Fragment Shader
    varying vec2 v_vTexcoord;
    varying vec4 v_vColour;
    
    uniform sampler2D u_sampler;
    
    // Parameters for edge detection
    uniform float u_resolution_x;
    uniform float u_resolution_y;
    
    void main()
    {
        // Texture coordinates
        vec2 texCoord = v_vTexcoord;
    
        // Define pixel size based on resolution
        vec2 pixel = vec2(1.0 / u_resolution_x, 1.0 / u_resolution_y);
    
        // Sobel operator kernels
        float Gx = 0.0;
        float Gy = 0.0;
    
        // Sample surrounding pixels
        for(int i = -1; i <= 1; i++) {
            for(int j = -1; j <=1; j++) {
                vec2 offset = vec2(float(i), float(j)) * pixel;
                float gray = texture2D(u_sampler, texCoord + offset).r; // Assuming grayscale
    
                Gx += gray * float(j) * (i == 0 ? 1.0 : 2.0);
                Gy += gray * float(i) * (j == 0 ? 1.0 : 2.0);
            }
        }
    
        // Calculate gradient magnitude
        float edge = length(vec2(Gx, Gy));
    
        // Normalize edge value
        edge = clamp(edge * 4.0, 0.0, 1.0); // Adjust multiplier as needed
    
        // Apply gradient shading based on edge proximity
        // The closer to the edge, the darker the color
        vec3 originalColor = texture2D(u_sampler, texCoord).rgb;
        vec3 shadedColor = originalColor * (1.0 - edge * 0.5); // Adjust 0.5 for shading intensity
    
        gl_FragColor = vec4(shadedColor, texture2D(u_sampler, texCoord).a);
    }
    

    Explanation:

    • Sobel Operator: The fragment shader uses the Sobel operator to detect edges by calculating the gradient magnitude (Gx and Gy) at each pixel.

    • Gradient Shading: After detecting edges, it darkens the original color based on the proximity to these edges. The edge value determines how much to darken the pixel, creating a gradient effect.

    • Parameters:

      • u_resolution_x and u_resolution_y help calculate the size of each pixel based on the sprite's resolution.

    4. Implementing the GML Code

    Now, integrate the shaders with your GameMaker object using GML.

    a. Loading the Shaders

    Create Event of Your Object:

    // Create Event
    
    // Load shaders
    vertex_shader = shader_get_builtin_vertex(shader_flat); // Using built-in flat vertex shader
    fragment_shader = shader_create("
    // Fragment Shader Code goes here
    \n
    varying vec2 v_vTexcoord;
    varying vec4 v_vColour;
    
    uniform sampler2D u_sampler;
    
    // Parameters for edge detection
    uniform float u_resolution_x;
    uniform float u_resolution_y;
    
    void main()
    {
        // Texture coordinates
        vec2 texCoord = v_vTexcoord;
    
        // Define pixel size based on resolution
        vec2 pixel = vec2(1.0 / u_resolution_x, 1.0 / u_resolution_y);
    
        // Sobel operator kernels
        float Gx = 0.0;
        float Gy = 0.0;
    
        // Sample surrounding pixels
        for(int i = -1; i <= 1; i++) {
            for(int j = -1; j <=1; j++) {
                vec2 offset = vec2(float(i), float(j)) * pixel;
                float gray = texture2D(u_sampler, texCoord + offset).r; // Assuming grayscale
    
                Gx += gray * float(j) * (i == 0 ? 1.0 : 2.0);
                Gy += gray * float(i) * (j == 0 ? 1.0 : 2.0);
            }
        }
    
        // Calculate gradient magnitude
        float edge = length(vec2(Gx, Gy));
    
        // Normalize edge value
        edge = clamp(edge * 4.0, 0.0, 1.0); // Adjust multiplier as needed
    
        // Apply gradient shading based on edge proximity
        // The closer to the edge, the darker the color
        vec3 originalColor = texture2D(u_sampler, texCoord).rgb;
        vec3 shadedColor = originalColor * (1.0 - edge * 0.5); // Adjust 0.5 for shading intensity
    
        gl_FragColor = vec4(shadedColor, texture2D(u_sampler, texCoord).a);
    }
    ");
    
    // Check for shader compilation errors
    if (fragment_shader == -1) {
        show_error("Failed to compile fragment shader.", true);
    }
    
    // Set shader parameters
    shader_set(fragment_shader);
    shader_set_uniform_f(shader_get_uniform(fragment_shader, "u_resolution_x"), sprite_width);
    shader_set_uniform_f(shader_get_uniform(fragment_shader, "u_resolution_y"), sprite_height);
    shader_reset();
    

    Explanation:

    • Shader Loading: The vertex shader uses GameMaker’s built-in flat vertex shader. The fragment shader is created using shader_create with the GLSL code provided earlier.

    • Error Handling: It’s crucial to check if the shader compiled successfully to avoid runtime errors.

    • Uniforms: The shader requires the resolution of the sprite to calculate pixel offsets accurately. These are set using shader_set_uniform_f.

    b. Drawing the Sprite with Shader

    Draw Event of Your Object:

    // Draw Event
    
    // Set the shader
    shader_set(fragment_shader);
    
    // Set shader parameters (in case sprite size changes)
    shader_set_uniform_f(shader_get_uniform(fragment_shader, "u_resolution_x"), sprite_width);
    shader_set_uniform_f(shader_get_uniform(fragment_shader, "u_resolution_y"), sprite_height);
    
    // Draw the sprite
    draw_self();
    
    // Reset the shader to default
    shader_reset();
    

    Explanation:

    • Applying the Shader: Before drawing the sprite, the shader is set, and the required uniforms are updated to match the current sprite dimensions.

    • Drawing the Sprite: draw_self() draws the sprite with the shader effect applied.

    • Resetting the Shader: It’s good practice to reset the shader after drawing to prevent affecting other objects.

    c. Cleaning Up

    Cleanup Event of Your Object:

    // Cleanup Event
    
    // Free shaders to prevent memory leaks
    if (shader_exists(fragment_shader)) {
        shader_delete(fragment_shader);
    }
    if (shader_exists(vertex_shader)) {
        shader_delete(vertex_shader);
    }
    

    Explanation:

    • Memory Management: Shaders consume GPU resources. Always ensure they are deleted when no longer needed to prevent memory leaks.

    5. Final Touches and Optimization

    a. Handling Colored Sprites

    The provided shader assumes the sprite is in grayscale for edge detection. If your sprite is colored, you might need to convert it to grayscale within the shader before applying the Sobel operator.

    Modify Fragment Shader for Colored Sprites:

    // Convert to grayscale
    float gray = dot(originalColor, vec3(0.299, 0.587, 0.114));
    

    Replace the grayscale sampling in the Sobel operator with the above conversion.

    b. Adjusting Sensitivity

    • Edge Sensitivity: Tweak the multiplier in edge = clamp(edge * 4.0, 0.0, 1.0); to make edge detection more or less sensitive.

    • Shading Intensity: Adjust the value in shadedColor = originalColor * (1.0 - edge * 0.5); to increase or decrease the shading effect.

    c. Performance Considerations

    • Shader Optimization: Complex shaders can impact performance, especially on lower-end devices. Optimize by minimizing calculations within loops and using precision qualifiers where appropriate.

    • Sprite Size: Larger sprites require more processing in shaders. Ensure sprites are optimized in size and resolution based on game requirements.


    Complete Example

    Here's a consolidated example incorporating all the elements discussed above.

    Vertex Shader (vertex_shader.glsl):

    // Vertex Shader
    attribute vec3 in_Position;
    attribute vec4 in_Colour;
    attribute vec2 in_TextureCoord;
    
    varying vec2 v_vTexcoord;
    varying vec4 v_vColour;
    
    void main()
    {
        v_vTexcoord = in_TextureCoord;
        v_vColour = in_Colour;
        gl_Position = vec4(in_Position.xy, 0.0, 1.0);
    }
    

    Fragment Shader (fragment_shader.glsl):

    // Fragment Shader
    varying vec2 v_vTexcoord;
    varying vec4 v_vColour;
    
    uniform sampler2D u_sampler;
    
    // Parameters for edge detection
    uniform float u_resolution_x;
    uniform float u_resolution_y;
    
    void main()
    {
        // Texture coordinates
        vec2 texCoord = v_vTexcoord;
    
        // Define pixel size based on resolution
        vec2 pixel = vec2(1.0 / u_resolution_x, 1.0 / u_resolution_y);
    
        // Sobel operator kernels
        float Gx = 0.0;
        float Gy = 0.0;
    
        // Sample surrounding pixels
        for(int i = -1; i <= 1; i++) {
            for(int j = -1; j <=1; j++) {
                vec2 offset = vec2(float(i), float(j)) * pixel;
                vec4 sampleColor = texture2D(u_sampler, texCoord + offset);
                float gray = dot(sampleColor.rgb, vec3(0.299, 0.587, 0.114)); // Grayscale conversion
    
                // Sobel Gx
                if (i == -1 && j == -1) { Gx += gray * -1.0; }
                if (i == -1 && j ==  0) { Gx += gray *  0.0; }
                if (i == -1 && j ==  1) { Gx += gray *  1.0; }
    
                if (i ==  0 && j == -1) { Gx += gray * -2.0; }
                if (i ==  0 && j ==  0) { Gx += gray *  0.0; }
                if (i ==  0 && j ==  1) { Gx += gray *  2.0; }
    
                if (i ==  1 && j == -1) { Gx += gray * -1.0; }
                if (i ==  1 && j ==  0) { Gx += gray *  0.0; }
                if (i ==  1 && j ==  1) { Gx += gray *  1.0; }
    
                // Sobel Gy
                if (i == -1 && j == -1) { Gy += gray * -1.0; }
                if (i == -1 && j ==  0) { Gy += gray * -2.0; }
                if (i == -1 && j ==  1) { Gy += gray * -1.0; }
    
                if (i ==  0 && j == -1) { Gy += gray *  0.0; }
                if (i ==  0 && j ==  0) { Gy += gray *  0.0; }
                if (i ==  0 && j ==  1) { Gy += gray *  0.0; }
    
                if (i ==  1 && j == -1) { Gy += gray *  1.0; }
                if (i ==  1 && j ==  0) { Gy += gray *  2.0; }
                if (i ==  1 && j ==  1) { Gy += gray *  1.0; }
            }
        }
    
        // Calculate gradient magnitude
        float edge = length(vec2(Gx, Gy));
    
        // Normalize edge value
        edge = clamp(edge * 0.2, 0.0, 1.0); // Adjust multiplier as needed
    
        // Apply gradient shading based on edge proximity
        vec3 originalColor = texture2D(u_sampler, texCoord).rgb;
        vec3 shadedColor = originalColor * (1.0 - edge * 0.5); // Adjust 0.5 for shading intensity
    
        gl_FragColor = vec4(shadedColor, texture2D(u_sampler, texCoord).a);
    }
    

    GML Code for the Object:

    // Create Event
    
    // Load shaders
    vertex_shader = shader_get_builtin_vertex(shader_flat); // Using GameMaker's built-in flat vertex shader
    
    // Read the fragment shader code from file or define it as a string
    // For illustration, we're defining it as a string. In practice, load it from a file.
    var fragment_code = "
    varying vec2 v_vTexcoord;
    varying vec4 v_vColour;
    
    uniform sampler2D u_sampler;
    
    // Parameters for edge detection
    uniform float u_resolution_x;
    uniform float u_resolution_y;
    
    void main()
    {
        // Texture coordinates
        vec2 texCoord = v_vTexcoord;
    
        // Define pixel size based on resolution
        vec2 pixel = vec2(1.0 / u_resolution_x, 1.0 / u_resolution_y);
    
        // Sobel operator kernels
        float Gx = 0.0;
        float Gy = 0.0;
    
        // Sample surrounding pixels
        for(int i = -1; i <= 1; i++) {
            for(int j = -1; j <=1; j++) {
                vec2 offset = vec2(float(i), float(j)) * pixel;
                vec4 sampleColor = texture2D(u_sampler, texCoord + offset);
                float gray = dot(sampleColor.rgb, vec3(0.299, 0.587, 0.114)); // Grayscale conversion
    
                // Sobel Gx
                if (i == -1 && j == -1) { Gx += gray * -1.0; }
                if (i == -1 && j ==  0) { Gx += gray *  0.0; }
                if (i == -1 && j ==  1) { Gx += gray *  1.0; }
    
                if (i ==  0 && j == -1) { Gx += gray * -2.0; }
                if (i ==  0 && j ==  0) { Gx += gray *  0.0; }
                if (i ==  0 && j ==  1) { Gx += gray *  2.0; }
    
                if (i ==  1 && j == -1) { Gx += gray * -1.0; }
                if (i ==  1 && j ==  0) { Gx += gray *  0.0; }
                if (i ==  1 && j ==  1) { Gx += gray *  1.0; }
    
                // Sobel Gy
                if (i == -1 && j == -1) { Gy += gray * -1.0; }
                if (i == -1 && j ==  0) { Gy += gray * -2.0; }
                if (i == -1 && j ==  1) { Gy += gray * -1.0; }
    
                if (i ==  0 && j == -1) { Gy += gray *  0.0; }
                if (i ==  0 && j ==  0) { Gy += gray *  0.0; }
                if (i ==  0 && j ==  1) { Gy += gray *  0.0; }
    
                if (i ==  1 && j == -1) { Gy += gray *  1.0; }
                if (i ==  1 && j ==  0) { Gy += gray *  2.0; }
                if (i ==  1 && j ==  1) { Gy += gray *  1.0; }
            }
        }
    
        // Calculate gradient magnitude
        float edge = length(vec2(Gx, Gy));
    
        // Normalize edge value
        edge = clamp(edge * 0.2, 0.0, 1.0); // Adjust multiplier as needed
    
        // Apply gradient shading based on edge proximity
        vec3 originalColor = texture2D(u_sampler, texCoord).rgb;
        vec3 shadedColor = originalColor * (1.0 - edge * 0.5); // Adjust 0.5 for shading intensity
    
        gl_FragColor = vec4(shadedColor, texture2D(u_sampler, texCoord).a);
    }
    ";
    
    // Create the fragment shader
    fragment_shader = shader_create(vertex_shader, fragment_code);
    
    // Check if shader compiled successfully
    if (fragment_shader == -1) {
        show_error("Failed to compile fragment shader.", true);
    }
    
    // Set initial shader parameters
    shader_set(fragment_shader);
    shader_set_uniform_f(shader_get_uniform(fragment_shader, "u_resolution_x"), sprite_width);
    shader_set_uniform_f(shader_get_uniform(fragment_shader, "u_resolution_y"), sprite_height);
    shader_reset();
    

    Draw Event:

    // Draw Event
    
    // Set the shader
    shader_set(fragment_shader);
    
    // Update shader uniforms in case of sprite scaling or resizing
    shader_set_uniform_f(shader_get_uniform(fragment_shader, "u_resolution_x"), sprite_width);
    shader_set_uniform_f(shader_get_uniform(fragment_shader, "u_resolution_y"), sprite_height);
    
    // Draw the sprite with shader applied
    draw_self();
    
    // Reset shader to default to avoid affecting other draw calls
    shader_reset();
    

    Cleanup Event:

    // Cleanup Event
    
    // Delete shaders to free resources
    if (shader_exists(fragment_shader)) {
        shader_delete(fragment_shader);
    }
    if (shader_exists(vertex_shader)) {
        shader_delete(vertex_shader);
    }
    

    Conclusion

    By leveraging GLSL shaders within GameMaker Studio, you can efficiently detect enclosed or tight spaces within sprites and apply dynamic gradient shading based on these detections. This method ensures real-time performance and allows for flexible adjustments to the shading effect.

    Key Takeaways:

    • Shaders Over GML for Pixel Operations: For tasks involving per-pixel manipulation like edge detection and gradient shading, shaders are the optimal choice due to their performance benefits.

    • Flexibility: Shaders can be adjusted to tweak the sensitivity of edge detection and the intensity of shading, allowing for customizable visual effects.

    • Resource Management: Always manage shader resources carefully to prevent memory leaks, especially when dynamically creating and deleting shaders.

    Feel free to customize the shaders further to match the specific aesthetic requirements of your game. Happy coding!