Skip to content
TheChatArchiveThe Chat Archive
o3-mini· OpenAIHugging Face LMArena Human Preference 140K

create a python project the gets the inventory of all cloudwatch log groups with a given…

  1. User

    create a python project the gets the inventory of all cloudwatch log groups with a given config yaml of accounts and regions. It should execute from a jenkinsfile which passes the credentials to python using WithCredentials. the config is loaded in Jenkinsfile and passed to python which loops through the config

  2. Assistant

    Below is an example solution. In our example, the repository contains a Python script (e.g. main.py), a YAML configuration file (e.g. config.yaml), and a Jenkinsfile that loads the config and passes it—along with AWS credentials obtained via Jenkins’ withCredentials step—to the Python script. (In this example we assume a single set of AWS credentials is used across the accounts in the config. If you need to use different credentials per account, you’d have to modify the approach accordingly.)


    Project Structure

    .
    ├── Jenkinsfile
    ├── config.yaml
    ├── main.py
    └── requirements.txt
    

    1. config.yaml

    This file defines a list of accounts and, for each, the regions where you want to collect CloudWatch log groups:

    accounts:
      - name: "account-1"
        regions:
          - "us-east-1"
          - "us-west-2"
      - name: "account-2"
        regions:
          - "eu-west-1"
    

    Note: Here, the “name” is informational. In a real-world scenario you might want to include additional data such as role ARN to assume (if you need to switch credentials for each account).


    2. main.py

    This Python script does the following:

    • Reads the YAML configuration passed as an argument.
    • For each account/region, it creates a boto3 session using the AWS credentials available on the environment (which are injected by Jenkins).
    • Uses the CloudWatch Logs client to list log groups (using pagination).
    • Prints the collected inventory.

    Make sure to install the boto3 and pyyaml packages (see requirements.txt below).

    #!/usr/bin/env python3
    import os
    import sys
    import boto3
    import yaml
    
    def get_log_groups(client):
        """Return all CloudWatch Log Groups using the describe_log_groups paginator."""
        paginator = client.get_paginator('describe_log_groups')
        log_groups = []
        for page in paginator.paginate():
            log_groups.extend(page.get('logGroups', []))
        return log_groups
    
    def process_region(region):
        """Create a boto3 client for CloudWatch Logs in the given region."""
        # AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
        # are expected to be in the environment (provided by Jenkins withCredentials).
        session = boto3.Session(
            aws_access_key_id=os.environ.get("AWS_ACCESS_KEY_ID"),
            aws_secret_access_key=os.environ.get("AWS_SECRET_ACCESS_KEY"),
            aws_session_token=os.environ.get("AWS_SESSION_TOKEN"),
            region_name=region
        )
        client = session.client('logs')
        return get_log_groups(client)
    
    def main(config_yaml_string):
        try:
            config = yaml.safe_load(config_yaml_string)
        except Exception as e:
            print(f"Error loading YAML config: {e}")
            sys.exit(1)
    
        results = {}
        for account in config.get("accounts", []):
            account_name = account.get("name", "unknown-account")
            regions = account.get("regions", [])
            account_results = {}
            print(f"\nProcessing account: {account_name}")
    
            for region in regions:
                print(f"  - Querying region: {region}")
                try:
                    log_groups = process_region(region)
                    account_results[region] = log_groups
                    print(f"    Found {len(log_groups)} log group(s)")
                except Exception as err:
                    print(f"    Error processing region {region}: {err}")
                    account_results[region] = f"Error: {err}"
    
            results[account_name] = account_results
    
        print("\nInventory Results:")
        print(yaml.dump(results, default_flow_style=False))
    
    if __name__ == '__main__':
        if len(sys.argv) != 2:
            print("Usage: python main.py '<yaml_config>'")
            sys.exit(1)
        # The YAML configuration is given as a string argument.
        yaml_config_string = sys.argv[1]
        main(yaml_config_string)
    

    3. requirements.txt

    List the required dependencies:

    boto3
    PyYAML
    

    4. Jenkinsfile

    Below is an example Declarative Pipeline that:

    • Checks out the source.
    • Reads the YAML configuration from the repository.
    • Uses Jenkins’ withCredentials to inject AWS credentials into the environment.
    • Executes the Python script, passing the YAML configuration as a command‐line argument.

    Note: In this example we assume you’ve set up an AWS credential in Jenkins (with id "aws-credentials-id") that supplies the AWS access key, secret key, and (optionally) session token.

    pipeline {
        agent any
    
        stages {
            stage('Checkout') {
                steps {
                    checkout scm
                }
            }
            stage('Run CloudWatch Log Groups Inventory') {
                steps {
                    // Reads the content of config.yaml into a Groovy variable.
                    script {
                        def configContent = readFile(file: 'config.yaml').trim()
                        // Escape single quotes (if any) for safe use in the shell command.
                        configContent = configContent.replace("'", "'\\''")
                        // Execute the Python script with AWS credentials injected.
                        withCredentials([
                            [
                                $class: 'AmazonWebServicesCredentialsBinding',
                                credentialsId: 'aws-credentials-id', // update with your credentials id
                                accessKeyVariable: 'AWS_ACCESS_KEY_ID',
                                secretKeyVariable: 'AWS_SECRET_ACCESS_KEY',
                                sessionTokenVariable: 'AWS_SESSION_TOKEN'
                            ]
                        ]) {
                            // Call Python passing the YAML config as a single-quoted string.
                            sh "python3 main.py '${configContent}'"
                        }
                    }
                }
            }
        }
    }
    

    Usage

    1. Set up your Jenkins credentials:
      Create an AWS credentials entry in Jenkins (with id aws-credentials-id) that contains your AWS access key, secret key, and optional session token.

    2. Install Dependencies:
      Make sure that the Python dependencies (listed in requirements.txt) are installed. For example:

      pip install -r requirements.txt
      
    3. Run from Jenkins:
      When you trigger the Jenkins job (which uses the Jenkinsfile from your repository), it will:

      • Check out the repository.
      • Read the config.yaml file.
      • Inject AWS credentials.
      • Run the Python script that connects to CloudWatch Logs and prints the inventory of log groups for each account/region.

    This sample project demonstrates how to:

    • Pass external configuration (loaded by Jenkins) into Python.
    • Use Jenkins’ withCredentials to securely pass AWS credentials.
    • Loop through multiple regions (and accounts) to gather CloudWatch logs inventory using boto3.

    You can further expand this project by adding error handling, output formatting, or extending the configuration to support more advanced use cases.