Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

I am trying to add the ability to sign and verify images in RKE2 with Cosign in the…

  1. User

    I am trying to add the ability to sign and verify images in RKE2 with Cosign in the simplest way possible. This is in an offline environment so it doesn't typically have direct access to the internet (although I can enable an HTTPS_PROXY in my terminal to enable this temporarily) Please take me through the necessary steps. Note that this is running on RHEL 8, and should use sigstore PolicyController

  2. Assistant

    To add image signing and verification to your RKE2 cluster using Cosign and the Sigstore PolicyController on RHEL 8 in an offline environment, you'll need to perform several steps. Below is a comprehensive guide to help you through the process.

    Overview of Steps:

    1. Install Cosign on RHEL 8
    2. Generate Signing Keys with Cosign
    3. Sign Container Images with Cosign
    4. Set Up a Local Container Image Registry
    5. Load Signed Images into the Local Registry
    6. Install Sigstore Cosigned (PolicyController) in Your RKE2 Cluster
    7. Configure Cosigned Policies for Image Verification
    8. Test Image Deployment and Verification

    Prerequisites:

    • RHEL 8 Server with RKE2 installed.
    • Access to the RKE2 Cluster (kubectl configured and pointing to your cluster).
    • Internet Access via HTTPS_PROXY (temporarily) for downloading necessary tools and images.
    • Air-Gapped Environment Considerations: Since you're in an offline environment, you'll need to download necessary resources and transfer them to your offline environment.

    Step 1: Install Cosign on RHEL 8

    Cosign is a tool for signing and verifying container images. You'll need to install it on your RHEL 8 machine.

    Steps:

    1. Set Up HTTPS_PROXY (if needed):

      If your terminal doesn't have internet access, temporarily set up the HTTPS_PROXY environment variable.

      export HTTPS_PROXY="http://your-proxy-server:port"
      
    2. Download the Latest Cosign Binary:

      Visit the Cosign GitHub Releases page to find the latest version. Replace vX.Y.Z with the latest version number.

      VERSION="vX.Y.Z"
      wget https://github.com/sigstore/cosign/releases/download/${VERSION}/cosign-linux-amd64
      
    3. Install Cosign:

      chmod +x cosign-linux-amd64
      sudo mv cosign-linux-amd64 /usr/local/bin/cosign
      
    4. Verify Installation:

      cosign version
      

    Step 2: Generate Signing Keys with Cosign

    In an offline environment, you'll need to use key-pair signing (not keyless).

    Steps:

    1. Generate Key Pair:

      cosign generate-key-pair
      

      This command will generate two files in your current directory:

      • cosign.key (private key)
      • cosign.pub (public key)
    2. Secure the Keys:

      • Private Key (cosign.key): Keep this secure and don't share it.
      • Public Key (cosign.pub): This will be shared with Cosigned for verification.

    Step 3: Sign Container Images with Cosign

    You'll need to sign your container images before they are deployed.

    Steps:

    1. Load or Build Your Container Image:

      Ensure that your image is built and tagged appropriately.

      docker build -t myregistry.local/myimage:tag .
      
    2. Push Image to Local Registry (we'll set this up in the next step).


    Step 4: Set Up a Local Container Image Registry

    Since you are in an offline environment, you need a local registry to host your images and their signatures.

    Steps:

    1. Deploy a Local Registry:

      docker run -d -p 5000:5000 --name registry registry:2
      
    2. Tag and Push Your Image to the Local Registry:

      docker tag myregistry.local/myimage:tag localhost:5000/myimage:tag
      docker push localhost:5000/myimage:tag
      

      Note: Ensure that your RKE2 cluster nodes can access this registry (localhost:5000). You may need to adjust the registry URL or make it accessible over your network.

    3. Configure RKE2 to Trust the Insecure Registry (if needed):

      Edit the RKE2 configuration to allow insecure registries.

      sudo vi /etc/rancher/rke2/registries.yaml
      

      Add:

      mirrors:
        "localhost:5000":
          endpoint:
            - "http://localhost:5000"
      

      Restart RKE2:

      sudo systemctl restart rke2-server
      

    Step 5: Sign the Images with Cosign

    Now, sign the image that you've pushed to the local registry.

    Steps:

    1. Sign the Image:

      cosign sign -key cosign.key localhost:5000/myimage:tag
      

      This command attaches the signature to the image in the registry.


    Step 6: Install Sigstore Cosigned (PolicyController) in Your RKE2 Cluster

    Cosigned is the admission controller that enforces image signature verification in your cluster.

    Steps:

    1. Download Cosigned Deployment Manifests:

      Since you are offline, you need to get the deployment manifests and images ahead of time.

      • Download the manifests from the Sigstore Cosign GitHub.

        wget https://raw.githubusercontent.com/sigstore/cosign/main/install/cosigned.yaml
        
    2. Download Cosigned Image:

      On a machine with internet access, pull the cosigned image.

      docker pull ghcr.io/sigstore/cosigned:<version>
      

      Replace <version> with the appropriate version tag.

    3. Save the Image to a Tar File:

      docker save ghcr.io/sigstore/cosigned:<version> -o cosigned.tar
      
    4. Transfer the Image and Manifests to Your Offline Environment:

      Use a USB drive or secure copy to transfer cosigned.tar and cosigned.yaml to your offline environment.

    5. Load the Cosigned Image into Your Local Registry:

      docker load -i cosigned.tar
      docker tag ghcr.io/sigstore/cosigned:<version> localhost:5000/sigstore/cosigned:<version>
      docker push localhost:5000/sigstore/cosigned:<version>
      
    6. Modify the Cosigned Deployment Manifest:

      Edit cosigned.yaml to point to your local registry.

      # In cosigned.yaml, find the Deployment and update the image:
      image: localhost:5000/sigstore/cosigned:<version>
      
    7. Deploy Cosigned to Your Cluster:

      kubectl apply -f cosigned.yaml
      
    8. Verify the Deployment:

      kubectl get pods -n cosigned-system
      

    Step 7: Configure Cosigned Policies for Image Verification

    You'll need to inform Cosigned about your public key and define policies for image verification.

    Steps:

    1. Create a Kubernetes Secret with Your Public Key:

      kubectl create namespace cosign-system
      kubectl create secret generic cosign-public-key \
        -n cosign-system \
        --from-file=cosign.pub
      
    2. Create a ClusterImagePolicy:

      Prepare a clusterimagepolicy.yaml file:

      apiVersion: policy.sigstore.dev/v1alpha1
      kind: ClusterImagePolicy
      metadata:
        name: require-cosign-signatures
      spec:
        images:
        - glob: "localhost:5000/*"
        authorities:
        - key:
            secretRef:
              name: cosign-public-key
              namespace: cosign-system
      

      This policy requires that any image from localhost:5000/* must be signed with the provided public key.

    3. Apply the Policy:

      kubectl apply -f clusterimagepolicy.yaml
      

    Step 8: Test Image Deployment and Verification

    Test whether unsigned images are blocked and signed images are allowed.

    Steps:

    1. Attempt to Deploy an Unsigned Image:

      Prepare a deployment manifest unsigned-deployment.yaml:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: unsigned-app
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: unsigned-app
        template:
          metadata:
            labels:
              app: unsigned-app
          spec:
            containers:
            - name: unsigned-app
              image: localhost:5000/unsigned-image:tag
      

      Apply the deployment:

      kubectl apply -f unsigned-deployment.yaml
      

      This should be blocked by Cosigned, and you should see an admission error.

    2. Attempt to Deploy a Signed Image:

      Prepare a deployment manifest signed-deployment.yaml:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: signed-app
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: signed-app
        template:
          metadata:
            labels:
              app: signed-app
          spec:
            containers:
            - name: signed-app
              image: localhost:5000/myimage:tag
      

      Apply the deployment:

      kubectl apply -f signed-deployment.yaml
      

      This should be allowed, and the pod should start running.

    3. Verify the Pod Status:

      kubectl get pods
      

    Additional Considerations

    • Network Accessibility:

      Ensure that all your RKE2 cluster nodes can access the local registry at localhost:5000. You might need to set up DNS or use the actual IP address.

    • Certificates and TLS for the Registry:

      For production environments, it's recommended to secure your registry with TLS certificates.

    • Updating Cosigned Policies:

      You can refine your ClusterImagePolicy to match specific images or add additional keys as needed.

    • Key Management:

      Always protect your private keys and manage them securely. Rotate keys if necessary.

    • Testing in a Non-Production Environment:

      Before rolling out to production, test this setup thoroughly in a staging environment.


    Summary

    By following these steps, you can set up image signing and verification in your RKE2 cluster using Cosign and Cosigned (PolicyController) from Sigstore, even in an offline environment. This ensures that only trusted and signed container images are deployed in your cluster, enhancing your security posture.


    If you encounter any issues during the setup, please provide specific error messages or logs, and I'll be happy to help troubleshoot further.