I am trying to add the ability to sign and verify images in RKE2 with Cosign in the…
- User
I am trying to add the ability to sign and verify images in RKE2 with Cosign in the simplest way possible. This is in an offline environment so it doesn't typically have direct access to the internet (although I can enable an HTTPS_PROXY in my terminal to enable this temporarily) Please take me through the necessary steps. Note that this is running on RHEL 8, and should use sigstore PolicyController
- Assistant
To add image signing and verification to your RKE2 cluster using Cosign and the Sigstore PolicyController on RHEL 8 in an offline environment, you'll need to perform several steps. Below is a comprehensive guide to help you through the process.
Overview of Steps:
- Install Cosign on RHEL 8
- Generate Signing Keys with Cosign
- Sign Container Images with Cosign
- Set Up a Local Container Image Registry
- Load Signed Images into the Local Registry
- Install Sigstore Cosigned (PolicyController) in Your RKE2 Cluster
- Configure Cosigned Policies for Image Verification
- Test Image Deployment and Verification
Prerequisites:
- RHEL 8 Server with RKE2 installed.
- Access to the RKE2 Cluster (
kubectlconfigured and pointing to your cluster). - Internet Access via HTTPS_PROXY (temporarily) for downloading necessary tools and images.
- Air-Gapped Environment Considerations: Since you're in an offline environment, you'll need to download necessary resources and transfer them to your offline environment.
Step 1: Install Cosign on RHEL 8
Cosign is a tool for signing and verifying container images. You'll need to install it on your RHEL 8 machine.
Steps:
-
Set Up HTTPS_PROXY (if needed):
If your terminal doesn't have internet access, temporarily set up the
HTTPS_PROXYenvironment variable.export HTTPS_PROXY="http://your-proxy-server:port" -
Download the Latest Cosign Binary:
Visit the Cosign GitHub Releases page to find the latest version. Replace
vX.Y.Zwith the latest version number.VERSION="vX.Y.Z" wget https://github.com/sigstore/cosign/releases/download/${VERSION}/cosign-linux-amd64 -
Install Cosign:
chmod +x cosign-linux-amd64 sudo mv cosign-linux-amd64 /usr/local/bin/cosign -
Verify Installation:
cosign version
Step 2: Generate Signing Keys with Cosign
In an offline environment, you'll need to use key-pair signing (not keyless).
Steps:
-
Generate Key Pair:
cosign generate-key-pairThis command will generate two files in your current directory:
cosign.key(private key)cosign.pub(public key)
-
Secure the Keys:
- Private Key (
cosign.key): Keep this secure and don't share it. - Public Key (
cosign.pub): This will be shared with Cosigned for verification.
- Private Key (
Step 3: Sign Container Images with Cosign
You'll need to sign your container images before they are deployed.
Steps:
-
Load or Build Your Container Image:
Ensure that your image is built and tagged appropriately.
docker build -t myregistry.local/myimage:tag . -
Push Image to Local Registry (we'll set this up in the next step).
Step 4: Set Up a Local Container Image Registry
Since you are in an offline environment, you need a local registry to host your images and their signatures.
Steps:
-
Deploy a Local Registry:
docker run -d -p 5000:5000 --name registry registry:2 -
Tag and Push Your Image to the Local Registry:
docker tag myregistry.local/myimage:tag localhost:5000/myimage:tag docker push localhost:5000/myimage:tagNote: Ensure that your RKE2 cluster nodes can access this registry (
localhost:5000). You may need to adjust the registry URL or make it accessible over your network. -
Configure RKE2 to Trust the Insecure Registry (if needed):
Edit the RKE2 configuration to allow insecure registries.
sudo vi /etc/rancher/rke2/registries.yamlAdd:
mirrors: "localhost:5000": endpoint: - "http://localhost:5000"Restart RKE2:
sudo systemctl restart rke2-server
Step 5: Sign the Images with Cosign
Now, sign the image that you've pushed to the local registry.
Steps:
-
Sign the Image:
cosign sign -key cosign.key localhost:5000/myimage:tagThis command attaches the signature to the image in the registry.
Step 6: Install Sigstore Cosigned (PolicyController) in Your RKE2 Cluster
Cosigned is the admission controller that enforces image signature verification in your cluster.
Steps:
-
Download Cosigned Deployment Manifests:
Since you are offline, you need to get the deployment manifests and images ahead of time.
-
Download the manifests from the Sigstore Cosign GitHub.
wget https://raw.githubusercontent.com/sigstore/cosign/main/install/cosigned.yaml
-
-
Download Cosigned Image:
On a machine with internet access, pull the cosigned image.
docker pull ghcr.io/sigstore/cosigned:<version>Replace
<version>with the appropriate version tag. -
Save the Image to a Tar File:
docker save ghcr.io/sigstore/cosigned:<version> -o cosigned.tar -
Transfer the Image and Manifests to Your Offline Environment:
Use a USB drive or secure copy to transfer
cosigned.tarandcosigned.yamlto your offline environment. -
Load the Cosigned Image into Your Local Registry:
docker load -i cosigned.tar docker tag ghcr.io/sigstore/cosigned:<version> localhost:5000/sigstore/cosigned:<version> docker push localhost:5000/sigstore/cosigned:<version> -
Modify the Cosigned Deployment Manifest:
Edit
cosigned.yamlto point to your local registry.# In cosigned.yaml, find the Deployment and update the image: image: localhost:5000/sigstore/cosigned:<version> -
Deploy Cosigned to Your Cluster:
kubectl apply -f cosigned.yaml -
Verify the Deployment:
kubectl get pods -n cosigned-system
Step 7: Configure Cosigned Policies for Image Verification
You'll need to inform Cosigned about your public key and define policies for image verification.
Steps:
-
Create a Kubernetes Secret with Your Public Key:
kubectl create namespace cosign-system kubectl create secret generic cosign-public-key \ -n cosign-system \ --from-file=cosign.pub -
Create a ClusterImagePolicy:
Prepare a
clusterimagepolicy.yamlfile:apiVersion: policy.sigstore.dev/v1alpha1 kind: ClusterImagePolicy metadata: name: require-cosign-signatures spec: images: - glob: "localhost:5000/*" authorities: - key: secretRef: name: cosign-public-key namespace: cosign-systemThis policy requires that any image from
localhost:5000/*must be signed with the provided public key. -
Apply the Policy:
kubectl apply -f clusterimagepolicy.yaml
Step 8: Test Image Deployment and Verification
Test whether unsigned images are blocked and signed images are allowed.
Steps:
-
Attempt to Deploy an Unsigned Image:
Prepare a deployment manifest
unsigned-deployment.yaml:apiVersion: apps/v1 kind: Deployment metadata: name: unsigned-app spec: replicas: 1 selector: matchLabels: app: unsigned-app template: metadata: labels: app: unsigned-app spec: containers: - name: unsigned-app image: localhost:5000/unsigned-image:tagApply the deployment:
kubectl apply -f unsigned-deployment.yamlThis should be blocked by Cosigned, and you should see an admission error.
-
Attempt to Deploy a Signed Image:
Prepare a deployment manifest
signed-deployment.yaml:apiVersion: apps/v1 kind: Deployment metadata: name: signed-app spec: replicas: 1 selector: matchLabels: app: signed-app template: metadata: labels: app: signed-app spec: containers: - name: signed-app image: localhost:5000/myimage:tagApply the deployment:
kubectl apply -f signed-deployment.yamlThis should be allowed, and the pod should start running.
-
Verify the Pod Status:
kubectl get pods
Additional Considerations
-
Network Accessibility:
Ensure that all your RKE2 cluster nodes can access the local registry at
localhost:5000. You might need to set up DNS or use the actual IP address. -
Certificates and TLS for the Registry:
For production environments, it's recommended to secure your registry with TLS certificates.
-
Updating Cosigned Policies:
You can refine your
ClusterImagePolicyto match specific images or add additional keys as needed. -
Key Management:
Always protect your private keys and manage them securely. Rotate keys if necessary.
-
Testing in a Non-Production Environment:
Before rolling out to production, test this setup thoroughly in a staging environment.
Summary
By following these steps, you can set up image signing and verification in your RKE2 cluster using Cosign and Cosigned (PolicyController) from Sigstore, even in an offline environment. This ensures that only trusted and signed container images are deployed in your cluster, enhancing your security posture.
If you encounter any issues during the setup, please provide specific error messages or logs, and I'll be happy to help troubleshoot further.