I want to create a secure eks cluster using terraform and I want to have prometheus,…
- User
I want to create a secure eks cluster using terraform and I want to have prometheus, grafana, argocd added. Would you be able to write the terraform config?
- Assistant
Yes, I can help you write a Terraform configuration to create a secure Amazon Elastic Kubernetes Service (EKS) cluster with Prometheus, Grafana, and ArgoCD installed. Below is a step-by-step guide along with code snippets to help you set up the infrastructure securely.
Overview
We'll accomplish the following:
- Set Up Providers
- Create a VPC
- Create IAM Roles and Policies
- Create the EKS Cluster and Node Groups
- Configure the Kubernetes Provider
- Deploy Prometheus, Grafana, and ArgoCD using Helm
- Implement Security Best Practices
Step 1: Set Up Providers
# providers.tf provider "aws" { region = "us-west-2" # Change to your desired AWS region } provider "kubernetes" { host = module.eks.cluster_endpoint cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) token = data.aws_eks_cluster_auth.eks.token } provider "helm" { kubernetes { host = module.eks.cluster_endpoint cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) token = data.aws_eks_cluster_auth.eks.token } }
Step 2: Create a VPC
Use the official AWS VPC module to create a virtual private cloud.
# vpc.tf module "vpc" { source = "terraform-aws-modules/vpc/aws" version = "3.19.0" # Use the latest compatible version name = "secure-eks-vpc" cidr = "10.0.0.0/16" azs = ["us-west-2a", "us-west-2b", "us-west-2c"] public_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"] private_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"] enable_nat_gateway = true single_nat_gateway = true tags = { Environment = "production" Terraform = "true" } }
Step 3: Create IAM Roles and Policies
Define IAM roles for the EKS cluster and node groups with least privilege.
# iam.tf resource "aws_iam_role" "eks_cluster_role" { name = "eks-cluster-role" assume_role_policy = data.aws_iam_policy_document.eks_cluster_assume_role_policy.json } data "aws_iam_policy_document" "eks_cluster_assume_role_policy" { statement { effect = "Allow" principals { type = "Service" identifiers = ["eks.amazonaws.com"] } actions = ["sts:AssumeRole"] } } resource "aws_iam_role_policy_attachment" "eks_cluster_AmazonEKSClusterPolicy" { role = aws_iam_role.eks_cluster_role.name policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy" } resource "aws_iam_role" "eks_node_role" { name = "eks-node-role" assume_role_policy = data.aws_iam_policy_document.eks_node_assume_role_policy.json } data "aws_iam_policy_document" "eks_node_assume_role_policy" { statement { effect = "Allow" principals { type = "Service" identifiers = ["ec2.amazonaws.com"] } actions = ["sts:AssumeRole"] } } resource "aws_iam_role_policy_attachment" "eks_node_AmazonEKSWorkerNodePolicy" { role = aws_iam_role.eks_node_role.name policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy" } resource "aws_iam_role_policy_attachment" "eks_node_AmazonEC2ContainerRegistryReadOnly" { role = aws_iam_role.eks_node_role.name policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" } resource "aws_iam_role_policy_attachment" "eks_node_AmazonEKS_CNI_Policy" { role = aws_iam_role.eks_node_role.name policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy" }
Step 4: Create the EKS Cluster and Node Groups
Use the official EKS module to create the cluster and node groups.
# eks.tf module "eks" { source = "terraform-aws-modules/eks/aws" version = "18.29.0" # Use the latest compatible version cluster_name = "secure-eks-cluster" cluster_version = "1.24" # Use the desired Kubernetes version vpc_id = module.vpc.vpc_id subnet_ids = module.vpc.private_subnets cluster_endpoint_private_access = true # Enable private access cluster_endpoint_public_access = false # Disable public access manage_aws_auth = true cluster_iam_role_name = aws_iam_role.eks_cluster_role.name cluster_iam_role_arn = aws_iam_role.eks_cluster_role.arn node_groups = { default = { desired_capacity = 2 max_capacity = 3 min_capacity = 2 instance_types = ["t3.medium"] subnets = module.vpc.private_subnets iam_role_arn = aws_iam_role.eks_node_role.arn tags = { Name = "eks-node" } } } tags = { Environment = "production" Terraform = "true" } } # data sources to get cluster info data "aws_eks_cluster" "eks" { name = module.eks.cluster_id } data "aws_eks_cluster_auth" "eks" { name = module.eks.cluster_id }
Step 5: Configure the Kubernetes Provider
Ensure the Kubernetes provider uses the cluster details.
# kubernetes.tf provider "kubernetes" { host = data.aws_eks_cluster.eks.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data) token = data.aws_eks_cluster_auth.eks.token load_config_file = false }
Step 6: Deploy Prometheus, Grafana, and ArgoCD using Helm
Create Namespaces
Define Kubernetes namespaces for your applications.
# namespaces.tf resource "kubernetes_namespace" "monitoring" { metadata { name = "monitoring" } } resource "kubernetes_namespace" "argocd" { metadata { name = "argocd" } }Install Helm Charts
Use the Helm provider to install the charts.
# helm_releases.tf resource "helm_release" "kube_prometheus_stack" { name = "kube-prometheus-stack" chart = "kube-prometheus-stack" repository = "https://prometheus-community.github.io/helm-charts" namespace = kubernetes_namespace.monitoring.metadata[0].name create_namespace = false version = "45.9.0" # Specify the desired chart version values = [ file("helm_values/kube_prometheus_stack_values.yaml") ] } resource "helm_release" "argo_cd" { name = "argo-cd" chart = "argo-cd" repository = "https://argoproj.github.io/argo-helm" namespace = kubernetes_namespace.argocd.metadata[0].name create_namespace = false version = "5.28.2" # Specify the desired chart version values = [ file("helm_values/argocd_values.yaml") ] }Helm Values Files
Create the directory
helm_valuesand add the respective values files.kube_prometheus_stack_values.yamlgrafana: adminPassword: "YourSecurePassword" service: type: ClusterIP alertmanager: alertmanagerSpec: storage: volumeClaimTemplate: spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 5Giargocd_values.yamlserver: ingress: enabled: false service: type: ClusterIP configs: secret: argocdServerAdminPassword: "$2a$10$YourBcryptHashedPassword" # Use bcrypt hash of a secure passwordNote: Replace
"YourSecurePassword"and"$2a$10$YourBcryptHashedPassword"with secure passwords.
Step 7: Implement Security Best Practices
Private Networking
- Private Endpoints: Already configured by setting
cluster_endpoint_public_access = falseandcluster_endpoint_private_access = truein the EKS module. - Private Subnets: Worker nodes are placed in private subnets.
Security Groups
- The EKS module automatically creates security groups. Ensure they are configured to allow necessary traffic and restrict unwanted access.
Network Policies
- Deploy a network policy engine like Calico.
# calico.tf resource "helm_release" "calico" { name = "calico" chart = "tigera-operator" repository = "https://projectcalico.docs.tigera.io/charts" namespace = "tigera-operator" create_namespace = true version = "3.24.1" # Specify the desired chart version set { name = "installation.calicoNetwork.ipPools[0].cidr" value = "10.0.0.0/16" # Use your cluster CIDR } }RBAC and IAM
- RBAC: Ensure Kubernetes RBAC is properly configured.
- IAM Roles for Service Accounts (IRSA): Use IRSA to assign IAM roles to Kubernetes service accounts.
- Least Privilege: IAM roles and policies are already set up with minimal required permissions.
Secrets Management
- Use AWS Secrets Manager or a tool like HashiCorp Vault.
- Use Kubernetes secrets or external secrets operator to fetch secrets securely.
Encrypt Kubernetes Secrets
- Enable envelope encryption for Kubernetes secrets using AWS KMS.
# kms.tf resource "aws_kms_key" "eks" { description = "KMS key for EKS secrets encryption" deletion_window_in_days = 7 } resource "aws_eks_cluster" "cluster" { # ... other configurations ... encryption_config { resources = ["secrets"] provider { key_arn = aws_kms_key.eks.arn } } }Note: Adjust the EKS cluster resource if using the module.
Logging and Monitoring
- Enable cluster control plane logs.
# logs.tf module "eks" { # ... other configurations ... enable_control_plane_logs = ["api", "audit", "authenticator", "controllerManager", "scheduler"] }
Full Example
For a full example, combine all the code snippets into their respective
.tffiles. Remember to:- Replace placeholder values (e.g., passwords, regions).
- Review and adjust versions for providers and Helm charts.
- Ensure all resources have appropriate tags for resource management.
Applying the Configuration
-
Initialize Terraform:
terraform init -
Review the Plan:
terraform plan -
Apply the Configuration:
terraform apply
Final Notes
- Testing: Always test in a non-production environment first.
- Costs: Be aware of AWS costs associated with the resources.
- Updates: Regularly update the versions of Terraform, providers, and Helm charts to incorporate security patches and improvements.
- Security Audits: Periodically audit your infrastructure for security compliance.
Let me know if you need further assistance or have questions about specific parts of the configuration.