Leveraging CDK for Cross-Account Load Balancing to Expose Amazon EKS Pods
- Itay Melamed
- Apr 1, 2024
- 5 min read
Introduction
Security is of utmost importance in cloud infrastructure management. However, orchestrating secure communication across multiple AWS accounts can present challenges. In this tutorial, we'll explore a solution leveraging VPC Endpoints to establish secure communication between an EKS cluster and a public ALB across different AWS accounts, while maintaining stringent security protocols.
Problem Statement
Picture a scenario where two distinct AWS accounts exist, each with its own VPC. Due to security constraints, direct VPC peering or transit gateway connections between these VPCs are not permissible. In Account A, there's an EKS cluster responsible for hosting crucial workloads. Access to the pods within this cluster is governed via an ingress mechanism, with traffic being directed through an NLB (Network Load Balancer). In Account B, a public Application Load Balancer (ALB) needs to route traffic to the EKS NLB securely.
Solution Components Explanation:
Workload Account (Account A):
Amazon EKS Cluster: Hosts the essential workloads.
NLB (Network Load Balancer): Routes traffic to the EKS ingress.
VPC Endpoint Service: Established to associate with the NLB.
Ingress Account (Account B):
Public Application Load Balancer (ALB): Routes traffic to the EKS NLB.
VPC Endpoint: Connects to the VPC Endpoint service in the Workload Account.
Why Separate Components between Ingress and Workload Accounts? Segmenting components between the Ingress and Workload Accounts ensures proper isolation and security enforcement. This division mitigates unauthorized access and potential breaches, offering greater control over resource access and minimizing the attack surface.
Importance of Separating Network Between Ingress Account and EKS Cluster Workload
Securing the network between the Ingress Account and the EKS cluster is critical due to security risks associated with exposing critical components to the public internet. Direct exposure could lead to DDoS attacks, unauthorized access, and data leaks. Leveraging VPC Endpoints establishes a private and secure communication channel, effectively mitigating these risks.
Addressing Cross-Account Challenge
As the VPC Endpoint service name of the NLB cluster resides in a different account, creating the VPC Endpoint service in the Ingress Account becomes necessary. To achieve this, AWS CodePipeline from a shared services account facilitates seamless data sharing between accounts.
Architecture Overview
Our architecture comprises two AWS accounts: the Workload Account (Account A) housing the EKS cluster and the Ingress Account (Account B) containing the public ALB. Each account has its own VPC, ensuring isolation and security.
Tutorial: Deploying the Solution Using CDK (Cloud Development Kit)
In the subsequent sections, we'll provide a detailed walkthrough of implementing this solution using CDK. CDK enables infrastructure definition as code, simplifying management and deployment of intricate architectures across multiple AWS accounts.
By following this tutorial, you'll gain insights into establishing secure communication between services across different AWS accounts.
Step 1:
To create a pipeline stage for deploying the workload stack, you can follow these steps:
from aws_cdk import (
Stage,
aws_ssm as ssm
)
from constructs import Construct
from stacks.network_stack import NetworkStack
from stacks.workload_stack import WorkloadStack
class PipelineStageWorkload(Stage):
def __init__(self, scope: Construct, id: str, env_name: str, **kwargs):
super().__init__(scope, id, **kwargs)
workload_stack = WorkloadStack(
self, 'Workload-Stack', env_name=env_name)
network_account = NetworkStack(
self, 'Workload-Stack', env_name=env_name)
Step 2:
Initialize and Deploy Stack. Initialize the CDK app, create an instance of the PipelineStack, and deploy it.
Here's the code snippet implementing the above steps:
from aws_cdk import (
App, Stack,
pipelines as pipelines,
aws_codecommit as code_commit
)
from deploy import PipelineStageNetwork, PipelineStageWorkload
from constructs import Construct
class PipelineStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
repository = code_commit.Repository.from_repository_name(
self, 'repository', repository_name="cdk-workload")
pipeline = pipelines.CodePipeline(
self, 'Workload-Pipeline',
pipeline_name='Workload-Pipeline',
cross_account_keys=True,
use_change_sets=False,
synth=pipelines.ShellStep("Synth",
input=pipelines.CodePipelineSource.code_commit(
repository, 'main'),
commands=[
"npm install -g aws-cdk",
"pip install -r requirements.txt",
"npm install cdk-nag",
"cdk synth"
]
)
)
deploy_workload_stage = pipeline.add_stage(
PipelineStageWorkload(self, 'DeployWorkload', env_name='dev', env={'account': 'workloadaccount', 'region': 'il-central-1'}))
deploy_ingress_stage = pipeline.add_stage(
PipelineStageNetwork(self, 'DeployIngress', env_name='dev', env={'account': 'ingressaccount', 'region': 'il-central-1'}))
# Initialize and Call Class PipelineStack
app = App()
PipelineStack(app, 'Workload-Pipeline-Stack',
env={'account': 'youraccount', 'region': 'il-central-1'})
app.synth()
Step 3:
Create stack for the workload, craete vpc endpoint service for ingress nlb and save it in ssm parameter store so ALB in network ingress accoiunt can use it:
from aws_cdk import (
aws_ec2 as ec2,
Stack,
aws_autoscaling as autoscaling,
aws_iam as iam,
aws_eks as eks,
aws_ssm as ssm,
aws_elasticloadbalancingv2 as elbv2)
from constructs import Construct
from aws_cdk.lambda_layer_kubectl_v29 import KubectlV29Layer
import requests
class WorkloadStack(Stack):
def __init__(self, scope: Construct, construct_id: str, env_name: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
KubectlV29Layer(self, "KubectlV29Layer")
# provisioning a cluster
cluster = eks.Cluster(self, "Workload",
version=eks.KubernetesVersion.V1_29
)
# Adding service account for ALB Ingress Controller
alb_service_account = cluster.add_service_account("aws-alb-ingress-controller-sa",
name="aws-load-balancer-controller",
namespace="kube-system")
# Fetching IAM policy for ALB Ingress Controller
aws_alb_controller_policy_url = 'https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json'
policy_json = requests.get(aws_alb_controller_policy_url).text
policy_statements = iam.PolicyStatement.from_json(policy_json)[
'Statement']
for statement in policy_statements:
alb_service_account.add_to_principal_policy(
iam.PolicyStatement.from_json(statement))
# Adding Helm chart for ALB Ingress Controller
cluster_helm = cluster.add_helm_chart("aws-load-balancer-controller-helm-chart",
chart="eks/aws-load-balancer-controller",
repository="https://aws.github.io/eks-charts",
release="aws-load-balancer-controller",
version="2.2.0",
namespace="kube-system",
values={
"clusterName": cluster.cluster_name,
"serviceAnnoations": {
"alb.ingress.kubernetes.io/tags": "Name=alb-contoller"
},
"serviceAccount": {
"create": False,
"name": "aws-load-balancer-controller"
}
})
nlb = elbv2.NetworkLoadBalancer.from_lookup(
self, 'NlbLookUp', load_balancer_tags={"Name": "alb-contoller"})
nlb.add_depends_on(cluster_helm)
vpc_endpoint_service = ec2.VpcEndpointService(self, "EndpointService",
vpc_endpoint_service_load_balancers=[nlb], acceptance_required=False,
allowed_principals=[iam.ArnPrincipal(
"arn:aws:iam::123456789012:root")]
)
ssm.StringParameter(self, "AlbControllerVpcES", parameter_name="/workload/alb-contoller-ingress-vpc-endpoint-service",
string_value=vpc_endpoint_service.vpc_endpoint_service_name)Step 4:
Now that we created the ingress and have its vpc endpoint service saved as a parameter, we can create in the network account a vpc endpoint point for the vpc endpoint service and ALB that target to vpc endpoint ips:
from aws_cdk import (
aws_ec2 as ec2,
Stack,
aws_elasticloadbalancingv2 as elbv2,
custom_resources as cs
)
from constructs import Construct
import boto3
import time
class NetworkStack(Stack):
def __init__(self, scope: Construct, construct_id: str, env_name: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
# Assume role in workload account
assumed_role_session = boto3.Session().client('sts').assume_role(
RoleArn='arn:aws:iam::WORKLOAD_ACCOUNT_ID:role/ROLE_NAME',
RoleSessionName='AssumedRoleSession'
)
ssm_client = assumed_role_session.client('ssm')
# Get parameter value
parameter_value = ssm_client.get_parameter(
Name='/workload/alb-contoller-ingress-vpc-endpoint-service')['Parameter']['Value']
vpc = ec2.Vpc(self, "VPC",
vpc_name="WorkloadVpc",
max_azs=2,
ip_addresses=ec2.IpAddresses.cidr("10.10.0.0/16"),
# configuration will create 2 groups in 2 AZs = 4 subnets.
subnet_configuration=[
ec2.SubnetConfiguration(
subnet_type=ec2.SubnetType.PUBLIC,
name="Public",
cidr_mask=24
), ec2.SubnetConfiguration(
subnet_type=ec2.SubnetType.PRIVATE_ISOLATED,
name="DB",
cidr_mask=24
)
]
)
vpc_endpoint_sg = ec2.SecurityGroup(
self, 'SecurityGroup', vpc=vpc, allow_all_outbound=True)
vpc_endpoint_sg.add_ingress_rule(
ec2.Peer.any_ipv4(), ec2.Port.tcp(443))
vpc_endpoint = ec2.InterfaceVpcEndpoint(self, "IngressVpcEndpoint",
service=ec2.InterfaceVpcEndpointService(
parameter_value),
vpc=vpc,
private_dns_enabled=True,
security_groups=[
vpc_endpoint_sg],
subnets=ec2.SubnetSelection(
subnet_type=ec2.SubnetType.PUBLIC)
)
vpc_endpoint_ips = []
for eni_id in vpc_endpoint.vpc_endpoint_network_interface_ids:
eni = cs.AwsCustomResource(
self, 'DescribeNetworkInterfaces',
on_create=cs.AwsSdkCall(
service='ec2',
action='describeNetworkInterfaces',
parameters={
'NetworkInterfaceId.N': [eni_id]
},
physical_resource_id=str(time.time())
)
)
vpc_endpoint_ips.append(eni.get_data(
'NetworkInterfaces.0.PrivateIpAddress').to_string())
# Create ALB
alb = elbv2.ApplicationLoadBalancer(self, "MyALB",
vpc=vpc,
internet_facing=True
)
# Create target group
target_group = elbv2.ApplicationTargetGroup(self, "AlbTargetGroup",
vpc=vpc,
port=80,
target_type=elbv2.TargetType.IP,
targets=[vpc_endpoint_ips],
protocol=elbv2.ApplicationProtocol.HTTP
)
# Add listener to ALB
listener = alb.add_listener("MyListener",
port=80,
open=True,
default_target_groups=[target_group]
)
Step 5:
Now we can deploy the pipeline we have created in step 2. Once it deployed it will run 2 stages to deploy resources in workload account and ingress account.
Conclusion
In this tutorial, we've explored a comprehensive solution for establishing secure communication between an Amazon EKS cluster and a public ALB across different AWS accounts. By leveraging VPC Endpoints, we've addressed the challenge of orchestrating secure communication while maintaining stringent security protocols and ensuring proper isolation between the Ingress and Workload Accounts. This approach not only mitigates security risks associated with exposing critical components to the public internet but also offers greater control over resource access and minimizes the attack surface. Through a step-by-step guide using AWS CDK, we've demonstrated how to implement this solution effectively, providing insights into deploying intricate architectures across multiple AWS accounts with ease and confidence.
Comments