AKS Private Clusters are finally GA!

In my previeous post on AKS Security, I talked about creating Private Clusters as a next step in your efforts to secure your Azure managed Kubernetes (they were still in Preview a couple of weeks earlier)

Private AKS clusters have all their control plane components, including the cluster’s Kubernetes API service, in a private RFC1918 network space. This limits access and keeps all traffic within Azure’s networks. You can lock down access to the API to specific VNets. Without this feature, the cluster’s API has a public IP address and all traffic to it, including from the cluster’s node pools, goes over public network.

Note that private clusters do have some limitations, you can learn more here

Note: The private cluster option can only be selected at AKS cluster creation time.

Architecture and Concepts

aks-archi-private

Enabling AKS Private Clusters using –enable-private-cluster, adds the following components to the MC_* nodes resource group:

aks-private-endpoint

Azure Private Endpoint

This is an endpoint added on the Subnet of AKS nodes and provides a private IP form this subnet to the API Server

Private DNS Zone

An A record of the API server FQDN is dynamically added to this DNS Zone pointing to the private IP of the endpoint above

Traffic from the private Endpoint to the API server goes through the Microsoft backbone thus using Azure Private Link

Options for connecting to the private cluster

The API server endpoint has no public IP address. To access the API server, you will need to use a VM that has access to the AKS cluster’s Azure Virtual Network (VNet). There are several options for establishing network connectivity to the private cluster.

  • Create a VM in the same Azure VNet as the AKS cluster.
  • Use a VM in a separate network and configure Vnet peering. This is what we will illustrate in the following Demo
  • Use an Express Route or VPN connection.

Demo

Let’s create a Private AKS Cluster and connect to its API Server from a jumpbox VM in peered Vnet

Prerequisites

  • The Azure CLI version 2.2.0 or later

Creating a Private AKS Cluster

The scripts for this demo are available on my Github adeelku/aks-private.

Executing ./scripts/deploy-all.sh script will automatically provision the following resources using variables that you define in ./scripts/params.sh:

  • Networking
    • Create the Users Vnet and subnet
    • Create the AKS Vnet and subnet
    • Create peering between both Vnets
  • Create AKS Private Cluster
  • Configure Private DNS Link tUsers Vnet
    • Enable jumbox to resolve API Server’s Private Endpoint IP
  • Jumbox VM
    • Create a subnet for the VM in the Users Vnet
    • Create the Linux Jumpbox VM
    • SSH to the VM

params.sh

## AKS
LOCATION="canadacentral"
RG_NAME="aks-private-rg"
CLUSTER_NAME="aks-private"
NODE_SIZE="Standard_B2s"
NODE_COUNT="1"
NODE_DISK_SIZE="30"
VERSION="1.16.7"
CNI_PLUGIN="kubenet"

## Networking
AKS_VNET="vnet-private-aks"
AKS_VNET_RG="net-private-rg"
AKS_VNET_CIDR="10.10.0.0/16"
AKS_SNET="aks-subnet"
AKS_SNET_CIDR="10.10.0.0/24"
USERS_VNET="vnet-users"
USERS_RG="users-rg"
USERS_VNET_CIDR="10.100.0.0/16"
USERS_SNET="users-subnet"
USERS_SNET_CIDR="10.100.100.0/24"

##Peering
VNET_SOURCE_RG=$AKS_VNET_RG
VNET_SOURCE=$AKS_VNET
VNET_DEST_RG=$USERS_RG
VNET_DEST=$USERS_VNET

## Jumpbox VM
VM_NAME="vm-jumpbox"
VM_IMAGE="UbuntuLTS"
VM_SIZE="Standard_B1s"
VM_OSD_SIZE="32"
VM_RG=$USERS_RG
VM_VNET=$USERS_VNET
VM_SNET="jumpbox-subnet"
VM_SNET_CIDR="10.100.110.0/28"
VM_PUBIP="vm-jumpbox-pip"

deploy-all.sh

##!/usr/bin/env bash
set -e
. ./params.sh

echo "configuring Networking"
## create Resource Group for Users VNet
az group create --name $USERS_RG --location $LOCATION

## Create USERS VNet and SubNet
az network vnet create \
    -g $USERS_RG \
    -n $USERS_VNET --address-prefix $USERS_VNET_CIDR \
    --subnet-name $USERS_SNET --subnet-prefix $USERS_SNET_CIDR

## create Resource Group for AKS VNet
az group create --name $AKS_VNET_RG --location $LOCATION

## Create AKS VNet and SubNet
az network vnet create \
    -g $AKS_VNET_RG \
    -n $AKS_VNET --address-prefix $AKS_VNET_CIDR \
    --subnet-name $AKS_SNET --subnet-prefix $AKS_SNET_CIDR

echo ""
echo "configuring Peering"
VNET_SOURCE_ID=$(az network vnet show \
    --resource-group $VNET_SOURCE_RG \
    --name $VNET_SOURCE \
    --query id -o tsv)
VNET_DEST_ID=$(az network vnet show \
    --resource-group $VNET_DEST_RG \
    --name $VNET_DEST \
    --query id -o tsv)

az network vnet peering create \
    --resource-group $VNET_SOURCE_RG -n "${VNET_SOURCE}-to-${VNET_DEST}" \
    --vnet-name $VNET_SOURCE \
    --remote-vnet $VNET_DEST_ID \
    --allow-vnet-access

az network vnet peering create \
    --resource-group $VNET_DEST_RG -n "${VNET_DEST}-to-${VNET_SOURCE}" \
    --vnet-name $VNET_DEST \
    --remote-vnet $VNET_SOURCE_ID \
    --allow-vnet-access

echo ""
echo "configuring Private AKS"
## get subnet info
echo "Getting Subnet ID"
AKS_SNET_ID=$(az network vnet subnet show \
  --resource-group $AKS_VNET_RG \
  --vnet-name $AKS_VNET \
  --name $AKS_SNET \
  --query id -o tsv)

### create private aks cluster
echo "Creating Private AKS Cluster RG"
az group create --name $RG_NAME --location $LOCATION
echo "Creating Private AKS Cluster"
az aks create --resource-group $RG_NAME --name $CLUSTER_NAME \
  --kubernetes-version $VERSION \
  --location $LOCATION \
  --enable-private-cluster \
  --node-vm-size $NODE_SIZE \
  --load-balancer-sku standard \
  --node-count $NODE_COUNT --node-osdisk-size $NODE_DISK_SIZE \
  --network-plugin $CNI_PLUGIN \
  --vnet-subnet-id $AKS_SNET_ID \
  --docker-bridge-address 172.17.0.1/16 \
  --dns-service-ip 10.2.0.10 \
  --service-cidr 10.2.0.0/24 

## Configure Private DNS Link to Jumpbox VM
echo ""
echo "Configuring Private DNS Link to Jumpbox VM"

noderg=$(az aks show --name $CLUSTER_NAME \
    --resource-group $RG_NAME \
    --query 'nodeResourceGroup' -o tsv) 

dnszone=$(az network private-dns zone list \
    --resource-group $noderg \
    --query [0].name -o tsv)

az network private-dns link vnet create \
    --name "${USERS_VNET}-${USERS_RG}" \
    --resource-group $noderg \
    --virtual-network $VNET_DEST_ID \
    --zone-name $dnszone \
    --registration-enabled false

echo ""
echo "configuring Jumbox VM"
## create subnet for vm
echo "Creating Jumpbox subnet"
az network vnet subnet create \
    --name $VM_SNET \
    --resource-group $USERS_RG \
    --vnet-name $VM_VNET \
    --address-prefix $VM_SNET_CIDR

## get subnet info
echo "Getting Subnet ID"
SNET_ID=$(az network vnet subnet show \
  --resource-group $USERS_RG \
  --vnet-name $VM_VNET \
  --name $VM_SNET \
  --query id -o tsv)

## create public ip
echo "Creating VM public IP"
az network public-ip create \
    --resource-group $VM_RG \
    --name $VM_PUBIP \
    --allocation-method dynamic \
    --sku basic

## create vm
echo "Creating the VM"
az vm create \
    --resource-group $VM_RG \
    --name $VM_NAME \
    --image $VM_IMAGE \
    --size $VM_SIZE \
    --os-disk-size-gb $VM_OSD_SIZE \
    --subnet $SNET_ID \
    --public-ip-address $VM_PUBIP \
    --admin-username azureuser \
    --generate-ssh-keys

## connect to vm
PUBLIC_IP=$(az network public-ip show -n $VM_PUBIP -g $VM_RG --query ipAddress -o tsv)
ssh azureuser@$PUBLIC_IP -i ~/.ssh/id_rsa

Accessing the API Server privately

When the cluster and its supporting resources are created, you will automatically SSH to the jumbox using your SSH Key

  • install AZ CLI

Follow these instructions to install AZ CLI on the jumpbox

  • Install Kubectl
sudo az aks install-cli
  • Configure Connection to the API server
az aks get-credentials -g <aks-rg> -n <cluster-name>
  • check the connection
kubectl get nodes

The following example output shows the single node created in the previous steps. Make sure that the status of the node is Ready:

NAME                       STATUS   ROLES   AGE     VERSION
aks-nodepool1-31718369-0   Ready    agent   6m44s   v1.16.7