In the continuous endeavour to secure AKS environments in the Enterprise, Customers encounter use cases that are not covered by a Security Solution, such as Azure Defender for Containers, or Third pary Container Security tools, neither by Azure Built-in Kubernetes Policy definitions.

In this post, I will walk you through the steps you can follow to create and assign a Custom AKS Azure Policy Definition on your Azure scope [Subscription or Management Group]

Please Note that Custom Azure Policy Definitions for Kubernetes are in Public Preview at the time of writing this post.

Before we dive into all this, let’s understand how Azure Policy add-on for AKS works, especially when dealing with Kubernetes Objects.

Azure Policy add-on for AKS extends Gatekeeper v3, the admission controller webhook for Open Policy Agent (OPA), to apply enforcements and safeguards on the Organisation’s AKS clusters centrally and consistently. Azure Policy provides the capability to manage and report on the compliance state of all Kubernetes clusters from one place. The add-on enables the following functions:

Checks with Azure Policy service for policy assignments to the cluster.
Deploys policy definitions into the cluster as constraint templates and constraint custom resources.
Reports auditing and compliance details back to Azure Policy service.

Use case

In this exemple, we will create a Custom Policy that aims at restricting the use of wildcards in Kubernetes Roles and ClusterRoles - Resources and Verbs.

This is a control which is found in the CIS Benchmark for AKS and that needs to be implemented within its Guidelines for those who want to be compliant with it’s Guidelines.

We will go through the following steps:

Create a Constraint Template
Create an Azure Policy Custom Definition and embed in it the Constraint Template
Assign and test the Policy

All samples of code used for this post can be found in this Github Repo, covering AKS native Security, under the following folder: az-custom-aks-policy that is dedicated to this Post

Constraint Template

The constraint template file we will be using in this example looks like this:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8sblockwildcardsinrolesandclusterroles
  annotations:
    description: Block wildcards in Roles and ClusterRole Verbs.
spec:
  crd:
    spec:
      names:
        kind: k8sblockwildcardsinrolesandclusterroles
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sblockwildcardsinrolesandclusterroles

        violation[{"msg": msg, "details": {}}]{
        input.review.object.kind == "Role"
        input.review.object.rules[_].verbs[_] == "*"
        msg := "Wildcards are prohibited in Roles Verbs"
        }
        
        violation[{"msg": msg, "details": {}}]{
        input.review.object.kind == "Role"
        input.review.object.rules[_].resources[_] == "*"
        msg := "Wildcards are prohibited in Roles Resources"
        }

        violation[{"msg": msg, "details": {}}]{
        input.review.object.kind == "ClusterRole"
        input.review.object.rules[_].verbs[_] == "*"
        msg := "Wildcards are prohibited in ClusterRoles Verbs"
        }
        
        violation[{"msg": msg, "details": {}}]{
        input.review.object.kind == "ClusterRole"
        input.review.object.rules[_].resources[_] == "*"
        msg := "Wildcards are prohibited in ClusterRoles Resources"
        }

Note: The use of the API version v1beta1 instead of v1 in the Constraint Template above, it seems that this is a limitation of the Preview Feature of Custom Azure Policy Definition for Kubernetes.

You cannot add these type of Constraint Templates manually into your AKS Cluster, as you can only do it through Azure Policy. If you try, you will receive an error message similar to the following one:

Admission Errors

Azure Policy Custom Definition

To use the Constraint Template, we will need to encode it in base64 to be able to embed it into the Azure Policy Custom Definition Json:

cd az-custom-aks-policy/block-wildcards-in-roles
base64 -w0 constraint.yml

YXBpVmVyc2lvbjogdGVtcGxhdGVzLmdhdGVrZWVwZXIuc2gvdjFiZXRhMQpraW5kOiBDb25zdHJhaW50VGVtcGxhdGUKbWV0YWRhdGE6CiAgbmFtZTogazhzYmxvY2t3aWxkY2FyZHNpbnJvbGVzYW5kY2x1c3RlcnJvbGVzCiAgYW5ub3RhdGlvbnM6CiAgICBkZXNjcmlwdGlvbjogQmxvY2sgd2lsZGNhcmRzIGluIFJvbGVzIGFuZCBDbHVzdGVyUm9sZSBWZXJicy4Kc3BlYzoKICBjcmQ6CiAgICBzcGVjOgogICAgICBuYW1lczoKICAgICAgICBraW5kOiBrOHNibG9ja3dpbGRjYXJkc2lucm9sZXNhbmRjbHVzdGVycm9sZXMKICB0YXJnZXRzOgogICAgLSB0YXJnZXQ6IGFkbWlzc2lvbi5rOHMuZ2F0ZWtlZXBlci5zaAogICAgICByZWdvOiB8CiAgICAgICAgcGFja2FnZSBrOHNibG9ja3dpbGRjYXJkc2lucm9sZXNhbmRjbHVzdGVycm9sZXMKCiAgICAgICAgdmlvbGF0aW9uW3sibXNnIjogbXNnLCAiZGV0YWlscyI6IHt9fV17CiAgICAgICAgaW5wdXQucmV2aWV3Lm9iamVjdC5raW5kID09ICJSb2xlIgogICAgICAgIGlucHV0LnJldmlldy5vYmplY3QucnVsZXNbX10udmVyYnNbX10gPT0gIioiCiAgICAgICAgbXNnIDo9ICJXaWxkY2FyZHMgYXJlIHByb2hpYml0ZWQgaW4gUm9sZXMgVmVyYnMiCiAgICAgICAgfQogICAgICAgIAogICAgICAgIHZpb2xhdGlvblt7Im1zZyI6IG1zZywgImRldGFpbHMiOiB7fX1dewogICAgICAgIGlucHV0LnJldmlldy5vYmplY3Qua2luZCA9PSAiUm9sZSIKICAgICAgICBpbnB1dC5yZXZpZXcub2JqZWN0LnJ1bGVzW19dLnJlc291cmNlc1tfXSA9PSAiKiIKICAgICAgICBtc2cgOj0gIldpbGRjYXJkcyBhcmUgcHJvaGliaXRlZCBpbiBSb2xlcyBSZXNvdXJjZXMiCiAgICAgICAgfQoKICAgICAgICB2aW9sYXRpb25beyJtc2ciOiBtc2csICJkZXRhaWxzIjoge319XXsKICAgICAgICBpbnB1dC5yZXZpZXcub2JqZWN0LmtpbmQgPT0gIkNsdXN0ZXJSb2xlIgogICAgICAgIGlucHV0LnJldmlldy5vYmplY3QucnVsZXNbX10udmVyYnNbX10gPT0gIioiCiAgICAgICAgbXNnIDo9ICJXaWxkY2FyZHMgYXJlIHByb2hpYml0ZWQgaW4gQ2x1c3RlclJvbGVzIFZlcmJzIgogICAgICAgIH0KICAgICAgICAKICAgICAgICB2aW9sYXRpb25beyJtc2ciOiBtc2csICJkZXRhaWxzIjoge319XXsKICAgICAgICBpbnB1dC5yZXZpZXcub2JqZWN0LmtpbmQgPT0gIkNsdXN0ZXJSb2xlIgogICAgICAgIGlucHV0LnJldmlldy5vYmplY3QucnVsZXNbX10ucmVzb3VyY2VzW19dID09ICIqIgogICAgICAgIG1zZyA6PSAiV2lsZGNhcmRzIGFyZSBwcm9oaWJpdGVkIGluIENsdXN0ZXJSb2xlcyBSZXNvdXJjZXMiCiAgICAgICAgfSAgICA=

The base64 encoded string can now be added to our Azure Policy Custom Definition file, that would look like this:

{
    "properties": {
        "displayName": "Custom Kubernetes - Restrict Wildcards in Verbs and Resources for Roles and ClusterRoles",
        "policyType": "Custom",
        "mode": "Microsoft.Kubernetes.Data",
        "description": "Custom Kubernetes - Restrict Wildcards in Verbs and Resources for Roles and ClusterRoles.",
        "metadata": {
            "version": "1.0.0",
            "category": "Kubernetes"
        },
        "parameters": {
            "effect": {
                "type": "String",
                "metadata": {
                    "displayName": "Effect",
                    "description": "'Audit' allows a non-compliant resource to be created or updated, but flags it as non-compliant. 'Deny' blocks the non-compliant resource creation or update. 'Disabled' turns off the policy."
                },
                "allowedValues": [
                    "audit",
                    "Audit",
                    "deny",
                    "Deny",
                    "disabled",
                    "Disabled"
                ],
                "defaultValue": "Audit"
            },
            "excludedNamespaces": {
                "type": "Array",
                "metadata": {
                    "displayName": "Namespace exclusions",
                    "description": "List of Kubernetes namespaces to exclude from policy evaluation. System namespaces \"kube-system\", \"gatekeeper-system\" and \"azure-arc\" are always excluded by design."
                },
                "defaultValue": [
                    "kube-system",
                    "gatekeeper-system",
                    "azure-arc"
                ]
            }
        },
        "policyRule": {
            "if": {
                "field": "type",
                "in": [
                    "Microsoft.Kubernetes/connectedClusters",
                    "Microsoft.ContainerService/managedClusters"
                ]
            },
            "then": {
                "effect": "[parameters('effect')]",
                "details": {
                    "templateInfo": {
                        "sourceType": "Base64Encoded",
                        "content": "YXBpVmVyc2lvbjogdGVtcGxhdGVzLmdhdGVrZWVwZXIuc2gvdjFiZXRhMQpraW5kOiBDb25zdHJhaW50VGVtcGxhdGUKbWV0YWRhdGE6CiAgbmFtZTogazhzYmxvY2t3aWxkY2FyZHNpbnJvbGVzYW5kY2x1c3RlcnJvbGVzCiAgYW5ub3RhdGlvbnM6CiAgICBkZXNjcmlwdGlvbjogQmxvY2sgd2lsZGNhcmRzIGluIFJvbGVzIGFuZCBDbHVzdGVyUm9sZSBWZXJicy4Kc3BlYzoKICBjcmQ6CiAgICBzcGVjOgogICAgICBuYW1lczoKICAgICAgICBraW5kOiBrOHNibG9ja3dpbGRjYXJkc2lucm9sZXNhbmRjbHVzdGVycm9sZXMKICB0YXJnZXRzOgogICAgLSB0YXJnZXQ6IGFkbWlzc2lvbi5rOHMuZ2F0ZWtlZXBlci5zaAogICAgICByZWdvOiB8CiAgICAgICAgcGFja2FnZSBrOHNibG9ja3dpbGRjYXJkc2lucm9sZXNhbmRjbHVzdGVycm9sZXMKCiAgICAgICAgdmlvbGF0aW9uW3sibXNnIjogbXNnLCAiZGV0YWlscyI6IHt9fV17CiAgICAgICAgaW5wdXQucmV2aWV3Lm9iamVjdC5raW5kID09ICJSb2xlIgogICAgICAgIGlucHV0LnJldmlldy5vYmplY3QucnVsZXNbX10udmVyYnNbX10gPT0gIioiCiAgICAgICAgbXNnIDo9ICJXaWxkY2FyZHMgYXJlIHByb2hpYml0ZWQgaW4gUm9sZXMgVmVyYnMiCiAgICAgICAgfQogICAgICAgIAogICAgICAgIHZpb2xhdGlvblt7Im1zZyI6IG1zZywgImRldGFpbHMiOiB7fX1dewogICAgICAgIGlucHV0LnJldmlldy5vYmplY3Qua2luZCA9PSAiUm9sZSIKICAgICAgICBpbnB1dC5yZXZpZXcub2JqZWN0LnJ1bGVzW19dLnJlc291cmNlc1tfXSA9PSAiKiIKICAgICAgICBtc2cgOj0gIldpbGRjYXJkcyBhcmUgcHJvaGliaXRlZCBpbiBSb2xlcyBSZXNvdXJjZXMiCiAgICAgICAgfQoKICAgICAgICB2aW9sYXRpb25beyJtc2ciOiBtc2csICJkZXRhaWxzIjoge319XXsKICAgICAgICBpbnB1dC5yZXZpZXcub2JqZWN0LmtpbmQgPT0gIkNsdXN0ZXJSb2xlIgogICAgICAgIGlucHV0LnJldmlldy5vYmplY3QucnVsZXNbX10udmVyYnNbX10gPT0gIioiCiAgICAgICAgbXNnIDo9ICJXaWxkY2FyZHMgYXJlIHByb2hpYml0ZWQgaW4gQ2x1c3RlclJvbGVzIFZlcmJzIgogICAgICAgIH0KICAgICAgICAKICAgICAgICB2aW9sYXRpb25beyJtc2ciOiBtc2csICJkZXRhaWxzIjoge319XXsKICAgICAgICBpbnB1dC5yZXZpZXcub2JqZWN0LmtpbmQgPT0gIkNsdXN0ZXJSb2xlIgogICAgICAgIGlucHV0LnJldmlldy5vYmplY3QucnVsZXNbX10ucmVzb3VyY2VzW19dID09ICIqIgogICAgICAgIG1zZyA6PSAiV2lsZGNhcmRzIGFyZSBwcm9oaWJpdGVkIGluIENsdXN0ZXJSb2xlcyBSZXNvdXJjZXMiCiAgICAgICAgfSAgICA="
                    },
                    "excludedNamespaces": "[parameters('excludedNamespaces')]",
                    "apiGroups": [
                        "rbac.authorization.k8s.io"
                    ],
                    "kinds": [
                        "Role",
                        "ClusterRole"
                    ]
                }
            }
        }
    }
}

Note:

The default effect of the policy is set to “Deny”, you can choose “Audit, if you wish.
This Policy is applied to Objects with Kind: Role or ClusterRole and the following Namespaces are excluded from inforcement by the policy:
- “kube-system”, “gatekeeper-system” and “azure-arc”

Assign and test the Policy

Add a Non-Compliant Role

Before proceeding with the assignment of the Custom Policy Definition, we will create a role that is non compliant prior to that, in order to test if it shows up in Azure Policy as *** Non Compliant*** after the assignement is performed.

To do so, provision and/or connect to an existing AKS Cluster and run the following command to create a dev namespace and add the K8s role to it:

cd k8s/rbac
kubectl create ns dev
kubectl apply -f role-dev-namespace.yaml

Create Custom Policy Definition in Azure Policy

Add the custom Policy Definition to your Azure environemt via the Azure portal

Navigate to Azure Policy, Definitions, and click on Add Definition Fill the form and paste the content of azpolicy.json in the Policy Rule section

Add Custom Definition

Assign the Custom Definition on the desired scope

Once the Definition has been added to Azure Policies, you can assign it on the desired Scope

Assign Custom Definition

The Parameters section would look like this:

Assign Custom Definition Params

After the Assignment is successful, we can proceed to testing the Policy

Test the Custom Policy

When the Assignement is complte - it takes about 30 minutes for the policy to be totally assigned on the scope, we will look at the Kubernetes Objects that has been created

kubectl get constrainttemplate k8sblockwildcardsinrolesandclusterroles

We can see the Constraint temaplate created by the Azure Policy in the cluster

Assign Custom Definition

Looking at the Constraint CRD [Custom Resource Definition]:

kubectl get k8sblockwildcardsinrolesandclusterroles.constraints.gatekeeper.sh

Constraint CRD

Non-Compliant showing in Azure Policy

if we navigate to the Policy section of the Subscription where the Policy was assigned, and where our AKS Cluster resides, we can see that a Non-Compliant status is shown for this cluster

Non Compliant

By clicking on Details, it will show the Non-Complient Objects

Non Compliant Details

Trying to create a Non-Compliant Role

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dev-user-full-access
  namespace: dev
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["batch"]
  resources:
  - jobs
  - cronjobs
  verbs: ["*"]

When we try to create a Role with Wildcards in Verbs or Resources, our action is denied by the Gatekeeperv3 Admission Control:

kubectl create ns test
kubectl apply -f k8s/rbac/role-test-namespace.yaml

deny-creation

Conclusion

In this post we took a closer look at how Custom Azure Policy Definitions for Kubernetes/AKS can be created and what are the components necessary for their creation, as well as what happens behind the scenes in your AKS Clusters covered by the Policy.

These Custom Defnitions can be powerfull tools to make AKS Environments more secure, and most importantly, compliant with Enterprise Regulatory and security Guidelines

Custom Azure Policy for Kubernetes [AKS]