Skip to main content

CIS Hardening Guide

This document provides prescriptive guidance for hardening a production installation of K3s. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Internet Security (CIS).

K3s has a number of security mitigations applied and turned on by default and will pass a number of the Kubernetes CIS controls without modification. There are some notable exceptions to this that require manual intervention to fully comply with the CIS Benchmark:

  1. K3s will not modify the host operating system. Any host-level modifications will need to be done manually.
  2. Certain CIS policy controls for NetworkPolicies and PodSecurityStandards (PodSecurityPolicies on v1.24 and older) will restrict the functionality of the cluster. You must opt into having K3s configure these by adding the appropriate options (enabling of admission plugins) to your command-line flags or configuration file as well as manually applying appropriate policies. Further details are presented in the sections below.

The first section (1.1) of the CIS Benchmark concerns itself primarily with pod manifest permissions and ownership. K3s doesn't utilize these for the core components since everything is packaged into a single binary.

Host-level Requirements​

There are two areas of host-level requirements: kernel parameters and etcd process/directory configuration. These are outlined in this section.

Ensure protect-kernel-defaults is set​

This is a kubelet flag that will cause the kubelet to exit if the required kernel parameters are unset or are set to values that are different from the kubelet's defaults.

Note: protect-kernel-defaults is exposed as a top-level flag for K3s.

Set kernel parameters​

Create a file called /etc/sysctl.d/90-kubelet.conf and add the snippet below. Then run sysctl -p /etc/sysctl.d/90-kubelet.conf.

vm.panic_on_oom=0
vm.overcommit_memory=1
kernel.panic=10
kernel.panic_on_oops=1

Kubernetes Runtime Requirements​

The runtime requirements to comply with the CIS Benchmark are centered around pod security (via PSP or PSA), network policies and API Server auditing logs. These are outlined in this section.

By default, K3s does not include any pod security or network policies. However, K3s ships with a controller that will enforce network policies, if any are created. K3s doesn't enable auditing by default, so audit log configuration and audit policy must be created manually. By default, K3s runs with the both the PodSecurity and NodeRestriction admission controllers enabled, among others.

Pod Security​

K3s v1.25 and newer support Pod Security Admissions (PSAs) for controlling pod security. PSAs are enabled by passing the following flag to the K3s server:

--kube-apiserver-arg="admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml"

The policy should be written to a file named psa.yaml in /var/lib/rancher/k3s/server directory.

Here is an example of a compliant PSA:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1beta1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "restricted"
      enforce-version: "latest"
      audit: "restricted"
      audit-version: "latest"
      warn: "restricted"
      warn-version: "latest"
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces: [kube-system, cis-operator-system]

Note: The Kubernetes critical additions such as CNI, DNS, and Ingress are run as pods in the kube-system namespace. Therefore, this namespace will have a policy that is less restrictive so that these components can run properly.

NetworkPolicies​

CIS requires that all namespaces have a network policy applied that reasonably limits traffic into namespaces and pods.

Network policies should be placed the /var/lib/rancher/k3s/server/manifests directory, where they will automatically be deployed on startup.

Here is an example of a compliant network policy.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: intra-namespace
  namespace: kube-system
spec:
  podSelector: {}
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            name: kube-system

With the applied restrictions, DNS will be blocked unless purposely allowed. Below is a network policy that will allow for traffic to exist for DNS.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-network-dns-policy
  namespace: <NAMESPACE>
spec:
  ingress:
  - ports:
    - port: 53
      protocol: TCP
    - port: 53
      protocol: UDP
  podSelector:
    matchLabels:
      k8s-app: kube-dns
  policyTypes:
  - Ingress

The metrics-server and Traefik ingress controller will be blocked by default if network policies are not created to allow access. Traefik v1 as packaged in K3s version 1.20 and below uses different labels than Traefik v2. Ensure that you only use the sample yaml below that is associated with the version of Traefik present on your cluster.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-metrics-server
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      k8s-app: metrics-server
  ingress:
  - {}
  policyTypes:
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-svclbtraefik-ingress
  namespace: kube-system
spec:
  podSelector: 
    matchLabels:
      svccontroller.k3s.cattle.io/svcname: traefik
  ingress:
  - {}
  policyTypes:
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-traefik-v121-ingress
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: traefik
  ingress:
  - {}
  policyTypes:
  - Ingress
---

info

Operators must manage network policies as normal for additional namespaces that are created.

API Server audit configuration​

CIS requirements 1.2.22 to 1.2.25 are related to configuring audit logs for the API Server. K3s doesn't create by default the log directory and audit policy, as auditing requirements are specific to each user's policies and environment.

The log directory, ideally, must be created before starting K3s. A restrictive access permission is recommended to avoid leaking potential sensitive information.

sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs

A starter audit policy to log request metadata is provided below. The policy should be written to a file named audit.yaml in /var/lib/rancher/k3s/server directory. Detailed information about policy configuration for the API server can be found in the Kubernetes documentation.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

Both configurations must be passed as arguments to the API Server as:

kube-apiserver-arg:
  - 'admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml'
  - 'audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log'
  - 'audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml'
  - 'audit-log-maxage=30'
  - 'audit-log-maxbackup=10'
  - 'audit-log-maxsize=100'

K3s must be restarted to load the new configuration.

sudo systemctl daemon-reload
sudo systemctl restart k3s.service

Configuration for Kubernetes Components​

The configuration below should be placed in the configuration file, and contains all the necessary remediations to harden the Kubernetes components.

protect-kernel-defaults: true
secrets-encryption: true
kube-apiserver-arg:
  - "enable-admission-plugins=NodeRestriction,EventRateLimit"
  - 'admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml'
  - 'audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log'
  - 'audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml'
  - 'audit-log-maxage=30'
  - 'audit-log-maxbackup=10'
  - 'audit-log-maxsize=100'
kube-controller-manager-arg:
  - 'terminated-pod-gc-threshold=10'
kubelet-arg:
  - 'streaming-connection-idle-timeout=5m'
  - "tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"

Manual Operations​

The following are controls that K3s currently does not pass by with the above configuration applied. These controls require manual intervention to fully comply with the CIS Benchmark.

Control 1.1.20​

Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual)

Details

Remediation K3s PKI certificate files are stored in /var/lib/rancher/k3s/server/tls/ with permission 644. To remediate, run the following command:

chmod -R 600 /var/lib/rancher/k3s/server/tls/*.crt

Control 1.2.9​

Ensure that the admission control plugin EventRateLimit is set

Details

Remediation Follow the Kubernetes documentation and set the desired limits in a configuration file. For this and other psa configuration, this documentation uses /var/lib/rancher/k3s/server/psa.yaml. Then, edit the K3s config file /etc/rancher/k3s/config.yaml and set the below parameters.

kube-apiserver-arg:
  - "enable-admission-plugins=NodeRestriction,EventRateLimit"
  - "admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml"

Control 1.2.11​

Ensure that the admission control plugin AlwaysPullImages is set

Details

Remediation Permissive, per CIS guidelines, "This setting could impact offline or isolated clusters, which have images pre-loaded and do not have access to a registry to pull in-use images. This setting is not appropriate for clusters which use this configuration." Edit the K3s config file /etc/rancher/k3s/config.yaml and set the below parameter.

kube-apiserver-arg:
  - "enable-admission-plugins=...,AlwaysPullImages,..."

Control 1.2.21​

Ensure that the --request-timeout argument is set as appropriate

Details

Remediation Permissive, per CIS guidelines, "it is recommended to set this limit as appropriate and change the default limit of 60 seconds only if needed". Edit the K3s config file /etc/rancher/k3s/config.yaml and set the below parameter if needed. For example,

kube-apiserver-arg:
  - "request-timeout=300s"

Control 4.2.13​

Ensure that a limit is set on pod PIDs

Details

Remediation Decide on an appropriate level for this parameter and set it, If using a K3s config file /etc/rancher/k3s/config.yaml, edit the file to set podPidsLimit to

kubelet-arg:
  - "pod-max-pids=<value>"

Control 5.X​

All the 5.X Controls are related to Kubernetes policy configuration. These controls are not enforced by K3s by default.

Refer to CIS 1.8 Section 5 for more information on how to create and apply these policies.

Conclusion​

If you have followed this guide, your K3s cluster will be configured to comply with the CIS Kubernetes Benchmark. You can review the CIS 1.8 Self-Assessment Guide to understand the expectations of each of the benchmark's checks and how you can do the same on your cluster.