Skip to main content

Secrets Encryption Config

K3s supports enabling secrets encryption at rest. When first starting the server, passing the flag --secrets-encryption will do the following automatically:

  • Generate an AES-CBC key
  • Generate an encryption config file with the generated key
  • Pass the config to the KubeAPI as encryption-provider-config
tip

Secrets-encryption cannot be enabled on an existing server without restarting it.
Use curl -sfL https://get.k3s.io | sh -s - server --secrets-encryption if installing from script, or other methods described in Configuration Options.

Example of the encryption config file:

{
"kind": "EncryptionConfiguration",
"apiVersion": "apiserver.config.k8s.io/v1",
"resources": [
{
"resources": [
"secrets"
],
"providers": [
{
"aescbc": {
"keys": [
{
"name": "aescbckey",
"secret": "xxxxxxxxxxxxxxxxxxx"
}
]
}
},
{
"identity": {}
}
]
}
]
}

Secrets Encryption Tool

Version Gate

Available as of v1.21.8+k3s1

K3s contains a utility tool secrets-encrypt, which enables automatic control over the following:

  • Disabling/Enabling secrets encryption
  • Adding new encryption keys
  • Rotating and deleting encryption keys
  • Reencrypting secrets
caution

Failure to follow proper procedure for rotating encryption keys can leave your cluster permanently corrupted. Proceed with caution.

Encryption Key Rotation

To rotate secrets encryption keys on a single-server cluster:

  • Start the K3s server with the flag --secrets-encryption
note

Starting K3s without encryption and enabling it at a later time is currently not supported.

  1. Prepare

    k3s secrets-encrypt prepare
  2. Kill and restart the K3s server with same arguments. If running K3s as a service:

    # If using systemd
    systemctl restart k3s
    # If using openrc
    rc-service k3s restart
  3. Rotate

    k3s secrets-encrypt rotate
  4. Kill and restart the K3s server with same arguments

  5. Reencrypt

    info

    K3s will reencrypt ~5 secrets per second.
    Clusters with large # of secrets can take several minutes to reencrypt.

    k3s secrets-encrypt reencrypt

Secrets Encryption Disable/Enable

After launching a server with --secrets-encryption flag, secrets encryption can be disabled.

To disable secrets encryption on a single-node cluster:

  1. Disable

    k3s secrets-encrypt disable
  2. Kill and restart the K3s server with same arguments. If running K3s as a service:

    # If using systemd
    systemctl restart k3s
    # If using openrc
    rc-service k3s restart
  3. Reencrypt with flags

    k3s secrets-encrypt reencrypt --force --skip

To re-enable secrets encryption on a single node cluster:

  1. Enable

    k3s secrets-encrypt enable
  2. Kill and restart the K3s server with same arguments

  3. Reencrypt with flags

    k3s secrets-encrypt reencrypt --force --skip

Secrets Encryption Status

The secrets-encrypt tool includes a status command that displays information about the current status of secrets encryption on the node.

An example of the command on a single-server node:

$ k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match

Active Key Type Name
------ -------- ----
* AES-CBC aescbckey

Another example on HA cluster, after rotating the keys, but before restarting the servers:

$ k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: hash does not match between node-1 and node-2

Active Key Type Name
------ -------- ----
* AES-CBC aescbckey-2021-12-10T22:54:38Z
AES-CBC aescbckey

Details on each section are as follows:

  • Encryption Status: Displayed whether secrets encryption is disabled or enabled on the node
  • Current Rotation Stage: Indicates the current rotation stage on the node.
    Stages are: start, prepare, rotate, reencrypt_request, reencrypt_active, reencrypt_finished
  • Server Encryption Hashes: Useful for HA clusters, this indicates whether all servers are on the same stage with their local files. This can be used to identify whether a restart of servers is required before proceeding to the next stage. In the HA example above, node-1 and node-2 have different hashes, indicating that they currently do not have the same encryption configuration. Restarting the servers will sync up their configuration.
  • Key Table: Summarizes information about the secrets encryption keys found on the node.
    • Active: The "*" indicates which, if any, of the keys are currently used for secrets encryption. An active key is used by Kubernetes to encrypt any new secrets.
    • Key Type: All keys using this tool are AES-CBC type. See more info here.
    • Name: Name of the encryption key.