Common Tasks

These sections provide information about common tasks that users and administrators may need to perform to manage SD Elements.

Retrieve Default Helm Chart Values

Whereas the Chart Files reference page provides more common chart configurations and their defaults, the SD Elements Helm chart’s default values file provides a more complete list of values that may be configured.

The helm show values command will print a Helm chart’s default values file to the console. See the Helm Docs Helm Show Values page for usage instructions.

Append > {filename}.yaml to save these values to a file on the client’s filesystem.

In the example below, the default values for the specified SD Elements version is saved to the user’s file system in the current directory.

$ helm show values sdelements-dev/sde --version {SDE_VERSION}

Retrieve Superuser Credentials for an Existing Instance

Username

Refer to sde.superuserEmail in the custom values file used to install SD Elements. If a username was not set, the default value was likely used. See Retrieve Default Helm Chart Values for instructions.

Password

The command below retrieves the Superuser password from the {RELEASE_NAME}-sde-secret and uses the base64 command to decode it.

The base64 command is only supported on Unix-based systems. Windows users must find an alternative method of decoding base64. 
# Superuser password
$ kubectl get secrets {RELEASE_NAME}-sde-secrets -n {NAMESPACE} --output jsonpath='{.data.SDE_SUPERUSER_PASSWORD}' | base64 -d

Retrieve Database Storage Usage

For SD Elements instances using the built-in {RELEASE_NAME}-database statefulset there are two methods to check storage usage.

Database Persistent Volume Usage

To retrieve the used and available space in the database’s persistent volume, exec into the database pod and use df.

$ kubectl exec -n {NAMESPACE} {RELEASE_NAME}-database-0 -- df -h
Defaulted container "postgresql" out of: postgresql, metrics, add-ro-user, init-chmod-data (init), shared-init (init)
Filesystem   Size Used Avail Use% Mounted on
overlay     100G  33G  68G 33% /
tmpfs      64M   0  64M  0% /dev
tmpfs      16G   0  16G  0% /sys/fs/cgroup
/dev/nvme0n1p1 100G  33G  68G 33% /tmp
/dev/nvme1n1  296G  29G 267G 10% /bitnami/postgresql
tmpfs      30G 1.5M  30G  1% /dev/shm
tmpfs      16G   0  16G  0% /proc/acpi
tmpfs      16G   0  16G  0% /sys/firmware

In the example above, the database persistent volume is using 33GB of the 100GB provisioned.

Postgres Database Size

To retrieve the size of the Postgres database, exec into the database pod and use du to check the /bitnami/postgresql/pgdata directory in kilobytes (KB).

kubectl exec -n {NAMESPACE} {RELEASE_NAME}-database-0 -- du -s /bitnami/postgresql/pgdata/
30219644	/bitnami/postgresql/pgdata/

In the example above, the /bitnami/postgresql/pgdata directory is 30219644KB (30.2GB).

Soft Maintenance Mode

Soft Maintenance Mode restricts access to SD Elements during maintenance activities. When enabled, the login page is redirected to a soft error page notifying that scheduled maintenance is taking place. This will allow these administrators to complete maintenance without concern for data integrity being compromised or users contacting internal support teams. Users included in the whitelist can access SD Elements regardless. Follow the steps below to configure worker pods for whitelisted users and enable Soft Maintenance Mode.

Configure Soft Maintenance Mode

Add the following environment variable to worker-10 pod configurations in values.yaml with a list of users that are in the allow list

worker:
    extraEnvVars:
    -   name: SDE_SOFT_MAINTENANCE_WHITELISTED_USERS
        value: user1@email.domain.com,user2@email.domain.com

Enable, Disable and check Status of Soft Maintenance Mode

  • For running soft maintenance mode commands, first exec into worker-10 pod by running the following

    # Get the worker-10 pod name
POD_NAME=$(kubectl get pods -o custom-columns=":metadata.name" | grep worker-10)
# Exec into the pod
kubectl exec -it ${POD_NAME} -- /bin/bash

  • To enable Soft Maintenance mode, run the following in the worker-10 pod

    # Run the enable command
/bin/sde.sh soft-maintenance on

  • To disable Soft Maintenance mode, run the following in the worker-10 pod

    # Run the disable command
/bin/sde.sh soft-maintenance off

  • To check status of Soft Maintenance mode, run the following in the worker-10 pod

    # Run command to check status
/bin/sde.sh soft-maintenance

Rotate the SECRET_KEY encryption key

SD Elements encrypts sensitive fields at rest (integration credentials, webhook URLs, SSO bind passwords, SAML key pairs, and similar) using the application secret key (sde.secretKey). Rotating the key re-encrypts every one of these fields under a new key.

You do not run the re-encryption by hand. When you deploy a new sde.secretKey with the previous key still present in sde.secretKeyFallbacks, SD Elements re-encrypts the data automatically during pod startup, under a database advisory lock so only one pod performs the rotation while the others wait. sde.secretKeyFallbacks supplies the old key that decrypts the existing data; the new sde.secretKey is what it is rewritten under.

Rotation rewrites every encrypted field in place and is not reversible. Take a database backup before you begin. Restoring that backup is the only clean recovery if the wrong previous key is supplied.
These steps use helm and kubectl directly. On a Container Lite (MicroK8s) appliance, prefix the commands with microk8s (for example microk8s helm upgrade …​) and use /etc/sde/custom.yaml as the values file. See the Container Lite guide for the appliance-specific commands.

Rotate the key

  1. Take a database backup.

  2. Prevent background writers from writing encrypted data under the old key during the rotation. A RollingUpdate keeps old-key pods running until the new pods are ready, and Soft Maintenance mode does not stop the Celery workers; so an old-key worker can write a field that the rotation has already passed. For the rotation deploy, either set the web and worker deployment strategy to Recreate (old pods stop before new ones start), or scale the workers to 0 and suspend the cronjobs for the duration.

  3. In your custom values file (values.custom.yaml), copy the current sde.secretKey into sde.secretKeyFallbacks, then set sde.secretKey to the new key:

    sde:
  secretKey: "{NEW_SECRET_KEY}"
  # the key you are rotating away from
  secretKeyFallbacks: "{CURRENT_SECRET_KEY}"

  4. Deploy. Startup re-encrypts the data automatically:

    $ helm upgrade {RELEASE_NAME} sdelements/sde -n {NAMESPACE} -f values.custom.yaml --version {SDE_VERSION}

  5. Wait for the rollout to complete and confirm the rotation finished before removing the old key. With sde.secretKeyFallbacks still set, exec into the worker-10 pod and run the check:

    POD_NAME=$(kubectl get pods -n {NAMESPACE} -o custom-columns=":metadata.name" | grep worker-10)
kubectl exec -it ${POD_NAME} -n {NAMESPACE} -- /bin/sde.sh rotate_encryption_keys --check

    A "no rows pending" result means every encrypted field is now under the new key. A "rotation pending" result means some rows were written under the old key during the rollout; re-deploy (or re-run the rotation) until the check reports nothing pending. Do not proceed until it is clean.

  6. Confirm the rollout fully completed before removing the old key. The --check above reflects only the key held by the pod you exec into; a pod that has not restarted still holds the previous key in memory and cannot read the re-encrypted data. Confirm that every web, worker, and cronjob pod restarted after the helm upgrade:

    kubectl get pods -n {NAMESPACE} --sort-by=.metadata.creationTimestamp

    No pod should predate the deploy. Restart any that lag and wait for them to become ready before continuing:

    $ kubectl rollout restart deployment {DEPLOYMENT_NAME} -n {NAMESPACE}

  7. Remove the old key: clear sde.secretKeyFallbacks in values.custom.yaml and deploy again:

    sde:
  secretKey: "{NEW_SECRET_KEY}"
  secretKeyFallbacks: ""
    $ helm upgrade {RELEASE_NAME} sdelements/sde -n {NAMESPACE} -f values.custom.yaml --version {SDE_VERSION}
sde.secretKeyFallbacks also validates data signed by Django itself; active user sessions, password-reset links, and similar tokens. Removing the old key (the final step) invalidates anything signed under it, so users are logged out and outstanding reset links stop working. This is expected; schedule the rotation accordingly.
If the check reports rows that decrypt under neither the current key nor the fallback, the value in sde.secretKeyFallbacks is not the key that data was encrypted with. Correct it and re-deploy. If the correct previous key is unavailable, restore the pre-rotation database backup.

results matching ""

    No results matching ""