Storage

Shared Object Storage

SD Elements makes use of Shared Object Storage via AWS S3 or an S3 compatible API object storage for sharing files between SD Elements microservices.

Requirements

  • An existing S3 bucket

  • An AWS IAM service account that has read/write access to the S3 bucket

  • The Access Key and Secret Key for the IAM service account

See Amazon S3: Allows read and write access to objects in an S3 Bucket for details on IAM policy configuration.

If you do not have access to AWS S3, see Alternative Configuration below for details.

S3 configuration

SD Elements can be configured to use S3 by modifying the follow section in your values.yaml overlay:

global:
  sharedStorage:
    bucketName: my-s3-bucket-name
    s3Url: https://s3.us-east-1.amazonaws.com
    s3AccessKey: AwsServiceAccountAccessKey
    s3SecretKey: AwsServiceAccountSecretKey

Enabling S3 Transfer Acceleration

To enable the use of S3 Transfer Acceleration in SD Elements when performing S3 operations, add the following environment in your values.yaml overlay:

worker:
  extraEnvVars:
    - name: S3_USE_ACCELERATE_ENDPOINT
      value: "true"

Alternative S3 Configuration

s3Url must be formatted in Amazon S3 Path-Style URL

You may wish to set up an IAM Policy to restrict service account to the specific S3 bucket

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:*"],
      "Resource": [
        "arn:aws:s3:::my-s3-bucket-name",
        "arn:aws:s3:::my-s3-bucket-name/*"
      ]
    }
  ]
}

If you are deploying in an environment without access to AWS S3 object storage, an alternative option is to use MinIO. If MinIO is already up and running in your environment, you can update the global.sharedStorage section to point to its endpoint:

global:
  sharedStorage:
    bucketName: my-bucket-name
    s3Url: http://minio-address:9000
    s3AccessKey: AccessKeyGoesHere
    s3SecretKey: SecretKeyGoesHere

Otherwise, SD Elements provides two ways to set up MinIO: 1. Using MinIO subchart within SD Elements. 2. Using MinIO Tenant subchart for MinIO Operator.

An explanation on how to set up MinIO with either of these methods is provided in the following sections.

Using MinIO subchart within SD Elements

In this scenario, you should configure both the global.sharedStorage and minio sections in your values.yaml overlay and ensure certain properties match.

minIO bucket naming conventions are the same as those of Amazon S3. See Amazon S3 bucket naming rules for more information.
minIO secretKey values must be at least 8 characters in length. 
global:
  sharedStorage:
    bucketName: my-bucket-name        # If using MinIO, ensure value matches a bucket in `minio` section
    s3Url: http://{release_name}-minio:9000
    s3AccessKey: AccessKeyGoesHere    # If using MinIO, ensure value matches `accessKey` in `minio` section
    s3SecretKey: SecretKeyGoesHere    # If using MinIO, ensure value matches `secretKey` in `minio` section
minio:
  enabled: true
  rootUser: admin
  rootPassword: Password
  persistence:
    storageClass: myStorageclassName
  buckets:
    - name: my-bucket-name            # should match global.sharedStorage.bucketName
      policy: none
      purge: false
  users:
    - accessKey: AccessKeyGoesHere    # should match global.sharedStorage.s3AccessKey
      secretKey: SecretKeyGoesHere    # should match global.sharedStorage.s3SecretKey
      policy: readwrite
  imagePullSecrets:
    - name: "security-compass-secret"

TLS can be enabled for minIO by providing the name of the secret containing the certificate and private key.

minio:
  ...
  tls:
    enabled: true
    certSecret: my-secret-name
    publicCrt: "tls.crt"
    privateKey: "tls.key"

If you do not have an external certificate secret, you may choose to use the self signed certificate provided by the Helm chart. In this configuration, SD Elements needs to be configured to trust third party CA certificates and the certificate added to the trust.

The name of the self-signed certificate is formatted based on the release name. 
global:
  thirdPartyCACertificates:
    enabled: true
    minioSelfSignedCertSecret: {release_name}-minio-server-tls-secrets
minio:
  ...
  tlsCreateSelfSigned: true
  tls:
    enabled: true
    certSecret: {release_name}-minio-server-tls-secrets

Alternatively, S3 certificate validation may be disabled.

worker:
  extraEnvVars:
    - name: AWS_S3_VERIFY
      value: "False"

Using MinIO Tenant subchart for MinIO Operator

In this scenario, you should configure both the global.sharedStorage and minio-tenant sections in your values.yaml overlay and ensure certain properties match.

This configuration is only compatible with SD Elements versions newer than 2023.4 and requires MinIO Operator to be already installed in the cluster. 
global:
  sharedStorage:
    bucketName: my-bucket-name
    s3Url: http://sdelements-minio-hl:9000
    s3AccessKey: AccessKeyGoesHere
    s3SecretKey: SecretKeyGoesHere
minio-tenant:
  enabled: true
  tenant:
    imagePullSecret:
      name: security-compass-secret
    env:
      - name: MINIO_BROWSER
        value: "off"                            # switch to "on" to enable the MinIO Console UI
    pools:
      - servers: 1                              # number of MinIO nodes running
        name: pool-0
        volumesPerServer: 1                     # number of volumes per MinIO node
        size: 100Gi                             # size of each volume
        storageClassName: myStorageclassName
    buckets:
      - name: my-bucket-name                    # should match global.sharedStorage.bucketName
    configuration:
      name: sdelements-minio-env-configuration  # should match the secret name below
  secrets:
    name: sdelements-minio-env-configuration
    accessKey: AccessKeyGoesHere                # should match global.sharedStorage.s3AccessKey
    secretKey: SecretKeyGoesHere                # should match global.sharedStorage.s3SecretKey

TLS can be enabled by providing the name of the secret containing the certificate and private key.

minio-tenant:
  ...
  tenant:
    ...
    certificate:
      externalCertSecret:
        - name: my-secret-name
          type: kubernetes.io/tls     # type can also be cert-manager.io/v1alpha2 or cert-manager.io/v1

For further details on how to create a TLS certificate, see MinIO’s documentation.

If you do not have an external certificate secret, you may choose to use the self-signed certificate generated by the Helm chart. In this configuration, SD Elements needs to disable checking S3 certificate validity.

minio-tenant:
  ...
  tenant:
    ...
    certificate:
      requestAutoCert: true
worker:
  extraEnvVars:
    - name: AWS_S3_USE_SSL
      value: "False"
In versions older than 2023.1, replace AWS_S3_VERIFY with AWS_S3_USE_SSL

AWS S3 endpoint configuration

When using AWS S3 as shared storage in an IPv6-only mode, the value for global.sharedStorage.s3Url must be adjusted to use dual-stack endpoints following the format s3.dualstack.AWS-REGION.amazonaws.com.

For instance, to access S3 in the us-east-1 region, the endpoint url should be updated to https://s3.dualstack.us-east-1.amazonaws.com.

global:
  sharedStorage:
    s3Url: https://s3.dualstack.us-east-1.amazonaws.com

Configuring an external database

This section has moved to Database.

results matching ""

    No results matching ""