Kubernetes Installation for Containerized Masking

This section describes how to utilize Delphix kubernetes images to deploy a containerized version of our Continuous Compliance Engine. Continuous Compliance and Masking are used interchangeably throughout these documents.

With a few small exceptions, Containerized Masking provides the same functionality and user experience as when deployed as a Virtual Machine Masking Engine.

Obtaining the images

Containerized Masking utilizes 3 integrated containers to deliver essentially the same masking experience of our Virtual Machine Masking Engine. The containerized form allows for rapid spin up / tear down of ephemeral engines to handle automated workflow deployments. The 3 containers are delivered in a compressed archive (.tar.gz) for convenience.

Licensed versions of these bundles are available for download from the download.delphix.com site. In the folder for each version are 2 files. One file is HTML with instructions very similar to this page that can be downloaded to provide an offline copy of the installation instructions. The second file is the masking_docker_images.tar.gz bundle which contains the container images.

Docker is employed to build the container images which produces a set of Open Source (OCI) images for each container. The intention is to make the contaiers as vendor independent as possible.

Setup

Containerized Masking is intended to be run as a pod on Kubernetes. The pod consists of three containers:

1. delphix-masking-app - Serves the application UI and API, and executes masking jobs.
2. delphix-masking-database - Stores various application configuration.
3. delphix-masking-proxy - Serves as a reverse proxy handling HTTP and HTTPS traffic for the UI and API.

The API and UI are served from internal ports 8080 and 8443. When deploying the application, the kubernetes config must provide a Service which directs external HTTP traffic to port 8080 and HTTPS traffic to port 8443 as shown in the example kubernetes-config.yaml file.

The pod also requires a single volume per instance. This storage should be attached to both the app container and the database container.

• This volume should be attached to the delphix-masking-database container at location /var/delphix/postgresql with a subpath of postgresql.
• This volume should be attached to the delphix-masking-app container twice. Once at location /var/delphix/masking/ with a subpath of masking and once at location /var/delphix/postgresql with a subpath of postgresql.

This volume should have at least 2GB of space for each container, though certain configurations may require significantly more space.

This storage volume should be created as a persistent volume. If it is not, masking job configurations will have to be recreated each time the pod is restarted. Also, certain diagnostic information captured in the logs will be lost when the pod is restarted unless the volume is persistent.

Because this volume is persistent, the pod should be deployed as a StatefulSet.

Network Management

The proxy container has built-in configurations to act as a reverse proxy. It is recommended that the main nginx.conf file remain unmodified; instead, modify the individual component configuration files that get incorporated into the main nginx.conf file through include statements (such as proxy.conf for the reverse proxy related configs and ssl.conf for HTTPS related configs).

To modify any nginx related files, such as config files or certificates and keys, an external volume should be bind mounted to the proxy container at /etc/config. During container startup, if the proxy container detects bind mounted files at the locations listed below, it will ignore the config files that are built into the proxy container's image and will instead use the mounted files.

HTTPS Certificates

If the proxy container does not detect an exernal certificate in the expected location, it will generate and use a self-signed certificate.

The expected locations of each file are shown below:

File	Description
/etc/config/nginx/nginx.conf	main configs file
/etc/config/nginx/proxy.conf	reverse proxy configs
/etc/config/nginx/ssl/ssl.conf	ssl configs
/etc/config/nginx/ssl/nginx.crt	ssl certificate
/etc/config/nginx/ssl/nginx.key	ssl private key
/etc/config/nginx/ssl/dhparam.pem	DH parameters file

OWASP CSRFGuard

The OWASP CSRFGuard product has been employed as part of the protections that are built-in to the Masking product. The supplied NginX proxy container rewrites a packet's Host header with the contents of the X-Forwarded-Host header, if it exists, so that CSRFGuard will accept proxied packets.

This results in a requirement. If the Pod is placed behind a proxy device which re-writes the Host header, that proxy must add an X-Forwarded-Host header containing the original host value.

Sample Configuration

The following configuration file shows an example of how Containerized Masking might be deployed. Details will vary based on use case, environment, and product version.

apiVersion: v1
kind: Service
metadata:
  name: delphix-masking
spec:
  type: NodePort
  selector:
    app: masking
  ports:
    - name: http
      port: 8080
      nodePort: 30080
    - name: https
      port: 8443
      nodePort: 30443
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: delphix-masking
spec:
  selector:
    matchLabels:
      app: masking
  serviceName: delphix-masking
  template:
    metadata:
      labels:
        app: masking
    spec:
      securityContext:
        runAsUser: 65436  # masking user
        runAsGroup: 50    # staff group
        fsGroup: 50
        #
        # Some volume providers, such as hostProvider, do not support fsGroup.
        # If you are using such a volume provider, use an init container to
        # change the ownership of each volume to 65436:50 and the permissions
        # to 775.
        #
        runAsNonRoot: true
      containers:
        - image: delphix-masking-database:6.0.16.0-c1
          name: mds
          ports:
            - containerPort: 5432
              name: mds
          volumeMounts:
            - name: masking-persistent-storage
              mountPath: /var/delphix/postgresql
              subPath: postgresql
        - image: delphix-masking-app:6.0.16.0-c1
          name: app
          ports:
            - containerPort: 8284
              name: http
          volumeMounts:
            - name: masking-persistent-storage
              mountPath: /var/delphix/masking
              subPath: masking
            - name: masking-persistent-storage
              mountPath: /var/delphix/postgresql
              subPath: postgresql
          startupProbe:
            httpGet:
              scheme: HTTPS
              path: /masking/api/system-information
              port: 8443
            failureThreshold: 30
            periodSeconds: 10
            timeoutSeconds: 10
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /masking/api/system-information
              port: 8443
            initialDelaySeconds: 300
            failureThreshold: 1
            periodSeconds: 10
            timeoutSeconds: 10
          readinessProbe:
            httpGet:
              scheme: HTTPS
              path: /masking/api/system-information
              port: 8443
            initialDelaySeconds: 30
            periodSeconds: 60
            timeoutSeconds: 10
        - image: delphix-masking-proxy:6.0.16.0-c1
          name: proxy
          ports:
          - containerPort: 8080
            name: http
          - containerPort: 8443
            name: https
  volumeClaimTemplates:
    - metadata:
        name: masking-persistent-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 4Gi

Deployment

Load the container images obtained from the download site into some Kubernetes container registry, then deploy the Masking Pod using a config file similar to the example provided above.

kubectl apply -f <path-to-config-file>

Debugging

In a support case, a Delphix Support engineer may ask for a support bundle containing diagnostic information. The preferred method of generating a support bundle is to use the API endpoints as shown in our document API Call for Generating a Support Bundle.

Please see our API Client Documentation for more information regarding using the API Client.

Generating and Retrieving a Support Bundle From the Command-line

However, if the API endpoints are not functioning properly or there are difficulties accessing them, a support bundle can be gathered by running the following command-line commands from the kubernetes layer of the node hosting the Pod: (kubernetes admin permissions are required to perform these actions)

$ kubectl exec -it <pod name> -c app -- /bin/bash /opt/delphix/masking/bin/generate_container_support_bundle.sh

The exact name of the tarball created by this command can then be found using kubectl exec. For example:

$ kubectl exec delphix-masking-0 -c app -- find /var/delphix/masking/ -name 'dlpx-support-*'
/var/delphix/masking/dlpx-support-4b3e2af2-1d00-43f5-b45b-c84dba62648a-20211201-18-21-53.tar.gz

The tarball can then be copied out of the pod using kubectl cp. For example:

$ kubectl cp delphix-masking-0:/var/delphix/masking/dlpx-support-4b3e2af2-1d00-43f5-b45b-c84dba62648a-20211201-18-21-53.tar.gz -c app dlpx-support-4b3e2af2-1d00-43f5-b45b-c84dba62648a-20211201-18-21-53.tar.gz

The tarball can then be provided to the Delphix Support engineer by uploading it to upload.delphix.com and adding the associated case number in the matching field.