What's New for Masking¶
6.0.5.0 Release¶
Character Mapping Algorithm:¶
Delphix is introducing a replacement for the Segment Mapping Algorithm, the Character Mapping Algorithm. The new Character Mapping Algorithm is built using the recently released algorithm SDK, and in most common configurations this new algorithm will be faster and require less memory than the existing segment mapping algorithm. In addition, this new version does not have a length limitation for the input string and can handle non-ASCII characters.
Certifications¶
- MySQL 8
- Postgres SQL 12
- DB2 LUW 11.5
- Oracle Database Cloud Services on Virtual Machines
- Oracle Database Cloud Services on Bare Metal
- Google Cloud SQL for PostgreSQL
- Google Cloud SQL for MySQL
- Google Cloud SQL for SQL Server
Default Api Version¶
Ability to specify the Masking API version to be used when the version is omitted from the base path of the Masking API request's URL.
New API Version¶
To reflect the API improvements mentioned above, the API version increased to 5.1.5 in this release. For a complete listing of version 5.1.5, see the Masking API Client page.
6.0.4.0 Release¶
Masking Job Memory Improvements¶
Memory management has been dramatically improved. Not only can jobs run with less memory, but the Masking Engine will also now ensure that jobs can only run if enough memory is available and that the engine cannot run out of memory.
Along with these changes, there are two new execution statuses: CANCELLED and QUEUED.
Extensible Connector Permissions Change¶
The first iteration of the Masking Extensible Connectors, supporting the ability to upload and use JDBC drivers, required that the permissions for each driver be enumerated at install time. Delphix has now replaced this mechanism with a fixed security policy blocking only the most dangerous permissions (specifically those that could inflict harm to the Masking Engine), removing the need for user management of permissions. It remains the case that the engine administrator must ensure that only trusted JDBC driver software is installed.
File Masking Performance¶
The performance of file masking has been significantly improved.
Builtin Extended Secure Lookup Framework¶
Delphix has added a builtin, configurable Secure Lookup Algorithm Framework, based on the Extended Algorithms feature (introduced in 6.0.3.0 release).
This framework provides better performance and new features when compared with the Legacy Secure Lookup Algorithms.
It allows configuring the case sensitivity of input values (true/false), and the case configuration of the output values:
Preserve Lookup File Case // i.e. as found in Lookup File
Preserve Input Case // i.e. preserve case of input value - UpperCase / LowerCase / Mixed
Force all Lowercase // forces output to LowerCase
Force all Uppercase // forces output to UpperCase
The algorithm instance (based on the new Secure Lookup Algorithm Framework) might be managed via the existing Algorithm API, similar to any other plugin algorithm. The GUI has been changed for configuring/editing Secure Lookup Algorithm. For details please see the Secure Lookup Algorithm Framework
Job Scheduler Removed:¶
As of this release, we have removed the Job Scheduler feature. The introduction of Masking’s REST API several releases ago allowed customers to schedule job executions using their preferred job scheduler. As a result, the integrated scheduler is seldom used.
Certifications¶
This release adds support for SQL Server 2017 and 2019.
Free Text Redaction Algorithm¶
The redaction strategies used in a free text redaction algorithm have been renamed to "Allowlist" and "Denylist".
New API Version¶
To reflect the API improvements mentioned above, the API version increased to 5.1.4 in this release. For a complete listing of version 5.1.4, see the Masking API Client page.
6.0.3.0 Release¶
Extended Algorithms¶
We introduced a new, radically simpler, method to create new masking algorithms. With the new framework, Delphix partners and customers can create and share new algorithms.
Extended algorithms and their related algorithm plugins can be managed through the following APIs:
| Group | Endpoints | Description |
|---|---|---|
| plugin | GET /plugin | Get all plugins |
| POST /plugin | Install plugin | |
| DELETE /plugin/{pluginId} | Delete plugin | |
| GET /plugin/{pluginId} | Get plugin detail by pluginId | |
| PUT /plugin/{pluginId} | Update plugin |
Existing algorithm API is extended with the following endpoints:
| Group | Endpoints | Description |
|---|---|---|
| algorithm | GET /algorithm/frameworks | Get all algorithm frameworks |
| GET /algorithm/frameworks/id/{frameworkId} | Get algorithm framework by frameworkId |
UI-based Environment Sync¶
Over the past several releases Delphix has introduced and refined the ability to synchronize objects between Masking Engines via the API. In 6.0.3, Delphix now supports importing and exporting environments via the UI.
Note
In this release, the deprecated XML import/export functionality has been removed. If you used the XML import/export feature in previous releases, you'll find the new Sync Environment feature to be a more robust and complete solution with complete API support in addition to being available in the UI.
New SQL Server JDBC Driver¶
The product switched from the jTDS JDBC driver to Microsoft's official open-source JDBC driver. This was done to obtain improved support for recent versions of SQL Server.
All SQL Server basic connectors will be converted transparently. If you used a SQL Server Advanced connector or a Generic connector using the jTDS driver, you will need to manually convert your JDBC URL to the Microsoft JDBC driver's format. To perform this conversion, please see the references for the jTDS parameters and the Microsoft JDBC parameters. Delphix Customer Support's upgrade validation checks will detect any SQL Server Advanced connectors and Generic connectors using the jTDS driver in your installation and they will notify you of the need to manually convert those connectors.
AzureSQL Managed Databases¶
This release is certified to be compatible with the following Azure SQL Managed Databases:
- Azure Database for PostgreSQL service
- Azure Database for MySQL service
- Azure Database for MariaDB service
- Azure Database for SQL
Note
You must enable support for non-TLS connections.
File Masking Performance¶
This release contains significant performance improvements for delimited and XML file masking.
New API Version¶
To reflect the API improvements mentioned above, the API version increased to 5.1.3 in this release. For a complete listing of version 5.1.3, see the Masking API Client page.
6.0.2.0 Release¶
Mainframe Data Set Improvements for Masking¶
This release delivers multiple quality-of-experience enhancements around mainframe masking workflows:
-
Mainframe Masking Performance: Anyone masking mainframe data sets may see a large improvement in performance.
-
Engine Sync Support for Mainframe: The Sync APIs and workflows now support mainframe objects: connectors, rule sets, jobs, and formats.
-
Mainframe Data Set Record Type APIs: This enhancement builds upon the recent release of Record Type APIs to include mainframe support. You will now be able to manage Mainframe data set record types via REST API, including redefine conditions. When masking a mainframe data set, the Masking Engine uses a mainframe data set format to interpret the data set's contents. A mainframe data set format has one default record type "All Record". If a mainframe data set format contains redefined fields, each redefined and redefines field will have a corresponding record type that holds the redefined condition for the redefined and redefines fields. Specifically, the following APIs were added:
| Group | Endpoints | Description |
|---|---|---|
| mainframeDatasetRecordType | GET /mainframe-dataset-record-types | Get all Mainframe Dataset record type |
| GET /mainframe-dataset-record-types/{mainframeDatasetRecordTypeId} | Get Mainframe Dataset record type by ID | |
| PUT /mainframe-dataset-record-types/{mainframeDatasetRecordTypeId} | Update Mainframe Dataset record type by ID |
For more information on redefine conditions, see the Managing a Mainframe Inventory section.
JDBC to Delimited Files Support¶
On-the-fly masking jobs with a JDBC source and delimited file target are now supported. This is targeted at users with data lake applications. This is targeted at users with data lake applications who wish to extract unmasked data using a JDBC connection and insert masked data back using a bulk file load mechanism.
Environment Sync Support for Masking¶
With this release, an entire environment is now syncable with a single operation via the Sync REST APIs. Previously, Sync users would have to export/import objects on an individual basis, the process now is far more streamlined. Note: Environment Sync APIs are the preferred way of handling environment export/import versus XML-based transfer.
New API Version¶
To reflect the API improvements mentioned above, the API version increased to 5.1.2 in this release. For a complete listing of version 5.1.2, see the Masking API Client page.
Certifications¶
This release adds support for Oracle 19c.
6.0.1.0 Release¶
Extended Connectors¶
Extended Connectors is a new feature that allows you to upload additional JDBC Drivers to the Delphix Masking engine. This enables masking data sources that are not natively supported by Delphix Masking. For more information, please refer to the Managing Extended Connectors section.
Sync for Tokenization and Reidentification Jobs¶
The Sync feature allows you to coordinate the operation of multiple engines. This release adds Sync support for Tokenization and Reidentification Jobs. For more information on the Sync feature, please refer to the Managing Multiple Engines for Masking section.
File Record Type APIs¶
When masking a delimited or fixed length file, the Masking Engine uses a file format to interpret the file's contents. Each format has one or more record types. In previous releases, these record types could only be created and managed through the graphical user interface. This release adds the ability to also create and manage file record types through the APIs. Specifically, the following APIs were added:
| Group | Endpoints | Description |
|---|---|---|
| recordType | GET /record-types | Get all record type |
| POST /record-types | Create record type | |
| DELETE /record-types/{recordTypeId} | Delete record type by ID | |
| GET /record-types/{recordTypeId} | Get record type by ID | |
| PUT /record-types/{recordTypeId} | Update record type | |
| recordTypeQualifier | GET /record-type-qualifiers | Get all record type qualifiers |
| POST /record-type-qualifiers | Create record type qualifier | |
| DELETE /record-type-qualifiers/{recordTypeQualifierId} | Delete record type qualifier by ID | |
| GET /record-type-qualifiers/{recordTypeQualifierId} | Get record type qualifier by ID | |
| PUT /record-type-qualifiers/{recordTypeQualifierId} | Update record type qualifier by ID |
Note that record types are only used for delimited and fixed length file formats. For more information on record types, see the Adding Record Types for Files section.
6.0.0.0 Release¶
Objects Names Requirements¶
In 6.0 we have added validations for objects names that can be created/renamed manually. For more information please refer to Naming Requirements. Please pay attention to the fact that enforcing these requirements might fail the import, sync, or upgrade from pre-6.0 release. Please refer to the following Knowledge Base Article KBA5096 on how to solve those failures.
Versioning Framework¶
6.0 marks the release of version 5.1 of the Masking API. For information on how the Masking API is versioned, please refer to the documentation here: Masking API Versioning Documentation
New API Endpoints¶
In 6.0 we have expanded the list of API endpoints to include:
| Group | Endpoints | Description |
|---|---|---|
| Application | DELETE /applications/{applicationId} | Delete application by ID |
| Mount Filesystem | GET /mount-filesystem | Get all mounts |
| POST /mount-filesystem | Create a mount | |
| GET /mount-filesystem/{mountId} | Get a mount by ID | |
| DELETE /mount-filesystem/{mountId} | Delete a mount by ID | |
| PUT /mount-filesystem/{mountId} | Update a mount by ID | |
| PUT /mount-filesystem/{mountId}/connect | Connect a mount by ID | |
| PUT /mount-filesystem/{mountId}/disconnect | Disconnect a mount by ID | |
| PUT /mount-filesystem/{mountId}/remount | Remount a mount by ID |
In addition to the new API endpoints, we have improved existing API endpoints. These improvements include:
- Addition of the applicationId field to the application model
- Replacement of the application field with an applicationId field in the Environment model
- Removal of the classification field from the domain model
- Addition of the rulesetType field to the Masking, Profiling, Reidentification, and Tokenization job models.
- Addition of mountName in the ConnectionInfo of a file connector and a mainframe dataset connector to use a filesystem mount point.
For more information on Delphix Masking APIs please see the API documentation.
NFS and CIFS Mounts¶
In previous releases, the Masking Engine has supported masking files via FTP or SFTP. In this release, we have added the ability for users to directly mount and mask a file system over NFS and CIFS. This should dramatically simplify the process of file masking. As with other Masking Engine objects, the Sync feature can be used to coordinate mount objects across multiple engines. For more information on the mount feature, please refer to the Managing Remote Mounts section.
5.3 Release¶
Synchronizing Masking Jobs and Universal Settings Across Engines¶
In 5.2 we introduced the ability to synchronize Masking Algorithms between engines to ensure consistent masking, regardless of the engine executing the masking. In 5.3 we are expanding the list of syncable objects to include:
- Masking Jobs
- Connectors
- Rulesets
- Domains
- File Formats
The sync of objects is possible through improvements to several sync API endpoints, including:
- GET /syncable-objects[?object_type=
] - POST /export
- POST /export-async
- POST /import
- POST/import-async
This expansion of syncable objects ensures that users can sync their Masking Jobs and all the objects necessary for that masking job to execute successfully - regardless of the masking engine it lives on, allowing for easier scaling of Delphix Masking across the enterprise. Please see Managing Multiple Masking Engines for more details.
Support for Kerberized Connections¶
In 5.2.4 we added support for Kerberos for our Oracle Masking Connector. In 5.3 we have expanded the list of connectors that support Kerberos to:
- SQL Server
- Sybase
To enable Kerberized connectors your engine must be configured properly and you must configure your masking Connectors for Kerberos. Kerberos can be enabled by going to the Advanced mode on Oracle, SQL Server and Sybase. Please see Managing Connectors for more details.
New API Endpoints¶
In 5.2 we released an all-new set of API endpoints allowing for the automation of many masking workflows. In 5.3 we have expanded this list of API endpoints around Algorithms, Users, Roles, File Upload, System Information, Login, Rulesets, and Connector. Below are the net new API endpoints:
| Group | Endpoints | Description |
|---|---|---|
| Algorithms | POST /algorithms | Create algorithm |
| DELETE /algorithms/{algorithmName} | Delete algorithm by name | |
| GET /algorithms/{algorithmName} | Get algorithm by name | |
| PUT /algorithms/{algorithmName} | Update algorithm by name | |
| PUT /algorithms/{algorithmName}/randomize-key | Randomize key by name | |
| Users | GET /users | Get all users |
| POST /users | Create user | |
| DELETE /users/{userId} | Delete user by ID | |
| GET /users/{userId} | Get user by ID | |
| PUT /users/{userId} | Update user by ID | |
| Roles | GET /roles | Get all roles |
| POST /roles | Create role | |
| DELETE /roles/{roleId} | Delete role by ID | |
| GET /roles/{roleId} | Get role by ID | |
| PUT /roles/{roleId} | Update role by ID | |
| Rulesets | PUT /database-rulesets/{databaseRulesetId}/bulk-table-update | Update the rule set’s tables |
| PUT /database-rulesets/{databaseRulesetId}/refresh | Refresh the rule set | |
| Connectors | POST /database-connectors/{databaseConnectorId}/test | Test a database connector |
| POST /database-connectors/test | Test an unsaved database connector | |
| POST /file-connectors/{fileConnectorId}/test | Test a file connector | |
| POST /file-connectors/test | Test an unsaved file connector | |
| Async Tasks | GET /async-tasks | Get all asyncTasks |
| GET /async-tasks/{asyncTaskId} | Get asyncTask by ID | |
| PUT /async-tasks/{asyncTaskId}/cancel | Cancel asyncTask by ID | |
| File Upload/Download | DELETE /file-uploads | Delete all file uploads |
| POST /file-uploads | Upload file | |
| GET /file-downloads/{fileDownloadId} | Download file | |
| System Information | GET /system-information | Get version, etc. |
| Login/Logout | PUT /logout | User logout |
| Executions | GET /execution-components | Status for a table, file, or Mainframe data set |
| Tokenization Job | GET /tokenization-jobs | Get all tokenization jobs |
| POST /tokenization-jobs | Create tokenization job | |
| DELETE /tokenization-jobs/{tokenizationJobid} | Delete tokenization job by ID | |
| GET /tokenization-jobs/{tokenizationJobid} | Get tokenization job by ID | |
| PUT /tokenization-jobs/{tokenizationJobid} | Update tokenization job by ID | |
| Re-identification Job | GET /reidentification-jobs | Get all re-identification jobs |
| POST /reidentification-jobs | Create re-identification job | |
| DELETE /reidentification-jobs/{reidentificationJobid} | Delete re-identification job by ID | |
| GET /reidentification-jobs/{reidentificationJobid} | Get re-identification job by ID | |
| PUT /reidentification-jobs/{reidentificationJobid} | Update re-identification job by ID | |
| Database Rulesets | PUT | Update Database Ruleset by ID |
In addition to the net new API endpoints, we have improved pre-existing API endpoints. Some of the improvements include:
- Addition of DB2 iSeries and Mainframe to connector endpoints.
- Addition of Kerberos configuration on Oracle, SQL Server, and Sybase connectors
- Ability to have ruleset refresh drop tables
- Support for XML file types
- Addition of dataType to column metadata
- Addition of isProfilerWritable field to file-field-metadata endpoints. This is now represented in the API as a new isProfilerWritable boolean field in the body of a file-field-metadata. When the isProfilerWritable field is set to true, the algorithm/domain assignment on a column can be overwritten by the profiler. When the field is false, it may not be overwritten.
- Addition of multipleProfilerCheck field to Profile Job endpoints. This feature is turned on using the boolean field in the body of a profile job. The job profiler normally stops profiling a column as soon as it flags a field as sensitive. If multipleProfilerCheck is true, the profiler will continue to scan the column for additional sensitive patterns. In the event that it finds more than one pattern, it will tag all the data domains found and apply 'one' standard algorithm for all those domains. The standard algorithm is ‘Null SL’ as of 5.3.4.0. This feature was formerly called ‘multi PHI’.
For more information on Delphix Masking APIs please see the API documentation. Please note that the previous generation of Masking APIs (commonly referred to as V4) is EOL and no longer supported in this release. All users are encouraged to migrate to the V5 APIs.