Skip to content

Free Text Redaction

A Free Text Redaction algorithm helps you remove sensitive data that appears in free-text columns such as “Notes.” This type of algorithm requires some expertise to use because you must set it to recognize sensitive data within a block of text.

One challenge is that individual words might not be sensitive on their own, but together they can be. The algorithm uses profiler sets to determine what information it needs to mask. You can decide which expressions the algorithm uses to search for material such as addresses. For example, you can set the algorithm to look for “St,” “Cir,” “Blvd,” and other words that suggest an address. You can also use pattern matching to identify potentially sensitive information. For example, a number that takes the form 123-45-6789 is likely to be a Social Security Number.

You can use a Free Text Redaction algorithm to show or hide information by displaying either a “denylist” or an “allowlist.”

Denylist – Designated material will be redacted (removed). For example, you can set a deny list to hide patient names and addresses. The deny list feature will match the data in the lookup file to the input file.

Allowlist – ONLY designated material will be visible. For example, if a drug company wants to assess how often a particular drug is being prescribed, you can use an allow list so that only the name of the drug will appear in the notes. The allow list feature enables you to mask data using both the lookup file and a profile set.

For either option, a list of words can be imported from an external text file, or alternatively, you can use Profiler Sets to match words based on regular expressions, defined within Profiler Expressions. You can also specify the redaction value that will replace the masked words. Regular expressions defined using Profiler Sets will match individual words within the input text, rather than phrases.

Creating a Free Text Redaction Algorithm via UI

  1. Enter an Algorithm Name.

  2. Enter a Description.

  3. Select the Deny List or Allow List radio button.

  4. Select Lookup File and enter Redaction Value OR/AND

  5. Select Profiler Sets from the drop-down menu and enter Redaction Value.

  6. Click Save.

For information on creating Free Text Redaction algorithms through the API, see API Calls for Creating Algorithms - Free Text Redaction.

Examples

  1. Create an Input file.

  2. Create an Input file using Notepad. Enter the following text:

    The customer Bob Jones is satisfied with the terms of the sales
    agreement. Please call to confirm at 718-223-7896.
    
  3. Save the file as txt.

  4. Create lookup file.

    1. Create a lookup file.

    2. Use Notepad to create a text file and save the file as a txt. Be sure to hit return after each field. The lookup flat file contains the following data:

      Bob
      Jones
      Agreement
      
Create an Algorithm

You will be prompted for the following information:

  1. For Algorithm Name, enter Denylist_Test1.

  2. For Description, enter Denylist Test.

  3. Select the Deny List radio button.

  4. Select LookUp File.

  5. Enter redaction value XXXX.

  6. Click Save.

Create Rule Set
  1. From the job page go to Rule Set and Click Create Rule Set.

  2. For Rule Set Name, enter Free_ Text_RS.

  3. From the Connector drop-down menu, select Free Text.

  4. Select the Input File by clicking the box next to your input file

  5. Click Save.

Create Masking Job
  1. Use Free_Text Rule Set

  2. Execute Masking job.

The results of the masking job will show the following:

The customer xxxx xxxx is satisfied with the terms
of the sales xxxx. Please call to confirm at 718-223-7896.

"Bob," "Jones," and "agreement" are redacted.