Skip to main content
All CollectionsProduct Updates
September 6th, 2023 - Automated Duplicate Filtering
September 6th, 2023 - Automated Duplicate Filtering
Kevin Yang avatar
Written by Kevin Yang
Updated over a week ago

Automated Filtering of duplicate records

You can now prevent duplicate records being appended to a dataset by defining one or more Match Keys that specify the rules. During the import process any uploaded records that are duplicate of a record in the same file or already existing in the Dataset will be filtered out and reported as a duplicate.

The automated filtering process works in the following way:

  1. Users create definitions for generating Match Keys for each dataset

  2. When data is imported Ipiphany will generate Match Keys for both the existing dataset records and the new records to be imported

  3. During the import process if any records that are to be imported have generated Match Keys that match other records in the import file or already existing in the Dataset, then these records are flagged as Duplicates and filtered out

  4. As shown below, any filtered Duplicates are reported in the import notification email and are available for download from the import request

For further details see Automated Duplicate filtering

Other Changes

  • The Import Summary Email has been enhanced to provide further details on reasons that records were excluded e.g. Duplicates, Translation Issues

Fixed Bugs/Issues

  • Allow datasets that fail but still show as Processing to be deleted or Reprocessed

  • Re-authentication error is not redirected to Re-enter Password screen

  • Handle double-quote variations cut and paste from Word in search expressions

  • Create new attribute when appending data from same source column fails

  • Google Translate returns some text in HTML-escaped format

  • New Connection not always mapping correctly to existing attributes

  • Improved wording of some validation error messages to reflect user friendly language

  • Link from Import Summary Email no longer displays Dataset Key Drivers screen

Did this answer your question?