Deduplicate Transform

Removes exact duplicate rows from your dataset. Duplicate rows are identified by exact, case-sensitive matches between values. For example, two strings with different capitalization do not match.

Basic Usage


Output: Rows that are exact duplicates of previous rows are removed from the dataset.


There are no parameters for this transform.


Matches and non-matches for Deduplicate Transform


For example, your dataset looks like the following, which contains three sets of very similar records. The second row of each set is different in one column than the previous one.

Joe Jones1/2/0388
joe jones1/2/0388
Jane Jackson2/3/0477
Jane JacksonFebruary 3, 200477
Jill Johns3/4/0566
Jill Johns3/4/0566.00



If you use deduplicate on this dataset, no rows are previewed. This preview indicates that no rows will be removed as duplicates. You might need to clean up the data before you can remove any duplicate rows.

Your first step should be get your capitalization consistent. Try the following:

set col:Name value:PROPER(Name)

All entries in the Name column now appear as proper names. Next, you can clean up the score column by normalizing numeric values to the same format. Try the following:

set col:Score value:NUMFORMAT(Score, '##.00')

The above transform normalizes the numeric formats to include two-digits after the decimal point always, which forces all numbers to be the same format. You can use the ## format string here, too.

Use the following to fix the Date column:

replace col:Date with:'2/3/04' on:'February 3, 2004'

Now, you can execute the deduplicate transform:



Joe Jones1/2/0388.00
Jane Jackson2/3/0477.00
Jill Johns3/4/0566.00

Send feedback about...

Google Cloud Dataprep Documentation