Dedupe Lines

Remove duplicate lines from any text. Choose to keep the first occurrence, keep the last, or keep only lines that appear exactly once (unique-only). Optional case-insensitive matching, trim whitespace before comparing. Useful when cleaning up exports, email lists, log lines, or feature flags.

Common cleaning patterns

The most common batch cleaning tasks:

• Email list dedup — vendor exports often duplicate addresses; one address, one row.
• Log line dedup — kill repeated identical errors before grepping.
• Feature flag list cleanup — multiple environments contributed; consolidate.
• Vocabulary list dedup — for language learning sets, dictionaries.
• CSV row dedup — when you want unique rows; combine with Sort Lines.

Modes in detail

• Keep first — input order preserved, first occurrence wins.
• Keep last — input order preserved, last occurrence wins. Useful when later entries override earlier (config files).
• Unique only — drop everything that appeared more than once. Useful for finding singletons.
• Show duplicates only — inverse of unique only; surfaces only the lines that recurred.

Common workflows

Clean an email list. Paste, dedupe, optionally lowercase first via Case Converter so Email@x.com and email@x.com collapse.

Find log fingerprints. Replace timestamps with placeholders, dedupe, count remaining lines — surfaces repeated error patterns.

Spot duplicates. Switch to “Show duplicates only” to find which lines are repeated; useful when auditing.

Last updated: 2025-01-15