TextToolboxTextToolbox
HomeText Cleaners & ModifiersRemove Duplicates from List

Remove Duplicates from List

Keep first-occurrence line items with explicit case, edge-space, NFC, output-trim and blank-removal choices.

Built by Saurabh Kumar

Quick Answer

Keep first-occurrence line items with explicit case, edge-space, NFC, output-trim and blank-removal choices. Input is limited to 1,000,000 UTF-16 units and 20,000 physical items; review meaningful whitespace and record structure before copying.

Keep the first original item for each matching comparison key. Case-sensitive matching and ignored edge whitespace are the defaults. Comparison settings change which items match; they do not alter retained spelling unless Trim retained output is explicitly selected.

0 input items; 0 retained; 0 duplicates removed; 0 blanks removed

Recommended Tools

Remove Empty LinesSort Lines ToolTrim Whitespace ToolChange Case ToolFind and Replace ToolFlip List ToolCount Words/Characters - Stats ToolAdd Prefix/Suffix - Line Editor

Keeping the First of Each Repeated Line

Paste a list and get it back with every repeated line removed. The tool keeps the first original item for each matching comparison key. Case-sensitive matching and ignored edge whitespace are the defaults. Comparison settings change which items match; they do not alter retained spelling unless Trim retained output is explicitly selected. Typical input is a plain list where first-occurrence order must survive; the settings also let you compare edge and case rules before dropping anything, and apply NFC when composed and decomposed text should match.

Paste a line-based list. Choose case sensitivity and whether edge whitespace belongs to the comparison key. Enable canonical NFC comparison only when equivalent composed and decomposed text should match. Select blank removal or output trimming separately, inspect the counts, then Copy or Download TXT.

How Comparison Keys Are Built

Physical CRLF, LF and CR separators are recognized and output uses LF. Each key optionally trims edges, normalizes to NFC and applies JavaScript lowercase; lowercase keys are normalized again when NFC is enabled. The first matching original item remains in order. Drop whitespace-only items runs before deduplication. Input count equals retained items plus duplicate removals plus explicit blank removals. A retained empty item can have an empty output string while the item count is one.

Reference: Unicode normalization forms. Browser textareas and clipboard handling may normalize line endings.

Four Settings, Four Outcomes

Settings and exact effects
Input and settingResult
Spaced apple, apple; ignore edges onFirst spaced apple retained
apple, Apple; case-sensitive onBoth retained
Composed and decomposed e with accent; NFC onFirst canonical-equivalent item retained
A, two blanks, B; Drop offOne blank representative retained

What Deduplication Does Not Decide

Lowercase comparison is not locale-specific case folding; it does not make every international spelling equivalent. Ignoring edges does not collapse interior spaces. NFC compares canonical equivalents without stripping accents or compatibility styles. This is whole-line matching, not CSV-column uniqueness, database validation, email deliverability or consent verification.

The list can hold up to 1,000,000 UTF-16 units and 20,000 physical items. Oversized input or a missing browser API clears the result with an error; the limits are size caps, not a speed promise. Empty output disables Copy and TXT export. Keep the original when removing repeats could drop wanted whitespace or record structure.

Leave output trimming off to retain the first original spelling and padding. Use exact matching when case or spaces carry meaning. Keep an original copy before enabling blank removal or changing comparison rules.

Use Sort Lines only if the retained first-occurrence order should change. Trim Whitespace changes line text separately from matching keys.