Deduplicate Lines
Remove duplicate lines from text. Keeps the first occurrence of each line, with optional case-insensitive matching and whitespace trimming.
Duplicates in, unique lines out
Keeps only the first occurrence of each line. Stable — original order is preserved.
apple banana apple cherry banana apple fig
apple banana cherry fig
What you'll use this for
Lists, logs, exports — anywhere duplicates sneak in and you need a clean unique set.
Log unique entries
Strip repeating error messages from a noisy log to see distinct events.
Email list cleanup
Remove repeated addresses pasted from multiple sources before a campaign.
CSV deduplication
Dedupe a column dumped from a database export. Pair with trim before compare.
Unique words
Combine with one-word-per-line input to produce a unique vocabulary list.
How to deduplicate lines
Paste your text
Drop the list, log, or column into the left editor. One entry per line.
Pick options
Toggle case-insensitive and trim if your duplicates differ only in case or surrounding spaces.
Click Dedupe
Or leave auto-dedupe on for live updates. Runs locally — no upload.
Copy or download
Copy to clipboard or save as unique.txt. Or jump to sort-lines to alphabetize.
Frequently asked questions
Yes. The first occurrence of each line stays in place; later duplicates are dropped. If you need an alphabetized output instead, pipe the result through the sort-lines tool.
Yes. Apple and apple count as different lines. Flip the Case-insensitive toggle to fold case before comparing.
Yes. No signup, no limits, no ads. Runs entirely in your browser.
The algorithm is O(n) using a hash set, so it scales linearly with input size. Multi-megabyte inputs finish in a fraction of a second on a modern laptop.
Not in one step — this tool preserves order. Run the result through sort-lines for a sorted unique list (the classic sort | uniq combination).
About deduplicating lines
Deduplication is the operation of reducing a multiset of lines to a set: each distinct value appears exactly once. The order-preserving variant — implemented here — keeps the first occurrence of each line and drops everything after, which is the behaviour of awk '!seen[$0]++' on the command line.
How it works
- Each line is hashed into a set as it's read.
- If the hash is already present, the line is skipped; otherwise it's appended to the output.
- Optional pre-processing (trim, lowercase) is applied to the hash key only — the original line is preserved in the output.
When to use which option
- Case-insensitive — emails, usernames, hostnames, anything where case shouldn't matter.
- Trim before compare — copy-pasted lists with stray indentation or trailing spaces.
- Auto-dedupe — leave on for live previews while you tweak the source text.