Remove Duplicate Lines
Delete repeated lines from a list, optionally ignoring case and surrounding whitespace.
Processed on your device — no upload
How to remove duplicate lines
- Paste your list, one item per line.
- Decide whether capitalisation and spacing should count as a difference.
- Copy the deduplicated list — the counter shows how many were removed.
How it works
The list is walked from top to bottom, and each line is added to a set of lines already seen. If a line is already in the set it is dropped; otherwise it is kept. That means the first occurrence of each value survives and the order is preserved — which is what you want when the list is ranked or chronological, and is not what sorting and deduplicating together would give you.
Two options decide what counts as a duplicate. Ignore capitalisation treats Ahmad@example.com and ahmad@example.com as the same — correct for email addresses, where the domain is case-insensitive and virtually every provider treats the local part that way too.
Trim before comparing ignores leading and trailing spaces. Lists pasted out of spreadsheets and PDFs are full of invisible trailing spaces, and without this option two visually identical lines are treated as different — which is the single most common reason a deduplication appears not to have worked.
Everything runs in your browser. Lists of email addresses and customer names are exactly the kind of data that should not be pasted into a site that uploads it.
A worked example
You have merged three event sign-up sheets into one list of 1,240 email addresses and need to send one invitation per person.
Paste the list with both options on. The status line reads lines in 1240, lines out 1104, removed 136. Those 136 were people who signed up twice.
Now turn Trim before comparing off and watch the removed count drop to 89. The difference — 47 addresses — are lines with trailing spaces from the spreadsheet export. They look identical on screen and are invisibly different to a computer. This is the option people forget, and it is why a list that “still has duplicates” usually does not.
Turn Ignore capitalisation off as well and the count drops again, to 61. Sara.Haddad@Gmail.com and sara.haddad@gmail.com are the same inbox, and only the case-insensitive comparison catches that.
The order is unchanged throughout: the first time each address appeared is where it stays. If you want it alphabetical afterwards, sort the result as a second step — doing it in that order lets you keep the earliest entry rather than an arbitrary one.
Questions
How do I get a unique list of emails?
Paste them one per line and turn on trim and ignore case. Addresses differing only in spacing or capitalisation collapse into one entry, which is the usual reason a list looks longer than it is.
Which duplicate is kept?
The first one. The list is processed top to bottom and the original order of the surviving lines is preserved, so ranked or chronological lists stay meaningful.
Is my list uploaded?
No. Deduplication runs in your browser. This matters here more than for most tools, because these lists are usually email addresses or customer names.
Why does it say there are no duplicates when I can see some?
Almost always trailing spaces. Turn on "trim before comparing" — lines exported from spreadsheets and PDFs frequently carry invisible whitespace that makes identical-looking lines different.
Should I ignore capitalisation?
For email addresses, yes. They are effectively case-insensitive. For anything case-sensitive such as passwords, product codes or identifiers, no.
Does it sort the list?
No, deliberately. Sorting would destroy any meaningful order. Use the sort tool afterwards if you want the result alphabetical.
What happens to blank lines?
By default they are removed entirely. Switch the blank lines option if your list uses them as separators you want to keep.
Can it handle a very long list?
Yes. Lookup is done with a hash set, so the work grows linearly — hundreds of thousands of lines process without a noticeable pause.
Can I see which lines were duplicated?
Not in this tool, which reports only the counts. To find them, sort the list first and duplicates will sit next to each other.
Is this the same as deleting or stripping duplicate lines?
Yes — delete duplicate lines and strip duplicate lines are the same request. The first occurrence of each line is kept in its original position, so the order you had is preserved rather than sorted.
Related Text tools
Word Counter
Words, characters, reading time
open$termiva case --to titleCase Converter
UPPER, lower, Title, camelCase
open$termiva unwrap --keep-paragraphsRemove Line Breaks
Unwrap text into flowing lines
open$termiva sort --alphaSort Lines
Alphabetical, numeric or random
open