Remove duplicate lines from text online
Clean a messy list in one pass: drop repeats, sort it, trim the whitespace and strip the blank rows.
Runs entirely in your browser. Your list is never uploaded.
How to use the duplicate line remover
- Paste your list into the top box with one item per line. Email addresses, keywords, URLs, product codes — anything that arrives one-per-row out of a spreadsheet or export.
- Choose your cleaning options. The first three are on by default because they are almost always wanted. Turn on "Treat case as the same" if Gmail.com and gmail.com should count as one entry.
- Pick a sort order if you want one. "Keep original order" preserves the first occurrence of each item in the sequence you pasted, which matters when the order carries meaning.
- Press Clean list. The three counters show how many lines went in, how many came out and how many were dropped, then copy the result or download it as a .txt file.
What you can use it for
Merging email or subscriber lists is the classic case. Two exports from different sources, pasted one after the other, will always contain overlap. Deduplicating before you import prevents double-sending and, on most email platforms, stops you paying twice for the same contact.
Keyword research generates enormous amounts of repetition. Pull suggestions from three tools and you will have the same long-tail phrases several times over. A dedupe pass with case-insensitive matching turned on typically cuts a 4,000-line export by a third before you start grouping.
It cleans up URL lists before a bulk operation. Sitemap extracts, crawl exports and redirect maps often carry duplicates and empty rows that break the import on the other end. Trimming spaces matters here too, since a trailing space makes an otherwise identical URL look unique.
Anyone auditing a CSV column can use it as a quick uniqueness check. Paste the column, clean it, and compare "lines in" against "lines out" — the gap tells you how many duplicate values are hiding in a field that was supposed to be unique.
Things to know
When duplicates are removed, the first occurrence is the one kept and every later copy is dropped. Combined with "Keep original order" this means a list arrives back in the same sequence, just thinner. Sorting is applied after deduplication, never before.
Trimming spaces changes what counts as a duplicate, which is usually the point. "apple " and "apple" are different strings to a computer; with trimming on they collapse into one. Turn trimming off if leading indentation is meaningful in your data, such as a nested outline.
Case-insensitive matching keeps the first version it encountered. If your list contains "Delhi" and then "delhi", you get "Delhi". If you want everything normalised to one case, run the result through the case converter afterwards.
The tool works on lines, so a comma-separated list on a single row is one line and nothing will be removed. Replace the commas with line breaks first — most spreadsheets will do this if you paste a column rather than a row.
Frequently asked questions
Yes. Every line is checked against everything seen before it, regardless of position. A repeat on line 900 of a line first seen on line 3 is caught. The list does not need to be sorted first.
Lists of 100,000 lines process in well under a second on a normal laptop, because the matching uses a hash set rather than repeated scanning. Very large pastes are limited more by how long your browser takes to render the text than by the deduplication itself.
Not in this tool — it returns the unique set rather than the duplicate set. If you need to see just the repeats, sort A to Z first without removing duplicates and the repeated entries will sit next to each other where you can spot them.
No. Lines are compared and copied as exact strings, with no encoding, escaping or lowercasing applied unless you switch on case-insensitive matching. What you paste is what you get back, minus the lines that were removed.