Remove Duplicate Lines
Clean text by removing duplicate lines and trimming whitespace.
About the Remove Duplicate Lines
Remove Duplicate Lines is the tool you reach for when you have a list — email addresses, product SKUs, log entries, URLs, hashtags, or any line-delimited data — and you need to deduplicate it without writing a one-off script. Paste your list, pick a mode, and get a clean output instantly. Four modes cover the common scenarios: keep the first occurrence of each duplicate, keep the last, show only the duplicates themselves, or count how often each repeated line appears.
Deduplication is one of the most common data-cleaning tasks in marketing, ops, and engineering. Mailing lists accumulate duplicate subscribers from form resubmissions. CSV exports from legacy systems often contain repeated rows. URL lists built up over months of research tend to have the same link entered multiple times. A reliable deduplicator turns ten minutes of manual scanning into a one-second operation.
The case-insensitive option matters because real-world data is messy. Email providers treat “User@Example.com” and “user@example.com” as the same address, but a case-sensitive deduplicator would keep both. Tagging systems that conflate “#Marketing” and “#marketing” inflate unique tag counts. Toggling case-insensitive comparison handles these without modifying your original data.
How It Works
The tool splits your input on newlines and walks the resulting array, maintaining a hash map (seen) that records whether each line has been encountered before. For each line, it checks the map: if the key is absent, the line is unique so far and gets added to the output; if present, the line is a duplicate. This is O(n) in time and O(unique) in space — efficient enough for lists of hundreds of thousands of lines.
“Keep first” mode outputs the lines in the order they were first seen. “Keep last” mode reverses the array first, deduplicates, then reverses back — this preserves the order of last appearance. “Show only duplicates” outputs the lines that appeared two or more times, in the order of their second occurrence. “Count duplicates” tallies how many times each repeated line appears and sorts the result by frequency descending.
Case-insensitive mode lowercases each line before comparison, so “Apple” and “apple” are treated as the same key. The original casing is preserved in the output (whichever occurrence was kept), but the comparison uses the lowercased form. Note that this is a simple lowercasing — locale-specific case folding (like Turkish dotted/dotless I) is not applied.
Worked Examples
The default input — eight lines including four duplicates — in “Remove duplicates (keep first)” mode produces five lines: apple, banana, cherry, date, elderberry. The duplicates (the second apple, the second banana, the second cherry) are stripped, and the order of first appearances is preserved. The summary reports “Removed 3 duplicate lines.”
Switch to “Count duplicates” mode and the same input produces “apple (2 times), banana (2 times), cherry (2 times)” sorted by frequency. This is useful when you want to know which items in a log file are repeating most often — for instance, when triaging error messages or analyzing user-agent strings.
Toggle case-insensitive comparison and paste a list like “Apple, apple, APPLE, Banana, banana” (one per line). In keep-first mode with case-insensitive on, you get “Apple, Banana” — the first-cased version of each is kept. With case-insensitive off, you get all five lines back because they are treated as distinct.
When to Use This Tool
- Cleaning email subscriber lists before importing into a CRM or ESP.
- Deduplicating URL lists gathered from multiple sources before crawling or scraping.
- Removing duplicate hashtag suggestions before pasting into a social media scheduling tool.
- Cleaning CSV exports that contain repeated rows due to join quirks or data-entry errors.
- Identifying repeated log entries to spot error patterns or stuck loops.
- Deduplicating SSH authorized_keys files when multiple admins have appended keys over time.
- Building a unique word list from a free-text export for vocabulary or keyword analysis.
Limitations & Disclaimer
This tool performs exact or case-insensitive line matching only. It does not perform fuzzy matching (so “John Smith” and “Jon Smith” are treated as distinct), phonetic matching, or near-duplicate detection. Trailing whitespace, line-ending differences (LF vs CRLF), and invisible characters (zero-width spaces, BOM) can cause lines that look identical to be treated as different. For large inputs (over ~500k lines), browser performance may degrade. See our disclaimer for full terms.
Frequently Asked Questions
Does the tool preserve the original order of lines?
In “keep first” mode, yes — the first occurrence of each unique line is kept in its original position relative to other unique lines. In “keep last” mode, the last occurrence is kept, also preserving relative order. “Show duplicates” outputs duplicates in the order of their second appearance.
Does it handle trailing whitespace differences?
By default, no. A line ending with a space and the same line without a trailing space are treated as different. To deduplicate considering only the trimmed content, run your input through a find-and-replace tool first to strip trailing whitespace, or pre-process with <code>trim()</code> in a spreadsheet.
Can I deduplicate case-insensitively but preserve original casing?
Yes. Toggle the case-insensitive option and the comparison uses lowercase, but the output preserves whichever occurrence was kept (the first or the last, depending on mode). This is the most common use case for email and tag deduplication.
What is the maximum input size?
There is no hard limit, but very large inputs (over a few hundred thousand lines) will make the browser UI laggy while processing. For lists over a million lines, use a command-line tool like <code>sort | uniq</code> or <code>awk '!seen[$0]++'</code> instead.
Does the tool work with Windows line endings (CRLF)?
Yes. The split uses <code>\n</code> which leaves a trailing <code>\r</code> on each line in CRLF input. This means a line “apple\r” and a line “apple” would be treated as different. For consistent deduplication of mixed line-ending input, normalize line endings first.
Is my list uploaded anywhere?
No. All processing happens in your browser. Email lists, customer IDs, and other sensitive data never leave your device, which makes this tool safe for GDPR-regulated and HIPAA-regulated data.
Last updated: July 21, 2026 · Author: HT99 Tools Editorial Team · Reviewed by: HT99 Tools Editorial Team