Remove duplicate lines from a list

Exact match or ignoring case

Remove duplicate lines keeps a line only when no line before it is identical, character for character. “Apple” and “apple” both stay, and so do “apple” and “apple ” with a space at the end. Use it when capitals carry meaning: product codes, file names on some systems, lines of code.

Remove duplicates, ignoring case compares the lines in small letters and ignores spaces at both ends. That is the one for email addresses, names, tags and keywords, where “Anna@Mail.com” and “anna@mail.com” are the same thing. The copy that stays is the first one, exactly as it was typed.

Neither choice ignores spaces inside a line or accents: “New  York” with two spaces and “Zoë” against “Zoe” still count as different lines. Run Collapse extra spaces from Clean first to catch the spacing ones.

Lines that carry more than the value you care about never match either: “Anna Smith <anna@mail.com>” in one list and the bare address in another are different lines. Pull the addresses out with Extract first, then remove the duplicates.

Where duplicate lines come from

• Merged lists: two exports of a mailing list, or sign-ups from several events, pasted one under the other.
• Keyword and tag research gathered from several tools that all suggest the same phrases.
• Error messages copied out of a log, where one line repeats hundreds of times and hides the rare ones.
• Columns copied from a spreadsheet, such as every city in an order sheet, when you want the list of distinct values.
• Reading lists and link collections built up over months.

To combine two lists into one without repeats, paste the second under the first and remove the duplicates. Compare the line count before and after to see how many went.

A column copied out of Excel or Google Sheets arrives as one value per line, which is exactly the shape this needs. Paste the cleaned list back into a column the same way and each value lands in its own cell.

Blank lines, order and sorting

Empty lines are never treated as duplicates, and neither are lines that hold only spaces, so a text with paragraphs keeps its gaps. If the list has empty lines you want gone, Remove empty lines sits in the same Lines menu.

The order never changes: each line that stays keeps its place. If you want the list in alphabetical order too, sort it before or after, and the result is the same either way. Sorting first has one advantage: repeats end up next to each other, so you can see what is about to go. A to Z ignores capitals, so “apple” and “Apple” land together.

Text Studio removes repeats but does not tally them. If you need to know how many times each line appeared, paste the list into a spreadsheet and count with COUNTIF.

Other ways on Windows

Excel's Remove Duplicates on the Data tab works on table columns and can compare several columns at once, which suits a sheet with many fields. PowerShell can do it for a text file with Get-Content list.txt | Select-Object -Unique, which keeps the first of each line and matches capitals exactly. Google Sheets has the same job under Data, Data cleanup, Remove duplicates. All of them are good tools when they are already open.

Text Studio is quicker for a list on the clipboard: paste, one click, and the result is in front of you to check, with the ignoring-case option one line down in the same menu. A list inside a Word document works the same way: copy it out, remove the duplicates here and paste it back, and each line returns as its own paragraph.