uniq Command
The uniq command in Linux is used to filter out or report repeated lines in a file. It is most effective when used on a sorted file, as it only detects adjacent duplicate lines. Therefore, it is often used in conjunction with the sort command.
Syntax
Description
uniq reads from the specified INPUT (or standard input if none is given) and writes to the specified OUTPUT (or standard output if none is given). It compares adjacent lines and, based on the options, either removes duplicates, reports only duplicates, or reports only unique lines.
Common uses include:
- Removing duplicate lines from a sorted file
- Counting the occurrences of each unique line
- Displaying only unique lines
- Displaying only duplicate lines
Common Options
| Option | Description |
|---|---|
-c, --count |
Prefix lines by the number of occurrences |
-d, --repeated |
Only print duplicate lines, one for each group of identical lines |
-u, --unique |
Only print unique lines |
-i, --ignore-case |
Ignore differences in case when comparing |
-s N, --skip-chars=N |
Skip N characters before comparison |
-w N, --check-chars=N |
Compare no more than N characters in lines |
Examples
Remove duplicate lines (requires sorted input)
Sorts 'words.txt' and then removes adjacent duplicate lines, printing the unique lines.
Count occurrences of each unique line
Sorts 'words.txt' and then counts the occurrences of each unique line, displaying the count before each line.
Show only unique lines (non-repeated)
Sorts 'words.txt' and then displays only the lines that are not repeated (appear only once).
Show only duplicate lines
Sorts 'words.txt' and then displays only the lines that have duplicates (appears more than once).
Ignore case when checking for duplicates
Sorts 'words.txt' ignoring case, then filters unique lines ignoring case.