Mastering the uniq Command in Linux: Essential Guide for Linux Users

When working with text files or processing data in Linux, the uniq command is a powerful yet often overlooked tool. It allows you to filter out or process duplicate lines, helping streamline workflows for system administrators, developers, and data analysts alike. In this guide, we’ll explore the full capabilities of uniq, from basic use to advanced options, specifically for Linux users.

For a step-by-step visual guide, watch this video.

What is the uniq Command in Linux?

The uniq command stands for “unique”, and as the name suggests, it helps filter out duplicate lines from a file or output. However, it’s important to note that uniq only works with sorted data—it does not remove duplicates unless they are adjacent. So, always remember to sort your data first if needed.

Basic Syntax of the uniq Command

The basic syntax for the uniq command is:

man uniq
uniq [options] [input_file] [output_file]

Let’s walk through a few practical examples to understand how it works.

1. Remove Duplicate Lines

Imagine you have a file called duplicates.txt containing the following data:

Alice
Bob
Alice
Charlie
Alice
Bob

To remove duplicate lines, you first need to sort the file and then pipe it to uniq:

sort duplicates.txt | uniq

Output:

Alice
Bob
Charlie

In this example, uniq removes duplicates, leaving only the first occurrence of each line. Sorting the file beforehand ensures that consecutive duplicates are identified.

2. Counting Duplicates with the -c Option

Sometimes, it’s useful to know how many times a particular line appears. You can use the -c option to display the count of each line:

sort duplicates.txt | uniq -c

Output:

3 Alice 
2 Bob 
1 Charlie

This feature is particularly useful when analyzing logs or debugging issues where duplicate entries are important to identify.

3. Show Only Duplicate Lines with the -d Option

If you’re specifically interested in lines that are duplicated, you can use the -d option:

sort duplicates.txt | uniq -d

Output:

Alice
Bob

This command isolates the repeated entries, which can be useful in various data processing tasks, such as identifying duplicate function names in code.

4. Show Only Unique Lines with the -u Option

On the other hand, if you want to display only the lines that are not duplicated, use the -u option:

sort duplicates.txt | uniq -u

Output:

Charlie

This is helpful for isolating unique records in datasets, like finding single instances of error messages or log entries.

5. Handling Null-Terminated Lines with the -z Option

In certain cases, you may work with files that use null characters as line terminators, often found in machine-readable logs. Use the -z option to handle these files:

sort -z duplicates.txt | uniq -z

While this is a less common scenario, it’s essential when working with structured data formats like CSVs or JSON files.

6. Comparing Specific Fields with -f and -s Options

You can compare specific fields or characters in each line using the -f and -s options. Consider this file log.txt:

2025-01-22 User1 Login Successful
2025-01-22 User2 Login Successful
2025-01-22 User1 Login Successful
2025-01-22 User3 Password Reset
2025-01-22 User3 Password Reset

Using the -f Option to Skip Fields

To ignore the first N fields (in this case, the date) and compare the remaining fields:

uniq -f 1 log.txt

Output:

2025-01-22 User1 Login Successful
2025-01-22 User2 Login Successful
2025-01-22 User3 Password Reset

Using the -s Option to Skip Characters

Alternatively, you can skip the first N characters (e.g., skip the date and a space) and compare from the 12th character onward:

uniq -s 11 log.txt

Output:

2025-01-22 User1 Login Successful
2025-01-22 User2 Login Successful
2025-01-22 User3 Password Reset

You can combine the -f and -s options for more granular comparisons.

7. Limiting Comparisons with the -w Option

If you need to limit the comparison to only the first N characters of each line, use the -w option. For example, with this data in data.txt:

Error: Failed to connect to server
Error: Failed to load configuration
Error: Failed to connect to database
Warning: Low disk space
Warning: High memory usage
Warning: Low disk space

To compare only the first 7 characters:

uniq -w 7 data.txt

Output:

Error: Failed to connect to server
Warning: Low disk space

This is especially useful for grouping logs or messages based on prefixes like error types or log levels.

Pro Tips for Linux Users

• Log Analysis: Pair uniq with tools like grep and awk to extract and analyze insights from logs like /var/log/secure or /var/log/httpd/access_log.
• Preprocessing: Always sort your files before using uniq, and use options like -n or -r for numerical or reverse sorting.
• Combine Commands: Use uniq in combination with awk, sed, or cut for powerful data manipulation.

Frequently Asked Questions (FAQs)

Q1: Do I always need to sort the file before using uniq?

Yes, uniq only considers consecutive duplicate lines, so sorting the data is crucial if duplicates are not already next to each other.

Q2: How can I count the number of occurrences of a line in my file?

Use the -c option to count occurrences: sort file.txt | uniq -c.

Q3: Can I remove only duplicate lines without affecting the unique ones?

Yes, you can use the -u option to show only unique lines: sort file.txt | uniq -u.

Q4: How do I process files with null-terminated lines?

Use the -z option to handle files with null-terminated lines: sort -z file.txt | uniq -z.

Q5: What’s the benefit of using the -w option in uniq?

The -w option allows you to compare only the first N characters, which is useful for grouping similar lines based on prefixes, such as log levels.

Conclusion

The uniq command is a versatile and essential tool for text processing in Linux. Whether you’re removing duplicates, counting occurrences, or filtering specific fields, it can greatly simplify your workflow.

For further learning, explore related tutorials on text processing and command-line tools for Linux.

See also:

List of monitoring tools 

Linux Blogs

AWS Cloud Blogs

Database Blogs

DevOps Blogs

Interview Questions & Answers

Docker Blogs

Google Cloud Blogs