[bash] How to use uniq

uniq

Remove duplicate lines.

UNIQ(1) BSD General Commands Manual UNIQ(1)

NAME
     uniq -- report or filter out repeated lines in a file

SYNOPSIS
     uniq [-c | -d | -u] [-i] [-f num] [-s chars] [input_file [output_file]]]

github

• The file in jupyter notebook format on github is here.

google colaboratory

• If you want to run it on google colaboratory here /uniq_nb.ipynb)

Environment

The author’s OS is macOS, and the options are different from those of Linux and Unix commands.

When you actually run the command, the leading ! and %%bash in the first line.

!sw_vers

ProductName: Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G95

!bash --version

GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.

Example usage

I use it a lot when I work in data analysis. Because duplicate and useless data is likely to be garbage. Usually, you can use

uniq <in file> <out file>

to remove duplicate lines and output the result to a separate file.

%%bash
echo "Preparing the file"
echo -e "123\n123\n123" > temp
echo -e "<before>"
cat temp

uniq temp temp2
echo -e "\n<after>"
cat temp2

Preparing the file
<before
123
123
123

<after
123