GNU parallel

Parallel

Developer(s)	GNU Project
Stable release	20160822[1] / August 22, 2016
Repository	git.savannah.gnu.org/cgit/parallel.git
Written in	Perl
Operating system	GNU
Type	Utility
License	GPLv3
Website	www.gnu.org/software/parallel/

GNU parallel is a command-line driven utility for Linux and other Unix-like operating systems which allows the user to execute shell scripts in parallel. GNU parallel is free software, written by Ole Tange in Perl. It is available under the terms of GPLv3.^[2]

Usage

Introduction video, Part 1

Introduction video, Part 2

The most common usage is to replace the shell loop, for example

    for x in `cat list` ; do 
        do_something "$x"
    done | process_output

to the form of

    cat list | parallel do_something | process_output

where the file list contains arguments for do_something and where process_output may be empty.

Scripts using parallel are often easier to read than scripts using pexec.

The program parallel features also

• grouping of standard output and standard error so the output of the parallel running jobs do not run together;
• retaining the order of output to remain the same order as input;
• dealing nicely with filenames containing special characters such as space, single quote, double quote, ampersand, and UTF-8 encoded characters;

By default, parallel runs as many jobs in parallel as there are CPU cores.

Examples

 find . -name "*.foo" | parallel grep bar

The above is the parallel equivalent to:

 find . -name "*.foo" -exec grep bar {} +

This searches in all files in the current directory and its subdirectories whose name end in .foo for occurrences of the string bar. The parallel command will work as expected unless a file name contains a newline. In order to avoid this limitation one may use:

 find . -name "*.foo" -print0 | parallel -0 grep bar

The above command uses the null character to delimit file names.

 find . -name "*.foo" | parallel -X mv {} /tmp/trash

The above command uses {} to tell parallel to replace {} with the argument list.

 find . -maxdepth 1 -type f -name "*.ogg" | parallel -X -r cp -v -p {} /home/media

The command above does the same as:

 cp -v -p *.ogg /home/media

however, the former command which uses find/parallel/cp is more resource efficient and will not halt with an error if the expansion of *.ogg is too large for the shell.

See also

• xargs
• pexec
• GNU Queue

References

1. Insivest, Ole (22 August 2016). "GNU Parallel 20160822 ('Og Nomekop') released". parallel (Mailing list).
2. "GNU Parallel". GNU.org.

External links

• Project page and manual page of GNU parallel