Andy Chu — 2018/05/24
xargsis a Unix command that runs another Unix command multiple times with different inputs.
-P 4tells it to run 4 parallel processes.
For productionizing data science and research (often CPU-intensive tasks).
Example: RAPPOR Data Flow
To run tests in parallel (especially with different test frameworks).
[oil spec test demo]
As a quick and dirty way to speed up anything else, including I/O-bound tasks.
cat tasks | xargs wget ...