Subprocess Popen To Run Commands (hdfs/hadoop)

I am trying to use subprocess.popen to run commands on my machine. This is what I have so far cmdvec = ['/usr/bin/hdfs', 'dfs', '-text', '/data/ds_abc/clickstream/{d_20151221-2300}

Solution 1:

In your example, passing the pipe | character as an argument to subprocess.Popen does not create a pipeline of processes the same way that it would in something like Bash. Instead, the pipe | character is being passed an argument to a single process.

Instead, you would need to chain together 2 separate subprocess.Popen calls to simulate a Bash-style pipeline. This documentation on the subprocess module contains more details.

https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline