By freshWoWer


2008-09-18 01:35:30 8 Comments

How can I call an external command (as if I'd typed it at the Unix shell or Windows command prompt) from within a Python script?

30 comments

@David Cournapeau 2008-09-18 01:39:35

Look at the subprocess modulein the standard library:

import subprocess
subprocess.run(["ls", "-l"])

The advantage of subprocess vs. system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).

The official documentation recommends the subprocess module over the alternative os.system():

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function [os.system()].

The Replacing Older Functions with the subprocess Module section in the subprocess documentation may have some helpful recipes.

Older versions of Python use call:

import subprocess
subprocess.call(["ls", "-l"])

@Kevin Wheeler 2015-09-01 23:17:07

Is there a way to use variable substitution? IE I tried to do echo $PATH by using call(["echo", "$PATH"]), but it just echoed the literal string $PATH instead of doing any substitution. I know I could get the PATH environment variable, but I'm wondering if there is an easy way to have the command behave exactly as if I had executed it in bash.

@SethMMorton 2015-09-02 20:38:24

@KevinWheeler You'll have to use shell=True for that to work.

@Murmel 2015-11-11 20:24:27

@KevinWheeler You should NOT use shell=True, for this purpose Python comes with os.path.expandvars. In your case you can write: os.path.expandvars("$PATH"). @SethMMorton please reconsider your comment -> Why not to use shell=True

@florisla 2017-07-12 10:02:16

The example calls ls -l but does not give access to its output (stdout is not accessible). I find that confusing -- you could use a command without stdout instead, such as touch.

@Braden Best 2017-07-24 23:13:35

Seems to be pretty clunky on Python3 + Windows. If I enter a filename with special characters like &, it will throw a FileNotFoundError. Even though the file is in the working directory where I executed python, and obviously does exist.

@Charlie Parker 2017-10-24 19:07:55

does call block? i.e. if I want to run multiple commands in a for loop how do I do it without it blocking my python script? I don't care about the output of the command I just want to run lots of them.

@sudo 2017-11-16 19:43:26

I understand using call for more advanced features, but I don't see anything wrong with using system if it does what you need.

@slehar 2018-06-16 17:15:48

To simplify at least conceptually:\n call("ls -l".split())

@Daniel F 2018-09-20 18:05:59

If you want to create a list out of a command with parameters, a list which can be used with subprocess when shell=False, then use shlex.split for an easy way to do this docs.python.org/2/library/shlex.html#shlex.split

@Daniel F 2018-09-20 18:15:30

subprocess also allows you to directly pipe two commands together, there's an example in the docs.

@Lie Ryan 2018-12-03 13:44:28

@CharlieParker: call, check_call, check_output, and run blocks. If you want non-blocking, use subprocess.Popen.

@Lie Ryan 2018-12-03 13:49:37

@sudo: you got it the other way around. system() is more advanced than Popen, it adds some friggin shell that is ready to screw you over when you least expects it. Which shell? Depends on what the user has as their SHELL env, which means the exact syntax it'll screw you is often out of your control. Popen is lower level, and is closer to what the underlying OS API looks like, and has much less surprising behavior.

@sudo 2018-12-04 03:03:56

@LieRyan I would use Popen if exact syntax were something to worry about, but if you're just running some program (maybe with a couple of constant arguments), system isn't an issue. Also, sometimes I do want the shell.

@pulse 2019-02-02 22:50:04

you forgot to say it needs python 3.5 at least. It doesn't work on python 3.4.3 for example, which is default for Ubuntu 14.04 LTS

@mark 2019-03-22 10:48:10

As already mentioned, this is only supported in recent versions of python. I think this should be stated in the answer

@tripleee 2019-03-23 12:09:23

The original answer had subprocess.call() so the person who updated the answer (not the original answerer!) should also explain this change in more detail.

@Chris Oliver 2019-08-19 00:07:07

You can use import and then the file name. For example, using the inbuilt module 'random': import random

@Pedro Lobito 2019-08-18 14:42:20

You can also use subprocess.getoutput() and subprocess.getstatusutput(), which are Legacy Shell Invocation Functions from the deprecated commands module, i.e.:

subprocess.getstatusoutput(cmd)

Return (exitcode, output) of executing cmd in a shell.


subprocess.getoutput(cmd)

Return output (stdout and stderr) of executing cmd in a shell.

@Zach Valenta 2019-07-01 20:46:19

Sultan is a recent-ish package meant for this purpose. Provides some niceties around managing user privileges and adding helpful error messages.

from sultan.api import Sultan

with Sultan.load(sudo=True, hostname="myserver.com") as sultan:
  sultan.yum("install -y tree").run()

@Cédric 2018-10-30 11:40:43

I wrote a small library to help with this use case:

https://pypi.org/project/citizenshell/

It can be installed using

pip install citizenshell

And then used as follows:

from citizenshell import sh
assert sh("echo Hello World") == "Hello World"

You can separate stdout from stderr and extract the exit code as follows:

result = sh(">&2 echo error && echo output && exit 13")
assert result.stdout() == ["output"]
assert result.stderr() == ["error"]
assert result.exit_code() == 13

And the cool thing is that you don't have to wait for the underlying shell to exit before starting processing the output:

for line in sh("for i in 1 2 3 4; do echo -n 'It is '; date +%H:%M:%S; sleep 1; done", wait=False)
    print ">>>", line + "!"

will print the lines as they are available thanks to the wait=False

>>> It is 14:24:52!
>>> It is 14:24:53!
>>> It is 14:24:54!
>>> It is 14:24:55!

More examples can be found at https://github.com/meuter/citizenshell

@Eli Courtwright 2008-09-18 13:11:46

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:

  1. os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example:

    os.system("some_command < input_file | another_command > output_file")  
    

    However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs. See the documentation.

  2. stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything. See the documentation.

  3. The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say:

    print subprocess.Popen("echo Hello World", shell=True, stdout=subprocess.PIPE).stdout.read()
    

    instead of:

    print os.popen("echo Hello World").read()
    

    but it is nice to have all of the options there in one unified class instead of 4 different popen functions. See the documentation.

  4. The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply waits until the command completes and gives you the return code. For example:

    return_code = subprocess.call("echo Hello World", shell=True)  
    

    See the documentation.

  5. If you're on Python 3.5 or later, you can use the new subprocess.run function, which is a lot like the above but even more flexible and returns a CompletedProcess object when the command finishes executing.

  6. The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.

The subprocess module should probably be what you use.

Finally please be aware that for all methods where you pass the final command to be executed by the shell as a string and you are responsible for escaping it. There are serious security implications if any part of the string that you pass can not be fully trusted. For example, if a user is entering some/any part of the string. If you are unsure, only use these methods with constants. To give you a hint of the implications consider this code:

print subprocess.Popen("echo %s " % user_input, stdout=PIPE).stdout.read()

and imagine that the user enters something "my mama didnt love me && rm -rf /" which could erase the whole filesystem.

@Jean 2015-05-26 21:16:27

Nice answer/explanation. How is this answer justifying Python's motto as described in this article ? fastcompany.com/3026446/… "Stylistically, Perl and Python have different philosophies. Perl’s best known mottos is " There’s More Than One Way to Do It". Python is designed to have one obvious way to do it" Seem like it should be the other way! In Perl I know only two ways to execute a command - using back-tick or open.

@phoenix 2015-10-07 16:37:18

If using Python 3.5+, use subprocess.run(). docs.python.org/3.5/library/subprocess.html#subprocess.run

@Evgeni Sergeev 2016-06-01 10:44:53

What one typically needs to know is what is done with the child process's STDOUT and STDERR, because if they are ignored, under some (quite common) conditions, eventually the child process will issue a system call to write to STDOUT (STDERR too?) that would exceed the output buffer provided for the process by the OS, and the OS will cause it to block until some process reads from that buffer. So, with the currently recommended ways, subprocess.run(..), what exactly does "This does not capture stdout or stderr by default." imply? What about subprocess.check_output(..) and STDERR?

@Charlie Parker 2017-10-24 19:08:43

which of the commands you recommended block my script? i.e. if I want to run multiple commands in a for loop how do I do it without it blocking my python script? I don't care about the output of the command I just want to run lots of them.

@Qback 2017-12-08 09:27:04

@phoenix I disagree. There is nothing preventing you from using os.system in python3 docs.python.org/3/library/os.html#os.system

@Pitto 2018-03-28 07:57:06

my mama didnt love me && rm -rf / will do nothing :D Probably my mama didnt love me || rm -rf / will be more dangerous :)

@Chris Arndt 2018-09-10 16:38:25

@Pitto yes, but that is not what gets executed by the example. Notice the echo in front of the string passed to Popen? So the full command will be echo my mama didnt love me && rm -rf /.

@tripleee 2018-12-03 06:00:46

This is arguably the wrong way around. Most people only need subprocess.run() or its older siblings subprocess.check_call() et al. For cases where these do not suffice, see subprocess.Popen(). os.popen() should perhaps not be mentioned at all, or come even after "hack your own fork/exec/spawn code".

@geckos 2019-03-31 12:21:50

If you are NOT using user input in the commands you can use this

from os import getcwd
from subprocess import check_output
from shlex import quote

def sh(command):
    return check_output(quote(command), shell=True, cwd=getcwd(), universal_newlines=True).strip()

And use it as

branch = sh('git rev-parse --abbrev-ref HEAD') 

shell=True will spawn a shell, so you can use pipe and such shell things sh('ps aux | grep python'). This is very very handy for running hardcoded commands and processing it's output. The universal_lines=True make sure the output is returned in a string instead of binary.

cwd=getcwd() will make sure that the command is run with the same working directory as the interpreter. This is handy for git commands to work like the git branch name example above.

Some recipes

  • free memory in megabytes: sh('free -m').split('\n')[1].split()[1]
  • free space on / in percent sh('df -m /').split('\n')[1].split()[4][0:-1]
  • cpu load sum(map(float, sh('ps -ef -o pcpu').split('\n')[1:])

But this isn't safe for user input, from the docs:

Security Considerations¶

Unlike some other popen functions, this implementation will never implicitly call a system shell. This means that all characters, including shell metacharacters, can safely be passed to child processes. If the shell is invoked explicitly, via shell=True, it is the application’s responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid shell injection vulnerabilities.

When using shell=True, the shlex.quote() function can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands.

Even using the shlex.quote() is good to keep a little paranoid when using user inputs on shell commands. One option is using a hardcode command to take some generic output and filtering by user input. Anyway using shell=False will make sure that only the exactly process that you want to execute will be executed or you get a No such file or directory error.

Also there is some performance impact on shell=True, from my tests it seems about 20% slower than shell=False (the default).

In [50]: timeit("check_output('ls -l'.split(), universal_newlines=True)", number=1000, globals=globals())
Out[50]: 2.6801227919995654

In [51]: timeit("check_output('ls -l', universal_newlines=True, shell=True)", number=1000, globals=globals())
Out[51]: 3.243950183999914

@Siddharth Satpathy 2019-02-05 00:13:13

There are two prominent ways in which one can execute shell commands using Python. Both the below mentioned examples show how one can get the name of present working directory (pwd) using Python. You can use any other Unix command in place of pwd.

1.> 1st method: One can use the os module from python, and system() function from there to execute shell commands in Python.

import os
os.system('pwd')

Output:

/Users/siddharth

1.> 2nd method: Another way is to use the subprocess module and call() function.

import subprocess
subprocess.call('pwd')

Output:

/Users/siddharth

@Farzad Vertigo 2019-01-29 05:11:40

As some of the answers were related to previous versions of python or were using os.system module I post this answer for people like me who intend to use subprocess in python 3.5+. The following did the trick for me on Linux:

import subprocess

#subprocess.run() returns a completed process object that can be inspected
c = subprocess.run(["ls", "-ltrh"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(c.stdout.decode('utf-8'))

As mentioned in the documentation,PIPE values are byte sequences and for properly showing them decoding should be considered. For later versions of python, text=True and encoding='utf-8' are added to kwargs of subprocess.run().

The output of the abovementioned code is:

total 113M
-rwxr-xr-x  1 farzad farzad  307 Jan 15  2018 vpnscript
-rwxrwxr-x  1 farzad farzad  204 Jan 15  2018 ex
drwxrwxr-x  4 farzad farzad 4.0K Jan 22  2018 scripts
.... # some other lines

@am5 2018-03-23 02:30:43

Often, I use the following function for external commands, and this is especially handy for long running processes. The below method tails process output while it is running and returns the output, raises an exception if process fails.

It comes out if the process is done using the poll() method on the process.

import subprocess,sys

def exec_long_running_proc(command, args):
    cmd = "{} {}".format(command, " ".join(str(arg) if ' ' not in arg else arg.replace(' ','\ ') for arg in args))
    print(cmd)
    process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    # Poll process for new output until finished
    while True:
        nextline = process.stdout.readline().decode('UTF-8')
        if nextline == '' and process.poll() is not None:
            break
        sys.stdout.write(nextline)
        sys.stdout.flush()

    output = process.communicate()[0]
    exitCode = process.returncode

    if (exitCode == 0):
        return output
    else:
        raise Exception(command, exitCode, output)

You can invoke it like this:

exec_long_running_proc(command = "hive", args=["-f", hql_path])

@sbk 2018-05-17 12:08:29

You'll get unexpected results passing an arg with space. Using repr(arg) instead of str(arg) might help by the mere coincidence that python and sh escape quotes the same way

@am5 2018-11-17 00:07:21

@sbk repr(arg) didn't really help, the above code handles spaces as well. Now the following works exec_long_running_proc(command = "ls", args=["-l", "~/test file*"])

@Valery Ramusik 2018-09-14 22:20:09

Invoke is a Python (2.7 and 3.4+) task execution tool & library. It provides a clean, high level API for running shell commands

>>> from invoke import run
>>> cmd = "pip install -r requirements.txt"
>>> result = run(cmd, hide=True, warn=True)
>>> print(result.ok)
True
>>> print(result.stdout.splitlines()[-1])
Successfully installed invocations-0.13.0 pep8-1.5.7 spec-1.3.1

@user9074332 2019-03-12 02:00:27

This is a great library. I was trying to explain it to a coworker the other day adn described it like this: invoke is to subprocess as requests is to urllib3.

@Honza Javorek 2013-04-11 17:17:53

With Standard Library

The Use subprocess module (Python 3):

import subprocess
subprocess.run(['ls', '-l'])

It is the recommended standard way. However, more complicated tasks (pipes, output, input, etc.) can be tedious to construct and write.

Note on Python version: If you are still using Python 2, subprocess.call works in a similar way.

ProTip: shlex.split can help you to parse the command for run, call, and other subprocess functions in case you don't want (or you can't!) provide them in form of lists:

import shlex
import subprocess
subprocess.run(shlex.split('ls -l'))

With External Dependencies

If you do not mind external dependencies, use plumbum:

from plumbum.cmd import ifconfig
print(ifconfig['wlan0']())

It is the best subprocess wrapper. It's cross-platform, i.e. it works on both Windows and Unix-like systems. Install by pip install plumbum.

Another popular library is sh:

from sh import ifconfig
print(ifconfig('wlan0'))

However, sh dropped Windows support, so it's not as awesome as it used to be. Install by pip install sh.

@abhi krishnan 2018-07-27 08:10:20

use the os module

import os
os.system("your command")

eg

import os
os.system("ifconfig")

@tripleee 2018-12-03 04:59:29

This duplicates a (slightly) more detailed answer from the previous April, which however also fails to point out the caveats.

@Samadi Salahedine 2018-04-30 13:47:17

It can be this simple:

import os
cmd = "your command"
os.system(cmd)

@tripleee 2018-12-03 05:02:26

This fails to point out the drawbacks, which are explained in much more detail in PEP-324. The documentation for os.system explicitly recommends avoiding it in favor of subprocess.

@dportman 2018-05-08 20:49:55

If you need to call a shell command from a Python notebook (like Jupyter, Zeppelin, Databricks, or Google Cloud Datalab) you can just use the ! prefix.

For example,

!ls -ilF

@rashok 2018-04-04 06:57:44

Calling an external command in Python

A simple way to call an external command is using os.system(...). And this function returns the exit value of the command. But the drawback is we won't get stdout and stderr.

ret = os.system('some_cmd.sh')
if ret != 0 :
    print 'some_cmd.sh execution returned failure'

Calling an external command in Python in background

subprocess.Popen provides more flexibility for running an external command rather than using os.system. We can start a command in the background and wait for it to finish. And after that we can get the stdout and stderr.

proc = subprocess.Popen(["./some_cmd.sh"], stdout=subprocess.PIPE)
print 'waiting for ' + str(proc.pid)
proc.wait()
print 'some_cmd.sh execution finished'
(out, err) = proc.communicate()
print 'some_cmd.sh output : ' + out

Calling a long running external command in Python in the background and stop after some time

We can even start a long running process in the background using subprocess.Popen and kill it after sometime once its task is done.

proc = subprocess.Popen(["./some_long_run_cmd.sh"], stdout=subprocess.PIPE)
# Do something else
# Now some_long_run_cmd.sh exeuction is no longer needed, so kill it
os.system('kill -15 ' + str(proc.pid))
print 'Output : ' proc.communicate()[0]

@Asav Patel 2017-10-24 23:30:20

I have written a wrapper to handle errors and redirecting output and other stuff.

import shlex
import psutil
import subprocess

def call_cmd(cmd, stdout=sys.stdout, quiet=False, shell=False, raise_exceptions=True, use_shlex=True, timeout=None):
    """Exec command by command line like 'ln -ls "/var/log"'
    """
    if not quiet:
        print("Run %s", str(cmd))
    if use_shlex and isinstance(cmd, (str, unicode)):
        cmd = shlex.split(cmd)
    if timeout is None:
        process = subprocess.Popen(cmd, stdout=stdout, stderr=sys.stderr, shell=shell)
        retcode = process.wait()
    else:
        process = subprocess.Popen(cmd, stdout=stdout, stderr=sys.stderr, shell=shell)
        p = psutil.Process(process.pid)
        finish, alive = psutil.wait_procs([p], timeout)
        if len(alive) > 0:
            ps = p.children()
            ps.insert(0, p)
            print('waiting for timeout again due to child process check')
            finish, alive = psutil.wait_procs(ps, 0)
        if len(alive) > 0:
            print('process {} will be killed'.format([p.pid for p in alive]))
            for p in alive:
                p.kill()
            if raise_exceptions:
                print('External program timeout at {} {}'.format(timeout, cmd))
                raise CalledProcessTimeout(1, cmd)
        retcode = process.wait()
    if retcode and raise_exceptions:
        print("External program failed %s", str(cmd))
        raise subprocess.CalledProcessError(retcode, cmd)

You can call it like this:

cmd = 'ln -ls "/var/log"'
stdout = 'out.txt'
call_cmd(cmd, stdout)

@Garfield 2012-07-25 06:51:50

Very simplest way to run any command and get the result back:

from commands import getstatusoutput

try:
    return getstatusoutput("ls -ltr")
except Exception, e:
    return None

@719016 2014-03-04 14:16:18

Is this going to be deprecated in python 3.0?

@tripleee 2018-12-03 05:06:54

Indeed, the commands documentation from Python 2.7 says it was deprecated in 2.6 and will be removed in 3.0.

@Facundo Casco 2011-04-28 20:29:29

If you need the output from the command you are calling, then you can use subprocess.check_output (Python 2.7+).

>>> subprocess.check_output(["ls", "-l", "/dev/null"])
'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

Also note the shell parameter.

If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory. However, note that Python itself offers implementations of many shell-like features (in particular, glob, fnmatch, os.walk(), os.path.expandvars(), os.path.expanduser(), and shutil).

@Bruno Bronosky 2018-01-30 18:18:54

Note that check_output requires a list rather than a string. If you don't rely on quoted spaces to make your call valid, the simplest, most readable way to do this is subprocess.check_output("ls -l /dev/null".split()).

@Atinc Delican 2010-01-08 21:11:30

There is another difference here which is not mentioned previously.

subprocess.Popen executes the <command> as a subprocess. In my case, I need to execute file <a> which needs to communicate with another program, <b>.

I tried subprocess, and execution was successful. However <b> could not communicate with <a>. Everything is normal when I run both from the terminal.

One more: (NOTE: kwrite behaves different from other applications. If you try the below with Firefox, the results will not be the same.)

If you try os.system("kwrite"), program flow freezes until the user closes kwrite. To overcome that I tried instead os.system(konsole -e kwrite). This time program continued to flow, but kwrite became the subprocess of the console.

Anyone runs the kwrite not being a subprocess (i.e. in the system monitor it must appear at the leftmost edge of the tree).

@Peter Mortensen 2018-06-03 20:14:54

What do you mean by "Anyone runs the kwrite not being a subprocess"?

@nimish 2008-09-18 01:37:24

import os
os.system("your command")

Note that this is dangerous, since the command isn't cleaned. I leave it up to you to google for the relevant documentation on the 'os' and 'sys' modules. There are a bunch of functions (exec* and spawn*) that will do similar things.

@Peter Mortensen 2018-06-03 17:10:31

What do you mean by "the command isn't cleaned"?

@nimish 2018-06-06 16:01:34

No idea what I meant nearly a decade ago (check the date!), but if I had to guess, it would be that there's no validation done.

@tripleee 2018-12-03 05:11:39

This should now point to subprocess as a slightly more versatile and portable solution. Running external commands is of course inherently unportable (you have to make sure the command is available on every architecture you need to support) and passing user input as an external command is inherently unsafe.

@nimish 2018-12-03 18:41:52

Note the timestamp on this guy: the "correct" answer has 40x the votes and is answer #1.

@user8468899 2018-01-31 17:42:38

As an example (in Linux):

import subprocess
subprocess.run('mkdir test.dir', shell=True)

This creates test.dir in the current directory. Note that this also works:

import subprocess
subprocess.call('mkdir test.dir', shell=True)

The equivalent code using os.system is:

import os
os.system('mkdir test.dir')

Best practice would be to use subprocess instead of os, with .run favored over .call. All you need to know about subprocess is here. Also, note that all Python documentation is available for download from here. I downloaded the PDF packed as .zip. I mention this because there's a nice overview of the os module in tutorial.pdf (page 81). Besides, it's an authoritative resource for Python coders.

@Nick Predey 2018-03-20 18:54:34

According to docs.python.org/2/library/…, "shell=True" may raise a security concern.

@user8468899 2018-03-21 19:49:03

@Nick Predley: noted, but "shell=False" doesn't perform the desired function. What specifically are the security concerns and what's the alternative? Please let me know asap: I do not wish to post anything which may cause problems for anyone viewing this.

@tripleee 2018-12-03 05:14:08

The basic warning is in the documentation but this question explains it in more detail: stackoverflow.com/questions/3172470/…

@mdwhatcott 2012-08-13 18:36:32

I quite like shell_command for its simplicity. It's built on top of the subprocess module.

Here's an example from the docs:

>>> from shell_command import shell_call
>>> shell_call("ls *.py")
setup.py  shell_command.py  test_shell_command.py
0
>>> shell_call("ls -l *.py")
-rw-r--r-- 1 ncoghlan ncoghlan  391 2011-12-11 12:07 setup.py
-rw-r--r-- 1 ncoghlan ncoghlan 7855 2011-12-11 16:16 shell_command.py
-rwxr-xr-x 1 ncoghlan ncoghlan 8463 2011-12-11 16:17 test_shell_command.py
0

@Aaron Hall 2017-10-18 16:37:52

Calling an external command in Python

Simple, use subprocess.run, which returns a CompletedProcess object:

>>> import subprocess
>>> completed_process = subprocess.run('python --version')
Python 3.6.1 :: Anaconda 4.4.0 (64-bit)
>>> completed_process
CompletedProcess(args='python --version', returncode=0)

Why?

As of Python 3.5, the documentation recommends subprocess.run:

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

Here's an example of the simplest possible usage - and it does exactly as asked:

>>> import subprocess
>>> completed_process = subprocess.run('python --version')
Python 3.6.1 :: Anaconda 4.4.0 (64-bit)
>>> completed_process
CompletedProcess(args='python --version', returncode=0)

run waits for the command to successfully finish, then returns a CompletedProcess object. It may instead raise TimeoutExpired (if you give it a timeout= argument) or CalledProcessError (if it fails and you pass check=True).

As you might infer from the above example, stdout and stderr both get piped to your own stdout and stderr by default.

We can inspect the returned object and see the command that was given and the returncode:

>>> completed_process.args
'python --version'
>>> completed_process.returncode
0

Capturing output

If you want to capture the output, you can pass subprocess.PIPE to the appropriate stderr or stdout:

>>> cp = subprocess.run('python --version', 
                        stderr=subprocess.PIPE, 
                        stdout=subprocess.PIPE)
>>> cp.stderr
b'Python 3.6.1 :: Anaconda 4.4.0 (64-bit)\r\n'
>>> cp.stdout
b''

(I find it interesting and slightly counterintuitive that the version info gets put to stderr instead of stdout.)

Pass a command list

One might easily move from manually providing a command string (like the question suggests) to providing a string built programmatically. Don't build strings programmatically. This is a potential security issue. It's better to assume you don't trust the input.

>>> import textwrap
>>> args = ['python', textwrap.__file__]
>>> cp = subprocess.run(args, stdout=subprocess.PIPE)
>>> cp.stdout
b'Hello there.\r\n  This is indented.\r\n'

Note, only args should be passed positionally.

Full Signature

Here's the actual signature in the source and as shown by help(run):

def run(*popenargs, input=None, timeout=None, check=False, **kwargs):

The popenargs and kwargs are given to the Popen constructor. input can be a string of bytes (or unicode, if specify encoding or universal_newlines=True) that will be piped to the subprocess's stdin.

The documentation describes timeout= and check=True better than I could:

The timeout argument is passed to Popen.communicate(). If the timeout expires, the child process will be killed and waited for. The TimeoutExpired exception will be re-raised after the child process has terminated.

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.

and this example for check=True is better than one I could come up with:

>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1

Expanded Signature

Here's an expanded signature, as given in the documentation:

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, 
shell=False, cwd=None, timeout=None, check=False, encoding=None, 
errors=None)

Note that this indicates that only the args list should be passed positionally. So pass the remaining arguments as keyword arguments.

Popen

When use Popen instead? I would struggle to find use-case based on the arguments alone. Direct usage of Popen would, however, give you access to its methods, including poll, 'send_signal', 'terminate', and 'wait'.

Here's the Popen signature as given in the source. I think this is the most precise encapsulation of the information (as opposed to help(Popen)):

def __init__(self, args, bufsize=-1, executable=None,
             stdin=None, stdout=None, stderr=None,
             preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
             shell=False, cwd=None, env=None, universal_newlines=False,
             startupinfo=None, creationflags=0,
             restore_signals=True, start_new_session=False,
             pass_fds=(), *, encoding=None, errors=None):

But more informative is the Popen documentation:

subprocess.Popen(args, bufsize=-1, executable=None, stdin=None,
                 stdout=None, stderr=None, preexec_fn=None, close_fds=True,
                 shell=False, cwd=None, env=None, universal_newlines=False,
                 startupinfo=None, creationflags=0, restore_signals=True,
                 start_new_session=False, pass_fds=(), *, encoding=None, errors=None)

Execute a child program in a new process. On POSIX, the class uses os.execvp()-like behavior to execute the child program. On Windows, the class uses the Windows CreateProcess() function. The arguments to Popen are as follows.

Understanding the remaining documentation on Popen will be left as an exercise for the reader.

@James Hirschorn 2018-10-16 18:05:17

A simple example of two-way communication between a primary process and a subprocess can be found here: stackoverflow.com/a/52841475/1349673

@tripleee 2018-12-03 05:16:05

The first example should probably have shell=True or (better yet) pass the command as a list.

@newtover 2010-02-12 10:15:34

Some hints on detaching the child process from the calling one (starting the child process in background).

Suppose you want to start a long task from a CGI-script, that is the child process should live longer than the CGI-script execution process.

The classical example from the subprocess module docs is:

import subprocess
import sys

# some code here

pid = subprocess.Popen([sys.executable, "longtask.py"]) # call subprocess

# some more code here

The idea here is that you do not want to wait in the line 'call subprocess' until the longtask.py is finished. But it is not clear what happens after the line 'some more code here' from the example.

My target platform was freebsd, but the development was on windows, so I faced the problem on windows first.

On windows (win xp), the parent process will not finish until the longtask.py has finished its work. It is not what you want in CGI-script. The problem is not specific to Python, in PHP community the problems are the same.

The solution is to pass DETACHED_PROCESS Process Creation Flag to the underlying CreateProcess function in win API. If you happen to have installed pywin32 you can import the flag from the win32process module, otherwise you should define it yourself:

DETACHED_PROCESS = 0x00000008

pid = subprocess.Popen([sys.executable, "longtask.py"],
                       creationflags=DETACHED_PROCESS).pid

/* UPD 2015.10.27 @eryksun in a comment below notes, that the semantically correct flag is CREATE_NEW_CONSOLE (0x00000010) */

On freebsd we have another problem: when the parent process is finished, it finishes the child processes as well. And that is not what you want in CGI-script either. Some experiments showed that the problem seemed to be in sharing sys.stdout. And the working solution was the following:

pid = subprocess.Popen([sys.executable, "longtask.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)

I have not checked the code on other platforms and do not know the reasons of the behaviour on freebsd. If anyone knows, please share your ideas. Googling on starting background processes in Python does not shed any light yet.

@maranas 2010-04-09 08:09:26

i noticed a possible "quirk" with developing py2exe apps in pydev+eclipse. i was able to tell that the main script was not detached because eclipse's output window was not terminating; even if the script executes to completion it is still waiting for returns. but, when i tried compiling to a py2exe executable, the expected behavior occurs (runs the processes as detached, then quits). i am not sure, but the executable name is not in the process list anymore. this works for all approaches (os.system("start *"), os.spawnl with os.P_DETACH, subprocs, etc.)

@Alexey Lebedev 2012-04-16 10:04:33

Windows gotcha: even though I spawned process with DETACHED_PROCESS, when I killed my Python daemon all ports opened by it wouldn't free until all spawned processes terminate. WScript.Shell solved all my problems. Example here: pastebin.com/xGmuvwSx

@jfs 2012-11-16 14:16:42

you might also need CREATE_NEW_PROCESS_GROUP flag. See Popen waiting for child process even when the immediate child has terminated

@ubershmekel 2014-10-27 21:01:35

I'm seeing import subprocess as sp;sp.Popen('calc') not waiting for the subprocess to complete. It seems the creationflags aren't necessary. What am I missing?

@newtover 2014-10-28 12:25:50

@ubershmekel, I am not sure what you mean and don't have a windows installation. If I recall correctly, without the flags you can not close the cmd instance from which you started the calc.

@ubershmekel 2014-10-30 05:45:20

I'm on Windows 8.1 and calc seems to survive the closing of python.

@SuperBiasedMan 2015-05-05 13:13:32

Is there any significance to using '0x00000008'? Is that a specific value that has to be used or one of multiple options?

@Eryk Sun 2015-10-27 00:27:30

The following is incorrect: "[o]n windows (win xp), the parent process will not finish until the longtask.py has finished its work". The parent will exit normally, but the console window (conhost.exe instance) only closes when the last attached process exits, and the child may have inherited the parent's console. Setting DETACHED_PROCESS in creationflags avoids this by preventing the child from inheriting or creating a console. If you instead want a new console, use CREATE_NEW_CONSOLE (0x00000010).

@Eryk Sun 2015-10-27 17:37:15

I didn't mean that executing as a detached process is incorrect. That said, you may need to set the standard handles to files, pipes, or os.devnull because some console programs exit with an error otherwise. Create a new console when you want the child process to interact with the user concurrently with the parent process. It would be confusing to try to do both in a single window.

@Dr_Zaszuś 2018-03-08 08:56:54

stdout=subprocess.PIPE will make your code hang up if you have long output from a child. For more details see thraxil.org/users/anders/posts/2008/03/13/…

@Charlie Parker 2019-02-24 19:05:29

is there not an OS-agnostic way to have the process run in the background?

@Charlie Parker 2019-02-24 19:38:51

your answer seems strange to me. I just opened a subprocess.Popen and nothing bad happened (not had to wait). Why exactly do we need to worry about the scenario you are pointing out? I'm skeptical.

@Tom Fuller 2016-10-29 14:02:50

There are lots of different libraries which allow you to call external commands with Python. For each library I've given a description and shown an example of calling an external command. The command I used as the example is ls -l (list all files). If you want to find out more about any of the libraries I've listed and linked the documentation for each of them.

Sources:

These are all the libraries:

Hopefully this will help you make a decision on which library to use :)

subprocess

Subprocess allows you to call external commands and connect them to their input/output/error pipes (stdin, stdout, and stderr). Subprocess is the default choice for running commands, but sometimes other modules are better.

subprocess.run(["ls", "-l"]) # Run command
subprocess.run(["ls", "-l"], stdout=subprocess.PIPE) # This will run the command and return any output
subprocess.run(shlex.split("ls -l")) # You can also use the shlex library to split the command

os

os is used for "operating system dependent functionality". It can also be used to call external commands with os.system and os.popen (Note: There is also a subprocess.popen). os will always run the shell and is a simple alternative for people who don't need to, or don't know how to use subprocess.run.

os.system("ls -l") # run command
os.popen("ls -l").read() # This will run the command and return any output

sh

sh is a subprocess interface which lets you call programs as if they were functions. This is useful if you want to run a command multiple times.

sh.ls("-l") # Run command normally
ls_cmd = sh.Command("ls") # Save command as a variable
ls_cmd() # Run command as if it were a function

plumbum

plumbum is a library for "script-like" Python programs. You can call programs like functions as in sh. Plumbum is useful if you want to run a pipeline without the shell.

ls_cmd = plumbum.local("ls -l") # get command
ls_cmd() # run command

pexpect

pexpect lets you spawn child applications, control them and find patterns in their output. This is a better alternative to subprocess for commands that expect a tty on Unix.

pexpect.run("ls -l") # Run command as normal
child = pexpect.spawn('scp foo [email protected]:.') # Spawns child application
child.expect('Password:') # When this is the output
child.sendline('mypassword')

fabric

fabric is a Python 2.5 and 2.7 library. It allows you to execute local and remote shell commands. Fabric is simple alternative for running commands in a secure shell (SSH)

fabric.operations.local('ls -l') # Run command as normal
fabric.operations.local('ls -l', capture = True) # Run command and receive output

envoy

envoy is known as "subprocess for humans". It is used as a convenience wrapper around the subprocess module.

r = envoy.run("ls -l") # Run command
r.std_out # get output

commands

commands contains wrapper functions for os.popen, but it has been removed from Python 3 since subprocess is a better alternative.

The edit was based on J.F. Sebastian's comment.

@Rajiv Sharma 2016-10-11 02:26:49

Here is calling an external command and return or print the command's output:

Python Subprocess check_output is good for

Run command with arguments and return its output as a byte string.

import subprocess
proc = subprocess.check_output('ipconfig /all')
print proc

@tripleee 2018-12-03 05:22:16

The argument should properly be tokenized into a list, or you should explicitly pass in shell=True. In Python 3.x (where x > 3 I think) you can retrieve the output as a proper string with universal_newlines=True and you probably want to switch to subproces.run()

@liuyip 2016-09-12 09:44:09

There are many ways to call a command.

  • For example:

if and.exe needs two parameters. In cmd we can call sample.exe use this: and.exe 2 3 and it show 5 on screen.

If we use a Python script to call and.exe, we should do like..

  1. os.system(cmd,...)

    • os.system(("and.exe" + " " + "2" + " " + "3"))
  2. os.popen(cmd,...)

    • os.popen(("and.exe" + " " + "2" + " " + "3"))
  3. subprocess.Popen(cmd,...)
    • subprocess.Popen(("and.exe" + " " + "2" + " " + "3"))

It's too hard, so we can join cmd with a space:

import os
cmd = " ".join(exename,parameters)
os.popen(cmd)

@tripleee 2018-12-03 05:25:03

os.popen should not be recommended and perhaps even mentioned any longer. The subpocess example should pass the arguments as a list instead of joining them with spaces.

@David Okwii 2016-06-24 11:29:00

Use:

import subprocess

p = subprocess.Popen("df -h", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
print p.split("\n")

It gives nice output which is easier to work with:

['Filesystem      Size  Used Avail Use% Mounted on',
 '/dev/sda6        32G   21G   11G  67% /',
 'none            4.0K     0  4.0K   0% /sys/fs/cgroup',
 'udev            1.9G  4.0K  1.9G   1% /dev',
 'tmpfs           387M  1.4M  386M   1% /run',
 'none            5.0M     0  5.0M   0% /run/lock',
 'none            1.9G   58M  1.9G   3% /run/shm',
 'none            100M   32K  100M   1% /run/user',
 '/dev/sda5       340G  222G  100G  69% /home',
 '']

@Swadhikar C 2016-06-17 09:14:24

In Windows you can just import the subprocess module and run external commands by calling subprocess.Popen(), subprocess.Popen().communicate() and subprocess.Popen().wait() as below:

# Python script to run a command line
import subprocess

def execute(cmd):
    """
        Purpose  : To execute a command and return exit status
        Argument : cmd - command to execute
        Return   : exit_code
    """
    process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    (result, error) = process.communicate()

    rc = process.wait()

    if rc != 0:
        print "Error: failed to execute command:", cmd
        print error
    return result
# def

command = "tasklist | grep python"
print "This process detail: \n", execute(command)

Output:

This process detail:
python.exe                     604 RDP-Tcp#0                  4      5,660 K

Related Questions

Sponsored Content

21 Answered Questions

[SOLVED] Does Python have a ternary conditional operator?

35 Answered Questions

[SOLVED] Check if a directory exists in a shell script

  • 2008-09-12 20:06:25
  • Grundlefleck
  • 2640780 View
  • 3476 Score
  • 35 Answer
  • Tags:   shell unix posix

28 Answered Questions

[SOLVED] How do I prompt for Yes/No/Cancel input in a Linux shell script?

18 Answered Questions

[SOLVED] Convert bytes to a string?

21 Answered Questions

[SOLVED] How to print without newline or space?

16 Answered Questions

[SOLVED] What are metaclasses in Python?

14 Answered Questions

[SOLVED] How do I set a variable to the output of a command in Bash?

13 Answered Questions

[SOLVED] Find current directory and file's directory

  • 2011-02-28 01:51:21
  • John Howard
  • 2546939 View
  • 1903 Score
  • 13 Answer
  • Tags:   python directory

20 Answered Questions

[SOLVED] Calling shell commands from Ruby

  • 2008-08-05 12:56:52
  • CodingWithoutComments
  • 498381 View
  • 998 Score
  • 20 Answer
  • Tags:   ruby shell interop

15 Answered Questions

[SOLVED] In the shell, what does " 2>&1 " mean?

  • 2009-05-03 22:57:00
  • Tristan Havelick
  • 997785 View
  • 2061 Score
  • 15 Answer
  • Tags:   bash shell unix redirect

Sponsored Content