By harperville


2016-05-05 18:44:58 8 Comments

My goal is simple: kick off rsync and DO NOT WAIT.

Python 2.7.9 on Debian

Sample code:

rsync_cmd = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}@{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1)
rsync_cmd2 = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}@{1}:'{2}' {3} &".format(remote_user, remote_server, file1, file1)
rsync_path = "/usr/bin/rsync"
rsync_args = shlex.split("-a -e 'ssh -i /home/mysuser/.ssh/id_rsa' {0}@{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1))
#subprocess.call(rsync_cmd, shell=True)     # This isn't supposed to work but I tried it
#subprocess.Popen(rsync_cmd, shell=True)    # This is supposed to be the solution but not for me
#subprocess.Popen(rsync_cmd2, shell=True)   # Adding my own shell "&" to background it, still fails
#subprocess.Popen(rsync_cmd, shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)  # This doesn't work
#subprocess.Popen(shlex.split(rsync_cmd))   # This doesn't work
#os.execv(rsync_path, rsync_args)           # This doesn't work
#os.spawnv(os.P_NOWAIT, rsync_path, rsync_args) # This doesn't work
#os.system(rsync_cmd2)                      # This doesn't work
print "DONE"

(I've commented out the execution commands only because I'm actually keeping all of my trials in my code so that I know what I've done and what I haven't done. Obviously, I would run the script with the right line uncommented.)

What happens is this...I can watch the transfer on the server and when it's finished, then I get a "DONE" printed to the screen.

What I'd like to have happen is a "DONE" printed immediately after issuing the rsync command and for the transfer to start.

Seems very straight-forward. I've followed details outlined in other posts, like this one and this one, but something is preventing it from working for me.

Thanks ahead of time.

(I have tried everything I can find in StackExchange and don't feel like this is a duplicate because I still can't get it to work. Something isn't right in my setup and need help.)

2 comments

@jfs 2016-05-05 20:15:36

Popen() starts a child process—it does not wait for it to exit. You have to call .wait() method explicitly if you want to wait for the child process. In that sense, all subprocesses are background processes.

On the other hand, the child process may inherit various properties/resources from the parent such as open file descriptors, the process group, its control terminal, some signal configuration, etc—it may lead to preventing ancestors processes to exit e.g., Python subprocess .check_call vs .check_output or the child may die prematurely on Ctrl-C (SIGINT signal is sent to the foreground process group) or if the terminal session is closed (SIGHUP).

To disassociate the child process completely, you should make it a daemon. Sometimes something in between could be enough e.g., it is enough to redirect the inherited stdout in a grandchild so that .communicate() in the parent would return when its immediate child exits.

@Charlie Parker 2019-02-23 21:45:38

is there a way to keep sending the running process data/commands?

@jfs 2019-02-24 14:54:01

@CharlieParker naturally, for example

@Viach Kakovskyi 2016-05-05 19:14:46

Here is verified example for Python REPL:

>>> import subprocess
>>> import sys
>>> p = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(100)'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT); print('finished')
finished

How to verify that via another terminal window:

$ ps aux | grep python

Output:

user           32820   0.0  0.0  2447684   3972 s003  S+   10:11PM   0:00.01 /Users/user/venv/bin/python -c import time; time.sleep(100)

@harperville 2016-05-05 19:24:19

This works for me in the interpreter as well, but not my script. I'll have to start tearing it down to nothing and build it back up. I'll also try Process() as recommended by @asav above.

@harperville 2016-05-05 19:30:23

I printed a message to the log after calling Popen as your example shows and I saw my message in the log. The issue is elsewhere that I have yet to find but as usual, your suggestion helped me determine Popen was not the issue. Thanks for the help! I am using p = subprocess(rsync_dict, close_fds=True) is it works as advertised.

@Viach Kakovskyi 2016-05-05 19:32:04

@harperville I feel like the reason of the behavior is in your command, which probably waits for some input from stdin.

@jfs 2018-06-12 18:45:26

use stdout=DEVNULL instead of stdout=PIPE otherwise the child process may stall when the OS pipe buffer fills up.

@Charlie Parker 2019-02-24 19:01:19

can you explain why your answer works?

@Charlie Parker 2019-02-24 19:16:01

the answer by jfs implies that just creating an instance of Popen makes it detached by itself or at least does not make your python script wait. If that is the case why does your answer seem much more complicated than just that? Also, in the original question you posted a link that references OS depedencies. Is your answer OS agnostic?

@jfs 2019-06-11 15:47:07

@CharlieParker I don't see the word "detach" in my answer. It is trivial to check that Popen() does not wait for the child process to exit, you don't have to take my word for it, you can read the Popen() docs (what it promises), test it (how it actually behaves in your environment). It depends, what additional steps might be required (you don't need to create a full daemon in every single case). For more details, you could follow the links in my answer.

Related Questions

Sponsored Content

13 Answered Questions

[SOLVED] How can I make a time delay in Python?

10 Answered Questions

[SOLVED] Does Python have a string 'contains' substring method?

33 Answered Questions

[SOLVED] How to get the current time in Python

  • 2009-01-06 04:54:23
  • user46646
  • 2760169 View
  • 2401 Score
  • 33 Answer
  • Tags:   python datetime time

20 Answered Questions

26 Answered Questions

[SOLVED] How can I safely create a nested directory?

21 Answered Questions

15 Answered Questions

[SOLVED] What are metaclasses in Python?

28 Answered Questions

[SOLVED] Finding the index of an item given a list containing it in Python

  • 2008-10-07 01:39:38
  • Eugene M
  • 3277766 View
  • 2697 Score
  • 28 Answer
  • Tags:   python list indexing

56 Answered Questions

[SOLVED] Calling an external command in Python

25 Answered Questions

[SOLVED] Is there a way to run Python on Android?

Sponsored Content