By Alok


2008-09-16 00:50:10 8 Comments

I have a text file on my local machine that is generated by a daily Python script run in cron.

I would like to add a bit of code to have that file sent securely to my server over SSH.

14 comments

@michael 2017-07-12 01:51:37

Using the external resource paramiko;

    from paramiko import SSHClient
    from scp import SCPClient
    import os

    ssh = SSHClient() 
    ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))
    ssh.connect(server, username='username', password='password')
    with SCPClient(ssh.get_transport()) as scp:
            scp.put('test.txt', 'test2.txt')

@Pradeep Pathak 2019-07-02 10:16:23

You can do something like this, to handle the host key checking as well

import os
os.system("sshpass -p password scp -o StrictHostKeyChecking=no local_file_path [email protected]:remote_path")

@Roberto Marzocchi 2018-01-04 15:40:07

a very simple approach is the following:

import os
os.system('sshpass -p "password" scp [email protected]:/path/to/file ./')

no python library are required (only os) and it works

@JavDomGom 2018-11-27 17:32:19

Try this if you wan't to use SSL certificates:

import subprocess

try:
    # Set scp and ssh data.
    connUser = 'john'
    connHost = 'my.host.com'
    connPath = '/home/john/'
    connPrivateKey = '/home/user/myKey.pem'

    # Use scp to send file from local to host.
    scp = subprocess.Popen(['scp', '-i', connPrivateKey, 'myFile.txt', '{}@{}:{}'.format(connUser, connHost, connPath)])

except CalledProcessError:
    print('ERROR: Connection to host failed!')

@Shawn 2018-11-16 19:47:12

You can use the vassal package, which is exactly designed for this.

All you need is to install vassal and do

from vassal.terminal import Terminal
shell = Terminal(["scp [email protected]:/home/foo.txt foo_local.txt"])
shell.run()

Also, it will save you authenticate credential and don't need to type them again and again.

@Jonno_FTW 2018-02-06 02:58:25

I used sshfs to mount the remote directory via ssh, and shutil to copy the files:

$ mkdir ~/sshmount
$ sshfs [email protected]:/path/to/remote/dst ~/sshmount

Then in python:

import shutil
shutil.copy('a.txt', '~/sshmount')

This method has the advantage that you can stream data over if you are generating data rather than caching locally and sending a single large file.

@Tony Meyer 2008-09-16 05:27:59

To do this in Python (i.e. not wrapping scp through subprocess.Popen or similar) with the Paramiko library, you would do something like this:

import os
import paramiko

ssh = paramiko.SSHClient() 
ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))
ssh.connect(server, username=username, password=password)
sftp = ssh.open_sftp()
sftp.put(localpath, remotepath)
sftp.close()
ssh.close()

(You would probably want to deal with unknown hosts, errors, creating any directories necessary, and so on).

@Jim Carroll 2009-10-01 13:28:50

paramiko has a nice sftp.put(self, localpath, remotepath, callback=None) function too, so you don't have to open write, and close each file.

@Drahkar 2012-09-14 23:47:12

I would note that SFTP is not the same thing as SCP.

@Tony Meyer 2012-09-20 21:50:58

@Drahkar the question asks for the file to be sent over SSH. That's what this does.

@doda 2012-10-09 14:49:09

Note: to make this work out of the box add ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) after instantiating ssh.

@Tom 2013-09-19 23:32:34

@doda only if you don't want to use the known_hosts file

@lpapp 2013-12-10 09:48:10

Any idea for an "scp" based solution? SFTP does not work with our server.

@Sekai 2014-12-31 16:09:37

Guys, is it safe to use the server password ? is there a possibility to use the Private Key ?

@Joel B 2015-04-24 16:38:17

It's certainly possible to use a private key: gist.github.com/batok/2352501

@Maviles 2016-07-24 20:12:13

Reached the same problem, but instead of "hacking" or emulating command line:

Found this answer here.

from paramiko import SSHClient
from scp import SCPClient

ssh = SSHClient()
ssh.load_system_host_keys()
ssh.connect('example.com')

with SCPClient(ssh.get_transport()) as scp:
    scp.put('test.txt', 'test2.txt')
    scp.get('test2.txt')

@Chuck 2017-12-10 01:24:13

Note - this relies on the scp package. As of Dec 2017, the latest update to that package was in 2015 and the version number is 0.1.x. Not sure I would want to add this dependency.

@Adriano_epifas 2018-08-02 14:26:05

so, it´s 2018 now and in may they released version 0.11.0. So, it seems SCPClient is not dead after all?

@Blair Conrad 2008-09-16 00:58:49

You'd probably use the subprocess module. Something like this:

import subprocess
p = subprocess.Popen(["scp", myfile, destination])
sts = os.waitpid(p.pid, 0)

Where destination is probably of the form [email protected]:remotepath. Thanks to @Charles Duffy for pointing out the weakness in my original answer, which used a single string argument to specify the scp operation shell=True - that wouldn't handle whitespace in paths.

The module documentation has examples of error checking that you may want to perform in conjunction with this operation.

Ensure that you've set up proper credentials so that you can perform an unattended, passwordless scp between the machines. There is a stackoverflow question for this already.

@Charles Duffy 2008-10-15 12:07:24

Using subprocess.Popen is the Right Thing. Passing it a string rather than an array (and using shell=True) is the Wrong Thing, as it means filenames with spaces don't work correctly.

@db42 2011-12-18 12:19:52

instead of "sts = os.waitpid(p.pid, 0)", we can use "sts = p.wait()".

@lpapp 2013-12-10 09:48:43

is there an execution free way of it with direct python API for this protocol?

@jfs 2014-03-28 11:10:21

subprocess.check_call(['scp', myfile, destination]) could be used instead since Python 2.5 (2006)

@jfs 2014-03-28 11:10:40

@LaszloPapp: see How to scp in python?

@lpapp 2014-03-28 11:45:04

@J.F.Sebastian: yep, I checked it out in December. My upvotes prove that, at least. :) Thanks for the follow-up though.

@jfs 2014-03-28 10:41:18

fabric could be used to upload files vis ssh:

#!/usr/bin/env python
from fabric.api import execute, put
from fabric.network import disconnect_all

if __name__=="__main__":
    import sys
    # specify hostname to connect to and the remote/local paths
    srcdir, remote_dirname, hostname = sys.argv[1:]
    try:
        s = execute(put, srcdir, remote_dirname, host=hostname)
        print(repr(s))
    finally:
        disconnect_all()

@jfs 2014-03-28 10:50:46

Calling scp command via subprocess doesn't allow to receive the progress report inside the script. pexpect could be used to extract that info:

import pipes
import re
import pexpect # $ pip install pexpect

def progress(locals):
    # extract percents
    print(int(re.search(br'(\d+)%$', locals['child'].after).group(1)))

command = "scp %s %s" % tuple(map(pipes.quote, [srcfile, destination]))
pexpect.run(command, events={r'\d+%': progress})

See python copy file in local network (linux -> linux)

@Michael 2008-09-16 01:32:08

There are a couple of different ways to approach the problem:

  1. Wrap command-line programs
  2. use a Python library that provides SSH capabilities (eg - Paramiko or Twisted Conch)

Each approach has its own quirks. You will need to setup SSH keys to enable password-less logins if you are wrapping system commands like "ssh", "scp" or "rsync." You can embed a password in a script using Paramiko or some other library, but you might find the lack of documentation frustrating, especially if you are not familiar with the basics of the SSH connection (eg - key exchanges, agents, etc). It probably goes without saying that SSH keys are almost always a better idea than passwords for this sort of stuff.

NOTE: its hard to beat rsync if you plan on transferring files via SSH, especially if the alternative is plain old scp.

I've used Paramiko with an eye towards replacing system calls but found myself drawn back to the wrapped commands due to their ease of use and immediate familiarity. You might be different. I gave Conch the once-over some time ago but it didn't appeal to me.

If opting for the system-call path, Python offers an array of options such as os.system or the commands/subprocess modules. I'd go with the subprocess module if using version 2.4+.

@Alok 2008-09-16 02:16:43

curious: what's the story on rsync vs. scp?

@pdq 2008-09-16 00:55:43

If you want the simple approach, this should work.

You'll want to ".close()" the file first so you know it's flushed to disk from Python.

import os
os.system("scp FILE [email protected]:PATH")
#e.g. os.system("scp foo.bar [email protected]:/path/to/foo.bar")

You need to generate (on the source machine) and install (on the destination machine) an ssh key beforehand so that the scp automatically gets authenticated with your public ssh key (in other words, so your script doesn't ask for a password).

ssh-keygen example

@Charles Duffy 2008-10-15 12:06:40

subprocess.Popen(["scp", filename, "%(user)[email protected]%(server)s:%(remotepath)s" % vars]).wait() is a much better approach than os.system(), which doesn't handle filenames with spaces correctly.

@KomodoDave 2012-08-15 11:11:51

@Charles What's the '%' notation you're using here called? I'd like to read the documentation on its functionality.

@KomodoDave 2012-08-15 11:18:50

Nevermind, found it. Search for "python format codes", or "python string formatting operations". The '%' operator is known as the "string formatting operator" or "string interpolation operator".

@jfs 2014-03-28 10:37:50

@CharlesDuffy: subprocess.check_call(['scp', srcfile, dest]) could be used since Python 2.5 instead of rc = Popen(..).wait(); if rc != 0: raise ..

@orezvani 2014-11-16 17:58:13

adding ssh key is not a good solution when it is not needed. For example, if you need a one time communication, you don't set up ssh key due to security reasons.

@Drew Olson 2008-09-16 00:56:31

Kind of hacky, but the following should work :)

import os
filePath = "/foo/bar/baz.py"
serverPath = "/blah/boo/boom.py"
os.system("scp "+filePath+" [email protected]:"+serverPath)

@user3521180 2019-04-29 17:30:11

error: serverPath is not defined

Related Questions

Sponsored Content

15 Answered Questions

[SOLVED] How to SSH to a VirtualBox guest externally through a host?

  • 2011-05-06 03:24:03
  • Jordan
  • 526805 View
  • 624 Score
  • 15 Answer
  • Tags:   ssh virtualbox

16 Answered Questions

[SOLVED] How do I copy a file in Python?

21 Answered Questions

[SOLVED] How do I list all files of a directory?

  • 2010-07-08 19:31:22
  • duhhunjonn
  • 3707634 View
  • 3474 Score
  • 21 Answer
  • Tags:   python directory

60 Answered Questions

[SOLVED] How do you split a list into evenly sized chunks?

9 Answered Questions

16 Answered Questions

[SOLVED] How to use SSH to run a shell script on a remote machine?

25 Answered Questions

[SOLVED] How can I safely create a nested directory?

37 Answered Questions

[SOLVED] How do I check whether a file exists without exceptions?

11 Answered Questions

[SOLVED] How to copy a folder from remote to local using scp?

4 Answered Questions

[SOLVED] Transferring files over SSH

  • 2008-12-05 12:46:03
  • Josh Maurice
  • 672975 View
  • 454 Score
  • 4 Answer
  • Tags:   ssh scp

Sponsored Content