By Dheeraj Gundra


2013-02-26 09:47:53 8 Comments

I want to copy a file in s3 bucket using python.

Ex : I have bucket name = test. And in the bucket, I have 2 folders name "dump" & "input". Now I want to copy a file from local directory to S3 "dump" folder using python... Can anyone help me?

12 comments

@Roman Orac 2019-02-08 15:22:27

Upload file to s3 within a session with credentials.

import boto3

session = boto3.Session(
    aws_access_key_id='AWS_ACCESS_KEY_ID',
    aws_secret_access_key='AWS_SECRET_ACCESS_KEY',
)
s3 = session.resource('s3')
# Filename - File to upload
# Bucket - Bucket to upload to (the top level directory under AWS S3)
# Key - S3 object name (can contain subdirectories). If not specified then file_name is used
s3.meta.client.upload_file(Filename='input_file_path', Bucket='bucket_name', Key='s3_output_key')

@Roelant 2019-03-07 21:17:31

What is the s3_output_key?

@Roman Orac 2019-03-08 07:50:46

It is the filename in the S3 bucket.

@Noufal Valapra 2019-08-22 11:10:50

Using boto3

import logging
import boto3
from botocore.exceptions import ClientError


def upload_file(file_name, bucket, object_name=None):
    """Upload a file to an S3 bucket

    :param file_name: File to upload
    :param bucket: Bucket to upload to
    :param object_name: S3 object name. If not specified then file_name is used
    :return: True if file was uploaded, else False
    """

    # If S3 object_name was not specified, use file_name
    if object_name is None:
        object_name = file_name

    # Upload the file
    s3_client = boto3.client('s3')
    try:
        response = s3_client.upload_file(file_name, bucket, object_name)
    except ClientError as e:
        logging.error(e)
        return False
    return True

For more:- https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html

@Willie Cheng 2018-12-03 08:23:35

For upload folder example as following code and S3 folder picture enter image description here

import boto
import boto.s3
import boto.s3.connection
import os.path
import sys    

# Fill in info on data to upload
# destination bucket name
bucket_name = 'willie20181121'
# source directory
sourceDir = '/home/willie/Desktop/x/'  #Linux Path
# destination directory name (on s3)
destDir = '/test1/'   'S3 Path

#max size in bytes before uploading in parts. between 1 and 5 GB recommended
MAX_SIZE = 20 * 1000 * 1000
#size of parts when uploading in parts
PART_SIZE = 6 * 1000 * 1000

access_key = 'MPBVAQ*******IT****'
secret_key = '11t63yDV***********HgUcgMOSN*****'

conn = boto.connect_s3(
        aws_access_key_id = access_key,
        aws_secret_access_key = secret_key,
        host = '******.org.tw',
        is_secure=False,               # uncomment if you are not using ssl
        calling_format = boto.s3.connection.OrdinaryCallingFormat(),
        )
bucket = conn.create_bucket(bucket_name,
        location=boto.s3.connection.Location.DEFAULT)


uploadFileNames = []
for (sourceDir, dirname, filename) in os.walk(sourceDir):
    uploadFileNames.extend(filename)
    break

def percent_cb(complete, total):
    sys.stdout.write('.')
    sys.stdout.flush()

for filename in uploadFileNames:
    sourcepath = os.path.join(sourceDir + filename)
    destpath = os.path.join(destDir, filename)
    print ('Uploading %s to Amazon S3 bucket %s' % \
           (sourcepath, bucket_name))

    filesize = os.path.getsize(sourcepath)
    if filesize > MAX_SIZE:
        print ("multipart upload")
        mp = bucket.initiate_multipart_upload(destpath)
        fp = open(sourcepath,'rb')
        fp_num = 0
        while (fp.tell() < filesize):
            fp_num += 1
            print ("uploading part %i" %fp_num)
            mp.upload_part_from_file(fp, fp_num, cb=percent_cb, num_cb=10, size=PART_SIZE)

        mp.complete_upload()

    else:
        print ("singlepart upload")
        k = boto.s3.key.Key(bucket)
        k.key = destpath
        k.set_contents_from_filename(sourcepath,
                cb=percent_cb, num_cb=10)

PS: For more reference URL

@Samuel Nde 2018-11-29 00:20:09

This is a three liner. Just follow the instructions on the boto3 documentation.

import boto3
s3 = boto3.resource(service_name = 's3')
s3.meta.client.upload_file(Filename = 'C:/foo/bar/baz.filetype', Bucket = 'yourbucketname', Key = 'baz.filetype')

Some important arguments are:

Parameters:

  • Filename (str) -- The path to the file to upload.
  • Bucket (str) -- The name of the bucket to upload to.
  • Key (str) -- The name of the that you want to assign to your file in your s3 bucket. This could be the same as the name of the file or a different name of your choice but the filetype should remain the same.

    Note: I assume that you have saved your credentials in a ~\.aws folder as suggested in the best configuration practices in the boto3 documentation.

  • @HassanSh__3571619 2019-01-25 00:04:49

    Thank you Nde Samuel, that worked with me...One thing that was additional required in my case was to have the bucket already been created, to avoid an error of ""The specified bucket does not exist"".

    @Samuel Nde 2019-01-29 22:11:27

    @HassanSh__3571619 I am glad it helped.

    @Martin 2018-11-13 19:59:12

    xmlstr = etree.tostring(listings,  encoding='utf8', method='xml')
    conn = boto.connect_s3(
            aws_access_key_id = access_key,
            aws_secret_access_key = secret_key,
            # host = '<bucketName>.s3.amazonaws.com',
            host = 'bycket.s3.amazonaws.com',
            #is_secure=False,               # uncomment if you are not using ssl
            calling_format = boto.s3.connection.OrdinaryCallingFormat(),
            )
    conn.auth_region_name = 'us-west-1'
    
    bucket = conn.get_bucket('resources', validate=False)
    key= bucket.get_key('filename.txt')
    key.set_contents_from_string("SAMPLE TEXT")
    key.set_canned_acl('public-read')
    

    @Nick 2018-11-13 21:01:39

    A text explanation with what your code does will be nice!

    @Boris 2017-11-03 15:17:29

    import boto3
    
    s3 = boto3.resource('s3')
    BUCKET = "test"
    
    s3.Bucket(BUCKET).upload_file("your/local/file", "dump/file")
    

    @venkat 2018-03-06 12:44:39

    can you explain this line s3.Bucket(BUCKET).upload_file("your/local/file", "dump/file")

    @Josh S. 2018-03-06 22:16:56

    @venkat "your/local/file" is a filepath such as "/home/file.txt" on the computer using python/boto and "dump/file" is a key name to store the file under in the S3 Bucket. See: boto3.readthedocs.io/en/latest/reference/services/…

    @seeiespi 2018-08-28 21:31:26

    It looks like the user has pre-configured AWS Keys, to do this open your anaconda command prompt and type aws configure, enter your info and you will automatically connect with boto3. Check boto3.readthedocs.io/en/latest/guide/quickstart.html

    @SriK 2018-12-27 22:30:38

    awesome. works perfectly. thanks!

    @barlaensdoonn 2019-03-03 20:03:14

    simplest solution IMO, just as easy as tinys3 but without the need for another external dependency. Also highly recommend setting up your AWS keys with aws configure ahead of time to make your life easier.

    @Tara Prasad Gurung 2019-08-01 10:12:22

    What happens when there are multiple profile in credentials. how to pass the specific credentials

    @vcarel 2015-06-29 10:00:18

    No need to make it that complicated:

    s3_connection = boto.connect_s3()
    bucket = s3_connection.get_bucket('your bucket name')
    key = boto.s3.key.Key(bucket, 'some_file.zip')
    with open('some_file.zip') as f:
        key.send_file(f)
    

    @cgseller 2015-06-29 22:53:34

    This will work, but for large .zip files you may need to use chunked. elastician.com/2010/12/s3-multipart-upload-in-boto.html

    @Leo Prince 2016-01-08 11:25:10

    Yes.. less complicated and commonly used practice

    @Simon 2016-06-24 18:57:58

    I tried this, it doesn't work, but k.set_contents_from_filename(testfile, cb=percent_cb, num_cb=10) does

    @vcarel 2016-06-27 09:37:53

    Are you on boto 2, latest? Anyway, set_contents_from_filename is an even simpler option. Go for it !

    @alfredox 2016-09-15 04:39:16

    Is uploading a folder with lots of files different?

    @Greg Sadetsky 2017-05-24 22:44:55

    key.set_contents_from_filename('some_file.zip') would also work here. See doc. The corresponding code for boto3 can be found here.

    @Pupper 2017-05-25 01:13:14

    I was getting 400 Bad Request from send_file() for some reason. set_contents_from_filename() worked fine on the other hand.

    @Shakti 2017-03-02 13:23:42

    import boto
    from boto.s3.key import Key
    
    AWS_ACCESS_KEY_ID = ''
    AWS_SECRET_ACCESS_KEY = ''
    END_POINT = ''                          # eg. us-east-1
    S3_HOST = ''                            # eg. s3.us-east-1.amazonaws.com
    BUCKET_NAME = 'test'        
    FILENAME = 'upload.txt'                
    UPLOADED_FILENAME = 'dumps/upload.txt'
    # include folders in file path. If it doesn't exist, it will be created
    
    s3 = boto.s3.connect_to_region(END_POINT,
                               aws_access_key_id=AWS_ACCESS_KEY_ID,
                               aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
                               host=S3_HOST)
    
    bucket = s3.get_bucket(BUCKET_NAME)
    k = Key(bucket)
    k.key = UPLOADED_FILENAME
    k.set_contents_from_filename(FILENAME)
    

    @Piyush S. Wanare 2017-02-01 08:19:25

    This will also work:

    import os 
    import boto
    import boto.s3.connection
    from boto.s3.key import Key
    
    try:
    
        conn = boto.s3.connect_to_region('us-east-1',
        aws_access_key_id = 'AWS-Access-Key',
        aws_secret_access_key = 'AWS-Secrete-Key',
        # host = 's3-website-us-east-1.amazonaws.com',
        # is_secure=True,               # uncomment if you are not using ssl
        calling_format = boto.s3.connection.OrdinaryCallingFormat(),
        )
    
        bucket = conn.get_bucket('YourBucketName')
        key_name = 'FileToUpload'
        path = 'images/holiday' #Directory Under which file should get upload
        full_key_name = os.path.join(path, key_name)
        k = bucket.new_key(full_key_name)
        k.set_contents_from_filename(key_name)
    
    except Exception,e:
        print str(e)
        print "error"   
    

    @Manish Mehra 2017-01-31 12:34:05

    from boto3.s3.transfer import S3Transfer
    import boto3
    #have all the variables populated which are required below
    client = boto3.client('s3', aws_access_key_id=access_key,aws_secret_access_key=secret_key)
    transfer = S3Transfer(client)
    transfer.upload_file(filepath, bucket_name, folder_name+"/"+filename)
    

    @colintobing 2017-08-03 02:41:22

    what is filepath and what is folder_name+filename? it's confusing

    @Manish Mehra 2017-08-29 11:47:51

    @colintobing filepath is path of file on cluster and folder_name/filename is the naming convention that you would want to have inside s3 bucket

    @Sebastian Wozny 2018-01-24 10:33:21

    wow, why is there 50 ways to do this...

    @Mark Amery 2018-02-20 16:30:59

    @ManishMehra The answer would be better if you edited it to clarify colintobing's point of confusion; it's non-obvious without checking the docs which parameters refer to local paths and which ones to S3 paths without checking the docs or reading the comments. (Once that's done, you can flag to have all the comments here purged, since they'll be obsolete.)

    @yvesva 2018-03-19 22:34:45

    aws_access_key_id and aws_secret_access_key can also be configured with the AWS CLI and stored out of the script so that `client = boto3.client('s3') can be called

    @Felipe Garcia 2013-02-26 11:04:56

    Try this...

    import boto
    import boto.s3
    import sys
    from boto.s3.key import Key
    
    AWS_ACCESS_KEY_ID = ''
    AWS_SECRET_ACCESS_KEY = ''
    
    bucket_name = AWS_ACCESS_KEY_ID.lower() + '-dump'
    conn = boto.connect_s3(AWS_ACCESS_KEY_ID,
            AWS_SECRET_ACCESS_KEY)
    
    
    bucket = conn.create_bucket(bucket_name,
        location=boto.s3.connection.Location.DEFAULT)
    
    testfile = "replace this with an actual filename"
    print 'Uploading %s to Amazon S3 bucket %s' % \
       (testfile, bucket_name)
    
    def percent_cb(complete, total):
        sys.stdout.write('.')
        sys.stdout.flush()
    
    
    k = Key(bucket)
    k.key = 'my test file'
    k.set_contents_from_filename(testfile,
        cb=percent_cb, num_cb=10)
    

    [UPDATE] I am not a pythonist, so thanks for the heads up about the import statements. Also, I'd not recommend placing credentials inside your own source code. If you are running this inside AWS use IAM Credentials with Instance Profiles (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html), and to keep the same behaviour in your Dev/Test environment, use something like Hologram from AdRoll (https://github.com/AdRoll/hologram)

    @cgseller 2015-06-29 22:51:15

    I would avoid the multiple import lines, not pythonic. Move the import lines to the top, and for the boto, you can use from boto.s3.connection import S3Connection ; conn = S3Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY); bucket = conn.create_bucket(bucketname...); bucket.new_key(keyname,...).set_contents_from_filename....

    @Alex Pavy 2018-06-21 09:02:12

    boto.s3.key.Key doesn't exist on 1.7.12

    @Oren Efron 2014-12-24 08:48:46

    I used this and it is very simple to implement

    import tinys3
    
    conn = tinys3.Connection('S3_ACCESS_KEY','S3_SECRET_KEY',tls=True)
    
    f = open('some_file.zip','rb')
    conn.upload('some_file.zip',f,'my_bucket')
    

    https://www.smore.com/labs/tinys3/

    @wordsforthewise 2016-10-12 03:10:49

    I don't think this works for large files. I had to use this: docs.pythonboto.org/en/latest/s3_tut.html#storing-large-data

    @wordsforthewise 2016-10-12 03:36:25

    @Halil Kaskavalci 2018-01-27 12:24:46

    Since tinys3 project is abandoned you should not use this. github.com/smore-inc/tinys3/issues/45

    @Russ 2019-05-06 14:02:26

    This flat out didn't work for me anymore in 2019. tinys3 isn't just abandoned... I don't think it works anymore. For anyone else who decides to try this, don't be surprised if you get 403 errors. A simple boto3.client solution (like Manish Mehra's answer) worked immediately, though.

    Related Questions

    Sponsored Content

    41 Answered Questions

    [SOLVED] How to merge two dictionaries in a single expression?

    22 Answered Questions

    [SOLVED] Importing files from different folder

    19 Answered Questions

    [SOLVED] How to import other Python files?

    20 Answered Questions

    [SOLVED] Running unittest with typical test directory structure

    • 2009-12-13 16:10:23
    • Major Major
    • 175019 View
    • 621 Score
    • 20 Answer
    • Tags:   python unit-testing

    21 Answered Questions

    [SOLVED] How do I list all files of a directory?

    • 2010-07-08 19:31:22
    • duhhunjonn
    • 3541892 View
    • 3474 Score
    • 21 Answer
    • Tags:   python directory

    25 Answered Questions

    [SOLVED] How can I safely create a nested directory?

    37 Answered Questions

    [SOLVED] How do I check whether a file exists without exceptions?

    10 Answered Questions

    26 Answered Questions

    [SOLVED] Find all files in a directory with extension .txt in Python

    • 2010-10-19 01:09:13
    • usertest
    • 1503874 View
    • 1043 Score
    • 26 Answer
    • Tags:   python file-io

    13 Answered Questions

    [SOLVED] Find current directory and file's directory

    • 2011-02-28 01:51:21
    • John Howard
    • 2589503 View
    • 1921 Score
    • 13 Answer
    • Tags:   python directory

    Sponsored Content