Boto File Upload To S3 Failing On Windows [errno: 10054]

I'm trying to upload a file to S3 using boto on a windows 7 machine, but i keep getting an error [Errno 10054] An existing connection was forcibly closed by the remote host My code

Solution 1:

@garnaat made a suggestion in the comments above that solved this for me. Thanks!! For some reason, connecting to the universal endpoint and then trying to upload specifically to the ap-southeast-2 S3 endpoint fails. But if we use the connect_to_region function to initiate the connection and specify the endpoint we want, everything works a-ok! Thanks again, and working example below.

from boto.s3 import connect_to_region
from boto.s3.connection import Location
from boto.s3.key import Key

conn = connect_to_region(Location.APSoutheast2,
                         aws_access_key_id=conf.Access_Key_ID,
                         aws_secret_access_key=conf.Secret_Key)
bucket = conn.lookup(bucket_name) # bucket is located in Location.APSoutheast2

k = Key(bucket)
k.key = 'akeynameformyfile'
k.set_contents_from_filename(source_path_of_file_to_upload)

Solution 2:

set_contents_from_filename accepts a string as the source for the file on S3. You are trying to upload an existing file. Try set_contents_from_file to specify an existing file to upload. Otherwise, read the contents of the file as pass in to set_contents_from_filename as a string.

Solution 3:

All of the above answers provide technical solutions but my first experience with this suggested that it was an environmental issue.

I have basically the same code as the original question. I had no problems the first time I ran it, and then ran into the same error message last night. I found this answer and many others but decided that since it (my code) worked well the first time (more than 20K files uploaded) I was probably having problems either on my machine or AWS was the source (and/or possibly some point in between).

I shutdown my computer, went home and when I restarted the this morning the original code worked fine.

One of the observations that led me to this conclusion is that while I was searching for the answer I noticed that many posts about the problem all seemed to be occurring at about the same time. This got me to thinking that maybe AWS was having problems. Again, I can't be sure if it was my computer, the AWS server or some traffic point - in-between but the original code ran fine when I went after it again this morning.

Here is the code I am using right now ( as you can see very similar to code in original question)

import boto
importglobAWS_KEY="mykey"
AWS_SECRET = "mySecret"
bucket_name = "myBucket"
numbers = [x for x in range(0,20000,500)]
to_upload = glob.glob('E:\MTurkTablesSelected\\*.htm')

s3 = boto.connect_s3(aws_access_key_id = AWS_KEY, aws_secret_access_key = AWS_SECRET)
bucket = s3.get_bucket(bucket_name)
for n, file_path in enumerate(to_upload):
    upload_key = file_path.split('\\')[-1]
    key = bucket.new_key(upload_key)
    key.set_contents_from_filename(file_path)
    key.set_acl('public-read')
    if n in numbers:
        print n

I am now more than 10,000 files into a large upload and not having any problems.