I'm trying to access objects from my S3 bucket with s3cmd with path style urls. This is no problem with the Java SDK like.
s3Client.setS3ClientOptions(S3ClientOptions.builder()
.setPathStyleAccess(true).build());
I want to do the same with s3cmd. I have set this up in my s3conf file:
host_base = s3.eu-central-1.amazonaws.com
host_bucket = s3.eu-central-1.amazonaws.com/%(bucket)s
This works for bucket listing with:
$ s3cmd ls
2016-08-24 12:36 s3://test
When trying to list all objects of a bucket I get the following error:
Traceback (most recent call last):
File "/usr/local/bin/s3cmd", line 2919, in <module>
rc = main()
File "/usr/local/bin/s3cmd", line 2841, in main
rc = cmd_func(args)
File "/usr/local/bin/s3cmd", line 120, in cmd_ls
subcmd_bucket_list(s3, uri)
File "/usr/local/bin/s3cmd", line 153, in subcmd_bucket_list
response = s3.bucket_list(bucket, prefix = prefix)
File "/usr/local/lib/python2.7/site-packages/S3/S3.py", line 297, in bucket_list
for dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params):
File "/usr/local/lib/python2.7/site-packages/S3/S3.py", line 324, in bucket_list_streaming
response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params)
File "/usr/local/lib/python2.7/site-packages/S3/S3.py", line 343, in bucket_list_noparse
response = self.send_request(request)
File "/usr/local/lib/python2.7/site-packages/S3/S3.py", line 1081, in send_request
conn = ConnMan.get(self.get_hostname(resource['bucket']))
File "/usr/local/lib/python2.7/site-packages/S3/ConnMan.py", line 192, in get
conn.c.connect()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 836, in connect
self.timeout, self.source_address)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 557, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno 8] nodename nor servname provided, or not known
Assuming that there is no other issue with your configuration, the value that you used for "host_bucket" is wrong.
It should be:
host_bucket = %(bucket)s.s3.eu-central-1.amazonaws.com
or
host_bucket = s3.eu-central-1.amazonaws.com
The second one will for "path style" to be used. But, if you are using amazon s3 and the first host_bucket value that I propose, s3cmd will automatically use dns-based or path-based buckets depending of what characters you are using in your bucket name.
Is it a particular reason why you would want to only use path-based style?
Related
While downloading azure blobs to local file system, I'm getting the following exception:
Client-Request-ID=99bdb0e4-2d1c-11e8-8fe6-00155dbf7128 Retry policy did not allow for a retry: Server-Timestamp=Wed, 21 Mar 2018 15:29:09 GMT, Server-Request-ID=1e7ab8f5-101e-0076-5329-c16a24000000, HTTP status code=404, Exception=The specified blob does not exist.
ErrorCode: BlobNotFound<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:1e7ab8f5-101e-0076-5329-c16a24000000Time:2018-03-21T15:29:09.6565984Z</Message></Error>.
Traceback (most recent call last):
File "C:\Program Files\Commvault\ContentStore\Automation\CloudApps\CloudAppsUtils\cahelper.py", line 483, in download_contents_azure
session.get_blob_to_path(container_name,fl,fl)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1817, in get_blob_to_path
timeout)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 2003, in get_blob_to_stream
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1971, in get_blob_to_stream
_context=operation_context)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1695, in _get_blob
operation_context=_context)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 354, in _perform_request
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 289, in _perform_request
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 275, in _perform_request
HTTPError(response.status, response.message, response.headers, response.body))
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\_error.py", line 111, in _http_error_handler
raise AzureHttpError(message, http_error.status)
azure.common.AzureMissingResourceHttpError: The specified blob does not exist.ErrorCode: BlobNotFound
<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:1e7ab8f5-101e-0076-5329-c16a24000000
Time:2018-03-21T15:29:09.6565984Z</Message></Error>
I'm using the following code to download blobs:
def download_contents_azure(self, account_name, account_key, content):
session=self.create_session_azure(account_name,account_key)
os.mkdir('in place')
os.chdir('in place')
for item in content:
# s = os.path.basename(item)
path_to_file = ("/".join(item.strip("/").split('/')[1:]))
container_name = Path(item).parts[1]
gen = session.list_blobs(container_name)
li = []
for i in gen:
li.append(i.name)
if path_to_file in li:
fl = os.path.basename(path_to_file)
print(fl)
c = self.splitall(item)
for i in range(1,len(c)-1):
if path.exists(c[i]) is False:
os.mkdir(c[i])
os.chdir(c[i])
session.get_blob_to_path(container_name,fl,fl)
for i in range(1,len(c)-1):
os.chdir("..")
else:
c = self.splitall(item)
for i in range(1,len(c)):
os.mkdir(c[i])
os.chdir(c[i])
generator = session.list_blobs(container_name,path_to_file+'/',delimiter='/')
for blob in generator:
bl = os.path.basename(blob.name)
session.get_blob_to_path(container_name,bl,bl)
I've a path in azure (/container/folder/subfolder).
I'm trying to download the structure and all the files under subfolder. the first file under the subfolder gets downloaded and then I've the above exception. Due to this, I'm unable to loop through and print the next items.
Thoughts?
Double check your line
session.get_blob_to_path(container_name,fl,fl)
You have fl for blob_name.
documentation here indicates second argument is blob_name while third is file path on file system where to download.
I think your blob name is wrong. Hence it does not exist.
NOTE: consider installing fiddler so you can look at the network traces. It allows you to see the raw request.
I'm trying to use my aws credentials file in boto but can't seem to get it to work. I'm new to python and boto so I'm looking at a bunch of stuff online trying to understand this.
All I'm trying to do right now is to just get all ec2 instances...here is my python code:
import boto
from boto import ec2
ec2conn = ec2.connection.EC2Connection(profile_name='profile_name')
ec2conn.get_all_instances()
when I run that, I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/ec2/connection.py", line 585, in get_all_instances
max_results=max_results)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/ec2/connection.py", line 681, in get_all_reservations
[('item', Reservation)], verb='POST')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/connection.py", line 1170, in get_list
response = self.make_request(action, params, path, verb)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/connection.py", line 1116, in make_request
return self._mexe(http_request)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/connection.py", line 913, in _mexe
self.is_secure)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/connection.py", line 705, in get_http_connection
return self.new_http_connection(host, port, is_secure)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/connection.py", line 747, in new_http_connection
connection = self.proxy_ssl(host, is_secure and 443 or 80)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/boto/connection.py", line 835, in proxy_ssl
ca_certs=self.ca_certificates_file)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 943, in wrap_socket
ciphers=ciphers)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 611, in __init__
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 840, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:661)
I've also tried ec2conn.get_all_reservations() but got the same result...
In boto3, I can do this which works:
import boto3
session = boto3.Session(profile_name='dev')
session = boto3.Session(profile_name='profile_name')
dev_ec2 = session.client('ec2')
dev_ec2.describe_instances()
------EDIT--------
So I found this link on stack...Recommended way to manage credentials with multiple AWS accounts? and what I did was exported my AWS_PROFILE var
export AWS_PROFILE="profile_nm"
that worked when I did this:
>>> import boto
>>> conn = boto.connect_s3()
>>> conn.get_all_buckets()
And I got all of the s3 buckets back...
but when I did the above to get all the ec2 instances back...i still got the ssl.SSLEOFError above. It seems to work with s3 but not ec2 now...So, is the way I get all the Ec2 instances wrong?
I want to upload files from s3 using boto 2.38 with python 3.4 and Django.
the two first retries doesn't work but the last worked fine (For the retries, it is executed manually not automatically.)
I get this error:
File "/usr/local/lib/python3.4/dist-packages/boto/s3/key.py", line 750, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/local/lib/python3.4/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
query_args=query_args
File "/usr/local/lib/python3.4/dist-packages/boto/s3/connection.py", line 664, in make_request
retry_handler=retry_handler
File "/usr/local/lib/python3.4/dist-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/usr/local/lib/python3.4/dist-packages/boto/connection.py", line 1030, in _mexe
raise ex
File "/usr/local/lib/python3.4/dist-packages/boto/connection.py", line 940, in _mexe
request.body, request.headers)
File "/usr/local/lib/python3.4/dist-packages/boto/s3/key.py", line 844, in sender
http_conn.send(chunk)
File "/usr/lib/python3.4/http/client.py", line 888, in send
self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
In the log, I have the right host name. By searching on the web I found that I must provide the host for the connection to s3 and as I have many buckets in many regions, I didn't specify the host (or should I consider all cases of regions ? I know that in Boto3 there is an improvement on this...):
bucket: 'NameOfTheBucket' is in eu-west-1
The big problem is that I can not reproduce the error so If anyone can help, I will be very grateful.
I added a part of my code:
s3 = connect_s3(
aws_access_key_id=xxxxx,
aws_secret_access_key=xxx,
is_secure=xxxxxx)
self.s3_bucket = s3.lookup(self.s3_bucket_name)
the problem is in this part of the code:
def put_s3_file_onepart(self, filename, key_name, headers):
key = self.s3_bucket.new_key(key_name)
key.BufferSize = CHUNK_SIZE
with open(self.abs_path(filename), 'rb') as fp:
key.md5, key.base64md5 = key.compute_md5(fp)
with open(self.abs_path(filename), 'rb') as fp:
key.send_file(fp, headers=headers) <==========
key.close()
In my opinion, it is related to chuncked file. Thanks for the help.
I'm trying to connect to an FTP server using TLS and upload a text file. The below code connects to the site just fine, but it's not uploading the file. Instead I'm getting the following error:
Traceback (most recent call last):
File "X:/HR & IT/Ryan/Python Scripts/ftps_connection_test.py", line 16, in <module>
ftps.storlines("STOR " + filename, open(filename,"r"))
File "C:\Python33\lib\ftplib.py", line 816, in storlines
with self.transfercmd(cmd) as conn:
File "C:\Python33\lib\ftplib.py", line 391, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "C:\Python33\lib\ftplib.py", line 756, in ntransfercmd
conn, size = FTP.ntransfercmd(self, cmd, rest)
File "C:\Python33\lib\ftplib.py", line 357, in ntransfercmd
resp = self.sendcmd(cmd)
File "C:\Python33\lib\ftplib.py", line 264, in sendcmd
return self.getresp()
File "C:\Python33\lib\ftplib.py", line 238, in getresp
raise error_perm(resp)
ftplib.error_perm: 550 The parameter is incorrect.
There's probably something really basic I'm missing, my code is below and any help is much appreciated.
import os
from ftplib import FTP_TLS as f
# Open secure connection
ftps = f("ftp.foo.com")
ftps.login(username,password)
ftps.prot_p()
# Create the test txt file to upload
filename = r"c:\path\to\file"
testFile = open(filename,"w")
testFile.write("Test file with test text")
testFile.close()
# Transfer testFile
ftps.storlines("STOR " + filename, open(filename,"r"))
# Quit connection
ftps.quit()
I have got the same error when trying to write upload a file to FTP server. In my case, the destination file name is not the correct format. It was something like
data_20180411T12:00:12.3435Z.txt
I renamed something like
data_20180411T120012_3435Z.txt. Then it worked.
filename = r"c:\path\to\file"
is the absolute path to a local file. This same value is being passed in the STOR command, i.e.
ftps.storlines("STOR " + filename, open(filename,"r"))
attempts to perform a STOR c:\path\to\file operation, however, it is unlikely that the path exists on the remote server, and the ftplib.error_perm exception would suggest that you don't have permission to write there (even if it does exist).
You could try this instead:
ftps.storlines("STOR " + os.path.basename(filename), open(filename,"r"))
which would issue a STOR file operation and upload the file to the default directory on the remote server. If you need to upload to a different path on the remote server, just add that to STOR.
I am trying to download a file from FTP using python. I was able to successfully move into the directory but can't download the file.
The command I use is ftp.retrbinary('master.idx', open(fname,'wb').write)
And error is below. It looks like the command is looking for MASTER.IDX instead of master.idx
The full path to the file I want to download is ftp://ftp.sec.gov/edgar/full-index/2011/QTR2/master.idx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/ftplib.py", line 406, in retrbinary
conn = self.transfercmd(cmd, rest)
File "/usr/lib/python2.7/ftplib.py", line 368, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib/python2.7/ftplib.py", line 331, in ntransfercmd
resp = self.sendcmd(cmd)
File "/usr/lib/python2.7/ftplib.py", line 244, in sendcmd
return self.getresp()
File "/usr/lib/python2.7/ftplib.py", line 219, in getresp
raise error_perm, resp
ftplib.error_perm: 500 MASTER.IDX not understood
I can't say why the name changes to uppercase. In any case, when using FTP, I make like this, it may help you:
server = "URL.of.server"
directory = "directory/where/the/file/is"
filename = "nameoffile.txt"
from ftplib import FTP
ftp = FTP(server) #Set server address
ftp.login() # Connect to server
ftp.cwd(directory) # Move to the desired folder in server
ftp.retrbinary('RETR ' + filename,open(filename, 'wb').write) # Download file from server
ftp.close() # Close connection
I think that it may be the 'RETR ', if you don't write, it the server may not understand what you want to do
use wget module of python instead. Here is an example snippet
import wget
fileloc = '/path/to/the/file/foo.txt'
wget.download(fileloc)