We Keep Coding

While downloading azure blobs to local file system an exception occurs

While downloading azure blobs to local file system, I'm getting the following exception:
Client-Request-ID=99bdb0e4-2d1c-11e8-8fe6-00155dbf7128 Retry policy did not allow for a retry: Server-Timestamp=Wed, 21 Mar 2018 15:29:09 GMT, Server-Request-ID=1e7ab8f5-101e-0076-5329-c16a24000000, HTTP status code=404, Exception=The specified blob does not exist.
ErrorCode: BlobNotFound<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:1e7ab8f5-101e-0076-5329-c16a24000000Time:2018-03-21T15:29:09.6565984Z</Message></Error>.
Traceback (most recent call last):
File "C:\Program Files\Commvault\ContentStore\Automation\CloudApps\CloudAppsUtils\cahelper.py", line 483, in download_contents_azure
session.get_blob_to_path(container_name,fl,fl)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1817, in get_blob_to_path
timeout)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 2003, in get_blob_to_stream
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1971, in get_blob_to_stream
_context=operation_context)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\blob\baseblobservice.py", line 1695, in _get_blob
operation_context=_context)
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 354, in _perform_request
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 289, in _perform_request
raise ex
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\storageclient.py", line 275, in _perform_request
HTTPError(response.status, response.message, response.headers, response.body))
File "C:\Program Files\Python36\lib\site-packages\azure\storage\common\_error.py", line 111, in _http_error_handler
raise AzureHttpError(message, http_error.status)
azure.common.AzureMissingResourceHttpError: The specified blob does not exist.ErrorCode: BlobNotFound
<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:1e7ab8f5-101e-0076-5329-c16a24000000
Time:2018-03-21T15:29:09.6565984Z</Message></Error>
I'm using the following code to download blobs:
def download_contents_azure(self, account_name, account_key, content):
session=self.create_session_azure(account_name,account_key)
os.mkdir('in place')
os.chdir('in place')
for item in content:
# s = os.path.basename(item)
path_to_file = ("/".join(item.strip("/").split('/')[1:]))
container_name = Path(item).parts[1]
gen = session.list_blobs(container_name)
li = []
for i in gen:
li.append(i.name)
if path_to_file in li:
fl = os.path.basename(path_to_file)
print(fl)
c = self.splitall(item)
for i in range(1,len(c)-1):
if path.exists(c[i]) is False:
os.mkdir(c[i])
os.chdir(c[i])
session.get_blob_to_path(container_name,fl,fl)
for i in range(1,len(c)-1):
os.chdir("..")
else:
c = self.splitall(item)
for i in range(1,len(c)):
os.mkdir(c[i])
os.chdir(c[i])
generator = session.list_blobs(container_name,path_to_file+'/',delimiter='/')
for blob in generator:
bl = os.path.basename(blob.name)
session.get_blob_to_path(container_name,bl,bl)
I've a path in azure (/container/folder/subfolder).
I'm trying to download the structure and all the files under subfolder. the first file under the subfolder gets downloaded and then I've the above exception. Due to this, I'm unable to loop through and print the next items.
Thoughts?

Double check your line
session.get_blob_to_path(container_name,fl,fl)
You have fl for blob_name.
documentation here indicates second argument is blob_name while third is file path on file system where to download.
I think your blob name is wrong. Hence it does not exist.
NOTE: consider installing fiddler so you can look at the network traces. It allows you to see the raw request.

Unable to ftp a file using python ftplib, but successful using curl

I receive the following error when uploading a small text file to an ftp site when using python's ftplib:
File "ftpuploader.py", line 13, in uploadFilePath
ftp.storbinary("STOR {}".format(filepath), file)
File "/usr/lib64/python2.7/ftplib.py", line 471, in storbinary
conn = self.transfercmd(cmd, rest)
File "/usr/lib64/python2.7/ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib64/python2.7/ftplib.py", line 358, in ntransfercmd
resp = self.sendcmd(cmd)
File "/usr/lib64/python2.7/ftplib.py", line 249, in sendcmd
return self.getresp()
File "/usr/lib64/python2.7/ftplib.py", line 224, in getresp
raise error_perm, resp
ftplib.error_perm: 553 Could not create file.
I am using the following code successfully when connecting to another system. I can log in, change directories, but am unable to create the file. I can load a file using either filezilla or a simple curl command, curl -T '/path/to/file' ftp://192.168.18.75 --user admin:password
ftp = FTP(address)
ftp.login(username, password)
ftp.cwd('/gui')
file = open(filepath, 'rb')
ftp.storbinary("STOR {}".format(filepath), file)
ftp.retrlines('LIST') # list directory contents
file.close()
ftp.quit()
any ideas?

You are passing a path to a local path to the STOR command (the same path you use with the open(filepath, 'rb')).
While with the CURL, you do not specify any path to the remote file. So the file gets uploaded to the current FTP working directory.
As already suggested in the comments by #acw1668, use the:
ftp.storbinary("STOR {}".format(os.path.basename(filepath)), file)

Python ftplib.error_perm 550: No such file or directory?

I've written a Python script that is part of my attempt to automate daily ftp transfers from my server. I've tested the script with a number of files and file types (html, mp3, png, jpg, etc.) and everything seems to work out fine so far.
However, when I try to download a simple text file, 'file.txt' (9 kb), the download fails, although I account for text files and switch from binary to text mode for the transfer. The following exception is thrown by ftplib:
ftplib.error_perm: 550 file.txt: No such file or directory
Here's my script:
from ftplib import FTP_TLS, error_perm
import os
def open_connection(server, user, pwd, work_dir=None):
global ftps
try:
ftps = FTP_TLS(host=server)
ftps.login(user=user, passwd=pwd)
ftps.prot_p() # switch to secure data connection
if work_dir != None:
ftps.cwd(work_dir)
else:
pass
except:
pass
def download_file(remote_path, local_path):
remote_file = os.path.basename(remote_path)
local_file_path = os.path.join(local_path, remote_file)
# differentiate between text and binary files
file_type, encoding = guess_type_and_encoding(remote_file)
# possibly needs a permission exception catch
if file_type.split("/")[0] == "text" and encoding == None:
# use text mode for transfer
local_file = open(local_file_path, 'w')
def callback(line): local_file.write(line + "\n")
ftps.retrlines("RETR " + remote_file, callback)
local_file.close()
else:
# use binary mode for transfer
local_file = open(local_file_path, 'wb')
ftps.retrbinary("RETR " + remote_file, local_file.write)
local_file.close()
return
def guess_type_and_encoding(filename):
from mimetypes import guess_type, add_type
add_type('text/x-python-win', '.pyw') # not in tables
mimetype, encoding = guess_type(filename, False) # allow extras
mimetype = mimetype or "?/?" # type unknown
return mimetype, encoding
open_connection(server, user, pwd, work_dir)
download_file("/files/dir/file.txt", "/Users/username/Desktop")
ftps.close()
I don't get why the error is raised!? The arguments 'remote_path' and 'local_path' are correctly provided. Both paths exist! 'file.txt' exists on the server under /files/dir and /Users/username/Desktop points to my desktop on OS X.
Here's the detailed ftplib error:
Traceback (most recent call last):
File "ftp2.py", line 138, in <module>
download_file("/files/dir/file.txt", "/Users/username/Desktop")
File "ftp2.py", line 93, in download_file
ftps.retrlines("RETR " + remote_file, callback)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 735, in retrlines
conn = self.transfercmd(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 710, in ntransfercmd
conn, size = FTP.ntransfercmd(self, cmd, rest)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 339, in ntransfercmd
resp = self.sendcmd(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 249, in sendcmd
return self.getresp()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 224, in getresp
raise error_perm, resp
ftplib.error_perm: 550 file.txt: No such file or directory
Any help is greatly appreciated.
Thanks. :)

Try to
replace remote_file
in ftps.retrlines("RETR " + remote_file, callback)
with remote_path.

Python: Uploading files FTP_TLS- "550 The parameter is incorrect"

I'm trying to connect to an FTP server using TLS and upload a text file. The below code connects to the site just fine, but it's not uploading the file. Instead I'm getting the following error:
Traceback (most recent call last):
File "X:/HR & IT/Ryan/Python Scripts/ftps_connection_test.py", line 16, in <module>
ftps.storlines("STOR " + filename, open(filename,"r"))
File "C:\Python33\lib\ftplib.py", line 816, in storlines
with self.transfercmd(cmd) as conn:
File "C:\Python33\lib\ftplib.py", line 391, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "C:\Python33\lib\ftplib.py", line 756, in ntransfercmd
conn, size = FTP.ntransfercmd(self, cmd, rest)
File "C:\Python33\lib\ftplib.py", line 357, in ntransfercmd
resp = self.sendcmd(cmd)
File "C:\Python33\lib\ftplib.py", line 264, in sendcmd
return self.getresp()
File "C:\Python33\lib\ftplib.py", line 238, in getresp
raise error_perm(resp)
ftplib.error_perm: 550 The parameter is incorrect.
There's probably something really basic I'm missing, my code is below and any help is much appreciated.
import os
from ftplib import FTP_TLS as f
# Open secure connection
ftps = f("ftp.foo.com")
ftps.login(username,password)
ftps.prot_p()
# Create the test txt file to upload
filename = r"c:\path\to\file"
testFile = open(filename,"w")
testFile.write("Test file with test text")
testFile.close()
# Transfer testFile
ftps.storlines("STOR " + filename, open(filename,"r"))
# Quit connection
ftps.quit()

I have got the same error when trying to write upload a file to FTP server. In my case, the destination file name is not the correct format. It was something like
data_20180411T12:00:12.3435Z.txt
I renamed something like
data_20180411T120012_3435Z.txt. Then it worked.

filename = r"c:\path\to\file"
is the absolute path to a local file. This same value is being passed in the STOR command, i.e.
ftps.storlines("STOR " + filename, open(filename,"r"))
attempts to perform a STOR c:\path\to\file operation, however, it is unlikely that the path exists on the remote server, and the ftplib.error_perm exception would suggest that you don't have permission to write there (even if it does exist).
You could try this instead:
ftps.storlines("STOR " + os.path.basename(filename), open(filename,"r"))
which would issue a STOR file operation and upload the file to the default directory on the remote server. If you need to upload to a different path on the remote server, just add that to STOR.

error 501 on ftp.retrbinary (ftplib)

I'm coding a script to check a FTP directory and download the new files. This is part of the code:
from ftplib import FTP
import os
ftp = FTP(ftp_lance)
ftp.login(login, password)
ftp.cwd('xxxxxx')
FTP_list = ftp.nlst()
lista_diferenca = [file for file in FTP_list if file not in local_list]
for file in lista_diferenca:
local_filename = os.path.join(cache, file)
ftp.retrbinary('REST ' + file, open(local_filename, 'wb').write)
When I run it, I get this error message:
Traceback (most recent call last):
File "D:\Scripts\Istari\Radagast\Radagast.py", line 44, in <module>
ftp.retrbinary('REST tabela14_pag5.pdf', open(local_filename, 'wb').write)
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 414, in retrbinary
conn = self.transfercmd(cmd, rest)
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 339, in ntransfercmd
resp = self.sendcmd(cmd)
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 249, in sendcmd
return self.getresp()
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 224, in getresp
raise error_perm, resp
error_perm: 501 Bad parameter. Numeric value required
I check several sites searching for this kind of error and find nothing. It seems that my retrbinaty is broken, but the arguments looks right (first the 'Rest' + file, and then the callback function).
Some idea about my error?

You need to specify the FTP command RETR, not REST:
ftp.retrbinary('RETR ' + file, open(local_filename, 'wb').write)