We Keep Coding

paramiko sftp script works on test server not on production

I'm a newb at python, so please excuse the hack job I created in order to transfer the contents of a folder into an ssh server.
The problem is that it works great in my test server, but as soon as run it against the actual server that I need to upload files for I receive the error below, and I'm not sure what it means.
I've googled it, but I can't figure it out, please help.
Thanks.
import paramiko
import glob
import os
from shutil import move
host = "192.168.1.87" #hard-coded
port = 22
password ="passwd" #hard-coded
username = "administator" #hard-coded
remotepath ='' #hard-coded
localpath = 'D:\\PH/PH_PROD\\PowerConnectInterf1_WINS\\bin\\data\\Sheex\\bc\\945\\'
#build filename array
os.chdir("D:/PH/PH_PROD/PowerConnectInterf1_WINS/bin/data/Sheex/bc/945")
filelist=[]
for files in glob.glob( "2016*" ):
f = open(files, 'r')
filelist.append(f.name)
f.close()
if (len(filelist)>0):
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
for s in filelist:
#print remotepath+s
sftp.put(localpath+s,remotepath+s)
#os.rename(localpath+s,localpath+"945back/"+s)
sftp.close()
transport.close()
#print 'Upload done.'
Error:
D:\Scripts>python mysftp.py
Traceback (most recent call last):
File "mysftp.py", line 37, in <module>
sftp.put(localpath+s,remotepath+s)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 676, in put
return self.putfo(fl, remotepath, file_size, callback, confirm)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 634, in put
fo
with self.file(remotepath, 'wb') as fr:
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 327, in ope
n
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 730, in _re
quest
return self._read_response(num)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 781, in _re
ad_response
self._convert_status(msg)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 807, in _co
nvert_status
raise IOError(errno.ENOENT, text)
IOError: [Errno 2] Invalid file ID

It sounds like the path you are writing on the server doesn't exist. You should check and create if doesn't exist.

Python: Uploading files FTP_TLS- "550 The parameter is incorrect"

I'm trying to connect to an FTP server using TLS and upload a text file. The below code connects to the site just fine, but it's not uploading the file. Instead I'm getting the following error:
Traceback (most recent call last):
File "X:/HR & IT/Ryan/Python Scripts/ftps_connection_test.py", line 16, in <module>
ftps.storlines("STOR " + filename, open(filename,"r"))
File "C:\Python33\lib\ftplib.py", line 816, in storlines
with self.transfercmd(cmd) as conn:
File "C:\Python33\lib\ftplib.py", line 391, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "C:\Python33\lib\ftplib.py", line 756, in ntransfercmd
conn, size = FTP.ntransfercmd(self, cmd, rest)
File "C:\Python33\lib\ftplib.py", line 357, in ntransfercmd
resp = self.sendcmd(cmd)
File "C:\Python33\lib\ftplib.py", line 264, in sendcmd
return self.getresp()
File "C:\Python33\lib\ftplib.py", line 238, in getresp
raise error_perm(resp)
ftplib.error_perm: 550 The parameter is incorrect.
There's probably something really basic I'm missing, my code is below and any help is much appreciated.
import os
from ftplib import FTP_TLS as f
# Open secure connection
ftps = f("ftp.foo.com")
ftps.login(username,password)
ftps.prot_p()
# Create the test txt file to upload
filename = r"c:\path\to\file"
testFile = open(filename,"w")
testFile.write("Test file with test text")
testFile.close()
# Transfer testFile
ftps.storlines("STOR " + filename, open(filename,"r"))
# Quit connection
ftps.quit()

I have got the same error when trying to write upload a file to FTP server. In my case, the destination file name is not the correct format. It was something like
data_20180411T12:00:12.3435Z.txt
I renamed something like
data_20180411T120012_3435Z.txt. Then it worked.

filename = r"c:\path\to\file"
is the absolute path to a local file. This same value is being passed in the STOR command, i.e.
ftps.storlines("STOR " + filename, open(filename,"r"))
attempts to perform a STOR c:\path\to\file operation, however, it is unlikely that the path exists on the remote server, and the ftplib.error_perm exception would suggest that you don't have permission to write there (even if it does exist).
You could try this instead:
ftps.storlines("STOR " + os.path.basename(filename), open(filename,"r"))
which would issue a STOR file operation and upload the file to the default directory on the remote server. If you need to upload to a different path on the remote server, just add that to STOR.

python paramiko module error with callback

I'm trying to use the paramiko module to copy a (big) file in my local network, and get the output to display a GtkProgressBar.
A part of my code is:
...
NetworkCopy.pbar.set_text("Copy of the file in the Pi...")
while gtk.events_pending(): # refresh the progress bar
gtk.main_iteration()
self.connection(transferred, toBeTransferred)
def connection(self, transferred, toBeTransferred):
sftp = self.sftp
fichier_pc = self.fichier_pc
chemin_pi = self.chemin_pi # var names are in french !
fichier = self.fichier
transferred = self.transferred
toBeTransferred = self.toBeTransferred
print "Transferred: {0}\tStill to send: {1}".format(transferred, toBeTransferred)
sftp.put(fichier_pc, chemin_pi + fichier, callback=self.connection)
In the terminal, I can see
Transferred: 0 Still to send: 3762398252
for a while, but after 10s I have this error:
File "network_copier.py", line 158, in connection
sftp.put(fichier_pc, chemin_pi + fichier, callback=self.connection)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 615, in put
return self.putfo(fl, remotepath, os.stat(localpath).st_size, callback, confirm)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 577, in putfo
fr.close()
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_file.py", line 67, in close
self._close(async=False)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_file.py", line 88, in _close
self.sftp._request(CMD_CLOSE, self.handle)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 689, in _request
return self._read_response(num)
File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 721, in _read_response
raise SSHException('Server connection dropped: %s' % (str(e),))
paramiko.SSHException: Server connection dropped:
I have the 1.12.2 version of paramiko, from this ppa
Thanks for your help
Edit: The solution is to use pexpect instead of paramiko. It's working with big files.
See here

Python FTP download not working

I am trying to download a file from FTP using python. I was able to successfully move into the directory but can't download the file.
The command I use is ftp.retrbinary('master.idx', open(fname,'wb').write)
And error is below. It looks like the command is looking for MASTER.IDX instead of master.idx
The full path to the file I want to download is ftp://ftp.sec.gov/edgar/full-index/2011/QTR2/master.idx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/ftplib.py", line 406, in retrbinary
conn = self.transfercmd(cmd, rest)
File "/usr/lib/python2.7/ftplib.py", line 368, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib/python2.7/ftplib.py", line 331, in ntransfercmd
resp = self.sendcmd(cmd)
File "/usr/lib/python2.7/ftplib.py", line 244, in sendcmd
return self.getresp()
File "/usr/lib/python2.7/ftplib.py", line 219, in getresp
raise error_perm, resp
ftplib.error_perm: 500 MASTER.IDX not understood

I can't say why the name changes to uppercase. In any case, when using FTP, I make like this, it may help you:
server = "URL.of.server"
directory = "directory/where/the/file/is"
filename = "nameoffile.txt"
from ftplib import FTP
ftp = FTP(server) #Set server address
ftp.login() # Connect to server
ftp.cwd(directory) # Move to the desired folder in server
ftp.retrbinary('RETR ' + filename,open(filename, 'wb').write) # Download file from server
ftp.close() # Close connection
I think that it may be the 'RETR ', if you don't write, it the server may not understand what you want to do

use wget module of python instead. Here is an example snippet
import wget
fileloc = '/path/to/the/file/foo.txt'
wget.download(fileloc)

'put' in SFTP using Paramiko

I've installed and written the following Paramiko which is unable to put the file. It is easily able to 'get' a file and execute ls commands on it.
#set username & password
username='runaway'
password='runaway'
port=22
source= '/Unzip.sh'
destination ='/var/mpx/www/http'
#SFTP
client.load_system_host_keys()
print " hostname =%s \n username=%s \n password=%s \n" (hostname,username,password)
t = paramiko.Transport((hostname, port))
t.connect(username=username,password=password)
sftp = paramiko.SFTPClient.from_transport(t)
sftp.put(source,destination)
#sftp.close()
#t.close()
Using a 'put' command gives the following error & stack trace -
File "upload_file.py", line 84, in ?
sftp.put(source,destination)
File "/usr/lib/python2.4/site-packages/paramiko/sftp_client.py", line 522, in put
fr = self.file(remotepath, 'wb')
File "/usr/lib/python2.4/site-packages/paramiko/sftp_client.py", line 221, in open
t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
File "/usr/lib/python2.4/site-packages/paramiko/sftp_client.py", line 572, in _request
return self._read_response(num)
File "/usr/lib/python2.4/site-packages/paramiko/sftp_client.py", line 619, in _read_response
self._convert_status(msg)
File "/usr/lib/python2.4/site-packages/paramiko/sftp_client.py", line 649, in _convert_status
raise IOError(text)
IOError: Failure
How do I overcome this?

The solution seemed very funny to me!
source= '/Unzip.sh'
destination ='/var/mpx/www/http/Unzip.sh'
Just modified the destination path to include the file name as well.
Didn't expect some error like this coming from a Python package.

This also occurs in 2.0.2 when you try to sftp.mkdir('/exists'):
Traceback (most recent call last):
...
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 380, in mkdir
self._request(CMD_MKDIR, path, attr)
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 730, in _request
return self._read_response(num)
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 781, in _read_response
self._convert_status(msg)
File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 811, in _convert_status
raise IOError(text)
IOError: Failure
This was my Python 2.7.9 fix:
try:
sftp.mkdir(remote_dir)
except IOError:
logging.debug('%s already exists.', remote_dir)