Using Python 2.6 and downloading files from an FTP server in passive mode, I found that retrbinary fails with a timeout if the source file is empty (0 bytes). Is this a bug or am I missing a configuration option?
ftp.retrbinary('RETR digital.conf', open('digital.conf','wb').write)
Downloading digital.conf
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "download.py", line 13, in run
ftp.retrbinary('RETR %s' % source, callback)
File "c:\Python26\lib\ftplib.py", line 398, in retrbinary
conn = self.transfercmd(cmd, rest)
File "c:\Python26\lib\ftplib.py", line 360, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "c:\Python26\lib\ftplib.py", line 337, in ntransfercmd
resp = self.getresp()
File "c:\Python26\lib\ftplib.py", line 216, in getresp
raise error_temp, resp
ftplib.error_temp: 421 Timeout
Other non-zero byte files transfer fine.
This is Your session idle time too long.You can file after the President into instantiate ftplib. Otherwise. Modify ftp software configuration.
For example, you use vsftpd, you can add the following configuration to vsftpd.conf:
idle_session_timeout=60000 # The default is 600 seconds
Related
Using parallel-ssh module I'm trying to run SSH commands using Natinve Client but getting SessionHandshakeError. And if I use Paramiko Client instead, everything works fine. I met the requirement of my_pkey.pub being in the same directory as my_pkey.
Here is my code which uses Native Client (changed real IPs to 'ip1' and 'ip2'):
from pssh.pssh2_client import ParallelSSHClient
pkey = os.path.dirname(os.path.abspath(__file__)) + '/my_pkey'
hosts = ['ip1', 'ip2']
client = ParallelSSHClient(hosts, user='root', pkey=pkey)
output = client.run_command('hostname')
for host, host_output in output.items():
for line in host_output.stdout:
print(line)
Getting this error:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 123, in _init
self.session.handshake(self.sock)
File "ssh2\session.pyx", line 81, in ssh2.session.Session.handshake
ssh2.exceptions.SessionHandshakeError: ('SSH session handshake failed with error code %s', -5)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 123, in _init
self.session.handshake(self.sock)
File "ssh2\session.pyx", line 81, in ssh2.session.Session.handshake
ssh2.exceptions.SessionHandshakeError: ('SSH session handshake failed with error code %s', -5)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 123, in _init
self.session.handshake(self.sock)
File "ssh2\session.pyx", line 81, in ssh2.session.Session.handshake
ssh2.exceptions.SessionHandshakeError: ('SSH session handshake failed with error code %s', -5)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/NazimokPP/Desktop/AnchorFree/QA-Automation/nodetest/nodetest.py", line 57, in <module>
main(args.server_domain, args.test_type)
File "C:/Users/NazimokPP/Desktop/AnchorFree/QA-Automation/nodetest/nodetest.py", line 45, in main
output = client.run_command('hostname')
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\pssh2_client.py", line 182, in run_command
encoding=encoding, use_pty=use_pty, timeout=timeout)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\base_pssh.py", line 91, in run_command
self.get_output(cmd, output)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\base_pssh.py", line 136, in get_output
(channel, host, stdout, stderr, stdin) = cmd.get()
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\greenlet.py", line 482, in get
self._raise_exception()
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\greenlet.py", line 159, in _raise_exception
reraise(*self.exc_info)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\_compat.py", line 33, in reraise
raise value.with_traceback(tb)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\pssh2_client.py", line 188, in _run_command
self._make_ssh_client(host)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\pssh2_client.py", line 313, in _make_ssh_client
allow_agent=self.allow_agent, retry_delay=self.retry_delay)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 107, in __init__
THREAD_POOL.apply(self._init)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\pool.py", line 325, in apply
return self.spawn(func, *args, **kwds).get()
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\event.py", line 385, in get
return self.get(block=False)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\event.py", line 375, in get
return self._raise_exception()
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\event.py", line 355, in _raise_exception
reraise(*self.exc_info)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\_compat.py", line 33, in reraise
raise value.with_traceback(tb)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\gevent\threadpool.py", line 211, in _worker
value = func(*args, **kwargs)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 126, in _init
return self._connect_init_retry(retries)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 116, in _connect_init_retry
return self._init(retries=retries)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 126, in _init
return self._connect_init_retry(retries)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 116, in _connect_init_retry
return self._init(retries=retries)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\pssh\ssh2_client.py", line 128, in _init
raise SessionError(msg, self.host, self.port, ex)
pssh.exceptions.SessionError: ('Error connecting to host %s:%s - %s', 'ip1', 22, SessionHandshakeError('SSH session handshake failed with error code %s', -5))
Process finished with exit code 1
Here is my code which uses Paramiko Client (changed real IPs to 'ip1' and 'ip2'):
from pssh.pssh_client import ParallelSSHClient
from pssh.utils import load_private_key
key_path = os.path.dirname(os.path.abspath(__file__)) + '/my_pkey'
pkey = load_private_key(key_path)
hosts = ['ip1', 'ip2']
client = ParallelSSHClient(hosts, user='root', pkey=pkey)
output = client.run_command('hostname')
for host, host_output in output.items():
for line in host_output.stdout:
print(line)
And it works. Here's the output (should I care about warnings?):
C:\Program Files (x86)\Python36-32\lib\site-packages\paramiko\ecdsakey.py:202: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
signature, ec.ECDSA(self.ecdsa_curve.hash_object())
C:\Program Files (x86)\Python36-32\lib\site-packages\paramiko\rsakey.py:110: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
algorithm=hashes.SHA1(),
ip1.hostname
ip2.hostname
Process finished with exit code 0
What am I doing wrong with Native Client?
This error was tracked down to the WinCNG back-end used for libssh2 on Windows - it does not support SHA-256 host key hashes which is now the default in recent versions of OpenSSH server.
The latest version of parallel-ssh, 1.6.0, fixes this issue by switching the Windows back-end to OpenSSL for better compatibility and to match the OSX and Linux binary wheels.
See release notes for more details.
Some explanations which I got from Panos in Google Groups thread. It didn't help me, but maybe it will be helpful for somebody else.
A -5 error is defined as a key exchange error in libssh2. It sounds
like the key type is not supported by libssh2 and paramiko shows
'ecdsakey.py' being used. ECDSA keys are not currently supported by
libssh2 (PR pending).
The warning are from paramiko itself, can't say if they matter.
Better exceptions for native client errors are being worked on for
next release.
_
So for a private key 'my_pkey', there should be a 'my_pkey.pub' in
same directory.
It may also be a case of the SSH server's key not being supported by
the native client (same limitations as user keys) which would explain
the key exchange error. Can check the type of key configured for the
server in /etc/ssh/sshd_config, HostKey entry. There should be at
least one non-ECDSA key configured, eg:
HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
If there is only a ECDSA key entry, the native client will not be able
to connect.
I want to download multiple files from FTP in python. the my code works when I just download 1 file, but not works for more than one!
import urllib
urllib.urlretrieve('ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/00/00/PMC1790863.tar.gz', 'file1.tar.gz')
urllib.urlretrieve('ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/00/00/PMC2329613.tar.gz', 'file2.tar.gz')
An error say:
Traceback (most recent call last):
File "/home/ehsan/dev_center/bigADEVS-bknd/daemons/crawler/ftp_oa_crawler.py", line 3, in <module>
urllib.urlretrieve('ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/00/00/PMC2329613.tar.gz', 'file2.tar.gz')
File "/usr/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/usr/lib/python2.7/urllib.py", line 245, in retrieve
fp = self.open(url, data)
File "/usr/lib/python2.7/urllib.py", line 213, in open
return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 558, in open_ftp
(fp, retrlen) = self.ftpcache[key].retrfile(file, type)
File "/usr/lib/python2.7/urllib.py", line 906, in retrfile
conn, retrlen = self.ftp.ntransfercmd(cmd)
File "/usr/lib/python2.7/ftplib.py", line 334, in ntransfercmd
host, port = self.makepasv()
File "/usr/lib/python2.7/ftplib.py", line 312, in makepasv
host, port = parse227(self.sendcmd('PASV'))
File "/usr/lib/python2.7/ftplib.py", line 830, in parse227
raise error_reply, resp
IOError: [Errno ftp error] 200 Type set to I
What should I do?
It is a bug in urllib in python 2.7. Reported here. The reason behind the same is explained here
Now, when a user tries to download the same file or another file from
same directory, the key (host, port, dirs) remains the same so
open_ftp() skips ftp initialization. Because of this skipping,
previous FTP connection is reused and when new commands are sent to
the server, server first sends the previous ACK. This causes a domino
effect and each response gets delayed by one and we get an exception
from parse227()
A possible solution is to clear the cache that may have been built up by previous calls. You may use the urllib.urlcleanup() method calls between your urlretrieve calls for the same, as mentioned here.
Hope this helps!
I use Duplicity to run backups from my local server to Amazon S3. This has been working fine for over a year. Three days ago, I started getting the following errors:
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/duplicity/backends/_boto_multi.py", line 204, in _upload
num_cb=max(2, 8 * bytes / (1024 * 1024))
File "/usr/lib/python2.7/site-packages/boto/s3/multipart.py", line 260, in upload_part_from_file
query_args=query_args, size=size)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1291, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 748, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 949, in _send_file_internal
query_args=query_args
File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
retry_handler=retry_handler
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1068, in make_request
retry_handler=retry_handler)
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 939, in _mexe
request.body, request.headers)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 842, in sender
http_conn.send(chunk)
File "/usr/lib64/python2.7/httplib.py", line 805, in send
self.sock.sendall(data)
File "/usr/lib64/python2.7/ssl.py", line 229, in sendall
v = self.send(data[count:])
File "/usr/lib64/python2.7/ssl.py", line 198, in send
v = self._sslobj.write(data)
error: [Errno 104] Connection reset by peer
These are still occurring even after I tried the following:
--added "s3-use-multiprocessing" to my script file
--added the following two lines to /etc/sysctl.conf:
net.ipv4.tcp_wmem = 4096 16384 512000
net.ipv4.tcp_rmem = 4096 87380 512000
-- ran sysctl -p to start using the above.
Three days ago, I started running Duplicity on a couple of other servers, backing up to a different bucket on the same account. That was when THIS server started reporting connection reset errors. The other servers are working fine, and all of them are using the same versions of Duplicity and Python. They are in different locations on different subnets, but that shouldn't make a difference.
The original chunk size on the problem server was 25MB. It's 250MB on the others. What else can I look for? I'm guessing Amazon is resetting the connection, but why single out this server?
I'm coding a script to check a FTP directory and download the new files. This is part of the code:
from ftplib import FTP
import os
ftp = FTP(ftp_lance)
ftp.login(login, password)
ftp.cwd('xxxxxx')
FTP_list = ftp.nlst()
lista_diferenca = [file for file in FTP_list if file not in local_list]
for file in lista_diferenca:
local_filename = os.path.join(cache, file)
ftp.retrbinary('REST ' + file, open(local_filename, 'wb').write)
When I run it, I get this error message:
Traceback (most recent call last):
File "D:\Scripts\Istari\Radagast\Radagast.py", line 44, in <module>
ftp.retrbinary('REST tabela14_pag5.pdf', open(local_filename, 'wb').write)
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 414, in retrbinary
conn = self.transfercmd(cmd, rest)
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 339, in ntransfercmd
resp = self.sendcmd(cmd)
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 249, in sendcmd
return self.getresp()
File "D:\Portable Python 2.7.6.1\App\lib\ftplib.py", line 224, in getresp
raise error_perm, resp
error_perm: 501 Bad parameter. Numeric value required
I check several sites searching for this kind of error and find nothing. It seems that my retrbinaty is broken, but the arguments looks right (first the 'Rest' + file, and then the callback function).
Some idea about my error?
You need to specify the FTP command RETR, not REST:
ftp.retrbinary('RETR ' + file, open(local_filename, 'wb').write)
I am trying to download a file from FTP using python. I was able to successfully move into the directory but can't download the file.
The command I use is ftp.retrbinary('master.idx', open(fname,'wb').write)
And error is below. It looks like the command is looking for MASTER.IDX instead of master.idx
The full path to the file I want to download is ftp://ftp.sec.gov/edgar/full-index/2011/QTR2/master.idx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/ftplib.py", line 406, in retrbinary
conn = self.transfercmd(cmd, rest)
File "/usr/lib/python2.7/ftplib.py", line 368, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib/python2.7/ftplib.py", line 331, in ntransfercmd
resp = self.sendcmd(cmd)
File "/usr/lib/python2.7/ftplib.py", line 244, in sendcmd
return self.getresp()
File "/usr/lib/python2.7/ftplib.py", line 219, in getresp
raise error_perm, resp
ftplib.error_perm: 500 MASTER.IDX not understood
I can't say why the name changes to uppercase. In any case, when using FTP, I make like this, it may help you:
server = "URL.of.server"
directory = "directory/where/the/file/is"
filename = "nameoffile.txt"
from ftplib import FTP
ftp = FTP(server) #Set server address
ftp.login() # Connect to server
ftp.cwd(directory) # Move to the desired folder in server
ftp.retrbinary('RETR ' + filename,open(filename, 'wb').write) # Download file from server
ftp.close() # Close connection
I think that it may be the 'RETR ', if you don't write, it the server may not understand what you want to do
use wget module of python instead. Here is an example snippet
import wget
fileloc = '/path/to/the/file/foo.txt'
wget.download(fileloc)