Im trying to read CSV files from ftp server in AppEngine, and I am able to connect to the ftp server. But it returned error when I tried to retrieve files.
Here is my code to read the CSV files from server:
import ftplib
import cStringIO
import csv
session = ftplib.FTP('myftpserver.com')
session.login('username','pwd')
session.set_pasv(False)
output = cStringIO.StringIO()
session.retrbinary('RETR myfile.csv', output.write)
csvfile = csv.reader(output.getvalue().splitlines(), delimiter=',')
for i, row in enumerate(csvfile):
print row
And here is the traceback of the error I am getting:
Traceback (most recent call last):
File "/home/vikas/apps/myproj/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/vikas/apps/myproj/admin/admin_actions.py", line 3528, in get_ftp_file
session.retrbinary('RETR myfile.csv', output.write)
File "/usr/lib/python2.7/ftplib.py", line 414, in retrbinary
conn = self.transfercmd(cmd, rest)
File "/usr/lib/python2.7/ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib/python2.7/ftplib.py", line 354, in ntransfercmd
sock = self.makeport()
File "/usr/lib/python2.7/ftplib.py", line 283, in makeport
for res in socket.getaddrinfo(None, 0, self.af, socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
File "/home/vikas/gcloud/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 318, in getaddrinfo
raise gaierror(EAI_NONAME, 'nodename nor servname provided, or not known')
gaierror: [Errno 8] nodename nor servname provided, or not known
I have no idea what I have done wrong, none of the commands like dir() nlst() etc is working, and the above error occurred as soon as I added them .
Alas, the socket simulation currently offered by App Engine isn't quite good enough to cover all use cases.
Here's a sample accessing a well-known public anonymous FTP server and fetching a small text file from it, just so everybody can reproduce and experiment...: in file getit.py, we have:
import ftplib
import cStringIO
def getit():
session = ftplib.FTP('ftp.mozilla.org')
session.login('anonymous','')
# session.set_pasv(False)
session.cwd('/pub/mozilla.org')
output = cStringIO.StringIO()
session.retrbinary('RETR README', output.write)
return output.getvalue()
if __name__ == '__main__':
print(getit())
This runs fine as stand-alone, python getit.py, whether you leave the set_pasv commented as here, or remove the comment.
To embed this in a GAE app, e.g:
import getit
class GetitPage(webapp2.RequestHandler):
def get(self): # pylint:disable-msg=invalid-name
try: result = getit.getit()
except Exception as e:
result = 'Error {}: {}'.format(type(e), e)
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write(result)
this works fine with the set_pasv left commented, but if you de-comment it, you get essentially the same exception you received.
So doing FTP to a server that forces you to active mode (doesn't support passive mode) is not going to work this way. However, that is a rather broken server -- could you get it fixed so that it supports the popular, default passive mode? Then your GAE app could work happily with it...!
Related
import paramiko
from socket import error as socket_error
import os
server =['10.10.0.1','10.10.0.2']
path='/home/test/'
for hostname in server:
try:
ssh_remote =paramiko.SSHClient()
ssh_remote.set_missing_host_key_policy(paramiko.AutoAddPolicy())
privatekeyfile = os.path.expanduser('~/.ssh/id')
mykey = paramiko.RSAKey.from_private_key_file(privatekeyfile, password='test123')
ssh_remote.connect(hostname, username = 'test1', pkey = mykey)
sftp=ssh_remote.open_sftp()
for i in sftp.listdir(path):
info = sftp.stat(i)
print info.st_size
except paramiko.SSHException as sshException:
print "Unable to establish SSH connection:{0}".format(hostname)
except socket_error as socket_err:
print "Unable to connect connection refused"
This is my code. I tried to get file size of remote server files. But below error was throwing. Can some please guide on this?
Error
Traceback (most recent call last):
File "<stdin>", line 15, in <module>
File "/usr/lib/python2.6/site-packages/paramiko/sftp_client.py", line 337, in stat
t, msg = self._request(CMD_STAT, path)
File "/usr/lib/python2.6/site-packages/paramiko/sftp_client.py", line 624, in _request
return self._read_response(num)
File "/usr/lib/python2.6/site-packages/paramiko/sftp_client.py", line 671, in _read_response
self._convert_status(msg)
File "/usr/lib/python2.6/site-packages/paramiko/sftp_client.py", line 697, in _convert_status
raise IOError(errno.ENOENT, text)
IOError: [Errno 2] No such file
SFTPClient.listdir returns file names only, not a full path. So to use the filename in another API, you have to add a path:
for i in sftp.listdir(path):
info = sftp.stat(path + "/" + i)
print info.st_size
Though that's inefficient. Paramiko knows the size already, you are just throwing the information away by using SFTPClient.listdir instead of SFTPClient.listdir_attr (listdir calls listdir_attr internally).
for i in sftp.listdir_attr(path):
print i.st_size
I'm trying to access a public FTP in my work, but I'm getting an error.
That's is the code that I used in my house.
from ftplib import FTP
ftp = FTP('ftp.cetip.com.br')
ftp.login()
ftp.cwd('/MediaCDI')
ftp.quit()
It's work fine in my home, but in my work I get this error:
Traceback (most recent call last):
File "C:\Users\TBMEPYG\Desktop\stack.py", line 3, in <module>
ftp = FTP('ftp.cetip.com.br')
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\ftplib.py", line 117, in __init__
self.connect(host)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\ftplib.py", line 152, in connect
source_address=self.source_address)
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\socket.py", line 722, in create_connection
raise err
File "C:\Users\TBMEPYG\AppData\Local\Continuum\Anaconda3\lib\socket.py", line 713, in create_connection
sock.connect(sa)
[Finished in 21.3s]
When I was developing a scraper in a HTTP website, I solved this problem for http using HTTPProxyAuth and requests. Just to illustrate, that's was the code:
from requests.auth import HTTPProxyAuth
import requests
user = 'xxxx'
password = 'yyyy'
credenciais = HTTPProxyAuth(user, password)
params = {'Dt_Ref': data, 'TpInstFinanceiro': 'CRI', 'Tipo':'1','saida':'txt'}
proxy_access = {'http':'proxy.mywork/accelerated_pac_base.pac'}
url = 'http://www.anbima.com.br/reune/reune_down.asp'
r = requests.post(url, data = params, proxies = proxy_access , auth = credenciais)
Anyone has any idea about what can I do?
Thanks
I have already been through these set of solutions for connecting IMAP under proxy
Information to add:
I am trying to write a python code that can fetch mails from gmails IMAP server using imapclient under http,https and socks proxy server of my academic insitute
When tried without any proxy handling, it used to give error
socket.error [101] network is unreachable
import imapclient
import pyzmail
imapObj = imapclient.IMAPClient('imap.gmail.com',ssl=True)
imapObj.login('***********#gmail.com','*********')
imapObj.select_folder('INBOX', readonly=True)
UIDs = imapObj.search(['SINCE 07-Jul-2016'])
for item in UIDs:
rawMessages = imapObj.fetch(item, ['BODY[]', 'FLAGS'])
message = pyzmail.PyzMessage.factory(rawMessages[item]['BODY[]'])
message.get_subject()
message.get_addresses('from')
message.get_addresses('to')
message.get_addresses('cc')
message.get_addresses('bcc')
message.text_part != None
message.text_part.get_payload().decode(message.text_part.charset)
message.html_part != None
message.html_part.get_payload().decode(message.html_part.charset)
imapObj.logout()
However, the process gives error as
File "mailtotext.py", line 16, in <module>
imapObj = imapclient.IMAPClient('imap.gmail.com',ssl=True)
File "/usr/local/lib/python2.7/dist-packages/imapclient/imapclient.py", line 152, in __init__
self._imap = self._create_IMAP4()
File "/usr/local/lib/python2.7/dist-packages/imapclient/imapclient.py",line 164, in _create_IMAP4
self._timeout)
File "/usr/local/lib/python2.7/dist-packages/imapclient/tls.py", line 153, in __init__
imaplib.IMAP4.__init__(self, host, port)
File "/usr/lib/python2.7/imaplib.py", line 172, in __init__
self.open(host, port)
File "/usr/local/lib/python2.7/dist-packages/imapclient/tls.py", line 158, in open
sock = socket.create_connection((host, port), self._timeout)
File "/usr/lib/python2.7/socket.py", line 571, in create_connection
raise err
socket.error: [Errno 101] Network is unreachable
I then followed the above mentioned link procedures owing to my institute proxy as http,https,socks
I have already set my system proxy settings as
http_proxy="http://10.3.100.207:8080/"
https_proxy="https://10.3.100.207:8080/"
ftp_proxy="ftp://10.3.100.207:8080/"
socks_proxy="socks://10.3.100.207:8080/"
and edited the code as
import imapclient
import pyzmail
import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4,"10.3.100.207",8080,True)
socket.socket = socks.socksocket
imapObj = imapclient.IMAPClient('imap.gmail.com',ssl=True)
imapObj.login('***********#gmail.com','*********')
imapObj.select_folder('INBOX', readonly=True)
UIDs = imapObj.search(['SINCE 07-Jul-2016'])
for item in UIDs:
rawMessages = imapObj.fetch(item, ['BODY[]', 'FLAGS'])
message = pyzmail.PyzMessage.factory(rawMessages[item]['BODY[]'])
message.get_subject()
message.get_addresses('from')
message.get_addresses('to')
message.get_addresses('cc')
message.get_addresses('bcc')
message.text_part != None
message.text_part.get_payload().decode(message.text_part.charset)
message.html_part != None
message.html_part.get_payload().decode(message.html_part.charset)
imapObj.logout()
But this process seems to freeze for very long time and finally I always need to make a keyboard interrupt. Its sure that it was freezed somewhere in socket files as per keyboard interrupt output.
Please help me through this, there are hardly solution to such problems over web. I have even tried tunneling but it isn't solving my problem or say making it worse (I might have not implemented it well :P)I would provide any other information and output if needed here
I'm trying to use ftps to send a file to a FTP server. Login and changing directory work:
import ftplib
ftps = ftplib.FTP_TLS('host','user','pwd')
ftps.set_pasv(True)
ftps.prot_p()
ftps.cwd('/target_directory')
however when I try to upload my file:
file = open(file, 'rb')
send_cmd = 'STOR file_name.txt'
ftps.storbinary(send_cmd, file)
file.close()
ftps.quit()
I get the following error:
File "/script/location/script.py", line 161, in <module>
ftps.storbinary(send_cmd,file)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 772, in storbinary
return self.voidresp()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 229, in voidresp
resp = self.getresp()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 222, in getresp
raise error_temp, resp
ftplib.error_temp: 425 Unable to build data connection: Operation not permitted
I've read that the 425 response code is often a result of being in active mode, which is why I included ftps.set_pasv(True) (although this is True by default).
I've also tried just listing the directory contents using ftps.retrlines('LIST') but get essentially the same error. I'm using Python 2.7.10. Any help would be greatly appreciated.
This is a reported bug in python: https://bugs.python.org/issue19500
You can apply the patch in a new class
class Explicit_FTP_TLS(ftplib.FTP_TLS):
"""Explicit FTPS, with shared TLS session"""
def ntransfercmd(self, cmd, rest=None):
conn, size = ftplib.FTP.ntransfercmd(self, cmd, rest)
if self._prot_p:
conn = self.context.wrap_socket(conn,
server_hostname=self.host,
session=self.sock.session)
return conn, size
I'm pretty new to python, so this very well be user error here, but I'm not quite sure what to do from here.
I'm trying to login to a zyxel gs2200 switch via ftp and download it's config file.
The command through cmd ftp is
get config X.log where X.log is whatever you decide to name it.
In python I can log in just fine, but I can not download a file without throwing an exception.
import ftplib
ftp = ftplib.FTP("my.ip.here")
ftp.login('user','Pass')
'230 Logged in'
ftp.retrbinary('RETR config', open('config', 'wb').write)
Traceback (most recent call last):
File "<pyshell#14>", line 1, in <module>
File "C:\Python27\lib\ftplib.py", line 398, in retrbinary
self.voidcmd('TYPE I')
File "C:\Python27\lib\ftplib.py", line 248, in voidcmd
self.putcmd(cmd)
File "C:\Python27\lib\ftplib.py", line 178, in putcmd
self.putline(line)
File "C:\Python27\lib\ftplib.py", line 173, in putline
self.sock.sendall(line)
File "C:\Python27\lib\socket.py", line 222, in meth
return getattr(self._sock,name)(*args)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
I've also tried opening a file before and then just calling it in the method, but it still gives the same error.
does anyone have any idea on how I can get these config files in python?
I Can tell you easy method to do this.
Ftp Login Url: ftp://username:password#hostname/
Username: your Username
Password: Your Password
To download a file from FTP server you could:
import urllib
urllib.urlretrieve('ftp://username:password#hostname/yourconfigname', 'yourconfigname')
In your case you can do something like this.
===============================================================
As for your problem the python error clearly indicating that you did not establish connection correctly(Always Close Connection) or May be your destination had already connected to maximum destinations thats why your target destination closing your connection.