We Keep Coding

How to fix invalid URL error while using http.client?

I am trying to hit an api using http.client of pyhton3 to simulate how I would do the same in a web browser.
However, http.client feels that url is inappropriate.
This is what I am trying to do.
import http.client
connection = http.client.HTTPSConnection("https://analyticsapi.zoho.com/api/EmailAddress/WorkspaceName/TableName?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=XML&ZOHO_API_VERSION=1.0&authtoken=************&ZOHO_IMPORT_TYPE=APPEND&ZOHO_AUTO_IDENTIFY=TRUE&ZOHO_ON_IMPORT_ERROR=ABORT&ZOHO_CREATE_TABLE=TRUE&ZOHO_FILE=/home/dev1/Desktop/Zoho/temporary.csv")
connection.request("GET", "/")
response = connection.getresponse()
print("Status: {} and reason: {}".format(response.status, response.reason))
connection.close()
And this is the error I am getting.
$ python3 pyToTestPushingCSV.py
Traceback (most recent call last):
File "/usr/lib/python3.5/http/client.py", line 798, in _get_hostport
port = int(host[i+1:])
ValueError: invalid literal for int() with base 10: '//analyticsapi.zoho.com/api/usename/ATable/InsideTable?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=XML&ZOHO_API_VERSION=1.0&authtoken=****
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "pyToTestPushingCSV.py", line 3, in <module>
connection = http.client.HTTPSConnection("https://analyticsapi.zoho.com/api/usename/ATable/InsideTable?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=XML&ZOHO_API_VERSION=1.0&authtoken=****************&ZOHO_IMPORT_TYPE=APPEND&ZOHO_AUTO_IDENTIFY=TRUE&ZOHO_ON_IMPORT_ERROR=ABORT&ZOHO_CREATE_TABLE=TRUE&ZOHO_FILE=/home/dev1/Desktop/Zoho/temporary.csv")
File "/usr/lib/python3.5/http/client.py", line 1233, in __init__
source_address)
File "/usr/lib/python3.5/http/client.py", line 762, in __init__
(self.host, self.port) = self._get_hostport(host, port)
File "/usr/lib/python3.5/http/client.py", line 803, in _get_hostport
raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
http.client.InvalidURL: nonnumeric port: '//analyticsapi.zoho.com/api/usename/ATable/InsideTable?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=XML&ZOHO_API_VERSION=1.0&authtoken=*************&ZOHO_IMPORT_TYPE=APPEND&ZOHO_AUTO_IDENTIFY=TRUE&ZOHO_ON_IMPORT_ERROR=ABORT&ZOHO_CREATE_TABLE=TRUE&ZOHO_FILE=/home/dev1/Desktop/Zoho/temporary.csv'
When I hit the URL with my Browser it gives a good response back in XML which in short, succeeds.
This is where I referred for documentation
Can you pin point where I went wrong?

As per documentation HTTPS support is only available if Python was compiled with SSL support (through the ssl module).
Also the default port for https connection is 443. The module seems to be hitting that port by default.

How can I correct an 'only one socket usage of each address error' caused by a process being spawned within a class?

Currently, I am trying to develop a server framework which passes messages from twitch to other machines on a local network. I have a class called server and below I have a rudimentary example which demonstrates the problem I am running into. The issue is that the twitch_socket is being created twice and bound to the address/port. My expected result is that the socket would be shared between the child processes of the Server class. How can I modify the class, or even get rid of it entirely, so that the Processes would be able to share sockets between them?
import multiprocessing
import socket
import re
from BotPass import PASSWORD
def send_message(socketobj, message):
'Sends a str as bytes through socket'
message = message.encode()
socketobj.sendall(message)
def recv_message(socketobj):
'Receives a str as bytes though socket'
return socketobj.recv(2048).decode()
class Server:
'Handles receiving messages from twitch and directs messages from clients'
twitch_socket = socket.socket()
twitch_socket.connect(('irc.chat.twitch.tv', 6667))
send_message(twitch_socket, 'PASS %s\r\n' % (PASSWORD))
send_message(twitch_socket, 'NICK %s\r\n' % ('squid_coin_bot'))
send_message(twitch_socket, 'JOIN #jtv\r\n')
send_message(twitch_socket, 'CAP REQ :twitch.tv/commands\r\n')
server_socket = socket.socket()
server_socket.bind(('', 9999))
work_queue = multiprocessing.Queue()
#Queue of messages from twitch
worker_queue = multiprocessing.Queue()
#Queue of free client socket objects
result_queue = multiprocessing.Queue()
#Queue of what to send back to twitch
def start():
accept_process = multiprocessing.Process(target=Server.accept_connections)
# *This is most likely where the issue is occurring*
accept_process.daemon = True
accept_process.start()
def accept_connections():
''
Server.server_socket.listen(10)
while 1:
(clientsocket, clientaddr) = Server.server_socket.accept()
# What I believe I am referencing here is the server socket which is inherent to the Server class
if re.match(r'192\.168\.\d{1,3}\.\d{1,3}', clientaddr[0])\
or clientaddr[0] == '127.0.0.1':
Server.worker_queue.put(clientsocket)
else:
clientsocket.close()
Server.start()
input()
Output in Console:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Program Files\Python36\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Program Files\Python36\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Program Files\Python36\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\twitch-market\server.py", line 18, in <module>
class Server:
File "C:\twitch-market\server.py", line 27, in Server
server_socket.bind(('', 9999))
OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

Add this socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
This is because the previous execution has left the socket in a TIME_WAIT state, and can’t be immediately reused.the SO_REUSEADDR flag tells the kernel to reuse a local socket in TIME_WAIT state, without waiting for its natural timeout to expire.

Connection to remote MySQL db from Python 3.4

I'm trying to connect to two MySQL databases (one local, one remote) at the same time using Python 3.4 but I'm really struggling. Splitting the problem into three:
Step 1: connect to the local DB. This is working fine
using PyMySQL. (MySQLdb isn't compatible with Python 3.4, of
course.)
Step 2: connect to the remote DB (which needs to
use SSH). I can get it to work from the Linux command prompt but not
from Python... see below.
Step 3: connect to both at the
same time. I think I'm supposed to use a different port for the
remote database so that I can have both connections at the same time
but I'm out of my depth here! If it's relevant then the two DBs will
have different names. And if this question isn't directly related,
please tell me and I'll post it separately.
Unfortunately I'm not really starting in the right place for a newbie... once I can get this working I can happily go back to basic Python and SQL but hopefully someone will take pity on me and give me a hand to get started!
For Step 2, my code is below. It seems to be quite close to the sshtunnel example which answers this question Python - SSH Tunnel Setup and MySQL DB Access - though that uses MySQLdb. For the moment I'm embedding the connection parameters – I'll move them to the config file once it's working properly.
import dropbox, pymysql, shlex, shutil, subprocess
from sshtunnel import SSHTunnelForwarder
import iot_config as cfg
def CloseLocalDB():
localcur.close()
localdb.close()
def CloseRemoteDB():
# Disconnect from the database
# remotecur.close()
# remotedb.close()
# Close the SSH tunnel
# ssh.close()
print("end of CloseRemoteDB function")
def OpenLocalDB():
global localcur, localdb
localdb = pymysql.connect(host=cfg.localdbconn['host'], user=cfg.localdbconn['user'], passwd=cfg.localdbconn['passwd'], db=cfg.localdbconn['db'])
localcur = localdb.cursor()
def OpenRemoteDB():
global remotecur, remotedb
with SSHTunnelForwarder(
('my_remote_site', 22),
ssh_username = "my_ssh_username",
ssh_private_key = "/etc/ssh/my_private_key.ppk",
ssh_private_key_password = "my_private_key_password",
remote_bind_address = ('127.0.0.1', 3308)) as server:
remotedb = None
#Following line gives an error if uncommented
# remotedb = pymysql.connect(host='127.0.0.1', user='remote_db_user', passwd='remote_db_password', db='remote_db_name', port=server.local_bind_port)
#remotecur = remotedb.cursor()
# Main program starts here
OpenLocalDB()
CloseLocalDB()
OpenRemoteDB()
CloseRemoteDB()
This is the error I'm getting:
2016-04-21 19:13:33,487 | ERROR | Secsh channel 0 open FAILED: Connection refused: Connect failed
2016-04-21 19:13:33,553 | ERROR | In #1 <-- ('127.0.0.1', 60591) to ('127.0.0.1', 3308) failed: ChannelException(2, 'Connect failed')
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 60591)
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/sshtunnel.py", line 286, in handle
src_address)
File "/usr/local/lib/python3.4/dist-packages/paramiko/transport.py", line 834, in open_channel
raise e
paramiko.ssh_exception.ChannelException: (2, 'Connect failed')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/socketserver.py", line 613, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.4/socketserver.py", line 669, in __init__
self.handle()
File "/usr/local/lib/python3.4/dist-packages/sshtunnel.py", line 296, in handle
raise HandlerSSHTunnelForwarderError(msg)
sshtunnel.HandlerSSHTunnelForwarderError: In #1 <-- ('127.0.0.1', 60591) to ('127.0.0.1', 3308) failed: ChannelException(2, 'Connect failed')
----------------------------------------
Traceback (most recent call last):
File "/home/pi/Documents/iot_pm2/iot_ssh_example_for_help.py", line 38, in <module>
OpenRemoteDB()
File "/home/pi/Documents/iot_pm2/iot_ssh_example_for_help.py", line 32, in OpenRemoteDB
remotedb = pymysql.connect(host='127.0.0.1', user='remote_db_user', passwd='remote_db_password', db='remote_db_name', port=server.local_bind_port)
File "/usr/local/lib/python3.4/dist-packages/pymysql/__init__.py", line 88, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/pymysql/connections.py", line 678, in __init__
self.connect()
File "/usr/local/lib/python3.4/dist-packages/pymysql/connections.py", line 889, in connect
self._get_server_information()
File "/usr/local/lib/python3.4/dist-packages/pymysql/connections.py", line 1190, in _get_server_information
packet = self._read_packet()
File "/usr/local/lib/python3.4/dist-packages/pymysql/connections.py", line 945, in _read_packet
packet_header = self._read_bytes(4)
File "/usr/local/lib/python3.4/dist-packages/pymysql/connections.py", line 981, in _read_bytes
2013, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
Thanks in advance.

Answering my own question because, with a lot of help from J.M. Fernández on Github, I have a solution: the example that I copied at the beginning uses port 3308 but port 3306 is the standard. Once I'd changed this it started working.

web connection with python

I have this code to create a webapp in my server:
import web
urls = (
'/update', 'Update',
)
app = web.application(urls, globals())
class Update:
print "hola"
if __name__=='__main__':
app.run()
When I try to execute:
python#ubuntu:~$ python prueba.py 8081
hola
http://0.0.0.0:8081/
Traceback (most recent call last):
File "prueba.py", line 21, in <module>
app.run()
File "/usr/local/lib/python2.6/dist-packages/web/application.py", line 311, in run
return wsgi.runwsgi(self.wsgifunc(*middleware))
File "/usr/local/lib/python2.6/dist-packages/web/wsgi.py", line 54, in runwsgi
return httpserver.runsimple(func, validip(listget(sys.argv, 1, '')))
File "/usr/local/lib/python2.6/dist-packages/web/httpserver.py", line 148, in runsimple
server.start()
File "/usr/local/lib/python2.6/dist-packages/web/wsgiserver/__init__.py", line 1753, in start
raise socket.error(msg)
socket.error: No socket could be created
Why is it happening?
Thank you in advance

The error message says that it couldn't create a listening socket on the specified port. Check if there is already a server running on port 8081.

Python BaseHTTPServer, how do I catch/trap "broken pipe" errors?

I build a short url translator engine in Python, and I'm seeing a TON of "broken pipe" errors, and I'm curious how to trap it best when using the BaseHTTPServer classes. This isn't the entire code, but gives you an idea of what I'm doing so far:
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import memcache
class clientThread(BaseHTTPRequestHandler):
def do_GET(self):
content = None
http_code,response_txt,long_url = \
self.ag_trans_url(self.path,content,'GET')
self.http_output( http_code, response_txt, long_url )
return
def http_output(self,http_code,response_txt,long_url):
self.send_response(http_code)
self.send_header('Content-type','text/plain')
if long_url:
self.send_header('Location', long_url)
self.end_headers()
if response_txt:
self.wfile.write(response_txt)
return
def ag_trans_url(self, orig_short_url, post_action, getpost):
short_url = 'http://foo.co' + orig_short_url
# fetch it from memcache
long_url = mc.get(short_url)
# other magic happens to look it up from db if there was nothing
# in memcache, etc
return (302, None, log_url)
def populate_memcache()
# connect to db, do lots of mc.set() calls
def main():
populate_memcache()
try:
port = 8001
if len(sys.argv) > 1:
port = int(sys.argv[1])
server = HTTPServer(('',port), clientThread)
#server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
print '[',str(datetime.datetime.now()),'] short url processing has begun'
server.serve_forever()
except KeyboardInterrupt,SystemExit:
print '^C received, shutting down server'
server.socket.close()
The code itself works great, but started throwing errors almost immediately when in production:
Traceback (most recent call last):
File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
self.handle()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 310, in handle_one_request
method()
File "/opt/short_url_redirector/shorturl.py", line 38, in do_GET
self.http_output( http_code, response_txt, long_url )
File "/opt/short_url_redirector/shorturl.py", line 52, in http_output
self.send_response(http_code)
File "/usr/lib/python2.5/BaseHTTPServer.py", line 370, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.5/BaseHTTPServer.py", line 376, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.5/socket.py", line 274, in write
self.flush()
File "/usr/lib/python2.5/socket.py", line 261, in flush
self._sock.sendall(buffer)
error: (32, 'Broken pipe')
The bulk of these errors seem to stem from having a problem calling the send_header() method where all I'm writing out is this:
self.send_header('Location', long_url)
So I'm curious where in my code to try to trap for this IO exception... do I write try/except calls around each of the self.send_header/self.end_headers/self.wfile.write calls? The other error I see from time to time is this one, but not sure which exception to watch to even catch this:
Traceback (most recent call last):
File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
self.handle()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 299, in handle_one_request
self.raw_requestline = self.rfile.readline()
File "/usr/lib/python2.5/socket.py", line 381, in readline
data = self._sock.recv(self._rbufsize)
error: (104, 'Connection reset by peer')

This appears to be a bug in SocketServer, see this link Python Bug: 14574
A fix (works for me in Python 2.7) is to override the SocketServer.StreamRequestHandler finish() method, something like this:
...
def finish(self,*args,**kw):
try:
if not self.wfile.closed:
self.wfile.flush()
self.wfile.close()
except socket.error:
pass
self.rfile.close()
#Don't call the base class finish() method as it does the above
#return SocketServer.StreamRequestHandler.finish(self)

The "broken pipe" exception means that your code tried to write to a socket/pipe which the other end has closed. If the other end is a web browser, the user could have stopped the request. You can ignore the traceback; it does not indicate a serious problem. If you want to suppress the message, you can put a try ... except block around all of the code in your http_output function, and log the exception if you like.
Additionally, if you want your HTTP server to process more than one request at a time, you need your server class to use one of the SocketServer.ForkingMixIn and SocketServer.ThreadingMixIn classes. Check the documentation of the SocketServer module for details.
Add: The "connection reset by peer" exception means that your code tried to read from a dead socket. If you want to suppress the traceback, you will need to extend the BaseHTTPServer class and override the handle_one_request method to add a try ... except block. You will need a new server class anyway, to implement the earlier suggestion about processing more than one request at a time.

In my application, the error didn't occur in finish(), it occurred in handle(). This fix catches the broken pipe errors:
class MyHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
...
def handle(self):
try:
BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
except socket.error:
pass