This has been vexing me for a while now. I am trying to create a very simple REST-like interface (without using third-party libraries, which I know are available).
The idea behind it is that I can have a directory, for example mylib, where I can drop in python files, like do_something.py and, by POSTing to http://localhost/do_something the code will spring into life and do something!
I think I have managed to get somewhere near to my goal with the following structure:
The contents of the files are as follows.
example.py
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import json, logging
from mylib import my_module
class MyRequestHandler (BaseHTTPRequestHandler):
# Send JSON responses
# -----------
def send_json(self, json_message, response_code=200):
self.send_response(response_code)
self.send_header('Content-type', 'application/json')
self.end_headers()
self.request.sendall(json.dumps(json_message).encode())
# Get JSON requests
# ----------
def get_json(self):
body = self.rfile.read(int(self.headers.get('Content-Length')))
if (body):
try:
receivedData = json.loads(body.decode())
except:
self.send_json({"Status": "Error", "Message": "Invalid JSON received"}, 400)
receivedData = None
else:
receivedData = None
return receivedData
# POST
# ---------
def do_POST(self):
module_to_call = (self.path).replace('/', '.')[1:]
if module_to_call.endswith('.'): # Remove trailing dot
module_to_call = module_to_call[:-1]
print("Path is: '" + module_to_call + "'")
# invoke function
module_to_call = getattr(self, module_to_call)
response = module_to_call()
self.send_json(response)
# GET
# --------
def do_GET(self):
pass
# -----------------------------------------------------------------------------
# Server startup code
# -------------------
def start_server():
# Begin serving
# -------------
port = 8003
server = HTTPServer(('', port), MyRequestHandler)
print(("Server now running on port {0} ...").format(port))
server.serve_forever()
# -----------------------------------------------------------------------------
# Start the Server
# ----------------
if __name__ == '__main__':
start_server()
my_module.py
def my_module():
print("Hello World!")
return{'Greeting': 'Hello World!'}
When I fire up the server and attempt to POST to http://localhost:8003/my_module, I get the following output:
Server now running on port 8003 ...
Path is: 'my_module'
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 59541)
Traceback (most recent call last):
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 313, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 341, in process_request
self.finish_request(request, client_address)
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 354, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 681, in __init__
self.handle()
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\http\server.py", line 422, in handle
self.handle_one_request()
File "C:\Users\Test\AppData\Local\Programs\Python\Python35-32\lib\http\server.py", line 410, in handle_one_request
method()
File ".\example.py", line 43, in do_POST
module_to_call = getattr(self, module_to_call)
AttributeError: 'MyRequestHandler' object has no attribute 'my_module'
----------------------------------------
This makes perfect sense, since 'MyRequestHandler" does not have an attribute "my_module"! What I can't wrap my head around, is how to fix this?
Should I pass "mylib" into MyRequestHandler? Should I perform the import within the class (but then the functionality would only be available within the class)?
I'm trying to keep things clean and simple, so that even a Python novice (like I seem to be!) can just write a standalone script, drop it into "mylib" and everything "just works". The novice could visit the web address of their script and have it magically run.
Any help or advice would be gratefully received.
Use the __import__() method:
temp = __import__('mylib', globals(), locals(), ['module_to_call'], -1)
response = temp.module_to_call()
I use 2.6 at work, and this is usually used by those using even 2.7 because the importlib module is far more robust in 3. If you are using 3 you can do the following:
from importlib import import_module
temp = import_module('mylib')
but now you have to use getattr to get the function you want to be called
func_to_call = getattr(temp, 'module_to_call')
response = func()
Or you can have a dictionary of functions in another module, but that will require a lot of work as that dictionary grows.
Related
I'm trying to use the IPC protocol with ZeroMQ in Python.
import sys
import time
from random import randint
import zmq
def main(url=None):
ctx = zmq.Context.instance()
publisher = ctx.socket(zmq.PUB)
if url:
publisher.bind(url)
else:
publisher.bind('ipc://var/run/fast-service')
# Ensure subscriber connection has time to complete
time.sleep(1)
# Send out all 1,000 topic messages
for topic_nbr in range(1000):
publisher.send_multipart([
b"%03d" % topic_nbr,
b"Save Roger",
])
if __name__ == '__main__':
main(sys.argv[1] if len(sys.argv) > 1 else None)
It gives the following error:
Traceback (most recent call last):
File "pathopub.py", line 43, in <module>
main(sys.argv[1] if len(sys.argv) > 1 else None)
File "pathopub.py", line 19, in main
publisher.bind("ipc://var/run/fast-service")
File "zmq/backend/cython/socket.pyx", line 547, in zmq.backend.cython.socket.Socket.bind
zmq.error.ZMQError: No such file or directory for ipc path "var/run/fast-service".
I don't understand why would this happen with the socket.bind() function, since in the documentation it says:
When binding a socket to a local address using zmq_bind() with the ipc transport, the endpoint shall be interpreted as an arbitrary string identifying the pathname to create.
which means that it is not necessary to supply an already created directory.
The URL scheme is ipc://. You need to add an absolute path /var/run/fast-service. So,
publisher.bind('ipc:///var/run/fast-service')
More generally the URL is ipc://<host>/<path>. You want local host, so that part is empty. A file system URL is similar, file:///home/foo/bar.txt references /home/foo/bar.txt on the local host.
I saw a tutorial on YouTube(I can't link it because I can't find it anymore),
So the code is supposed to detect devices that are connected to my Internet/Router, I don't understand a lot about how his(The person who made the tutorial) code works
I also got this error in my console:
File "c:/Users/j/Desktop/Connection-Detection.py", line 6, in
IP_NETWORK = config('IP_NETWORK')
File "C:\Users\j\AppData\Local\Programs\Python\Python38-32\lib\site-packages\decouple.py", line 199, in call
return self.config(*args, **kwargs)
File "C:\Users\j\AppData\Local\Programs\Python\Python38-32\lib\site-packages\decouple.py", line 83, in call
return self.get(*args, **kwargs)
File "C:\Users\j\AppData\Local\Programs\Python\Python38-32\lib\site-packages\decouple.py", line 68, in get
raise UndefinedValueError('{} not found. Declare it as envvar or define a default value.'.format(option))
decouple.UndefinedValueError: IP_NETWORK not found. Declare it as envvar or define a default value.
PS C:\Users\j\Desktop\python\login>
That's "Detection.py"
import sys
import subprocess
import os
from decouple import config
IP_NETWORK = config('IP_NETWORK')
IP_DEVICE = config('IP_DEVICE')
proc = subprocess.Popen(['ping', IP_NETWORK],stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline
if not line:
break
connected_ip = line.decode('utf-8').split()[3]
if connected_ip == IP_DEVICE:
subprocess.Popen(['say', 'Someone connected to network'])
You need to define an environment variable in same directory as the Detection.py file.
Steps
Install python-decouple - pip install python-decouple.
Create a file called .env
Open the .env file and paste the following into it.
IP_NETWORK=YOUR_IP_NETWORK
IP_DEVICE=YOUR_IP_DEVICE
Replace YOUR_IP_NETWORK and YOUR_IP_DEVICE with your IP_NETWORK and IP_DEVICE
I am trying to use WSGI on Windows Server to run a simple flask app. I keep running into the following error:
Error occurred while reading WSGI handler: Traceback (most recent call
last): File "c:\inetpub\wwwroot\test_site\wfastcgi.py", line 711, in
main env, handler = read_wsgi_handler(response.physical_path) File
"c:\inetpub\wwwroot\test_site\wfastcgi.py", line 568, in
read_wsgi_handler return env, get_wsgi_handler(handler_name) File
"c:\inetpub\wwwroot\test_site\wfastcgi.py", line 551, in
get_wsgi_handler raise ValueError('"%s" could not be imported' %
handler_name) ValueError: "app.app" could not be imported StdOut:
StdErr
For my site I configured a handler to call the FastCGIModule from Microsoft Web Platform installer
My app file looks as such:
from flask import Flask, request, jsonify
from analyzers import analyzer
import write_log
app = Flask(__name__)
#app.route("/")
def test():
return "Test load"
#app.route('/analyze', methods=['POST'])
def parse():
text = request.json['text']
name = request.json['name']
model = request.json['model']
try:
convert_flag = request.json['convert_flag']
except KeyError:
convert_flag = False
results= analyzer(text, name, model, convert_dose=convert_flag)
write_log.write_log(text, name, model, results)
return jsonify(results)
if __name__ == "__main__":
app.run()
If I comment out the custom import of my analyzer script and my write_log script along with the POST method things will run, so I know I must be messing something up there.
Does anybody have any suggestions?
Thanks in advance.
Paul
I had the same issue and the problem was with a third-party library. What's causing your problem is certainly something different, but here's something I did to identify my issue and may help you as well:
Open wfastcgi.py
Locate the method get_wsgi_handler (probably on line 519)
There's a try/except inside a while module_name statement
Add raise to the end of the except block and save the file, like this:
except ImportError:
...
raise
Access your website URL again and check your logs, they now should be more detailed about what caused the ImportError and will point you in the right direction to fix the issue
The documentation is really vague about subclassing the CommandLineApp, only mentioning one example:
class YourApp(cli.app.CommandLineApp):
def main(self):
do_stuff()
So with the information I've found I've pieced together this code:
#!/usr/bin/env python
import os
import sys
from cli.app import CommandLineApp
# Append the parent folder to the python path
sys.path.append(os.path.join(os.path.dirname(__file__), '../'))
import tabulardata
from addrtools import extract_address
class SplitAddressApp(CommandLineApp):
def main(self):
"""
Split an address from one column to separate columns.
"""
table = tabulardata.from_file(self.params.file)
def for_each_row(i, item):
addr = extract_address(item['Address'])
print '%-3d %-75s %s' % (i, item['Address'], repr(addr))
table.each(for_each_row)
def setup(self):
self.add_param('file', metavar='FILE', help='The data file.')
self.add_param(
'cols', metavar='ADDRESS_COLUMN', nargs='+',
help='The name of the address column. If multiple names are ' + \
'passed, each column will be checked for an address in order'
)
if __name__ == '__main__':
SplitAddressApp().run()
Which seems correct to me. The documentation gives no examples on how to handle the setup method or running the application when using subclassing. I get the error:
Traceback (most recent call last):
File "bin/split_address_column", line 36, in
SplitAddressApp().run()
File "/Users/tomas/.pythonbrew/venvs/Python-2.7.3/address_cleaner/lib/python2.7/site-packages/cli/app.py", line 440, in __init__
Application.__init__(self, main, **kwargs)
File "/Users/tomas/.pythonbrew/venvs/Python-2.7.3/address_cleaner/lib/python2.7/site-packages/cli/app.py", line 129, in __init__
self.setup()
File "bin/split_address_column", line 28, in setup
self.add_param('file', metavar='FILE', help='The data file.')
File "/Users/tomas/.pythonbrew/venvs/Python-2.7.3/address_cleaner/lib/python2.7/site-packages/cli/app.py", line 385, in add_param
action = self.argparser.add_argument(*args, **kwargs)
AttributeError: 'SplitAddressApp' object has no attribute 'argparser'
So presumably I'm doing something wrong, but what?
I figured it out. Reading the source of pyCLI it turns out that the setup function is quite important for the functionality of the whole library, while I thought it was just a function where I could put my setup code. argparser is created in cli.app.CommandLineApp.setup which means I at least have to call
cli.app.CommandLineApp.setup(self)
inside the setup function for it to even work. And now the code works perfectly!
I have a simple python script that updates that statuses of justin.tv streams in my database. It's a Django based web application. This script worked perfectly before I moved it to my production server, but now it has issues with timing out or freezing. I've solved the time out problem by adding try/except blocks and making the script retry, but I still can't figure out the freezing problem.
I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That's the same point where the socket.timeout exception occurs. Some times however, it just locks up for ever. I just can't picture a scenario where python would freeze infinitely. Here is the code for the script that freezes. I'm linking website.networkmanagers below, as well as oauth and the justin.tv python library that I'm using.
import sys, os, socket
LOG = False
def updateStreamInfo():
# Set necessary paths
honstreams = os.path.realpath(os.path.dirname(__file__) + "../../../")
sys.path.append(honstreams)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
# Import necessary moduels
from website.models import Stream, StreamInfo
from website.networkmanagers import get_manager, \
NetworkManagerReturnedErrorException
# Get all streams
streams = Stream.objects.all()
try:
# Loop through them
for stream in streams:
skipstream = False
print 'Checking %s...' % stream.name,
# Get the appropriate network manager and
manager = get_manager(stream.network.name)
# Try to get stream status up to 3 times
for i in xrange(3):
try:
streamOnline = manager.getStreamOnline(stream.name, LOG)
break
except socket.error as e:
code, message = e
# Retry up to 3 times
print 'Error: %s. Retrying...'
# If this stream should be skipped
if(skipstream):
print 'Can\'t connect! Skipping %s' % stream.name
continue
# Skip if status has not changed
if streamOnline == stream.online:
print 'Skipping %s because the status has not changed' % \
stream.name
continue
# Save status
stream.online = streamOnline
stream.save()
print 'Set %s to %s' % (stream.name, streamOnline)
except NetworkManagerReturnedErrorException as e:
print 'Stopped the status update loop:', e
if(__name__ == "__main__"):
if(len(sys.argv) > 1 and sys.argv[1] == "log"):
LOG = True
if(LOG): print "Logging enabled"
updateStreamInfo()
networkmanagers.py
oauth.py
JtvClient.py
Example of the script freezing
foo#bar:/.../honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
Checking cilantrogamer... Skipping cilantrogamer because the status has not changed
| <- caret sits here blinking infinitely
Interesting update
Every time it freezes and I send a keyboard interrupt, it's on the same line in socket.py:
root#husta:/home/honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
^CChecking cilantrogamer...
Traceback (most recent call last):
File "website/scripts/updateStreamStatus.py", line 64, in <module>
updateStreamInfo()
File "website/scripts/updateStreamStatus.py", line 31, in updateStreamInfo
streamOnline = manager.getStreamOnline(stream.name, LOG)
File "/home/honstreams/honstreams/website/networkmanagers.py", line 47, in getStreamOnline
return self.getChannelLive(channelName, log)
File "/home/honstreams/honstreams/website/networkmanagers.py", line 65, in getChannelLive
response = client.get('/stream/list.json?channel=%s' % channelName)
File "/home/honstreams/honstreams/website/JtvClient.py", line 51, in get
return self._send_request(request, token)
File "/home/honstreams/honstreams/website/JtvClient.py", line 90, in _send_request
return conn.getresponse()
File "/usr/lib/python2.6/httplib.py", line 986, in getresponse
response.begin()
File "/usr/lib/python2.6/httplib.py", line 391, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.6/socket.py", line 397, in readline
data = recv(1)
KeyboardInterrupt
Any thoughts?
Have you tried using another application to open that connection? Given that it's an issue in production, perhaps you don't have some firewall issues.
Down in JtvClient.py it uses httplib to handle the connection. Have you tried changing this to use httplib2 instead?
Other than that stab in the dark, I would add a lot of logging statements to this code in order to track what actually happens and where it gets stuck. Then I would make sure that the point where it gets stuck can timeout on the socket (which usually involves either monkeypatching or forking the codebase) so that stuff fails instead of hanging.
You said:
I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That's the same point where the socket.timeout exception occurs.
Wrong. It doesn't freeze on that line because that line is a function call which calls lots of other functions through several levels of other modules. So you do not yet know where the program freezes. Also, that line is NOT the point where the socket timeout occurs. The socket timeout will only occur on a low level socket operation like select or recv which is being called several times in the chain of activity triggered by getStreamOnline.
You need to trace your code in a debugger or add print statements to track down exactly where the hang occurs. It could possibly be an infinite loop in Python but is more likely to be a low-level call to an OS networking function. Until you find the source of the error, you can't do anything.
P.S. the keyboard interrupt is a reasonable clue that the problem is around line 90 in JtvClient.py, so put in some print statements and find out what happens. There may be a stupid loop in there that keeps calling getresponse, or you may be calling it with bad parameters or maybe the network server really is borked. Narrow it down to fewer possibilities.
It turns out this HTTP connection isn't passed a timeout in jtvClient.py
def _get_conn(self):
return httplib.HTTPConnection("%s:%d" % (self.host, self.port))
Changed the last line to
return httplib.HTTPConnection("%s:%d" % (self.host, self.port), timeout=10)
Which solved it