We Keep Coding

Getting an error while trying to pass arguments from command line

I am trying to create a python script file with multiple functions and pass 3 arguments to the main functions. These parameters are also passed onto other functions with the script. My code looks somewhat like this:
import benepar
import spacy
import json
import gc
import typer
from spacy.tokens import Doc,DocBin
from typing import Tuple, List
from pathlib import Path
def main(nlp_dir: Path,inp_File:Path,out_File:Path):
try:
seg_obj=load_nlp_object(nlp_dir)
print('Loaded nlp obj')
doc_train=DocBin().from_disk(inp_File)
docs=get_list_of_docs(doc_train,nlp_dir)
data=[]
except Exception as e:
print(e)
chunk_count=0
for d in docs:
try:
temp=seg_obj(d)
chunk_count=chunk_count+1
if chunk_count%5 == 0:
seg_obj=load_nlp_object(nlp_dir)
gc.collect()
#Other code
except Exception as e:
print(e)
#Saving linker data
for val in data:
with open(out_File,'a',encoding='utf8') as f:
json.dump(val,f)
f.write(",\n")
print('Data created successfully.')
def load_nlp_object(nlp_dir):
#Code
return nlp
def get_list_of_docs(bin_obj,nl):
n=load_nlp_object(nl)
dcs=list(bin_obj.get_docs(n.vocab))
return dcs
def benepar_split(doc: Doc) -> List[Tuple]:
#Code
return split_indices
if __name__=='__main__':
typer.run(main)
While running I get the following error:
Traceback (most recent call last):
File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\Project\create_linker_data.py", line 96, in <module>
typer.run(main)
File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\typer\main.py", line 864, in run
app()
File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\typer\main.py", line 214, in __call__
return get_command(self)(*args, **kwargs)
File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 1053, in main
rv = self.invoke(ctx)
File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\typer\main.py", line 500, in wrapper
return callback(**use_params) # type: ignore
TypeError: main() got an unexpected keyword argument 'inp_file'
I have debugged by code but I am unable to find the root cause. My command line command is python create_linker_data.py "Entity Linker\\ner_obj\\" "Named Entity Recognition\\sentiment_data\\3\\3.spacy" "Entity Linker\\data\\entity_linker_3.txt"

Python XMLRPC: Cannot allow None even after allow_none = True

I'm trying to write a simple application that communicates using RPCs. I'm using python 3.7's xmlrpc.
This is my server code
MY_ADDR = ("localhost", int(sys.argv[1]))
HOST_ADDR = ("localhost", int(sys.argv[2]))
class RpcServer(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.port = MY_ADDR[1]
self.addr = MY_ADDR[0]
# serve other hosts using this
self.server = SimpleXMLRPCServer((self.addr, self.port))
self.server.register_function(self.recv_ops)
def run(self):
self.server.serve_forever()
def recv_ops(self, sender, op):
print("Sender ", sender, " sent: ", op)
pass
And this is what I'm using as my client's code
def send_ops(host_addr, op):
# contact the other host using this
proxy_addr = "http://{addr}:{port}/".format(addr=host_addr[0], port=host_addr[1])
client_proxy = xmlrpc.client.ServerProxy(proxy_addr, allow_none=True)
resp = client_proxy.recv_ops(MY_ADDR, op)
...
send_ops(HOST_ADDR, ("d", ii, last_line[ii])) # THE RPC CALL I MAKE
Despite setting allow_none=True, I keep getting this:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "nb.py", line 102, in editor
send_ops(HOST_ADDR, ("d", ii, last_line[ii]))
File "nb.py", line 63, in send_ops
resp = client_proxy.recv_ops(MY_ADDR, op)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1112, in __call__
return self.__send(self.__name, args)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1452, in __request
verbose=self.__verbose
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1170, in single_request
return self.parse_response(resp)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1342, in parse_response
return u.close()
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 656, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'TypeError'>:cannot marshal None unless allow_none is enabled">
What's tripping my is that the server on the other side actually receives the message (without any None)
Sender ['localhost', 8001] sent: ['d', 4, 'o']
What am I missing here? Any help would be appreciated.
Thanks!

In your server class, add allow_none=True to your SimpleXMLRPCServer instantiation.
self.server = SimpleXMLRPCServer((self.addr, self.port), allow_none=True)
The allow_none and encoding parameters are passed on to xmlrpc.client and control the XML-RPC responses that will be returned from the server.

Can't initiate get request in web.py

I'm trying to run two servers using web.py and initiating calls from one to another. Both servers start normally but when I try to call a url the below stack trace is thrown.
import web
urls = (
'/ping', 'Ping',
'/acqlock/+(.*)', 'Acquire',
)
class MSA(web.application):
def run(self, port=8081, *middleware):
func = self.wsgifunc(*middleware)
return web.httpserver.runsimple(func, ('127.0.0.1', port))
app = MSA(urls, globals())
if __name__ == "__main__":
app.run(port=8081)
class Acquire:
def GET(self, resource_name):
print resource_name
response = app.request('http://127.0.0.1:8080/acqlock/' + resource_name, method='GET')
return response
But I keep getting this error after calling the /acqlock.
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\web\wsgiserver\__init__.py", line 1245, in communicate
req.respond()
File "C:\Python27\lib\site-packages\web\wsgiserver\__init__.py", line 775, in respond
self.server.gateway(self).respond()
File "C:\Python27\lib\site-packages\web\wsgiserver\__init__.py", line 2018, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "C:\Python27\lib\site-packages\web\httpserver.py", line 306, in __call__
return self.app(environ, xstart_response)
File "C:\Python27\lib\site-packages\web\httpserver.py", line 274, in __call__
return self.app(environ, start_response)
File "C:\Python27\lib\site-packages\web\application.py", line 279, in wsgi
result = self.handle_with_processors()
File "C:\Python27\lib\site-packages\web\application.py", line 249, in handle_with_processors
return process(self.processors)
File "C:\Python27\lib\site-packages\web\application.py", line 246, in process
raise self.internalerror()
File "C:\Python27\lib\site-packages\web\application.py", line 515, in internalerror
parent = self.get_parent_app()
File "C:\Python27\lib\site-packages\web\application.py", line 500, in get_parent_app
if self in web.ctx.app_stack:
AttributeError: 'ThreadedDict' object has no attribute 'app_stack'

Use requests library for this.
import requests
response = requests.request(method='GET', url ='http://127.0.0.1:8080/acqlock/' + resource_name)
Note: You have used port 8080 in url even though you have hosted the web.py in 8081

I wrote a project with tornado, but this exception is always in my log file

This is the error log:
[I 160308 11:09:59 web:1908] 200 GET /admin/realtime (117.93.180.216) 107.13ms
[E 160308 11:09:59 http1connection:54] Uncaught exception
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tornado/http1connection.py", line 238, in _read_message
delegate.finish()
File "/usr/local/lib/python3.4/dist-packages/tornado/httpserver.py", line 290, in finish
self.delegate.finish()
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 1984, in finish
self.execute()
File "/usr/local/lib/python3.4/dist-packages/blueware-1.0.10/blueware/hooks/framework_tornado/web.py", line 480, in _bw_wrapper__RequestDispatcher_execute
future = wrapped(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 2004, in execute
**self.handler_kwargs)
File "/usr/local/lib/python3.4/dist-packages/blueware-1.0.10/blueware/hooks/framework_tornado/web.py", line 448, in _bw_wrapper_RequestHandler___init___
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 185, in init
self.initialize(**kwargs)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 2714, in wrapper
self.redirect(url)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 671, in redirect
self.finish()
File "/usr/local/lib/python3.4/dist-packages/blueware-1.0.10/blueware/hooks/framework_tornado/web.py", line 309, in _bw_wrapper_RequestHandler_finish_
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 934, in finish
self.flush(include_footers=True)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 870, in flush
for transform in self._transforms:
TypeError: 'NoneType' object is not iterable
[I 160308 11:10:00 web:1908] 200 GET /admin/order?order_type=1&order_status=1&page=0&action=allreal (49.89.27.173) 134.53ms
Can anyone tell me how to solve this problem? Thank you very much

I assume that OneAPM (blueware agent) is compatible with your python and Tornado version, however it's can be tricky.
Solution
Move self.redirect(url) from your handler initialize method to get method, like this
class MyHandler(tornado.web.RequestHandler):
def get(self):
self.redirect('/some_url')
or use RedirectHandler.
Every action that could finish request needs to be called in context of http-verb method (get, post, put and so on). The common mistake is making authetication/authorization in __init__ or initialize.
More detail
In Tornado's source there is a note about _transforms that is initialized in the constructor with None and set in_execute (oversimplifying - after headers_received).
A transform modifies the result of an HTTP request (e.g., GZip encoding).
Applications are not expected to create their own OutputTransforms
or interact with them directly; the framework chooses which transforms
(if any) to apply.
Reproduce
Sample that triggers this error. I'm including this only as a cross-check that blueware is not the cause:
import tornado.ioloop
import tornado.web
class SomeHandler(tornado.web.RequestHandler):
def initialize(self, *args, **kwargs):
url = '/some'
self.redirect(url)
# ^ this is wrong
def get(self):
# redirect should be here
self.write("Hello")
def make_app():
return tornado.web.Application([
(r"/", SomeHandler),
])
if __name__ == "__main__":
app = make_app()
app.listen(8888)
tornado.ioloop.IOLoop.current().start()
And stacktrace:
ERROR:tornado.application:Uncaught exception
Traceback (most recent call last):
File "/tmp/py3/lib/python3.4/site-packages/tornado/http1connection.py", line 238, in _read_message
delegate.finish()
File "/tmp/py3/lib/python3.4/site-packages/tornado/httpserver.py", line 289, in finish
self.delegate.finish()
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 2022, in finish
self.execute()
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 2042, in execute
**self.handler_kwargs)
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 183, in __init__
self.initialize(**kwargs)
File "test.py", line 8, in initialize
self.redirect(url)
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 666, in redirect
self.finish()
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 932, in finish
self.flush(include_footers=True)
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 868, in flush
for transform in self._transforms:
TypeError: 'NoneType' object is not iterable

Error when serving html with WSGI

I am trying to make an application that serves a simple HTML form to the user and then calls a function when the user submits the form. It uses wsgiref.simple_server to serve the HTML. The server is encountering an error and I can't understand why. The code is as follows:
#!/usr/bin/python3
from wsgiref.simple_server import make_server
from wsgiref.util import setup_testing_defaults
import webbrowser # open user's web browser to url when server is run
from sys import exc_info
from traceback import format_tb
# Easily serves an html form at path_to_index with style at path_to_style
# Calls on_submit when the form is submitted, passing a dictionary with key
# value pairs { "input name" : submitted_value }
class SimpleServer:
def __init__(self, port=8000, on_submit=None, index_path="./index.html", css_path="./style.css"):
self.port = port
self.on_submit = on_submit
self.index_path = index_path
self.css_path = css_path
# Forwards request to proper method, or returns 404 page
def wsgi_app(self, environ, start_response):
urls = [
(r"^$", self.index),
(r"404$", self.error_404),
(r"style.css$", self.css)
]
path = environ.get("PATH_INFO", "").lstrip("/")
# Call another application if they called a path defined in urls
for regex, application in urls:
match = re.search(regex, path)
# if the match was found, return that page
if match:
environ["myapp.url_args"] = match.groups()
return application(environ, start_response)
return error_404(environ, start_response)
# Gives the user a form to submit all their input. If the form has been
# submitted, it sends the ouput of self.on_submit(user_input)
def index(self, environ, start_response):
# user_input is a dictionary, with keys from the names of the fields
user_input = parse_qs(environ['QUERY_STRING'])
# return either the form or the calculations
index_html = open(self.index_path).read()
body = index_html if user_input == {} else calculate(user_input)
mime_type = "text/html" if user_input == {} else "text/plain"
# return the body of the message
status = "200 OK"
headers = [ ("Content-Type", mime_type),
("Content-Length", str(len(body))) ]
start_response(status, headers)
return [body.encode("utf-8")]
def start_form(self):
httpd = make_server('', self.port, ExceptionMiddleware(self.wsgi_app))
url = "http://localhost:" + str(self.port)
print("Visit " + url)
# webbrowser.open(url)
httpd.serve_forever()
if __name__ == "__main__":
server = SimpleServer()
server.start_form()
When I run it, I get the error
127.0.0.1 - - [16/Dec/2014 21:15:57] "GET / HTTP/1.1" 500 0
Traceback (most recent call last):
File "/usr/lib/python3.4/wsgiref/handlers.py", line 138, in run
self.finish_response()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 180, in finish_response
self.write(data)
File "/usr/lib/python3.4/wsgiref/handlers.py", line 266, in write
"write() argument must be a bytes instance"
AssertionError: write() argument must be a bytes instance
127.0.0.1 - - [16/Dec/2014 21:15:57] "GET / HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 49354)
Traceback (most recent call last):
File "/usr/lib/python3.4/wsgiref/handlers.py", line 138, in run
self.finish_response()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 180, in finish_response
self.write(data)
File "/usr/lib/python3.4/wsgiref/handlers.py", line 266, in write
"write() argument must be a bytes instance"
AssertionError: write() argument must be a bytes instance
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/wsgiref/handlers.py", line 141, in run
self.handle_error()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 368, in handle_error
self.finish_response()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 180, in finish_response
self.write(data)
File "/usr/lib/python3.4/wsgiref/handlers.py", line 274, in write
self.send_headers()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 331, in send_headers
if not self.origin_server or self.client_is_modern():
File "/usr/lib/python3.4/wsgiref/handlers.py", line 344, in client_is_modern
return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/socketserver.py", line 305, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 331, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.4/socketserver.py", line 669, in __init__
self.handle()
File "/usr/lib/python3.4/wsgiref/simple_server.py", line 133, in handle
handler.run(self.server.get_app())
File "/usr/lib/python3.4/wsgiref/handlers.py", line 144, in run
self.close()
File "/usr/lib/python3.4/wsgiref/simple_server.py", line 35, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
This output doesn't actually include the script I am running, which I am confused about. Any thoughts?

Just to register the solution for this issue, the problem is with len() function.
str(len(body))
It calculate the wrong size and when return the server Content-Length, then it wait more bytes that needed.
Thus, always send bytes using a buffer with UTF-8, follow example:
from io import StringIO
stdout = StringIO()
print("Hello world!", file=stdout)
start_response("200 OK", [('Content-Type', 'text/plain; charset=utf-8')])
return [stdout.getvalue().encode("utf-8")]

Looking at your code I don't see a direct reason for this error. However, I would strongly advise that unless you're trying to learn how wsgi works (or implement your own framework), you should use an existing micro-framework. WSGI is NOT meant to be used directly by applications. It provides a very thin interface between Python and a web server.
A nice and light framework is bottle.py -- I use it for all Python webapps. But there are many many others, look for "Non Full-Stack Frameworks" in https://wiki.python.org/moin/WebFrameworks.
A nice advantage of bottle is that it's a single file, which makes it easy to distribute with your server.