PySpark+Flask+CherryPy - AttributeError: 'module' object has no attribute 'tree' - python

I'm trying to test how to integrate Flask with Spark model according to this tutorial https://www.codementor.io/spark/tutorial/building-a-web-service-with-apache-spark-flask-example-app-part2#/ . Here CherryPy is used for wsgi. Trouble is that when we are launching app via spark-submit, it shows such stack trace:
Traceback (most recent call last):
File "/home/roman/dev/python/flask-spark/cherrypy.py", line 43, in <module>
run_server(app)
File "/home/roman/dev/python/flask-spark/cherrypy.py", line 21, in run_server
cherrypy.tree.graft(app_logged, '/')
AttributeError: 'module' object has no attribute 'tree'
I have no idea where the trouble is. I think that it because of new/old version or something like that, but I'm not sure. I have used also python 3 instead of python 2, but it didn't help. Here is wsgi config:
import time, sys, cherrypy, os
from paste.translogger import TransLogger
from webapp import create_app
from pyspark import SparkContext, SparkConf
def init_spark_context():
# load spark context
conf = SparkConf().setAppName("movie_recommendation-server")
# IMPORTANT: pass aditional Python modules to each worker
sc = SparkContext(conf=conf, pyFiles=['test.py', 'webapp.py'])
return sc
def run_server(app):
# Enable WSGI access logging via Paste
app_logged = TransLogger(app)
# Mount the WSGI callable object (app) on the root directory
cherrypy.tree.graft(app_logged, '/')
# Set the configuration of the web server
cherrypy.config.update({
'engine.autoreload.on': True,
'log.screen': True,
'server.socket_port': 5432,
'server.socket_host': '0.0.0.0'
})
# Start the CherryPy WSGI web server
cherrypy.engine.start()
cherrypy.engine.block()
if __name__ == "__main__":
# Init spark context and load libraries
sc = init_spark_context()
dataset_path = os.path.join('datasets', 'ml-latest-small')
app = create_app(sc, dataset_path)
# start web server
run_server(app)

Traceback you've provided clearly shows that your app is trying to use a local module called cherrypy (/home/roman/dev/python/flask-spark/cherrypy.py) not the actual cherrypy library (which should be something like /path/to/your/python/lib/python-version/siteX.Y/cherrypy).
To solve this problem you can simply rename the local module to avoid conflicts.

Related

Import with and without main scope

I have two files, app.py and database.py in the same directory.
Primarily I have the following code snippets:
app.py
import database
db = "demo_database"
print(database.show_database_information())
database.py
from app import db
database_username = "root"
database_password = "password"
def show_database_information():
information = {}
information["filename"] = db
information["username"] = database_username
information["password"] = database_password
return information
When I try to run app.py I got the following error:
Traceback (most recent call last):
File "K:\PyPrac\circular_call\app.py", line 1, in <module>
import database
File "K:\PyPrac\circular_call\database.py", line 1, in <module>
from app import db
File "K:\PyPrac\circular_call\app.py", line 3, in <module>
print(database.show_database_information())
AttributeError: module 'database' has no attribute 'show_database_information'
Then I updated app.py and included __main__ check like below:
app.py
import database
db = "demo_database"
if __name__ == '__main__':
print(database.show_database_information())
Now it runs smoothly without any error.
I have several questions,
What is name of the error occurred in first scenario? Need explanation.
Why it runs after including __main__ scope?
What is the better approach of doing operations like this?
What I can understand are as below's. Maybe someone more expert can elaborate !
Import error.
if __name__ == '__main__': This condition is used to check whether a python module is being run directly or being imported.
If a module is imported, then it's __name__ is the name of the module instead of main. So, in such situations it is better to call if __name__ == '__main__':
Man!! You are creating a circular moment. Let me tell how.
import database # from app.py
But from database.py you imported db from app. That is creating a circular moment.
On the other hand,
if __name__ == '__main__':
this is making you database.py as a name of the module instead of __main__ that's why it's working. Nothing magical :)
UPDATE: Placed from app import db this line inside the function show_database_information()
This is the HOTFIX for you.

No module named core while importing flask enterprise

I wrote a python script to have a soap server with flask, it's exactly like what the documentation says:
from time import ctime
from flask import Flask
from flaskext.enterprise import Enterprise
if __name__ == '__main__':
app = Flask(__name__)
enterprise = Enterprise(app)
class DemoService(enterprise.SOAPService):
#enterprise.soap(_returns=enterprise._sp.String)
def get_time(self):
return ctime()
But when I run the program, it says:
Traceback (most recent call last):
File "D:/Workspace/src/flask_soap_server.py", line 3, in <module>
from flaskext.enterprise import Enterprise
File "C:\Python27\lib\site-packages\flaskext\enterprise.py", line 20, in <module>
from soaplib.core import Application
ImportError: No module named core
I also wrote a client to call this server's WSDL address:
from flaskext.enterprise import Enterprise
from flask import Flask
if __name__ == '__main__':
app = Flask(__name__)
enterprise = Enterprise(app)
client = enterprise.connect_to_soap_service('http://192.168.20.232:8000/_enterprise/soap?wsdl')
#app.route('/time')
def index():
time = client.service.get_time()
Soaplib (https://github.com/soaplib/soaplib):
This project is discontinued. Please head over to
github.com/arskom/spyne for the next version.
Flask Enterprise (https://pypi.python.org/pypi/Flask-Enterprise): latest release in 2011
Maybe it's time to move on to a better supported projects :)
There is a Spyne + Flask example: https://github.com/arskom/spyne/tree/master/examples/flask

KeyError with CherryPy WSGIServer serving static files

I'm trying to use CherryPy's WSGI server to serve static files, like in Using Flask with CherryPy to serve static files. Option 2 of the accepted answer there looks exactly like what I'd like to do, but I'm getting a KeyError when I try to use the static directory handler.
What I've tried:
>>>> import cherrypy
>>>> from cherrypy import wsgiserver
>>>> import os
>>>> static_handler = cherrypy.tools.staticdir.handler(section='/', dir=os.path.abspath('server_files')
>>>> d = wsgiserver.WSGIPathInfoDispatcher({'/': static_handler})
>>>> server = wsgiserver.CherryPyWSGIServer(('localhost', 12345), d)
>>>> server.start()
Then, when I try to access the server I'm getting a 500 response and the following error in the console:
KeyError('tools',)
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1353, in communicate
req.respond()
File "/Library/Python/2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 868, in respond
self.server.gateway(self).respond()
File "/Library/Python/2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 2267, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "/Library/Python/2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 2477, in __call__
return app(environ, start_response)
File "/Library/Python/2.7/site-packages/cherrypy/_cptools.py", line 175, in handle_func
handled = self.callable(*args, **self._merged_args(kwargs))
File "/Library/Python/2.7/site-packages/cherrypy/_cptools.py", line 102, in _merged_args
tm = cherrypy.serving.request.toolmaps[self.namespace]
KeyError: 'tools'
This is displayed twice for each time I try to hit anything that the server should be able to display. When I hooked up a Flask app to the server the Flask app worked as expected, but the static file serving still gave the same error.
What do I need to do to get the staticdir.handler to work?
I've tried various ways of getting this to work and up until today was also hitting the KeyError you have been seeing (among other issues).
I finally managed to get CherryPy to serve static alongside a Django app by adapting the code from this gist (included below).
import os
import cherrypy
from cherrypy import wsgiserver
from my_wsgi_app import wsgi
PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), 'public'))
class Root(object):
pass
def make_static_config(static_dir_name):
"""
All custom static configurations are set here, since most are common, it
makes sense to generate them just once.
"""
static_path = os.path.join('/', static_dir_name)
path = os.path.join(PATH, static_dir_name)
configuration = {static_path: {
'tools.staticdir.on': True,
'tools.staticdir.dir': path}
}
print configuration
return cherrypy.tree.mount(Root(), '/', config=configuration)
# Assuming your app has media on diferent paths, like 'c', 'i' and 'j'
application = wsgiserver.WSGIPathInfoDispatcher({
'/': wsgi.application,
'/c': make_static_config('c'),
'/j': make_static_config('j'),
'/i': make_static_config('i')})
server = wsgiserver.CherryPyWSGIServer(('0.0.0.0', 8070), application,
server_name='www.cherrypy.example')
try:
server.start()
except KeyboardInterrupt:
print "Terminating server..."
server.stop()
Hopefully wrapping a Flask app will be fairly similar.
The key for me was using the cherrypy.tree.mount on a dummy class, rather than trying to use the staticdir.handler directly.
For the curious - I used the code in the gist to customise a version of django-cherrypy's runcpserver management command, although in hindsight it would probably have been easier to create a new command from scratch.
Good luck (and thanks to Alfredo Deza)!

Python "name app = bottle.default_app() not defined" error

I'm using the Bottle framework for a simple application that I'm working on atm. I have my bottle library located in the folder "lib" and I call the bottle framework from the lib folder by "import lib.bottle". This is my folder structure:
lib
- bottle.py
- bottledaemon.py
- __init__.py
view
- log-in.tpl
mybottleapp.py
This is my code:
#!/usr/bin/env python
import lib.bottle
from lib.bottle import route, template, debug, static_file, TEMPLATE_PATH, error, auth_basic, get, post, request, response, run, view, redirect, SimpleTemplate, HTTPError
from lib.bottledaemon import daemon_run
import os
import ConfigParser
#######################
# Application Logic #
#######################
# This line of code is not recognised:
app = bottle.default_app()
##################
# Page Routing #
##################
##### LOG-IN PAGE #####
#route('/')
#view('log-in')
def show_page_index():
outout = 0
# Pathfix for Daemon mode
TEMPLATE_PATH.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "view")))
debug(mode=True)
# Pass to the daemon
if __name__ == "__main__":
daemon_run()
So it throws this error at me:
"name app = bottle.default_app() not defined"
If I remove this line "app = bottle.default_app()" the app works fine BUT I realy want to have it in there for programming purposes.
So what am I doing wrong? Is it maybe cuz I run the app in daemon mode or maybe I don't call it right from the lib folder?
Btw I also can't import ConfigParser. This maybe has a diffirent cause but I can't use it.
I think all you need to do is change this:
import lib.bottle
to this
import lib.bottle as bottle
Note: in my setup all I need to do is this:
import bottle
So it throws this error at me: name app = bottle.default_app() not defined
Lies
Your error is actually
Traceback (most recent call last):
File ..., line ..., in ...
app = bottle.default_app()
NameError: name 'bottle' is not defined
Because you did not define bottle. You defined lib.bottle. Either use your new name
app = lib.bottle.default_app()
or rename it:
import lib.bottle as bottle

Running python app on localhost

I'm trying to get a simple python script from the learnpythonthehardway tutorial to show up on my browser but keep encountering getting the following error:
$ python app.py
http://0.0.0.0:8080/
Traceback (most recent call last):
File "app.py", line 15, in <module>
app.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/web/application.py", line 310, in run
return wsgi.runwsgi(self.wsgifunc(*middleware))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/web/wsgi.py", line 54, in runwsgi
return httpserver.runsimple(func, validip(listget(sys.argv, 1, '')))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/web/httpserver.py", line 148, in runsimple
server.start()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/web/wsgiserver/__init__.py", line 1753, in start
raise socket.error(msg)
socket.error: No socket could be created
The script app.py is this:
import web
urls = (
'/', 'index'
)
app = web.application(urls, globals())
class index:
def GET(self):
greeting = "Hello World"
return greeting
if __name__ == "__main__":
app.run()
What should I try now? It actually worked to print Hello World on my browser once, but I've been meddling with it and now it's giving me error messages regardless of what I try. I don't know what these messages mean or how to fix them.
The problem is due to the way web.py loading works, you need to have the creation and running of the app only occur if this is the main entry point of the program.
Otherwise, the class loader in web.py will reload this file again later, and wind up spawning a new server when its intent was simply to load a class.
try this
if __name__ == '__main__' :
app = web.application(urls, globals())
app.run()
Source: No socket could be created (other causes)
Changing the port address could fix this. Probably 8080 might be used for something else. Idk im speculating. The following fixed the issue for me.
try
$ python app.py 8081

Categories