We Keep Coding

Setup of Flask architecture for machine learning pipeline

I want to setup a machine learning pipeline that is callable by flask but I am facing some issues, these links are for the documentations I have read so far:
https://exploreflask.com/en/latest/views.html#view-decorators
https://flask.palletsprojects.com/en/1.1.x/api/#flask.Flask
Let me explain the pipeline I have in mind:
pull a dataframe from a PostgreSQL database
encode said dataframe to make it ready for most algorithms
split up the data
feed to a pipeline and determine accuracy
store the model in a pickle file
What is working so far:
All parts are working as a regular script
I can just slap all the steps into one huge flask file with one decorator and it would run as well (my emergency solution)
The File Structure
The encoder script:
#Flask main thread
#makes flask start this part as application and not as module
app = Flask('encoder_module')
#app.route('/df_encoder')
def df_encoder(rng = 4):
encoding stuff
`return df`
The Pipeline script (random forest regressor here)
app = Flask('pipeline_module')
#app.route('/pipeline_rfr')
def pipeline_rfr():
pipeline stuff
`return grid_search_rfr`
The pickle module:
app = Flask('pickle_module')
#app.route('/store_reg_pickle')
def store_pickle():
"""
Usage of a Pickle Model -Storage of a trained Model
"""
model = grid_search_rfr
#specify file name in letter strings
model_file = "regression_model"
with open(model_file, mode='wb') as m_f:
pickle.dump(model, m_f)
print(f"Model saved in: {os.getcwd()}")
return model_file
The Main Flask File
#packages
from flask import Flask
from encoder_main_thread import df_encoder
from rfr_pipeline_function import pipeline_rfr
from pickle_call import store_pickle
app = Flask(__name__.split('.')[0])
#app.route('/regression_pipe')
#df_encoder
#pipeline_rfr
#store_reg_pickle
def regression_pipe():
`return 'pipeline done`
The problem iss that the return value of the encoder cannot be a dataframe, only a string, tuple, etc.
Is there a workaround for this?
I actually want it to be a flawless passing of the dataframe to the pipeline and eventually storing it in the pickle file which is then saved in the folder.
For some reason it cannot detect the pickle file import and throws following error:
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "C:\ANACONDA3\Scripts\flask-script.py", line 9, in <module>
sys.exit(main())
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 967, in main
cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 848, in run_command
app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 305, in __init__
self._load_unlocked()
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 330, in _load_unlocked
self._app = rv = self.loader()
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 388, in load_app
app = locate_app(self, import_name, name)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 240, in locate_app
__import__(module_name)
File "C:\Users\bill-\OneDrive\Dokumente\Docs Bill\TA_files\functions_scripts_storage\flask_test\flask_regression_pipeline.py", line 18, in <module>
#store_reg_pickle
NameError: name 'store_reg_pickle' is not defined
If you wish I could upload the entire scripts but that is a lot to look through and since it is working as a long regular pice of code, the mistake needs to be somewhere with my flask setup.

Error while building a RESTApi using flask_restful

Flask RESTApi newbie here
I am trying to build a RESTapi service in Flask (and I am trying to save the output as a .txt file) using flask_restful for a code of mine using the pydifact module as follows:
import datetime
from pydifact.message import Message
from pydifact.segments import Segment
from flask import Flask, request
from flask_restful import Resource, Api
app = Flask(__name__)
api = Api(app)
class RestAPI(Resource):
def get(self, ABSENDER_ID, EMPFÄNGER_ID, ERSTELLUNG_DATUM_ZEIT, REFERENCE):
MSCONS = Message()
def erstellung_datum_zeit(dt_time):
# Needed for the UNB segment
dt_time = dt_time.strftime('%Y%m%d%H%M')
return dt_time
def UNA_UNB_segment(absender_id, empfänger_id, erst_datum, ref):
MSCONS.add_segment(Segment('UNA', ":+.? '"))
MSCONS.add_segment(Segment('UNB', ['UNOC', '3'], [absender_id, '14'], [
empfänger_id, '500'], [erst_datum[2:8], erst_datum[8:]], ref, '', 'TL'))
ERSTELLUNG_DATUM_ZEIT = str(
erstellung_datum_zeit(datetime.datetime.now()))
UNA_UNB_segment(ABSENDER_ID, EMPFÄNGER_ID,
ERSTELLUNG_DATUM_ZEIT, REFERENCE)
result = MSCONS.serialize()
final_result = result
PATH_FOR_TXT = r'C:\Users\kashy\OneDrive\Desktop\Codes\mscons.txt'
textfile = open(PATH_FOR_TXT, 'w')
textfile.write(result)
textfile.close()
return {'result': final_result}
api.add_resource(
RestAPI,
'/RestAPI/<int:ABSENDER_ID>/<int:EMPFÄNGER_ID/<int:ERSTELLUNG_DATUM_ZEIT/<int:REFERENCE>')
if __name__ == '__main__':
app.run(debug=True)
The ABSENDER_ID, EMPFÄNGER_ID, ERSTELLUNG_DATUM_ZEIT, REFERENCE should all be user inputs and they should be all in string format.
When I do /RestAPI/<str:ABSENDER_ID>/<str:EMPFÄNGER_ID/<str:ERSTELLUNG_DATUM_ZEIT/<str:REFERENCE>, i get the following error:
C:\Users\kashy\OneDrive\Desktop\Codes\pydifact> & C:/Users/kashy/Anaconda3/envs/py36/python.exe c:/Users/kashy/OneDrive/Desktop/Codes/api.py
Traceback (most recent call last):
File "c:/Users/kashy/OneDrive/Desktop/Codes/api.py", line 44, in <module>
self.url_map.add(rule)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 1401, in add
rule.bind(self)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 730, in bind
self.compile()
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 790, in compile
_build_regex(self.rule if self.is_leaf else self.rule.rstrip("/"))
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 779, in _build_regex
convobj = self.get_converter(variable, converter, c_args, c_kwargs)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 738, in get_converter
raise LookupError("the converter %r does not exist" % converter_name)
LookupError: the converter 'str' does not exist
and when I do
/RestAPI/<int:ABSENDER_ID>/<int:EMPFÄNGER_ID/<int:ERSTELLUNG_DATUM_ZEIT/<int:REFERENCE>, I get the following error:
PS C:\Users\kashy\OneDrive\Desktop\Codes\pydifact> & C:/Users/kashy/Anaconda3/envs/py36/python.exe c:/Users/kashy/OneDrive/Desktop/Codes/api.py
Traceback (most recent call last):
File "c:/Users/kashy/OneDrive/Desktop/Codes/api.py", line 44, in <module>
'/RestAPI/<int:ABSENDER_ID>/<int:EMPFÄNGER_ID/<int:ERSTELLUNG_DATUM_ZEIT/<int:REFERENCE>')
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\flask_restful\__init__.py", line 382, in add_resource
self._register_view(self.app, resource, *urls, **kwargs)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\flask_restful\__init__.py", line 448, in _register_view
app.add_url_rule(rule, view_func=resource_func, **kwargs)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\flask\app.py", line 98, in wrapper_func
return f(self, *args, **kwargs)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\flask\app.py", line 1277, in add_url_rule
self.url_map.add(rule)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 1401, in add
rule.bind(self)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 730, in bind
self.compile()
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 790, in compile
_build_regex(self.rule if self.is_leaf else self.rule.rstrip("/"))
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 766, in _build_regex
for converter, arguments, variable in parse_rule(rule):
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\werkzeug\routing.py", line 226, in parse_rule
raise ValueError("malformed url rule: %r" % rule)
ValueError: malformed url rule: '/RestAPI/<int:ABSENDER_ID>/<int:EMPFÄNGER_ID/<int:ERSTELLUNG_DATUM_ZEIT/<int:REFERENCE>'
I am totally new to this and just started learning it using the Building a REST API using Python and Flask | Flask-RESTful tutorial.
Can anyone please tell me what is the mistake I am doing?

Your url routes have problem. In the first one, it should be string instead of str and in the second one you have a missing > at the end of int:EMPFÄNGER_ID and int:ERSTELLUNG_DATUM_ZEIT
Correct ones should be:
/RestAPI/<string:ABSENDER_ID>/<string:EMPFANGER_ID>/<string:ERSTELLUNG_DATUM_ZEIT>/<string:REFERENCE>
and
/RestAPI/<int:ABSENDER_ID>/<int:EMPFANGER_ID>/<int:ERSTELLUNG_DATUM_ZEIT>/<int:REFERENCE>
Note: I replaced Ä with A in urls above because that might also cause malformed url rule issue.

Google App Engine Python - YouTube API v3 - Insufficient Permission

I successfully duplicated the Tasks API from this video, but I am unable to successfully translate this format to using the YouTube API.
Here is my .py file:
import httplib2
import os
import sys
import jinja2
import webapp2
import logging
import pprint
from oauth2client import tools
from oauth2client.client import flow_from_clientsecrets
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.client import AccessTokenRefreshError
from oauth2client.tools import argparser, run_flow
from oauth2client.appengine import OAuth2Decorator
from apiclient.discovery import build
from google.appengine.ext.webapp import util
from google.appengine.ext.webapp import template
decorator = OAuth2Decorator(
client_id = '*my client ID*',
client_secret = '*my client secret*',
scope='https://www.googleapis.com/auth/youtube')
service = build("youtube", "v3")
class MainHandler(webapp2.RequestHandler):
#decorator.oauth_required
def get (self):
self.response.headers['Content-Type'] = 'text/plain'
channels_list = service.channels().list(
mine=True,
part="id"
).execute(http = decorator.http())
self.response.out.write (pprint.pformat(channels_list))
app = webapp2.WSGIApplication (
[
('/', MainHandler),
(decorator.callback_path, decorator.callback_handler()),
],
debug=True)
Here is my Traceback:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "C:\Users\...\testapp\oauth2client\appengine.py", line 733, in check_oauth
resp = method(request_handler, *args, **kwargs)
File "C:\Users\...\testapp\testapp.py", line 35, in get
).execute(http = decorator.http())
File "C:\Users\...\testapp\oauth2client\util.py", line 129, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Users\...\testapp\apiclient\http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 403 when requesting https://www.googleapis.com/youtube/v3/channels?part=id&mine=true&alt=json returned "Insufficient Permission">
INFO 2015-06-13 12:27:54,515 module.py:788] default: "GET / HTTP/1.1" 500 2278
I have checked and double-checked that I have both the YouTube Data API and YouTube Analytics API enabled this client ID. I have even disabled them and re-enabled to check, but I am still getting this error.
I am new to GAE and its methods, so maybe I am not understanding an error in the Traceback.
One note is that I was getting "ImportError: No module named..." for apiclient, httplib2, oauth2client, and uritemplate, so I moved those folders directly into my app file (and didn't get that error again). Not sure if moving those directly into the folder is causing errors.

Well I hope I didn't waste anyone's time, but if anyone else has this issue, I found this question, and although I didn't have a memcache issue, I revoked permission for my app from the user account I was using, refreshed the app and gave permission again and now it seems to be working.
I guess this was a Google-side issue. Thanks to anyone who spent the time to read my question.

GAESessions (Python) - "AttributeError: 'thread._local' object has no attribute 'current_session'."

So I've been trying to implement the Janrain Engage plugin to the application I'm working on (Google App Engine (Python 2.7)), using GAESessions as the sessions library.
Following the instructions given on the GAESessions page, I created the "gaesessions" folder (containing "__init__.py") as well as "appengine_config.py" in my application's root directory, as well as the relevant files to process the plugin.
Attempting to login via Janrain, however, threw me a 500 error and gave me this traceback in the GAE logs:
E 2013-03-25 07:06:55.535
'thread._local' object has no attribute 'current_session'
Traceback (most recent call last):
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1536, in __call__
rv = self.handle_exception(request, response, e)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~k-sketch-test/10.366190612177083948/rpx.py", line 56, in post
session = get_current_session()
File "/base/data/home/apps/s~k-sketch-test/10.366190612177083948/gaesessions/__init__.py", line 38, in get_current_session
return _tls.current_session
AttributeError: 'thread._local' object has no attribute 'current_session'
I have searched through the other posts regarding the "get_current_session()" issue, but they seem to be referencing 'local' instead of 'thread._local'.
Any ideas on what's going on here? Thanks in advance!
========
{root folder}/appengine_config.py
from gaesessions import SessionMiddleware
import os
COOKIE_KEY = '<hidden - was generated through os.urandom(64)>'
def webapp_add_wsgi_middleware(app):
from google.appengine.ext.appstats import recording
app = SessionMiddleware(app, cookie_key=COOKIE_KEY)
app = recording.appstats_wsgi_middleware(app)
return app
========

{root folder}/appengine_config.py Is the file appengine_config.py in the root location?
aaps/app.yaml
/main.py
/appengine_config.py
/gaesessions #folder
Can look a basic tutorial on gaesessions

Cannot run appengine-admin on dev_server

I've decided to try out this project:
http://code.google.com/p/appengine-admin/wiki/QuickStart
For the sake of the experiment, I took the demo guest-book shipped with App Engine. The import park look like this:
import cgi
import datetime
import wsgiref.handlers
from google.appengine.ext import db
from google.appengine.api import users
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google import appengine_admin
The db model and the admin look like this:
class Greeting(db.Model):
author = db.UserProperty()
content = db.StringProperty(multiline=True)
date = db.DateTimeProperty(auto_now_add=True)
class AdminGreeting(appengine_admin.ModelAdmin):
model = Greeting
listFields = ('author','content','date')
editFields = ('author','content','date')
appengine_admin.register(AdminGreeting)
Yet I get this exception, trying to run the site:
File "/home/<username>/python/google_appengine/google/appengine/tools/ dev_appserver.py", line 2875, in _HandleRequest
base_env_dict=env_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 387, in Dispatch
base_env_dict=base_env_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 2162, in Dispatch
self._module_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 2080, in ExecuteCGI
reset_modules = exec_script(handler_path, cgi_path, hook)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1976, in ExecuteOrImportScript
exec module_code in script_module.__dict__
File "/home/<username>/python/google_appengine/demos/guestbook/guestbook.py", line 37, in <module>
appengine_admin.register(AdminGreeting)
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 120, in register
modelAdminInstance = modelAdminClass()
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 64, in __init__
self._extractProperties(self.listFields, self._listProperties)
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 76, in _extractProperties
storage.append(PropertyWrapper(getattr(self.model, propertyName), propertyName))
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 17, in __init__
logging.info("Caching info about property '%s'" % name)
File "/usr/lib/python2.6/logging/__init__.py", line 1451, in info
root.info(*((msg,)+args), **kwargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1030, in info
self._log(INFO, msg, args, **kwargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1142, in _log
record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
File "/usr/lib/python2.6/logging/__init__.py", line 1117, in makeRecord
rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
File "/usr/lib/python2.6/logging/__init__.py", line 272, in __init__
from multiprocessing import current_process
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1736, in load_module
return self.FindAndLoadModule(submodule, fullname, search_path)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1638, in FindAndLoadModule
description)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1589, in LoadModuleRestricted
description)
File "/usr/lib/python2.6/multiprocessing/__init__.py", line 83, in <module>
import _multiprocessing
ImportError: No module named _multiprocessing
INFO 2009-04-25 23:34:27,628 dev_appserver.py:2934] "GET / HTTP/1.1" 500 -
Any idea what could have went wrong?

You appear to be using Python 2.6 (given that some of the messages are from files in /usr/lib/python2.6 ...!), but Google App Engine needs Python 2.5 (any 2.5.x will do for any versions of x), so you should install and use that to run the App Engine SDK.

Google App Engine only supports Python 2.5, and you're on a newer version.
By the look of your directories, you might be on a Linux (or is it a Mac?). On, say, Ubuntu you can "sudo apt-get install python2.5" (it won't affect your Python 2.6 at all), and then rather than:
<path-to-gae>/dev_appserver.py ...
do
python2.5 <path-to-gae>/dev_appserver.py ...
This is better than just blithely developing on 2.6 and deploying on 2.5 which is surely asking for hassles later on.

As others have said, this problem occurs in Python 2.6. I used the fix proposed in this comment in the App Engine issue tracker:
A quickfix is to create a file in your app's root named `_multiprocessing.py' with
the contents:
import multiprocessing
This way it's possible to import the _multiprocessing module.
It worked for me using Python 2.6.2
Cheers,
Kaji

To make this work on my local machine (with 2.6) and on GAE, I used:
import sys, logging
if sys.version[:3] == "2.6": logging.logMultiprocessing = 0

just do the following at the top of your something.py
import logging
logging.logMultiprocessing = 0

import logging
logging.logMultiprocessing = 0
Worked for me

It wired that I have been using GAE with python2.6(probably 2.6.1), and every thing worked fine.
But now I get the same _multiprocess import error.(python2.6.2).

import logging
logging.logMultiprocessing = 0
Worked for me too. Before uploading to GAE, comment those lines.