I want to setup a machine learning pipeline that is callable by flask but I am facing some issues, these links are for the documentations I have read so far:
https://exploreflask.com/en/latest/views.html#view-decorators
https://flask.palletsprojects.com/en/1.1.x/api/#flask.Flask
Let me explain the pipeline I have in mind:
pull a dataframe from a PostgreSQL database
encode said dataframe to make it ready for most algorithms
split up the data
feed to a pipeline and determine accuracy
store the model in a pickle file
What is working so far:
All parts are working as a regular script
I can just slap all the steps into one huge flask file with one decorator and it would run as well (my emergency solution)
The File Structure
The encoder script:
#Flask main thread
#makes flask start this part as application and not as module
app = Flask('encoder_module')
#app.route('/df_encoder')
def df_encoder(rng = 4):
encoding stuff
`return df`
The Pipeline script (random forest regressor here)
app = Flask('pipeline_module')
#app.route('/pipeline_rfr')
def pipeline_rfr():
pipeline stuff
`return grid_search_rfr`
The pickle module:
app = Flask('pickle_module')
#app.route('/store_reg_pickle')
def store_pickle():
"""
Usage of a Pickle Model -Storage of a trained Model
"""
model = grid_search_rfr
#specify file name in letter strings
model_file = "regression_model"
with open(model_file, mode='wb') as m_f:
pickle.dump(model, m_f)
print(f"Model saved in: {os.getcwd()}")
return model_file
The Main Flask File
#packages
from flask import Flask
from encoder_main_thread import df_encoder
from rfr_pipeline_function import pipeline_rfr
from pickle_call import store_pickle
app = Flask(__name__.split('.')[0])
#app.route('/regression_pipe')
#df_encoder
#pipeline_rfr
#store_reg_pickle
def regression_pipe():
`return 'pipeline done`
The problem iss that the return value of the encoder cannot be a dataframe, only a string, tuple, etc.
Is there a workaround for this?
I actually want it to be a flawless passing of the dataframe to the pipeline and eventually storing it in the pickle file which is then saved in the folder.
For some reason it cannot detect the pickle file import and throws following error:
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "C:\ANACONDA3\Scripts\flask-script.py", line 9, in <module>
sys.exit(main())
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 967, in main
cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 848, in run_command
app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 305, in __init__
self._load_unlocked()
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 330, in _load_unlocked
self._app = rv = self.loader()
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 388, in load_app
app = locate_app(self, import_name, name)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 240, in locate_app
__import__(module_name)
File "C:\Users\bill-\OneDrive\Dokumente\Docs Bill\TA_files\functions_scripts_storage\flask_test\flask_regression_pipeline.py", line 18, in <module>
#store_reg_pickle
NameError: name 'store_reg_pickle' is not defined
If you wish I could upload the entire scripts but that is a lot to look through and since it is working as a long regular pice of code, the mistake needs to be somewhere with my flask setup.
Related
I want to run great_expectation test suites against csv files in my ADLS Gen2. On my ADLS, I have a container called "data" in which I have a file at mypath/test/mydata.csv. I use a InferredAssetAzureDataConnector. I was able to create and test/validate the data source configuration but I believe there is a "silent" issue which was not caught.
The problem is that I cannot create a test suite based on this data source. When I run great_expectations suite new,
I select (3) to create the suite with the profiler, and then
select my newly created datasource, and then
instead of showing me the available files at the data source, it crashes with the following error (see below for full stacktrace):
TypeError: __init__() missing 1 required positional argument: 'data_asset_name'
When I execute this with a local data source (InferredAssetFilesystemDataConnector), it would show me the available files at the data source for selection in the CLI.
Does the error mean it cannot find the csv file on the ADLS and thus has no names to show? How do I fix this?
My Code to create the data source:
import great_expectations as ge
from great_expectations.cli.datasource import sanitize_yaml_and_save_datasource, check_if_datasource_name_exists
context = ge.get_context()
datasource_name = "my_datasource_name"
example_yaml = f"""
name: {datasource_name}
class_name: Datasource
execution_engine:
class_name: SparkDFExecutionEngine
azure_options:
account_url: https://<ACCOUNT-NAME>.blob.core.windows.net
credential: <ACCOUNT-KEY>
data_connectors:
default_inferred_data_connector_name:
class_name: InferredAssetAzureDataConnector
azure_options:
account_url: https://<ACCOUNT-NAME>.blob.core.windows.net
credential: <ACCOUNT-KEY>
container: data
name_starts_with: mypath/test
default_regex:
group_names:
- data_asset_name
pattern: (.csv)
default_runtime_data_connector_name:
class_name: RuntimeDataConnector
assets:
my_runtime_asset_name:
batch_identifiers:
- runtime_batch_identifier_name
"""
print(example_yaml)
# Test the yml:
context.test_yaml_config(yaml_config=example_yaml)
The output after creating the data source via the Jupyter Notebook:
Attempting to instantiate class from config...
Instantiating as a Datasource, since class_name is Datasource
Successfully instantiated Datasource
ExecutionEngine class name: SparkDFExecutionEngine
Data Connectors:
default_inferred_data_connector_name : InferredAssetAzureDataConnector
Available data_asset_names (0 of 0):
Unmatched data_references (0 of 0):[]
default_runtime_data_connector_name:RuntimeDataConnector
default_runtime_data_connector_name : RuntimeDataConnector
Available data_asset_names (1 of 1):
my_runtime_asset_name (0 of 0): []
Unmatched data_references (0 of 0):[]
<great_expectations.datasource.new_datasource.Datasource at 0x1cdc9e01f70>
Full error stack:
Traceback (most recent call last):
File "c:\coding\python38\lib\runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\coding\python38\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\coding\myrepo\venv\Scripts\great_expectations.exe\__main__.py", line 7, in <module>
File "C:\coding\myrepo\venv\lib\site-packages\great_expectations\cli\cli.py", line 190, in main
cli()
File "C:\coding\myrepo\venv\lib\site-packages\click\core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "C:\coding\myrepo\venv\lib\site-packages\click\core.py", line 1053, in main
rv = self.invoke(ctx)
File "C:\coding\myrepo\venv\lib\site-packages\click\core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\coding\myrepo\venv\lib\site-packages\click\core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\coding\myrepo\venv\lib\site-packages\click\core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\coding\myrepo\venv\lib\site-packages\click\core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "C:\coding\myrepo\venv\lib\site-packages\click\decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "C:\coding\myrepo\venv\lib\site-packages\great_expectations\cli\suite.py", line 151, in suite_new
_suite_new_workflow(
File "C:\coding\myrepo\venv\lib\site-packages\great_expectations\cli\suite.py", line 335, in _suite_new_workflow
raise e
File "C:\coding\myrepo\venv\lib\site-packages\great_expectations\cli\suite.py", line 279, in _suite_new_workflow
toolkit.add_citation_with_batch_request(
File "C:\coding\myrepo\venv\lib\site-packages\great_expectations\cli\toolkit.py", line 1020, in add_citation_with_batch_request
and BatchRequest(**batch_request)
TypeError: __init__() missing 1 required positional argument: 'data_asset_name'
I had a mistake in my regex ... with the following pattern it works flawlessly:
default_regex:
group_names:
- data_asset_name
pattern: (.*\.csv)
I'm trying to write a celery application that passes numpy arrays (or any arbitrary objects) to the workers. As far as I can tell, this requires serialization to occur via pickle (NB: I'm aware of the security implications but this isn't a concern in this case).
However, even after trying every possible way I could find to allow pickle as a serializer, I keep getting the following kombu exception:
kombu.exceptions.ContentDisallowed: Refusing to deserialize untrusted
content of type pickle (application/x-python-serialize)
My current files are currently:
# tasks.py
from celery import Celery
app = Celery(
'tasks',
broker='redis://localhost',
accept_content=['pickle'],
task_serializer='pickle'
)
#app.task
def adding(x, y):
return x + y
if __name__ == '__main__':
import numpy as np
adding.apply_async((np.array([1]), np.array([1])), serializer='pickle')
In addition I have a config file:
# celeryconfig.py
print('configuring...')
accept_content = ['pickle', 'application/x-python-serialize']
task_serializer = 'pickle'
result_serializer = 'pickle'
from kombu import serialization
serialization.register_pickle()
serialization.enable_insecure_serializers()
However, if I run the worker (celery -A tasks worker --loglevel=info) and then execute the code that makes an async call (python tasks.py), I get the following traceback. Am I missing something?
[2018-06-16 11:46:23,617: CRITICAL/MainProcess] Unrecoverable error: ContentDisallowed('Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)',)
Traceback (most recent call last):
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
self.blueprint.start(self)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 322, in start
blueprint.start(self)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/consufrom celery import Celery
mer/consumer.py", line 598, in start
c.loop(*c.loop_args())
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop
next(loop)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 354, in create_loop
cb(*cbargs)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/redis.py", line 1040, in on_readable
self.cycle.on_readable(fileno)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/redis.py", line 337, in on_readable
chan.handlers[type]()
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/redis.py", line 724, in _brpop_read
self.connection._deliver(loads(bytes_to_str(item)), dest)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver
callback(message)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 633, in _callback
return callback(message)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/messaging.py", line 624, in _receive_callback
return on_m(message) if on_m else self.receive(decoded, message)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 572, in on_task_received
callbacks,
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/strategy.py", line 136, in task_message_handler
if body is None and 'args' not in message.payload:
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/message.py", line 207, in payload
return self._decoded_cache if self._decoded_cache else self.decode()
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/message.py", line 192, in decode
self._decoded_cache = self._decode()
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/message.py", line 197, in _decode
self.content_encoding, accept=self.accept)
File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/serialization.py", line 253, in loads
raise self._for_untrusted_content(content_type, 'untrusted')
kombu.exceptions.ContentDisallowed: Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)
For anyone coming to this question:
The answer was to use the app.config_from_object method:
import celeryconfig
app.config_from_object(celeryconfig)
This is the error log:
[I 160308 11:09:59 web:1908] 200 GET /admin/realtime (117.93.180.216) 107.13ms
[E 160308 11:09:59 http1connection:54] Uncaught exception
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tornado/http1connection.py", line 238, in _read_message
delegate.finish()
File "/usr/local/lib/python3.4/dist-packages/tornado/httpserver.py", line 290, in finish
self.delegate.finish()
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 1984, in finish
self.execute()
File "/usr/local/lib/python3.4/dist-packages/blueware-1.0.10/blueware/hooks/framework_tornado/web.py", line 480, in _bw_wrapper__RequestDispatcher_execute
future = wrapped(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 2004, in execute
**self.handler_kwargs)
File "/usr/local/lib/python3.4/dist-packages/blueware-1.0.10/blueware/hooks/framework_tornado/web.py", line 448, in _bw_wrapper_RequestHandler___init___
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 185, in init
self.initialize(**kwargs)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 2714, in wrapper
self.redirect(url)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 671, in redirect
self.finish()
File "/usr/local/lib/python3.4/dist-packages/blueware-1.0.10/blueware/hooks/framework_tornado/web.py", line 309, in _bw_wrapper_RequestHandler_finish_
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 934, in finish
self.flush(include_footers=True)
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 870, in flush
for transform in self._transforms:
TypeError: 'NoneType' object is not iterable
[I 160308 11:10:00 web:1908] 200 GET /admin/order?order_type=1&order_status=1&page=0&action=allreal (49.89.27.173) 134.53ms
Can anyone tell me how to solve this problem? Thank you very much
I assume that OneAPM (blueware agent) is compatible with your python and Tornado version, however it's can be tricky.
Solution
Move self.redirect(url) from your handler initialize method to get method, like this
class MyHandler(tornado.web.RequestHandler):
def get(self):
self.redirect('/some_url')
or use RedirectHandler.
Every action that could finish request needs to be called in context of http-verb method (get, post, put and so on). The common mistake is making authetication/authorization in __init__ or initialize.
More detail
In Tornado's source there is a note about _transforms that is initialized in the constructor with None and set in_execute (oversimplifying - after headers_received).
A transform modifies the result of an HTTP request (e.g., GZip encoding).
Applications are not expected to create their own OutputTransforms
or interact with them directly; the framework chooses which transforms
(if any) to apply.
Reproduce
Sample that triggers this error. I'm including this only as a cross-check that blueware is not the cause:
import tornado.ioloop
import tornado.web
class SomeHandler(tornado.web.RequestHandler):
def initialize(self, *args, **kwargs):
url = '/some'
self.redirect(url)
# ^ this is wrong
def get(self):
# redirect should be here
self.write("Hello")
def make_app():
return tornado.web.Application([
(r"/", SomeHandler),
])
if __name__ == "__main__":
app = make_app()
app.listen(8888)
tornado.ioloop.IOLoop.current().start()
And stacktrace:
ERROR:tornado.application:Uncaught exception
Traceback (most recent call last):
File "/tmp/py3/lib/python3.4/site-packages/tornado/http1connection.py", line 238, in _read_message
delegate.finish()
File "/tmp/py3/lib/python3.4/site-packages/tornado/httpserver.py", line 289, in finish
self.delegate.finish()
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 2022, in finish
self.execute()
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 2042, in execute
**self.handler_kwargs)
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 183, in __init__
self.initialize(**kwargs)
File "test.py", line 8, in initialize
self.redirect(url)
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 666, in redirect
self.finish()
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 932, in finish
self.flush(include_footers=True)
File "/tmp/py3/lib/python3.4/site-packages/tornado/web.py", line 868, in flush
for transform in self._transforms:
TypeError: 'NoneType' object is not iterable
If I try to run a test via PyCharm, I get this exception
...bin/python /usr/local/pycharm-4.5.1/helpers/pycharm/utrunner.py ...src/foo/foo/tests/FooEditTest.py::FooEditTest::test_issue_add true
Testing started at 15:59 ...
Traceback (most recent call last):
File "/usr/local/pycharm-4.5.1/helpers/pycharm/utrunner.py", line 139, in <module>
module = loadSource(a[0])
File "/usr/local/pycharm-4.5.1/helpers/pycharm/utrunner.py", line 41, in loadSource
module = imp.load_source(moduleName, fileName)
File "...src/foo/foo/tests/FooEditTest.py", line 45, in <module>
from foo.views.issue.forward import forward
File "...src/foo/foo/views/issue/forward.py", line 29, in <module>
class ForwardForm(forms.Form):
File "...src/foo/foo/views/issue/forward.py", line 36, in ForwardForm
group=forms.ModelChoiceField(Group.objects.exclude(groupconfig__no_issue=True).extra(
File "...python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "...python2.7/site-packages/django/db/models/query.py", line 698, in exclude
return self._filter_or_exclude(True, *args, **kwargs)
File "...python2.7/site-packages/django/db/models/query.py", line 707, in _filter_or_exclude
clone.query.add_q(~Q(*args, **kwargs))
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1331, in add_q
clause, require_inner = self._add_q(where_part, self.used_aliases)
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1358, in _add_q
current_negated=current_negated, connector=connector)
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1182, in build_filter
lookups, parts, reffed_aggregate = self.solve_lookup_type(arg)
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1120, in solve_lookup_type
_, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1383, in names_to_path
field, model, direct, m2m = opts.get_field_by_name(name)
File "...python2.7/site-packages/django/db/models/options.py", line 416, in get_field_by_name
cache = self.init_name_map()
File "...python2.7/site-packages/django/db/models/options.py", line 445, in init_name_map
for f, model in self.get_all_related_m2m_objects_with_model():
File "...python2.7/site-packages/django/db/models/options.py", line 563, in get_all_related_m2m_objects_with_model
cache = self._fill_related_many_to_many_cache()
File "...python2.7/site-packages/django/db/models/options.py", line 577, in _fill_related_many_to_many_cache
for klass in self.apps.get_models():
File "...python2.7/site-packages/django/utils/lru_cache.py", line 101, in wrapper
result = user_function(*args, **kwds)
File "...python2.7/site-packages/django/apps/registry.py", line 168, in get_models
self.check_models_ready()
File "...python2.7/site-packages/django/apps/registry.py", line 131, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
Process finished with exit code 1
I use group=forms.ModelChoiceField(Group.objects.exclude(...))
This line gets executed during importing. The call to django.setup() was not done before.
I have no clue how to solve this:
Should I call django.setup()? But where to insert this line?
Avoid using MyModel.objects.filter(...) during import time? This would need a big refactoring, since we have several ModelChoiceFields.
You're currently running the tests using the PyCharm's Python unit test runner pycharm-4.5.1/helpers/pycharm/utrunner.py.
This test runner is great for low-level unit tests (subclassed from unittest.TestCase) that don't touch Django features like the ORM, but
if the tests rely on Django-specific things like the database, then your TestCases probably need to be subclassed from django.test.testcases.TestCase and you'll need to go through PyCharm's Django test manager django_test_manage.py, which takes care of setting up the test database, initialising the model registry, and so on.
Run > Edit Configurations > Add New Configuration (+ button) > Django Tests
Set the target to foo.foo.tests.FooEditTest and make sure to put DJANGO_SETTINGS_MODULE=... in the environment variables if it doesn't find your settings.
I faced the same problem while running test cases and solved it by adding this to test file at the beginning with import statements
import os
import sys
from django.core.handlers.wsgi import WSGIHandler
os.environ['DJANGO_SETTINGS_MODULE'] = 'myapp.settings'
application = WSGIHandler()
Please show your exact code (especially the extra() clause). Calling filter() or exclude() at import time is not bad because querysets are lazy but you could evaluate the queryset here that caused the exception.
Do not evaluate querysets during import time because import statements are executed only once: e.g., if a new group is created, it won't be available as a choice for you ModelChoiceField.
A bit hard to say without seeing your code, but I had a similar issue once. For me it was related to a module that was being imported by my models and that also contained an import of my models.
ModelA --- imports --> service -- imports --> ModelA
I solved this by moving the import in my service to the method that needed the import. So instead of putting it at the top of the service module, I limited the scope of the import to the method that needed this import, thus avoiding importing the models whilst initializing the service module. Fwew, I wish I could draw it for you :)
I've decided to try out this project:
http://code.google.com/p/appengine-admin/wiki/QuickStart
For the sake of the experiment, I took the demo guest-book shipped with App Engine. The import park look like this:
import cgi
import datetime
import wsgiref.handlers
from google.appengine.ext import db
from google.appengine.api import users
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google import appengine_admin
The db model and the admin look like this:
class Greeting(db.Model):
author = db.UserProperty()
content = db.StringProperty(multiline=True)
date = db.DateTimeProperty(auto_now_add=True)
class AdminGreeting(appengine_admin.ModelAdmin):
model = Greeting
listFields = ('author','content','date')
editFields = ('author','content','date')
appengine_admin.register(AdminGreeting)
Yet I get this exception, trying to run the site:
File "/home/<username>/python/google_appengine/google/appengine/tools/ dev_appserver.py", line 2875, in _HandleRequest
base_env_dict=env_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 387, in Dispatch
base_env_dict=base_env_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 2162, in Dispatch
self._module_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 2080, in ExecuteCGI
reset_modules = exec_script(handler_path, cgi_path, hook)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1976, in ExecuteOrImportScript
exec module_code in script_module.__dict__
File "/home/<username>/python/google_appengine/demos/guestbook/guestbook.py", line 37, in <module>
appengine_admin.register(AdminGreeting)
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 120, in register
modelAdminInstance = modelAdminClass()
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 64, in __init__
self._extractProperties(self.listFields, self._listProperties)
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 76, in _extractProperties
storage.append(PropertyWrapper(getattr(self.model, propertyName), propertyName))
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 17, in __init__
logging.info("Caching info about property '%s'" % name)
File "/usr/lib/python2.6/logging/__init__.py", line 1451, in info
root.info(*((msg,)+args), **kwargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1030, in info
self._log(INFO, msg, args, **kwargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1142, in _log
record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
File "/usr/lib/python2.6/logging/__init__.py", line 1117, in makeRecord
rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
File "/usr/lib/python2.6/logging/__init__.py", line 272, in __init__
from multiprocessing import current_process
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1736, in load_module
return self.FindAndLoadModule(submodule, fullname, search_path)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1638, in FindAndLoadModule
description)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1589, in LoadModuleRestricted
description)
File "/usr/lib/python2.6/multiprocessing/__init__.py", line 83, in <module>
import _multiprocessing
ImportError: No module named _multiprocessing
INFO 2009-04-25 23:34:27,628 dev_appserver.py:2934] "GET / HTTP/1.1" 500 -
Any idea what could have went wrong?
You appear to be using Python 2.6 (given that some of the messages are from files in /usr/lib/python2.6 ...!), but Google App Engine needs Python 2.5 (any 2.5.x will do for any versions of x), so you should install and use that to run the App Engine SDK.
Google App Engine only supports Python 2.5, and you're on a newer version.
By the look of your directories, you might be on a Linux (or is it a Mac?). On, say, Ubuntu you can "sudo apt-get install python2.5" (it won't affect your Python 2.6 at all), and then rather than:
<path-to-gae>/dev_appserver.py ...
do
python2.5 <path-to-gae>/dev_appserver.py ...
This is better than just blithely developing on 2.6 and deploying on 2.5 which is surely asking for hassles later on.
As others have said, this problem occurs in Python 2.6. I used the fix proposed in this comment in the App Engine issue tracker:
A quickfix is to create a file in your app's root named `_multiprocessing.py' with
the contents:
import multiprocessing
This way it's possible to import the _multiprocessing module.
It worked for me using Python 2.6.2
Cheers,
Kaji
To make this work on my local machine (with 2.6) and on GAE, I used:
import sys, logging
if sys.version[:3] == "2.6": logging.logMultiprocessing = 0
just do the following at the top of your something.py
import logging
logging.logMultiprocessing = 0
import logging
logging.logMultiprocessing = 0
Worked for me
It wired that I have been using GAE with python2.6(probably 2.6.1), and every thing worked fine.
But now I get the same _multiprocess import error.(python2.6.2).
import logging
logging.logMultiprocessing = 0
Worked for me too. Before uploading to GAE, comment those lines.