I'm having a module import issue.
using python 2.6 on ubuntu 10.10
I have a class that subclasses the daemon at http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/ . I created a python package with a module containing code that imports some models from a django project. The code works when used from a class, not subclassing the daemon. The structure looks something like:
my_module
__init__.py
- bin
- cfg
- core
__init__.py
collection.py
daemon.py
The ItemRepository code:
class ItemRepository(object):
"""This class provides an implementation to retrieve configuration data
from the monocle database
"""
def __init__(self, project_path):
if project_path is not None:
sys.path.append(project_path)
try:
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj.someapp.models import ItemConfiguration
except ImportError:
print "Could not import models from web app. Please ensure the\
PYTHONPATH is configured properly"
def get_scheduled_collectors(self):
"""This method finds and returns all ItemConfiguration objects
that are scheduled to run
"""
logging.info('At the error path: %s' % sys.path)
# query ItemConfigs from database
items = ItemConfiguration.objects.filter(disabled=False) #ERROR OCCURS AT THIS LINE
return [item for item in items if item.scheduled]
The daemon code (in /usr/local/bin/testdaemon.py):
import sys
from my_module.core.daemon import Daemon
from my_module.core.collection import ItemRepository
import logging
import time
class TestDaemon(Daemon):
default_conf = '/etc/echodaemon.conf'
section = 'echo'
def run(self):
while True:
logging.info('The echo daemon says hello')
ir = ItemRepository(project_path=self.project_path)
items = ir.get_scheduled_collectors() #TRIGGERS ERROR
logging.info('there are %d scheduled items' % len(items))
time.sleep(1)
if __name__ == '__main__':
TestDaemon().main()
The error I get is "NameError: global name 'my_module' is not defined" It get's past the import but then fails when trying to call a method on the object. I'm assuming it has to do with sys.path / PYTHONPATH, but I know my django project is on the path, as I've printed it out. Nothing so far in the python docs or Learning Python has helped yet. Does anyone have any insights or know of a good reference to module imports?
UPDATE:
Now I've attempted to simplify the problem to make it easier to understand. Now I have a directory structure that looks like:
/home
/username
/django
/someproj
/web
models.py
/my_module
daemon.py
I have set the $PYTHONPATH variable in /etc/bash.bashrc to '/home/username/django'.
Inside the testdaemon.py file the imports look like:
import logging
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj.web.models import ItemConfiguration
But now I get an ImportError: No module named 'someproj'. So then I appended the path.
import sys
sys.path.append('/home/username/django')
import logging
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj.web.models import ItemConfiguration
And now the ImportError says: No module named 'web'. Here's the traceback:
Traceback (most recent call last):
File "testdaemon.py", line 77, in <module>
TestDaemon('/tmp/testdaemon.pid').run()
File "testdaemon.py", line 47, in run
scheduled_items = [item for item in ItemConfiguration.objects.filter(disabled=False) if collector.scheduled]
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 141, in filter
return self.get_query_set().filter(*args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 550, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 568, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 1128, in add_q
can_reuse=used_aliases)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 1026, in add_filter
negate=negate, process_extras=process_extras)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 1179, in setup_joins
field, model, direct, m2m = opts.get_field_by_name(name)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/options.py", line 291, in get_field_by_name
cache = self.init_name_map()
File "/usr/local/lib/python2.6/dist-packages/django/db/models/options.py", line 321, in init_name_map
for f, model in self.get_all_related_m2m_objects_with_model():
File "/usr/local/lib/python2.6/dist-packages/django/db/models/options.py", line 396, in get_all_related_m2m_objects_with_model
cache = self._fill_related_many_to_many_cache()
File "/usr/local/lib/python2.6/dist-packages/django/db/models/options.py", line 410, in _fill_related_many_to_many_cache
for klass in get_models():
File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 167, in get_models
self._populate()
File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 61, in _populate
self.load_app(app_name, True)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 76, in load_app
app_module = import_module(app_name)
File "/usr/local/lib/python2.6/dist-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
ImportError: No module named web
So going from the earlier comments I tried adding:
import sys
sys.path.append('/home/username/django')
import logging
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj import web
from someproj.web import models
from someproj.web.models import ItemConfiguration
But that didn't help. So I created a very simple file:
#!/usr/bin/python
import logging
import time
import sys
sys.path.append('/home/username/django')
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj.web.models import ItemConfiguration
if __name__ == '__main__':
print sys.path
items = ItemConfiguration.objects.all()
for item in items:
print item
And this works! Which really only further confuses me. So now I'm thinking maybe it has to do with the daemon. It uses os.fork() and I'm not sure if the path is still set. This is why I set the $PYTHONPATH variable in the /etc/bash.bashrc file.
Any more insights? I really need the daemon, I don't have much of a choice as I need a long running process.
With from my_module.core.daemon import Daemon you do not actually bind the loaded module my_module to a variable. Use import my_module
just before or after your other imports to do that.
Explained in code:
>>> from xml import dom
>>> xml
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'xml' is not defined
>>> import xml
>>> xml
<module 'xml' from '/usr/lib/python2.6/xml/__init__.pyc'>
It ended up being that I needed to reference the fully qualified name of my app in my Django project's settings.py file in the INSTALLED_APPS setting. "someproj.web" instead of just "web". Using the shorter version works fine within a Django project, just not so well from the outside.
you get me confused what do you mean "your Django project" is your module "my_module" is part of an higher package? something like this
django_project
|
|my_module
__init__.py
- bin
- cfg
- core
__init__.py
collection.py
daemon.py
if so , and if like you said django_project is in PYTHONPATH , so you should just import my_module like this:
from django_project.my_module.core.daemon import Daemon
by the way it's preferable to import module no class neither function like this:
from django_project.my_module.core import daemon
and use it like this:
daemon.Daemon()
Related
https://github.com/CodeHostedHere/channel_response_bot
I want to create a db instance and share it between my two modules "messages" and "slash_commands". Extensions.py contains the code to create this and is imported in both modules. I would like to import the Flask(app) from app.py but it results in an error.
Traceback (most recent call last):
File "app.py", line 1, in <module>
from slash_commands import slash_cmds_bp
File "/opt/channel_response_bot/slash_commands/__init__.py", line 5, in <module>
from . import views
File "/opt/channel_response_bot/slash_commands/views.py", line 10, in <module>
from extensions import db, bolt_app, client, handler
File "/opt/channel_response_bot/extensions.py", line 12, in <module>
from app import flask_app
File "/opt/channel_response_bot/app.py", line 2, in <module>
from messages import messages_bp
File "/opt/channel_response_bot/messages/__init__.py", line 5, in <module>
from . import views
File "/opt/channel_response_bot/messages/views.py", line 10, in <module>
from extensions import db, bolt_app, client, handler
ImportError: cannot import name 'db' from 'extensions' (/opt/channel_response_bot/extensions.py)
I also can't do an import from __init__.py in the same level using from . import app as this results in ImportError: attempted relative import with no known parent package
I've been battling blueprints and database issues a lot so my my head is completely fried on this one, I'll need some serious ELI5 to get me across the line
Try using:
from ..extensions import db, bold_app, client, handler
Here is what I did:
I wrapped my 12 Django apps in an apps folder in my Django project.
I added sys.path.append(os.path.join(BASE_DIR, 'apps')) in settings.py (before INSTALLED_APPS declaration) sothat I don't need to specify the apps folder when importing my apps.
I changed every 12 apps.py files to set name = apps.myappname
INSTALLED_APPS are still defined with myappname (not apps.myappname)
Here is what I have so far:
Code works normally
Unit tests, when run app by app, 100% works for the 12 apps.
Unit tests, when run as a whole, at some point of execution, fail with errors like this one:
ImportError: Failed to import test module: apps.badges.tests
Traceback (most recent call last):
File "/usr/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/vagrant/fugo/fugoproj/apps/badges/tests.py", line 10, in <module>
from .factories import BadgeFactory
File "/vagrant/fugo/fugoproj/apps/badges/factories.py", line 6, in <module>
from .models import Badge
File "/vagrant/fugo/fugoproj/apps/badges/models.py", line 51, in <module>
models.Model):
File "/home/ubuntu/fenv/lib/python3.6/site-packages/django/db/models/base.py", line 118, in __new__
"INSTALLED_APPS." % (module, name)
RuntimeError: Model class apps.badges.models.Badge doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.
Here is the badges/factories.py file in error:
from datetime import timedelta
import factory
from django.utils import timezone
from .models import Badge
# some code
On every test which fails, it seems related to the relative import for the model (from .models import SomeModel). If I replace this with from myappname.models import SomeModel, this solve the error.
Any idea on something wrong in my code? I'd like to keep my relative imports.
How to explain it does not fail when running tests app by app?
Thanks.
I would like to note that the following error only occurs when ran through a celery worker.
with the following command in the terminal:
celery -A MarketPlaceTasks worker --loglevel=info
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 218, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 398, in __protected_call__
return self.run(*args, **kwargs)
File "/home/nick/mpcrawler2/MarketPlaceTasks.py", line 65, in get_item_data
logger, request, run_data, store_config, app_config = setup_task(payload)
File "/home/nick/mpcrawler2/MarketPlaceTasks.py", line 33, in setup_task
store_config = ConfigReader.read_store_config(request.Store)
File "/home/nick/mpcrawler2/shared/ConfigReader.py", line 22, in read_store_config
from singletons.StoreData import StoreData
File "/home/nick/mpcrawler2/singletons/StoreData.py", line 3, in <module>
from models.StoreConfig import StoreConfig
File "/home/nick/mpcrawler2/models/StoreConfig.py", line 3, in <module>
from enums.MpStores import MpStore
ImportError: No module named enums.MpStores
I have all my enums in a separate module. The module looks like this, and is located inside the same directory as the project:
Whenever I run the project via pycharm or terminal everything seems to work as intended.
The worker's starting point looks like this:
from celery import Celery
app = Celery('tasks', broker='*some ampq address here*')
The __init__.py file is empty. The enum files look like this:
from enum import Enum
# noinspection SpellCheckingInspection
class MpStore(Enum):
somevalue = 1
someothervalue = 2
etc = 3
As I'm using Python 2.7 I'm using enum34 that was installed using pip.
Please let me know if there's anything else I should provide in the question.
Well it seems like some sort of work around but following the advice in this answer:
How to accomplish relative import in python
I moved most of the project inside a "main" module containing all of them. and then i was able to:
instead of from enums.MpStore import MpStore
I now use from stuff.enums.MpStore import MpStore "stuff" being the new module name.
I would love to hear of a better way though...
I have managed to run a simple script located in a 'scripts' folder using 'runscript' from django-extensions. The technique is described here [1] (https://django-extensions.readthedocs.org/en/latest/runscript.html).
The script will run successfully as follows:
python manage.py runscript mysimplescript --script-args Testing 123
Should I now move the script to a sub-folder, then it appears I need to specify the full path to the sub-folder:
python manage.py runscript scripts.myfolder.mysimplescript --script-args Testing 123
I have written a python script which runs successfully when called directly from the command line within its own project folder. The complex script uses a number of additional scripts located within a plethora of sub-folders.
I now want to call this script within django using django-extensions' 'runscript', which means it's being invoked from the 'mysite/' folder where 'manage.py' is located. The script 'mycomplexscript.py' is not in the 'scripts/' folder but relocated to a separate 'myapps/' project folder, along with the sub-folders and scripts.
I've shown the structure below that includes 'mysite/' and some of the sub-folders/files within 'myapp/', notably the module 'myapp/common/types_':
mysite/
manage.py
mysite/
__init__.py
settings.py
urls.py
wsgi.py
scripts
__init__.py
mysimplescript.py
myapps/
__init__.py
mycomplexscript.py
common/
__init__.py
comparison.py
serialisation/
__init__.py
serialisation.py
xml.py
types_/
__init__.py
enum.py
Django-extensions requires that 'mycomplexscript.py' must incorporate a 'run()' function. Line #6 imports a class from one of the project sub-folders:
def run(*script_args):
import os
import sys
import time
from common.serialisation import XML
When calling the script, the error occurs because the module 'common.types_' cannot be located.
python manage.py runscript myapps.mycomplexscript --script-args Testing 123
/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/base.py:265: RemovedInDjango110Warning: OptionParser usage for
Django management commands is deprecated, use ArgumentParser instead
RemovedInDjango110Warning)
Exception while running run() in 'myapps.mycomplexscript'
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/email_notifications.py", line 63, in
run_from_argv
super(EmailNotificationCommand, self).run_from_argv(argv)
File "/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/email_notifications.py", line 75, in execute
super(EmailNotificationCommand, self).execute(*args, **options)
File "/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/utils.py", line 57, in inner
ret = func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/commands/runscript.py", line 172, in handle
run_script(mod, *script_args)
File "/usr/lib/python2.7/site-packages/django_extensions-1.6.1-py2.7.egg/
django_extensions/management/commands/runscript.py", line 79, in run_script
mod.run(*script_args)
File "/usr/tester/SwFact/isg_cid-system_test_current/my_site/myapps/
mycomplexscript.py", line 9, in run
from myapps.common.serialisation import XML
File "/usr/tester/SwFact/isg_cid- system_test_current/my_site/myapps/
common/__init__.py", line 7, in <module>
from comparison import Comparison
File "/usr/tester/SwFact/isg_cid-system_test_current/my_site/myapps/
common/comparison.py", line 7, in <module>
from common.types_ import Enum
ImportError: No module named common.types_
The problematic Line #7 in 'comparison.py' is as follows:
from common.types_ import Enum
The error can be fixed if the full path to the root of the project is stated explicitly (+++):
from apps.common.types_ import Enum
This means that I would have to edit all the files and change the paths accordingly, which I'd like to avoid, if possible.
This link [2]:(Python: Referencing another project) describes the referencing another project from the command line, so I realize that this may nothing to do with the fact that I'm trying to do this by using django-extensions' 'runscript'.
Is there a way that I can resolve the path discrepancy, without having to modify all the files? Would setting PYTHONPATH be the answer, as suggested here [3]:(How to use PYTHONPATH)? I checked but 'echo $PYTHONPATH' currently doesn't return anything.
Apologies for the long-winded description.
Thanks in advance.
I tried sys.path.append("my/path") as recommended by link above and [4]:(Python sys.path - appending PYTHONPATH) and it seemed to work, so all the paths included in 'from' are now recognized without having to set the full path, as I described previously here (+++).
myapps/mycomplexscript:
# Procedures
def run(*script_args):
import os
import sys
import time
sys.path.append('/usr/tester/SwFact/isg_cid-system_test_current/
my_site/myapps')
from common.serialisation import XML
from common.task import Subscriber
from test_ import TestHost
from common.filesystem import Directory, File
from common.tables import Column, Condition, CSVTable, Group
from string import Template
from datetime import date, time, datetime, timedelta
from operator import itemgetter
print script_args
So the result when running the script from 'mysite/':
python manage.py runscript myapps.mycomplexscript --script-args Testing 123
/usr/lib/python2.7/site-packages/Django-1.9.2-py2.7.egg/django/core/
management/base.py:265: RemovedInDjango110Warning: OptionParser usage for
Django management commands is deprecated, use ArgumentParser instead
RemovedInDjango110Warning)
('Testing', '123')
It's not ideal to have to hard-code the path this inside the script, so perhaps it can be passed in as one of the arguments. The only problem with this is that imports appear at the top of the file, before any of the command line arguments are processed within the complex script however at least I know it works in principle.
I have a xml parser that will store that data into a MySQL database, as part of a django app. The parser is intended to run daily, capturing some tests outputs:
`dbsync.py`:
1 #!/usr/bin/env python
2 import os
3 os.environ['DJANGO_SETTINGS_MODULE'] = 'autotester.settings'
4
5 import xml_parser
6 from models import *
The Django project is called autotester and the app is called autoreporter.
When I execute
python dbsync.py
I'm getting:
Traceback (most recent call last):
File "autoreporter/dbsync.py", line 6, in <module>
from models import *
File "/root/autotester/autoreporter/models.py", line 1, in <module>
from django.db import models
File "/usr/lib/python2.7/dist-packages/django/db/models/__init__.py", line 5, in <module>
from django.db.models.query import Q
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 17, in <module>
from django.db.models.deletion import Collector
File "/usr/lib/python2.7/dist-packages/django/db/models/deletion.py", line 4, in <module>
from django.db.models import signals, sql
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/__init__.py", line 4, in <module>
from django.db.models.sql.subqueries import *
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/subqueries.py", line 12, in <module>
from django.db.models.sql.query import Query
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 22, in <module>
from django.db.models.sql import aggregates as base_aggregates_module
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/aggregates.py", line 9, in <module>
ordinal_aggregate_field = IntegerField()
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 116, in __init__
self.db_tablespace = db_tablespace or settings.DEFAULT_INDEX_TABLESPACE
File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 54, in __getattr__
self._setup(name)
File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 49, in _setup
self._wrapped = Settings(settings_module)
File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 132, in __init__
% (self.SETTINGS_MODULE, e)
ImportError: Could not import settings 'autotester.settings' (Is it on sys.path? Is there an import error in the settings file?): No module named autotester.settings
How can I have DJANGO_SETTINGS_MODULE set, so I can execute the dbsync.py script properly?
Two issues
You need to import models from your app, so provided that your Django project is in the PYTHONPATH you can simply import the app models by:
from autoreporter.models import *
or do a relative import, if dbsync.py(looks from your traceback) is in the app directory.
from .models import *
Add Django project to PYTHONPATH. You need to have the Django project in the PYTHONPATH to it be accessed from your dbsync.py(provided it is in your django app directory), quick easy fix for this would be to do the following in your code to dbsync.py.
import sys
import os
##get project directory
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
##get project parent directory and add to python path using sys.path.append
SYS_PATH = os.path.dirname(BASE_DIR)
if SYS_PATH not in sys.path:
sys.path.append(SYS_PATH)
os.environ['DJANGO_SETTINGS_MODULE'] = 'autotester.settings'
I run a few scripts using the ORM without the rest of Django. I wouldn't set the environment in the script.
Since Django 1.7 you need to import your models, and run django setup.
I would set the environment as you do for the main Django application. Then at the start of your script add these lines. Those should set up the Django environment so it can access your models the same way a standard Django app will.
import django
django.setup()
import xml_parser
from .models import *