I have the following script and it's meant to be a standalone Django script so I can run python my_script.py from the command line. It used to work with Django 1.8, after I upgrade to Django 1.11, I'm getting the following error:
Traceback (most recent call last):
File "app.py", line 8, in <module>
django.setup()
File "C:\python27\lib\site-packages\django-1.11.5-py2.7.egg\django\__init__.py", line 22, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "C:\python27\lib\site-packages\django-1.11.5-py2.7.egg\django\conf\__init__.py", line 56, in __getattr__
self._setup(name)
File "C:\python27\lib\site-packages\django-1.11.5-py2.7.egg\django\conf\__init__.py", line 41, in _setup
self._wrapped = Settings(settings_module)
File "C:\python27\lib\site-packages\django-1.11.5-py2.7.egg\django\conf\__init__.py", line 110, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "C:\python27\lib\importlib\__init__.py", line 37, in import_module
__import__(name)
ImportError: Import by filename is not supported.
This is my python script
# standalone django setup
import os, sys, logging, django
prj_dir = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
logging.basicConfig(level=logging.INFO)
logging.info("PRJ_DIR: %s" % prj_dir)
sys.path.append(prj_dir)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "%s.settings" % prj_dir.split("/")[-1])
django.setup()
...
...
It seems you're trying to split a file path by the forward slash / while running your script on Windows where path separator is the backslash \. Use os.path.basename instead of manually dealing with the stings (.split("/")[-1]):
>>> import os
>>> os.path.basename(r'/home/user/project')
'project'
>>> os.path.basename(r'c:\users\user\project')
'project'
In comparison:
>>> r'/home/user/project'.split("/")[-1]
'project' # works
>>> r'c:\users\user\project'.split("/")[-1]
'c:\\users\\user\\project' # error
Don't reinvent the wheel. There is a cool manage command on django extensions that lets you run scripts on the django context.
https://django-extensions.readthedocs.io/en/latest/runscript.html
Others have pointed out an issue with your path manipulation, however the cause of your actual import exception is that you are setting DJANGO_SETTINGS_MODULE to the name of a file.
It needs to be the name of a Python module. In other words, your settings file should end in .py and you should pass the name of it (without .py), or the importable path to it - exactly as you would type it in an import statement.
If your settings are in my_settings.py, then you would set the variable to my_settings.
Here is a simplified example of a one-page django application taken from the excellent Lightweight Django book's excerpt:
import sys
from django.conf import settings
settings.configure(
DEBUG=True,
SECRET_KEY='thisisthesecretkey',
ROOT_URLCONF=__name__,
MIDDLEWARE_CLASSES=(
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
),
)
from django.conf.urls import url
from django.http import HttpResponse
def index(request):
return HttpResponse('Hello World')
urlpatterns = (
url(r'^$', index),
)
if __name__ == "__main__":
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
The error is in
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "%s.settings" % prj_dir.split("/")[-1])
If you run this script from the same folder as manage.py, it will return the name of the manage.py folder instead settings.py folder resulting managepy_folder.settings
when it should output settingspy_folder.settings to access the app
Test with :
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settingspy_folder.settings")
if it that was the problem, if you want use the same algorithm or something similar in multiple projects, you need a pattern to modify the current folder name to indicate the settings.py folders name, using the same folders name in both wouldn't require modification.
if that doesn't fix please post your settings.py file
If your script is in the same folder as settings.py, the code should be like this:
import os, sys, logging, django
prj_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
logging.basicConfig(level=logging.INFO)
logging.info("PRJ_DIR: %s" % prj_dir)
sys.path.append(prj_dir)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "{}.settings".format(os.path.basename(prj_dir)))
django.setup()
else, if your script is in the parent folder of settings.py:
import os, sys, logging, django
prj_dir = os.path.dirname(os.path.abspath(__file__))
logging.basicConfig(level=logging.INFO)
logging.info("PRJ_DIR: %s" % prj_dir)
sys.path.append(prj_dir)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "{}.settings".format(os.path.basename(prj_dir)))
django.setup()
in the line:
os.environ.setdefault(
"DJANGO_SETTINGS_MODULE",
"myproject.settings") # should be like in import command
)
Just use python module path: like "myproject.settings" instead of path like /opt/myproject/settings" or "c:\myproject\settings.py"
Related
I have a Python API running on Google's Cloud Endpoints framework.
I'm trying to implement a runner for my unittests, using the dev_appserver.
I've copied and adapted what Google is offering about this, just edited a bit to make it run the devappserver all-in-one.
This is what I've done so far:
import argparse
import os
import sys
import unittest
def fixup_paths(path):
"""Adds GAE SDK path to system path and appends it to the google path
if that already exists."""
try:
import google
google.__path__.append("{0}/google".format(path))
except ImportError:
pass
sys.path.insert(0, path)
def main(sdk_path, test_path, test_pattern):
# If the SDK path points to a Google Cloud SDK installation
# then we should alter it to point to the GAE platform location.
if os.path.exists(os.path.join(sdk_path, 'platform/google_appengine')):
sdk_path = os.path.join(sdk_path, 'platform/google_appengine')
# Make sure google.appengine.* modules are importable.
fixup_paths(sdk_path)
# Make sure all bundled third-party packages are available.
import dev_appserver
dev_appserver.fix_sys_path()
# Fix google shits
from google.appengine.tools.devappserver2 import devappserver2
# Start the devappserver
server_instance = devappserver2.DevelopmentServer()
server_instance.start()
# Discover and run tests.
suite = unittest.loader.TestLoader().discover(test_path, test_pattern)
suite_result = unittest.TextTestRunner(verbosity=2).run(suite)
server_instance.stop()
return suite_result
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument(
'sdk_path',
help='The path to the Google App Engine SDK or the Google Cloud SDK.')
parser.add_argument(
'--test-path',
help='The path to look for tests, defaults to the current directory.',
default=os.getcwd())
parser.add_argument(
'--test-pattern',
help='The file pattern for test modules, defaults to *_test.py.',
default='test_*.py')
args = parser.parse_args()
result = main(args.sdk_path, args.test_path, args.test_pattern)
if not result.wasSuccessful():
sys.exit(1)
But i'm struggling with a strange import error.
I've been searching through Github but I was unable to find anything:
Traceback (most recent call last):
File "runner.py", line 65, in <module>
result = main(args.sdk_path, args.test_path, args.test_pattern)
File "runner.py", line 34, in main
from google.appengine.tools.devappserver2 import devappserver2
File "/Users/toto/Libraries/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 27, in <module>
from google.appengine.tools.devappserver2 import api_server
File "/Users/toto/Libraries/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 79, in <module>
from google.appengine.tools.devappserver2 import wsgi_server
File "/Users/toto/Libraries/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 48, in <module>
from cherrypy import wsgiserver
File "/usr/local/lib/python2.7/site-packages/cherrypy/__init__.py", line 73, in <module>
from ._cptools import default_toolbox as tools, Tool
File "/usr/local/lib/python2.7/site-packages/cherrypy/_cptools.py", line 33, in <module>
from cherrypy.lib import auth_basic, auth_digest
File "/usr/local/lib/python2.7/site-packages/cherrypy/lib/auth_digest.py", line 27, in <module>
from six.moves.urllib.request import parse_http_list, parse_keqv_list
ImportError: cannot import name parse_http_list
I can't figure out how to fix that import thing :(
The error means that the package six you have installed does not include parse_http_list. Try upgrading the six package:
pip install six==1.14.0
It could be that the parent package then throws an error, then upgrade that package as well.
I have a twisted tac file (twisted_service.py) with a code:
from twisted.application import service
# application.py file in the same dir
from .application import setup_reactor
class WebsocketService(service.Service):
def startService(self):
service.Service.startService(self)
setup_reactor()
application = service.Application("ws")
ws_service = WebsocketService()
ws_service.setServiceParent(application)
Here is application.py file, which sets up the reactor:
# -*- coding: utf-8 -*-
from twisted.web.server import Site
from twisted.web.static import Data
from twisted.internet import reactor, defer
from autobahn.twisted.resource import WebSocketResource
from autobahn.twisted.websocket import WebSocketServerFactory
from txsni.snimap import SNIMap
from txsni.maputils import Cache
from txsni.snimap import HostDirectoryMap
from twisted.python.filepath import FilePath
from tools.database.async import pg_conn
from tools.database import makedsn
from tools.config import main_db
from tools.modules.external import flask_setup
import tools.config as config
import websockethandlers as wsh
from pytrapd import TrapsListener
PROTOCOLMAP = {
'portcounters': wsh.PortCounters,
'eqcounters': wsh.EquipmentCounters,
'settings': wsh.Settings,
'refresh': wsh.Refresher,
'montraps': wsh.TrapsMonitoring,
'fdbs': wsh.FdbParser,
'portstate': wsh.PortState,
'cable': wsh.CableDiagnostic,
'eqcable': wsh.EquipmentCableDiagnostic,
'igmp': wsh.Igmp,
'ipmac': wsh.IpMac,
'lldp': wsh.LLDPParser,
'alias': wsh.AliasSetup,
'ping': wsh.Ping
}
#defer.inlineCallbacks
def setup_reactor():
flask_setup()
yield pg_conn.connect(makedsn(main_db))
root = Data("", "text/plain")
for key in PROTOCOLMAP:
factory = WebSocketServerFactory("wss://localhost:%s" % config.ws_port)
factory.protocol = PROTOCOLMAP[key]
resource = WebSocketResource(factory)
root.putChild(key, resource)
site = Site(root)
context_factory = SNIMap(
Cache(HostDirectoryMap(FilePath(config.certificates_directory)))
)
reactor.listenSSL(config.ws_port, site, context_factory)
traps_listener = TrapsListener()
traps_listener.listen_traps(config.trap_ip)
traps_listener.listen_messages(config.fifo_file)
if __name__ == '__main__':
setup_reactor()
import sys
from twisted.python import log
log.startLogging(sys.stdout)
reactor.run()
I use twistd -noy twisted_service.py command to run the twisted service. It has been working for twisted 16.3.2 version. After upgrade to any next version I got the error:
Unhandled Error
Traceback (most recent call last):
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/app.py", line 662, in run
runApp(config)
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/scripts/twistd.py", line 25, in runApp
_SomeApplicationRunner(config).run()
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/app.py", line 380, in run
self.application = self.createOrGetApplication()
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/app.py", line 445, in createOrGetApplication
application = getApplication(self.config, passphrase)
--- <exception caught here> ---
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/app.py", line 456, in getApplication
application = service.loadApplication(filename, style, passphrase)
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/application/service.py", line 412, in loadApplication
application = sob.loadValueFromFile(filename, 'application')
File "/home/kalombo/.virtualenvs/dev/local/lib/python2.7/site-packages/twisted/persisted/sob.py", line 177, in loadValueFromFile
eval(codeObj, d, d)
File "twisted_service.py", line 3, in <module>
from .application import setup_reactor
exceptions.ImportError: No module named application
How should I run the twisted or import module properly?
What the answer I found nearly yours. like the follows:
import os
import sys
sys.path = [os.path.join(os.getcwd(), '.'), ] + sys.path
just add the current working directory to the sys.path.
But I have not found more better method.... I think this not very good.
I found the answer here http://twistedmatrix.com/pipermail/twisted-python/2016-September/030783.html
It is a new feature in Twisted 16.4.0. In previous versions twistd script automatically added working dir to system paths, from 16.4.0 version i have to added it manually. It can be added something like this in twisted_service.py file:
import os
import sys
TWISTED_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.append(TWISTED_DIR)
I'm writing a program that enables my Django application to consume a REST API. My problem is that Django rejects the program if I run it as a .py file, but accepts it via shell. The program is divided into two files: one that interacts with the API, and another that interacts with Django. The main folder structure is as follows:
/root folder
[rest of Django project]
-__init__.py
-test.py
-mid_side.py
-teamup_api/
-__init__.py
-teamup.py
where test.py is the file that starts the program, mid_side.py is the Django-side part, and teamup.py is the API-side part. Below is the specific code I'm having trouble with.
test.py
import django
from mid_side import dates
from datetime import datetime
start = datetime(2017,7,28)
end = datetime(2017,8,19)
test = dates(start,end)
print(test)
Where dates is a method from mid_side that accepts two datetime objects as parameters.
According to Django, the problem emerges in mid_side.py, as it imports models from an app in the main Django project. The specific line Django has issues with is from events.models import Service, where events is an app Django uses and Services is a model. The traceback is
Traceback (most recent call last):
File "test.py", line 2, in <module>
from mid_side import dates
File "/home/brian/Github/nydkc11/nydkcd11/mid_side.py", line 3, in <module>
from events.models import Service
File "/home/brian/Github/nydkc11/nydkcd11/events/models.py", line 1, in <module>
from embed_video.fields import EmbedVideoField
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/fields.py", line 6, in <module>
from .backends import detect_backend, UnknownIdException, \
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/backends.py", line 17, in <module>
from .settings import EMBED_VIDEO_BACKENDS, EMBED_VIDEO_TIMEOUT, \
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/settings.py", line 7, in <module>
'embed_video.backends.SoundCloudBackend',
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/django/conf/__init__.py", line 39, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting EMBED_VIDEO_BACKENDS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
I know for a fact the traceback concerns an app my project uses, django-embed-video. What I don't understand is why Django is indicating it as a problem considering a) Service doesn't use fields from django-embed-video, b) I successfully deployed the application on my project earlier. My questions are thus as follows:
1) Why does this problem occur in the .py file and not in shell?
2) What is causing Django to output the traceback?
Below is relevant code from the program. Please notify me if additional clarifications are required. Thank you!
*I decided not to include teamup.py as this is more of a Django-side problem, but I will include it if requested.
mid_side.py
from dateutil.parser import parse
from events.models import Service
import datetime
from teamup_api.teamup import *
def dates(start,end):
event_list = day_query(start.year, start.month, start.day, end.year, end.month, end.day)['events']
service_objects = []
for event in event_list:
service_objects.append(Service(
title=event['title'],
school=none_parse(event['who']),
location=none_parse(event['location']),
start_time=parse(event['start_dt']),
end_time=parse(event['end_dt']),
all_day=event['all_day'],
description=none_parse(event['notes'])
))
return service_objects
models.py
class Service(models.Model):
title = models.CharField(max_length=100)
school = models.CharField(max_length=100)
location = models.CharField(max_length=100)
start_time = models.DateTimeField()
end_time = models.DateTimeField()
all_day = models.BooleanField()
description = models.TextField()
def __str__(self):
return self.title
UPDATE: I've been running 'shell' through manage.py shell, and 'console' via natively running a python file (e.g. python [file].py)
My problem was probably that I was using Django resources in the native python environment. As Chris indicates in the comments, creating a custom manage.py command instead of a .py start file should suffice.
I have a basic parser app I'm building in Python. I monitors a folder and imports files when they are dropped there. I have a MongoDB that I'm trying to save the imports to. There's almost nothing to it. The problem happens when I try to include one of my class/mongo-document files. I'm sure it's a simple syntax issue I don't understand. I have all my requirements installed, and I'm running this in an virtual env. This is my first python app though, so it's likely something I'm not seeing.
My file structure is
application.py
requirements.txt
__init__.py
-services
parser.py
__init__.py
-models
hl7message.py
__init__.py
Here is application.py
from mongoengine import connect
import os, os.path, time
from services import parser
db = connect('testdb')
dr = 'C:\\Imports\\Processed'
def processimports():
while True:
files = os.listdir(dr)
print(str(len(files)) + ' files found')
for f in files:
msg = open(dr + '\\' + f).read().replace('\n', '\r')
parser.parse_message(msg)
print('waiting')
time.sleep(10)
processimports()
requirements.txt
mongoengine
hl7
parser.py
import hl7
from models import hl7message
def parse_message(message):
m = hl7.parse(str(message))
h = hl7message()
hl7message.py
from utilities import common
from application import db
import mongoengine
class Hl7message(db.Document):
message_type = db.StringField(db_field="m_typ")
created = db.IntField(db_field="cr")
message = db.StringField(db_field="m")
If I don't include the hl7message class in the parser.py it runs fine, but as soon as I include it I get the error, so I'm sure it has something to do with that file. The error message though isn't to helpful. I don't know if I've got myself into some kind of include loop or something.
Sorry, stack trace is below
Traceback (most recent call last):
File "C:/OneDrive/Dev/3/Importer/application.py", line 3, in <module>
from services import parser
File "C:\OneDrive\Dev\3\Importer\services\parser.py", line 2, in <module>
from models import hl7message
File "C:\OneDrive\Dev\3\Importer\models\hl7message.py", line 2, in <module>
from application import db
File "C:\OneDrive\Dev\3\Importer\application.py", line 23, in <module>
processimports()
File "C:\OneDrive\Dev\3\Importer\application.py", line 17, in processimports
parser.parse_message(msg)
AttributeError: module 'services.parser' has no attribute 'parse_message'
This is a circular import issue. Application.py imports parser, which imports h17 which imports h17message, which imports application which runs processimports before the whole code of the parser module has been run.
It seems to me that service modules should not import application. You could create a new module common.py containing the line db = connect('testdb') and import db from common both in application.py and in h17message.
When i run the same code in django shell it works fine for me. but when I fire up the Python interpreter (Python 2) to check some things, I get the an error when I try importing - from django.contrib.auth.models import User
import os
import django
django.setup()
import smtplib
import sys
sys.path.insert(0, "/edx/app/edxapp/edx-platform/lms/djangoapps/courseware")
import dateutil.relativedelta
from django.conf import settings
from django.contrib.auth.models import User
from opaque_keys.edx.keys import CourseKey
from courseware.models import StudentModule
from datetime import datetime
def PCSurvey_email():
for student in StudentModule.objects.filter(module_type="PCSurvey"):
two_months_date = datetime.now().date() - dateutil.relativedelta.relativedelta(months=2)
created = student.created.date()
if created == two_months_date :
user_id = student.student_id
user_email = User.objects.get(id = user_id).email
FROM = "user#example.com"
TO = [user_email] # must be a list
SUBJECT = "Hello!"
TEXT = "This message was sent with Python's smtplib."
# Prepare actual message
message = """\
From: %s
To: %s
Subject: %s
%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)
# Send the mail
server = smtplib.SMTP('outlook.net')
server.sendmail(FROM, TO, message)
server.quit()
#deleting the module
smod = StudentModule.objects.get(module_type="PCSurvey", course_id = student.course_id, student_id= student.student_id)
smod.delete()
The error that I get is
Traceback (most recent call last):
File "/edx/app/edxapp/edx-platform/lms/djangoapps/courseware/PCSurvey_email.py", line 4, in <module>
django.setup()
File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 22, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 53, in __getattr__
self._setup(name)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 39, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting LOGGING_CONFIG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
Any help on this is very much appreciated.
Django needs to be setup, in order to run fine, when you run it through manage.py shell, manage.py takes care or setting it up for you, but doing it via a python script, needs manual setup.
You also seem to have used your defined models, to be able to import them into your python script, you need to add the path to the root folder of your project to the current python path.
Finally, you need to tell django where your settings file is (before setting up your django), in your manage.py file, you should have something like this :
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
Go make it a constant, name it * DEFAULT_SETTINGS_MODULE *, so you now have:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", DEFAULT_SETTINGS_MODULE)
Now you need to import the constant into your script and tell django (By setting an env var) where it should look for the settings file.
So in all:
import sys, os
sys.path.insert(0, "/path/to/parent/of/courseware") # /home/projects/my-djproj
from manage import DEFAULT_SETTINGS_MODULE
os.environ.setdefault("DJANGO_SETTINGS_MODULE", DEFAULT_SETTINGS_MODULE)
import django
django.setup()
This way you're setup up just fine.
For new users coming on this question.
If you don't want to setup django settings file manually as described wonderfully by #SpiXel
You can run python manage.py shell from the project directory which will basically link the settings file that django needs automatically.
When you open a shell, django automatically sets itself up.
With Python interpreter you have to do this manually:
import django
django.setup()
Just a simple solution all can try!
I tried to run my custom script inside shell only by using the following way:
python manage.py shell < /path_to_script