I have JSON data coming through various external API's into my Django project. I have two apps, one called 'products' and other 'extract'. Through product app, have created the database and rendered the views required. Objective of extract app is to parse the JSON data and create/update the fields in the Django database. I have this code saved in extract/views.py as shown below -
import json
import urllib2
from products.models import Product
url = " .........."
.............
for i in data[results]:
Product.objects.get_or_create (...........)
The issue I am having is this code is working well in python shell. It is extracting all the JSON data and updating the database. But, when I run the script on the command line outside of Python shell, it is giving error - 'Import error: No module named products.models'.
Django requires you to run python manage.py shell in order for the product app's models and Django internals to be initialized.
Check out this blog post by Stavros Korokithakis to convert your code into a standalone script.
I believe the error is with the directory set up. Make sure all your migrations are done as well as the initial folder has the paths to your apps. Possible you need to include the paths to each app in the other's settings.
Related
Introduction:
I am learning Django as I develop my first project using PyCharm with Python 3.8 and Postgresql.
I have a problem with the settings.
Environment description:
My aim is to have a Django project for a web site connected to a database (let's call it "web_db"), which is fed at regular intervals with data from an external source (another database, let's call it "source_db").
For this ETL process from one database to the other one, I am developing a Python module called "source2web". I read the content of source_db using psycopg2.
I am developing everything inside one PyCharm project.
Now, since I will be using Django ORM to access the content of web_db from the Django application server, I concluded I might as well use Django ORM in source2web to load my content into web_db.
This means I am trying to share some code between the Django project and source2web: the ORM models of Django.
source2web is simply launched as a python module ( python source2web.py).
Inside my PyCharm project, I have a src folder inside which are all my python modules, with a tree like this (folders are noted with parenthesis) :
(src)
|--(project_name)
|--(module1)
| |--source2web.py
|
|--(django)
|--(django_project)
|--(django_project)
|--(another_app)
The folders django_project/django_project and django_project/another_app have been created classicaly by Django with the commands django-admin startproject django_project and python3 manage.py startapp another_app.
The ORM models I want to share are the model classes inside the file django_project/another_app/models.py
The problem:
I tinkered to find a way to use the Django ORM models from source2web. I thought I had it : inside the code of source2web, I added these lines :
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django
django.setup()
But then, I realize that the Django server and the standalone module source2web can both work, but not with the exact same content inside project_name.django.django_project.django_project.settings.py : the name of another_app in the list of INSTALLED_APPS must be different.
Django requires the app to be listed as just another_app
source2web requires the app to be listed as project_name.django.django_project.another_app
I am sure I could find a 'dirty way' to solve this, but I also think I am not the first one to face this problem (sharing Django ORM code between a Django project and a standalone program) and I hope there is cleaner way to manage this.
What I can think of is duplicating the module project_name.django.django_project.django_project.settings and then just change the settings call inside source2web ( os.environ['DJANGO_SETTINGS_MODULE'] = ... ) but I never like duplicating code and maybe there is a cleaner than that ?
I have found at least one decently clean workaround :
In the project_name.django.django_project.django_project.settings file, I have left the value of INSTALLED_APPS that work with Django, i.e., the list of default apps created with the Django Project plus another_app.
Then in the source2web module, I have changed my code for initializing Django ORM :
In module source2web, I added 2 lines :
import project_name.django.django_project.django_project.settings as djangosettings
djangosettings.INSTALLED_APPS = ['project_name.django.django_project.another_app',]
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django`
django.setup()
This way, I leave untouched the settings for the Django server, and only override what's necessary in the source2web module (i.e. the value of INSTALLED_APPS, getting rid at the same time of all the default apps I don't need for my ETL process).
It works and seems robust enough.
Still, it brought me to question the whole project structure :
Because obviously, the whole problem is the result of the PyCharm project considering for source root src/, while the Django project considers for root project_name/django/django_project/ .
There should be a way to reconcile them, but how ?
Noob here with Django. I have the following folder structure for a Django app, which is inside the main project folder.
my_app/
__init__.py
admin.py
apps.py
migrations/
__init__.py
models.py
tests.py
views.py
I have a command line python script I wrote to fetch a JSON file and parse it to displays very specific information. it's using requests library for JSON and data parsing.
My question is how do I integrate my script into Django app. specifically how to bring the logic of it and to place under which file? My thinking is to create another file and import them into views. and pass them into render function - this maybe not the right and Django way, but kinda stuck there. Oh and I don't use any DB, the script uses a text file and writes to it as well.
I place a folder called services in my Django app and for each non-django stuff I add a folder. But that is basically just convenience, Django is not posing restrictions on you here.
Since you are including your script in the views, it is supposed that:
Your JSON file does not change frequently
It is not an issue to have multiple requests if your Django app is spawned many times (e.g. if you use uwsgi or gunicorn)
If such is the case, any Pythonic solution will be fine.
If 1 does not apply, it may be the case that you should implement your own middleware that, upon request,
fetches the JSON file if it is expired (e.g. if the last fetch happened long time ago)
adds the JSON file content in the request: this way you do not have to disclosure where you are keeping your JSON file
If you have multiple instances, you can do more or less the same as 1, but you might decide to store the JSON value in a separate storage, together with its expiration. You might also configure uwsgi/guincorn to fetch the JSON file at startup: this way you will not fetch the JSON file multiple times at startup. Then your middleware will do the work to keep it up-to-date, if necessary.
I want to integrate my existing code into django. How can I access models.py from external code.I tried putting it in the same application directory but it did not work.
Case Scenario: Having a web parsing code that takes data from the web and prints it. Here I am integrating it with django so as to save that data in the db and print it out in the web page. I'm trying the input to get from the webpage and when I hit submit it triggers the parsing code and gives output on the page.
I have the Django server up and running, created the apps (startapp), have the table created via models.py, have the urls.py setup. Here my question is how or where do I put my external code to make it trigger to access models .py, will it execute by itself when I start the server and hit the submit button ( i know i have to configure this one).
There are a few lines you need to include in your script if you want to use Django in "standalone" mode. See the documentation here: https://docs.djangoproject.com/en/1.10/topics/settings/#calling-django-setup-is-required-for-standalone-django-usage .
Another option would be to turn your script into a Django management command which you would then call with manage.py.
I'm trying to get the html generated by web2py from a different module, so I can turn it into a pdf file:
html = response.render('path/myview.html', context_dict)
I've included this at the top of my file:
sys.path.append('/home/www-data/web2py')
from gluon import *
However, I'm still getting:
NameError: global name 'response' is not defined
Maybe there's an easier way to do this I haven't thought of?
You can execute web2py templates using the gluon.template.render function, but if the template was designed to be executed within a web2py application, it may include references to some web2py environment objects (e.g., HTML helper objects, the T translation object, the request and response objects, etc.), which would not be available when calling the render function outside of a web2py HTTP request.
If proper execution of the template requires a web2py environment, you can build one using the gluon.shell.env function:
import os
import gluon.shell
context_dict = {...} # The context to be passed to the template.
app = 'myapp'
app_path = os.path.join(os.path.dirname(os.path.abspath(gluon.shell.__file__)),
'..', '..', 'applications', app)
w2p_env = gluon.shell.env(app, dir=app_path, import_models=True)
context_dict.update(**w2p_env) # Update context with web2py environment objects.
response = w2p_env['response']
html = response.render(os.path.join(app_path, 'views', 'mycontroller', 'myview.html'),
context=context_dict)
gluon.shell.env returns a dictionary containing all the web2py execution environment objects. If import_models=True, as in the code above, it will also execute the application's models (and therefore, any objects defined in the models will also be included in the returned dictionary).
By updating context_dict with the full web2py execution environment, we ensure that any web2py objects referenced in the template (and any templates it extends or includes) will be available.
The w2p_env environment also includes a response object, so we can use the response.render method to render the template. Alternatively, as noted above, we could have used the gluon.template.render function.
An alternative approach is to simply run your Python script in the environment of your web2py application:
python web2py.py -S myapp -M -R '/path/to/myscript.py'
With this approach, myscript.py can make reference to any web2py environment objects that would be available from within the application code (no need to import anything or explicitly build a web2py environment).
Also, note that you can exclude the -M if you don't need the application's models to be run.
I was able to get it to work by adding my python script into the web2py "modules" folder. After doing this I had to modify my import statements that referenced the other files in the "modules" directory:
from applications.myapp.modules.mymodule import mystuff
from gluon import *
html = current.response.render('myview.html', context_dict)
Then to run it from the command line I used:
python /home/www-data/web2py/web2py.py -S myapp -R 'path/to/myscript.py' As described by Anthony
I plan to call this module from a cron job once it's completed. I assume the web2py cron will use the web2py environment, which won't require the same execution script I've used to test.
My site is named ficosa. I have a Django server set up with a sqlite database that contains a table named core_Data. Now, I am developing a python file named serverMQTT.py that should insert data into that sqlite database. This file is outside Django so in order to import the Django models from ficosa site I call django.setup()
import django
from django.conf import settings
from ficosa import settings as fsettings
settings.configure(default_settings=fsettings, DEBUG=True)
django.setup()
Now this script or any imported module can use any part of Django it needs.
from core.models import Data
However, I am having an error:
AttributeError: 'module' object has no attribute 'LOGGING_CONFIG'
I would be gratefull if sombody could help me
That's not how you configure settings in a standalone app. Note what the documentation says:
Be aware that if you do pass in a new default module, it entirely replaces the Django defaults, so you must specify a value for every possible setting that might be used in that code you are importing.
Presumably, your fsettings module only contains database settings. In which case, just override that one thing:
settings.configure(DATABASES=fsettings.DATABASES, DEBUG=True)