We Keep Coding

Django settings: sharing ORM with a standalone module

Introduction:
I am learning Django as I develop my first project using PyCharm with Python 3.8 and Postgresql.
I have a problem with the settings.
Environment description:
My aim is to have a Django project for a web site connected to a database (let's call it "web_db"), which is fed at regular intervals with data from an external source (another database, let's call it "source_db").
For this ETL process from one database to the other one, I am developing a Python module called "source2web". I read the content of source_db using psycopg2.
I am developing everything inside one PyCharm project.
Now, since I will be using Django ORM to access the content of web_db from the Django application server, I concluded I might as well use Django ORM in source2web to load my content into web_db.
This means I am trying to share some code between the Django project and source2web: the ORM models of Django.
source2web is simply launched as a python module ( python source2web.py).
Inside my PyCharm project, I have a src folder inside which are all my python modules, with a tree like this (folders are noted with parenthesis) :
(src)
|--(project_name)
|--(module1)
| |--source2web.py
|
|--(django)
|--(django_project)
|--(django_project)
|--(another_app)
The folders django_project/django_project and django_project/another_app have been created classicaly by Django with the commands django-admin startproject django_project and python3 manage.py startapp another_app.
The ORM models I want to share are the model classes inside the file django_project/another_app/models.py
The problem:
I tinkered to find a way to use the Django ORM models from source2web. I thought I had it : inside the code of source2web, I added these lines :
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django
django.setup()
But then, I realize that the Django server and the standalone module source2web can both work, but not with the exact same content inside project_name.django.django_project.django_project.settings.py : the name of another_app in the list of INSTALLED_APPS must be different.
Django requires the app to be listed as just another_app
source2web requires the app to be listed as project_name.django.django_project.another_app
I am sure I could find a 'dirty way' to solve this, but I also think I am not the first one to face this problem (sharing Django ORM code between a Django project and a standalone program) and I hope there is cleaner way to manage this.
What I can think of is duplicating the module project_name.django.django_project.django_project.settings and then just change the settings call inside source2web ( os.environ['DJANGO_SETTINGS_MODULE'] = ... ) but I never like duplicating code and maybe there is a cleaner than that ?

I have found at least one decently clean workaround :
In the project_name.django.django_project.django_project.settings file, I have left the value of INSTALLED_APPS that work with Django, i.e., the list of default apps created with the Django Project plus another_app.
Then in the source2web module, I have changed my code for initializing Django ORM :
In module source2web, I added 2 lines :
import project_name.django.django_project.django_project.settings as djangosettings
djangosettings.INSTALLED_APPS = ['project_name.django.django_project.another_app',]
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django`
django.setup()
This way, I leave untouched the settings for the Django server, and only override what's necessary in the source2web module (i.e. the value of INSTALLED_APPS, getting rid at the same time of all the default apps I don't need for my ETL process).
It works and seems robust enough.
Still, it brought me to question the whole project structure :
Because obviously, the whole problem is the result of the PyCharm project considering for source root src/, while the Django project considers for root project_name/django/django_project/ .
There should be a way to reconcile them, but how ?

How can I call a web2py url from an outside module?

I'm trying to get the html generated by web2py from a different module, so I can turn it into a pdf file:
html = response.render('path/myview.html', context_dict)
I've included this at the top of my file:
sys.path.append('/home/www-data/web2py')
from gluon import *
However, I'm still getting:
NameError: global name 'response' is not defined
Maybe there's an easier way to do this I haven't thought of?

You can execute web2py templates using the gluon.template.render function, but if the template was designed to be executed within a web2py application, it may include references to some web2py environment objects (e.g., HTML helper objects, the T translation object, the request and response objects, etc.), which would not be available when calling the render function outside of a web2py HTTP request.
If proper execution of the template requires a web2py environment, you can build one using the gluon.shell.env function:
import os
import gluon.shell
context_dict = {...} # The context to be passed to the template.
app = 'myapp'
app_path = os.path.join(os.path.dirname(os.path.abspath(gluon.shell.__file__)),
'..', '..', 'applications', app)
w2p_env = gluon.shell.env(app, dir=app_path, import_models=True)
context_dict.update(**w2p_env) # Update context with web2py environment objects.
response = w2p_env['response']
html = response.render(os.path.join(app_path, 'views', 'mycontroller', 'myview.html'),
context=context_dict)
gluon.shell.env returns a dictionary containing all the web2py execution environment objects. If import_models=True, as in the code above, it will also execute the application's models (and therefore, any objects defined in the models will also be included in the returned dictionary).
By updating context_dict with the full web2py execution environment, we ensure that any web2py objects referenced in the template (and any templates it extends or includes) will be available.
The w2p_env environment also includes a response object, so we can use the response.render method to render the template. Alternatively, as noted above, we could have used the gluon.template.render function.
An alternative approach is to simply run your Python script in the environment of your web2py application:
python web2py.py -S myapp -M -R '/path/to/myscript.py'
With this approach, myscript.py can make reference to any web2py environment objects that would be available from within the application code (no need to import anything or explicitly build a web2py environment).
Also, note that you can exclude the -M if you don't need the application's models to be run.

I was able to get it to work by adding my python script into the web2py "modules" folder. After doing this I had to modify my import statements that referenced the other files in the "modules" directory:
from applications.myapp.modules.mymodule import mystuff
from gluon import *
html = current.response.render('myview.html', context_dict)
Then to run it from the command line I used:
python /home/www-data/web2py/web2py.py -S myapp -R 'path/to/myscript.py' As described by Anthony
I plan to call this module from a cron job once it's completed. I assume the web2py cron will use the web2py environment, which won't require the same execution script I've used to test.

Where to run Python Code in Django once the file is uploaded?

My django application has a file uploader which uploads to a specific location in my local system.
It redirects to a new html page which shows successful message after upload is done.
Once the file is uploaded I need to do some processing of csv files.
I have a python code which does the processing.
Now my question is, where do I put the python file in the django project and how do i call it to be run once the upload is done?
Any help is appreciable
Thanks in advance

You can place it anywhere you like, it's just Python. Maybe in a csv_processing.py if it fits in a single module, or as a completely independent library if it's more. Django doesn't have an opinion on this.
The best way to run it is by doing it asynchronously using Celery.

Make sure the file is within a python package, you do this by adding init.py to the directory, documentation here. In accordance to Django convention; you would place the file within the app that needs to use it, or within another app you would name utils, documentation here.

Question:
I need the file to be run completely after the application uploads the file.
Answer:
new = Storage()
new.file = request.FILES['file']
new.save()
Now we have the database id. (When file object saved into database it emits the id).
originalObj = Storage.objects.get(pk=new.id)
Now you can import the csv file and do modification here.

The best way to write initial settings data for custom CMS in Django

I'm writing my own CMS and have the situation where the some initial settings information should be written in the database. I don't like the idea to write some XML/Json file with fixture data which will be imported when I will run syncdb.
What I'm thinking about is that I can create some cms_init.py file and run it before manage.py syncdb. In this file I need to setup environment then and after that with the usage of the models I can write my custom data to database.
Another way - to have method in admin side, for example initialize() and the url for it. It will store some variable and will never run second time and in this function I just call the models I need and that's it.
Why I'm looking for the solution, because I need dynamic initial data writing, which will depend on settings.py and from other module's settings each time and I don't want to rewrite the database's initial file every time I run the new project.
Any ideas?

Don't do it automatically/implicitly, make the user do it explicitly with a manage.py command, e.g.
python manage.py your_cms init
When running your CMS functionality check or rather try-except if the initialization has taken place and if not remind the user to run the init command first.
Doing it explicitly is safer and not a big inconvenience for the user since it must only be done once.

Changing admin classes in runtime

I want to change admin classes programmatically without restarting server. I want to for example change list displays of a model in runtime. Now it only changes when I restart the server... Example (Versionadmin is an extention of modeladmin):
admin.site.unregister(model)
class YourModelAdmin(VersionAdmin):
list_display = new_list_display
admin.site.register(model, YourModelAdmin)
This works if I run it in admin.py, but if I run it when the admin site is already setup, nothing changes. Any idea how to go about this?

Have you tried reloading the module after you make changes?
Python 3+
import importlib.reload as reload
reload(admin)
Python 2.7+
reload(admin)