We Keep Coding

How do you add environment variables to a python scrapy project? dotenv didn't work

I'm having trouble incorporating an IP address into a format string in my Python Scrapy project. I was trying to use python-dotenv to store sensitive information, such as server IPs, in a .env file and load it into my project, instead of hardcoding it.
I added python-dotenv to the settings.py file of my Scrapy project, but when I run a function that should use the values stored in os, I get an error saying that it can't detect dotenv. Can someone help me understand why this is happening and how to properly incorporate an IP address in a format string using python-dotenv in a Python Scrapy project?

It is difficult to help figure out what you might be doing wrong with the limited information you have provided. However here is a general example of how one might use python-dotenv with a scrapy settings.py file.
First create a .env file and add your IP
.env
IPADDRESS = SUPERSECRETIPADDRESS
Finally in your settings.py file you need to import dotenv and then run dotenv.load_dotenv() and then you can get the ip address from the environment variable.
settings.py
import os
import dotenv
dotenv.load_dotenv()
IP_ADDRESS_SCRAPY_SETTING = os.environ["IPADDRESS"]
print(IP_ADDRESS_SCRAPY_SETTING)
output:
SUPERSECRETIPADDRESS
Note: Make sure that the .env file is in the same directory as the settings.py file or it is in one of it's parent/ancestor directories.
So if your settings.py file is in
/home/username/scrapy_project/scrapy_project/settings.py
then that means the .env file can be in one of the following:
/home/username/scrapy_project/scrapy_project/.env
/home/username/scrapy_project/.env
/home/username/.env
/home/.env
/.env
Otherwise it will not be able to find the file.

Set DJANGO_SETTINGS_MODULE as an Environment Variable in Windows permanently

How can I permanently set the environmental variable DJANGO_SETTINGS_MODULE on WINDOWS on a permanent basis and be done with it?
I mean
Win Button + Pause/Break Button
This leads to Control Panel\System and Security\System
Click Advanced System Settings
Click Environment Variables
There are two boxes the first is titled User variables and the second System variables
On the System variables click the New Button
For variable name put in DJANGO_IMPORT_SETTINGS
XXX--> WHAT DO I PUT IN VARIABLE VALUE TO SET IT ONCE AND FOR ALL?
In the Django Site on this issue it states:
DJANGO_SETTINGS_MODULE
When you use Django, you have to tell it which settings you’re using. Do this by using an environment variable, DJANGO_SETTINGS_MODULE.
The value of DJANGO_SETTINGS_MODULE should be in Python path syntax,e.g. mysite.settings. Note that the settings module should be on the Python import search path.
What does it mean ...should be in Python path syntax e.g. mysite.settings... ?
I have a certain directory where my Python is located:
C:\Python27
I have a certain directory where my Django is located: C:\Python27\Lib\site-packages\django
What does this mysite means. What directory is it meanning C:\Something......
Can you put this variable once and for all or you have to constantly change it for every project (I hope not!)
And what does this suspiciously line means Note that the settings module should be on the Python import search path.
All I want it to set the DJANGO_SETTINGS_MODULE environmental variable and be done once and for all from this hassle
EDIT
In order to work, Django just has to be pointed at a valid settings file, and by default it
looks for an environment variable named DJANGO_SETTINGS_MODULE to tell it where to find the
settings. The value of this variable should be the Python import path of the settings file, such
as cms.settings.
--> What king of directory is this: cms.settings? In windows every directory starts with a hard drive as C:\Something...... How can you start a directory like this in Windows?
EDIT_2
Excerpt from a book
PROBLEM
Environment variable DJANGO_SETTINGS_MODULE is undefined.
SOLUTION
Run the command python manage.py shell rather than python.
MY QUESTION --> ON WHAT DIRECTORY?///CAN YOU SET IT FOR ONCE OR IS IT DIFFERENT PER PROJECT?
MY PROJECT IS STRUCTURED LIKE THIS
C:\Python27\pysec-master(file)
|__local_settings.py
|__manage.py
|__settings.py
|__C:\Python27\pysec(file)
|__ __init__.py
|__example.py
|__models.py
|__xbrl.py
|__xbrl_fundamentals.py
I am trying to run models.py and I have a settings.py in the C:\Python27\pysec-master
You can find an exact copy here.
MAYBE_IMPORTANT_EDIT
I have a file called manage.py in my project which has these contents
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
Does this has to do anything on setting the variable? Do I need to set here here inside the loop?
EDIT
For the command in the IDLE from django.db import settings do i need to set a directory for the PYTHON_MODULE_SETTINGS like C:\Python27\Lib\site-packages\django\db ?

Okay, don't be so frustrated. Let's look at this step by step:
Python path syntax:
In Python, when you split your code base across modules, you qualify the name of the import with the name of the module. Let's say your project is structured like this:
my_project
|__utils
| |____init__.py
| |__file_utils.py
|__my_module
|____init__.py
|__main.py
In your main.py if you want to access methods you have defined in file_utils.py you add an import statement in your main.py like this:
import utils.file_utils.read_file
assuming read_file is the method you want to import into main.py. This way of importing modules where you have a . separating every module is referred as python path syntax.
PYTHONPATH:
In the above example, the import statement would work only if the Python interpreter knows where to look for the first module namely the utils. Only when it finds utils can it find file_utils and read_file. You specify the list of all the paths you want the interpreter to look into in the environment variable PYTHONPATH. So in order to have an import statement like above in your code, you have to make sure that the full path to your project my_project is in PYTHONPATH. Assuming my_project is in C:\AMAZEBALLS_CODE\my_project you should have C:\AMAZEBALLS_CODE in your PYTHONPATH
DJANGO_SETTINGS_MODULE:
Now let's suppose your my_project is actually a Django application. A Django application needs a settings file where you specify a whole bunch of things. In order to instruct Django which settings file to look into you specify it in DJANGO_SETTINGS_MODULE. Assuming this is your project structure:
my_project
|__utils/
| |____init__.py
| |__file_utils.py
|__my_module/
| |____init__.py
| |__main.py
|__site_settings/
|__dev_settings.py
|__production_settings.py
myroject.site_settings is the module Django has to look into for the settings file. And myroject.site_settings.dev_settings is the value you need to be setting to the DJANGO_SETTINGS_MODULE in the above case. When documentation says cms.settings or mysite.settings they mean cms or mysite is the name of your project and settings.py is the name of your settings file.
Now let's look at your question:
Can you permanently set it in the environment variables of Windows? Sure.
Is it the right way? No.
Because if you want to create another application tomorrow in another location, you will have to edit this in the environment variables section of Windows. Also, it is a practice to use a different settings file for development environment and another one for production. So setting it at one place with one value makes it inflexible. But if you are aware of all of the above and sure you are gonna be using just the one settings file set DJANGO_SETTINGS_MODULE to myproject.site_settings.dev_settings in the env variables section.
Hope this helps!
EDIT:
Looks like you are putting your pysec-master project in C:\Python27. Do not put your projects in the python installation. Create a settings.py file in your project and set DJANGO_SETTINGS_MODULE to pysec-master.settings

You said:
All i want it to set the DJANGO_SETTINGS_MODULE environmental variable
and be done once and for all from this hassle
If you don't want to go through a tedious procedure every time, you don't have to make it permanent, you only need to make the procedure automatic.
Create a bat and put set DJANGO_SETTINGS_MODULE='xxx.settings' inside.
If you are using virtualenv, you can set the environment variable in bin/activate.bat.

Unable to get a setting from settings file in django

I've a setting say gi in settings.py file. I've created a separate python file (i.e. I'm not using it in views) and used import statement as:
from django.conf import settings
but when I try to access settings.gi, it says that 'Settings' object has no attribute 'gi'. What's missing? :s

From the Django docs on creating your own settings states:
Setting names are in all uppercase.
Try renaming the setting to GI.

The project must be added on the PYTHONPATH and the DJANGO_SETTINGS_MODULE has to be defined.
Please refer to the commands mentioned below.
export PYTHONPATH=PATH_OF_PROJECT
export DJANGO_SETTINGS_MODULE=settings

Django settings.py Error: Import by filename is not supported

I am running Django in a virtual environment (using virtualenv), and I'm trying to add a custom development environment settings file to simplify app configuration when I'm developing. My plan was to do this with two lines of code
if os.environ.get('DEVELOPMENT', None):
from login import settings_dev
I've also tried import settings_def and from login.settings_dev import *. My settings_dev.py file is sitting in the same directory as my settings.py file and my app is sitting in a folder called login. When I run python login/manage.py syncdb I get this error:
Error: Import by filename is not supported.
My searching keeps bringing up DJANGO_SETTINGS_MODULE (though I'm not sure how it plays into all this - first Django app :]), so just an FYI it is set in my settings.py file like so:
os.environ['DJANGO_SETTINGS_MODULE'] = 'login.settings'
I've also tried exporting it in my terminal, but I get the same error.
Does anyone know how I can fix this/what I'm doing wrong here?

Make sure while passing relative address of file to use "." instead of "/".
I faced the same error what I actually did
"music/urls"
But it should be
"music.urls"

In the original settings.py, at the very end:
try:
from settings_dev import *
except ImportError:
pass
Create settings_dev.py in the same directory as settings.py, and in it, add these two lines at the very top:
import sys
globals().update(vars(sys.modules['settings']))
Now add whatever development settings you want in this file.

I had similar error in runserver command execution and finally I've found that this error raises because of python version incompatibility by the django version installed. There is two versions of python on my system and I had running django server by the wrong one. Hope it could be helpful to someone.

django manage.py settings default

I have a settings.py file and a dev_settings.py file that I use to override some values for dev purposes. Everytime I run the ./manage.py command, I have to specify --settings=whatever.local_settings. This becomes very tedious to do every time and I am trying to find a way to force manage.py to load my dev_settings.py file every by default so that I don't have to type that long argument every time I want to run a command.
I have tried setting DJANGO_SETTINGS_MODULE, however, it appears that manage.py overrides this option.
Is it possible to make this happen or am I doomed to always specify that argument?

manage.py sets path to settings for you, that's why it's ignoring DJANGO_SETTINGS_MODULE (it's basically just script that wraps around django-admin.py).
There are 2 easy ways to fix your problem:
set DJANGO_SETTINGS_MODULE and use django-admin.py to run all commands instead of manage.py. This is even better if you use vitualenv.
copy manage.py and name it local.py (that's the name in my case) and rename all settings mentions to dev_settings.
For example:
#!/usr/bin/env python
from django.core.management import execute_manager
import imp
try:
import settings_local
except ImportError:
import sys
sys.stderr.write("Error: Can't find the file 'settings_local.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n" % __file__)
sys.exit(1)
if __name__ == "__main__":
execute_manager(settings_local)
You can run all commands by ./local.py now.

The way this is typically done is you have settings.py with all settings that are common between environments (things like INSTALLED_APPS, etc.). Then, you have something like settings_local.py, that defines settings particular to the environment in context. You then import settings_local.py in settings.py.
# settings.py
from settings_local import *
settings.py gets added to your source code repository, but settings_local.py does not. (However, you would normally add something like settings_local.py.example to the repo.)
When you first move your app over to production, for example, you pull down the code base from your repo. You then copy settings_local.py.example to settings_local.py and make any necessary environment specific changes.
You then have separate settings_local.py files in each environment, and it all just works.

You can make a bash alias by adding these lines to your .bash_profile file:
mymanage()
{
python manage.py $1 --settings=settings_debug
}
alias mng=mymanage
Then when you run this command:
mng runserver
settings_debug.py file will be used for settings.

You can use django-admin.py with that environment variable. Commands are interchangeable, only django-admin.py doesn't override the variable you're trying to use.

If a settings file is common to all installation, you can just import it e.g.
from settings_local import *
but usually settings_local are changed and tweaked per installation and as my installation script directly copy files to target sites (without worrying what is local what is not), which mean settings_local may get overwritten, to avoid that I just keep settings_local in parent folder of the installation target and manually import it in settings.py e.g.
local_settings_file = os.path.join(prevFolder, "settings_local.py")
if os.path.exists(local_settings_file):
execfile(local_settings_file)