We Keep Coding

Current working directory of PyPi package is root user folder? How to make cwd the package's directory?

I have a custom PyPi package. It is installed under Pyhon\Python38\Lib\site-packages\myCustomPackage.
In the __init__ code for myCustomPackage, I perform a few different directory operations, which failed to find the correct files and directories which reside in the Pyhon\Python38\Lib\site-packages\myCustomPackage folder.
I looked at the output of os.getcwd() and it showed the cwd to be C:\Users\TestUser, which is the root Windows user folder.
I would like the root folder to be the myCustomPackage folder.
For example, the file \myCustomPackage\__init__.py would contain
import os
class myCustomPackage():
def __init__(self):
print(os.getcwd())
If I run:
from myCustomPackage import myCustomPackage
theInstance = myCustomPackage()
The output is:
C:\Users\TestUser
How can I change that to be C:\Users\TestUser\AppData\Local\Programs\Python\Python38\Lib\site-packages\myCustomPackage?
Note : I would want it to be dynamic. No hard coding, in case the python version changes or the Windows user changes.

To get the directory path of the current module, you can use the built-in __file__.
To set the cwd to the module directory, use:
import os
import sys
from pathlib import Path
class myCustomPackage():
def __init__(self):
module_directory = Path(__file__).parent
os.chdir(module_directory)
print(os.getcwd())

My solution was the following function:
import site
import os
import traceback
def changeCWD(self):
try:
sitePackages = site.getsitepackages()
site_packages_dir = sitePackages[1]
top_module_dir = site_packages_dir + os.path.sep + "myCustomPackage"
os.chdir(top_module_dir)
return True
except:
self.myLogger.error("An error occurred in changeCWD")
tb = traceback.format_exc()
self.myLogger.exception(tb)
return False

Using DJANGO_SETTINGS_MODULE in a script in subfolder

In order to populate the database of my Django application, I created a small script that reads a CSV (a list of filenames) and creates objects accordingly:
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
import django
django.setup()
import csv
import sys
from myapp.models import Campaign, Annotation
campaign_name = "a_great_name"
path_to_csv = "filenames.csv"
with open(path_to_csv) as f:
reader = csv.reader(f)
filenames = [i[0] for i in reader]
new_campaign = Campaign.objects.create(name=campaign_name)
for i in filenames:
new_annotation = Annotation(
campaign=new_campaign,
asset_loc = i)
new_annotation.save()
I saved this script at the root of my project: myrepo/populator.py
and it worked fine.
… until I decided to move it into a subfolder of my project (it’s a bit like an admin tool that should rarely be used): myrepo/useful_tools/import_filenames/populator.py
Now, when I try to run it, I get this error: ModuleNotFoundError: No module named 'myproject'
Sorry for the rookie question, but I’m having a hard time understanding why this happens exactly and as a consequence, how to fix it. Can anybody help me?

Yes, the problem arose when you changed the script location because then the root directory was not in the sys.path list anymore. You just need to append the root directory to the aforementioned list in order to be able to load the settings.py file of your project.
If you have the script now in myrepo/useful_tools/import_filenames/populator.py, then you will need to go back three folders down in your folders tree to be back in your root directory. For this purpose, we can do a couple of tricks using the os module.
import os
import sys
sys.path.append(os.path.abspath(os.path.join(__file__, *[os.pardir] * 3)))
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
import django
django.setup()
print('loaded!')
# ... all your script's logic

Is there a better way to set sys.path.append in python project?

I have a simple python project with the following directory structure:
sample-project/
main/
config/
config.yaml
docker/
Dockerfile
tests/
__init__.py
sometests.py
config.py
run.py
__init__.py
requirements.txt
README.rst
In the config.py file, I have:
import yaml
class Config:
def __init__(self):
with open("config/config.yaml", 'r') as ymlfile:
cfg = yaml.load(ymlfile)
self.host = cfg["mysql"]["host"]
self.database = cfg["mysql"]["database"]
self.user = cfg["mysql"]["user"]
self.password = cfg["mysql"]["password"]
my_config = Config()
In the run.py file, I have an import statement as follows:
from main.config import my_config
when I run using command line: python3.5 run.py, I get the error:
from main.config import my_config
ImportError: No module named 'main'
But when I add this in run.py imports it works:
import sys
sys.path.append('/home/tomas/sample-project/')
Is there any better way to make it work rather than give absolute path or any other way? Please give some example :)

Generally, never ever touch sys.path from within your program.
Since main/ (not the best name) contains an __init__.py, we'll consider it a package, so let's make it runnable as one. This way imports will be considered correctly.
Add main/__main__.py with e.g.
from main.run import run
if __name__ == "__main__":
run()
Then run your app with
python -m main
instead of
python main/main.py

when mentioning below
from main.config import my_config
keep dir path also
eg:
from sample-project/main.config import my_config
assuming you are executing from /home/tomas

Although I share AKX's sentiment that editing your path is something to avoid I would generally suggest the following method:
import sys
from os.path import dirname
abs_dir = dirname(__file__)
sys.path.append(abs_dir)
The __file__ variable provides access to the location of the file in which it is accessed. This allows you to get an absolute path to any file in your project.

How do I get the location of the entry module in python?

I know I can use __file__ and os.path.* to figure out the path of the current module. What about the entry module?
I'm trying to load an ini file into ConfigParser from a module based on the hostname, but I want it to be relative to the entry assembly, not necessarily the working directory. Is this possible?

If by entry module you mean the program's entrypoint this is usually sys.argv[0]. So you could do something like this:
Example:
import os
import sys
def main():
basedir = os.path.dirname(sys.argv[0])
print "base directory: {0:s}".format(basedir)
if __name__ == "__main__":
main()

Reload all modules in a directory

I need to reload all the python modules within a specified directory.
I've tried something like this:
import sys, os
import_folder = "C:\\myFolder"
sys.path.insert( 0 , import_folder )
for dir in os.listdir(import_folder):
name = os.path.splitext(dir)[0]
ext = os.path.splitext(dir)[1]
if ext == ".py":
import( eval(name) )
reload( eval(name) )
Anyone know how to do this correctly?

import os # we use os.path.join, os.path.basename
import sys # we use sys.path
import glob # we use glob.glob
import importlib # we use importlib.import_module
import_folder = 'C:\\myFolder'
sys.path.append(import_folder) # this tells python to look in `import_folder` for imports
for src_file in glob.glob(os.path.join(import_folder, '*.py')):
name = os.path.basename(src_file)[:-3]
importlib.import_module(name)
reload(sys.modules[name])
importlib.import_module(name)
There is the code. Now to the semantics of the whole thing: using importlib makes this a little bit more normal, but it still promotes some bugs. You can see that this breaks for source files in a subdirectory. What you should probably do is: import the package, (import the whole folder), and use the . operator like so:
import sys # we use sys.path
sys.path.append('C:\\')
import myFolder
...
myFolder.func1(foo)
myFolder.val
bar = myFolder.Class1()
Perhaps you should take a look at the documentation for modules, but don't forget to update the path to include the parent of the folder you want to import.