We Keep Coding

Proper way of getting path of main script from library in Python

I'm writing a Python library which which you can load an object from a file and do something with it. For convenience, I'd like to make it so that people can provide three kinds of paths:
A path starting with a "/", which will be interpreted as an absolute path
A path starting with a "~/", which will be interpreted as relative to the user's home directory (for which I plan to use os.path.expanduser)
A path starting with neither, which would be interpreted as relative to the directory of the top-level script that is importing my library.
My question is: what is the best way of approaching the third case? The two ways I have come across on Stack Overflow are both kind of klugey:
1)
import __main__
if hasattr(__main__, "__file__"):
script_dir = os.path.dirname(os.path.abspath(__main__.__file__))
import inspect
frame_info = inspect.stack()[-1]
mod = inspect.getmodule(frame_info[0])
script_dir = os.path.dirname(mod.__file__)
Obviously both of these will fail in the case that we're running from an interactive terminal or something, and in that case I would want to fall back to just treating it as an absolute path.
Anyway, I get the sense that using inspect like this in a library is frowned upon, but the other way seems klugey as well. Is there a better way to do this that I'm unaware of?

Use the built-in package pathlib's getcwd method. It defaults to the root folder of the top package.
Will not work if you change the working directory. Although, an unprefixed path is common practice to be relative to the working directory.
import pathlib
print(pathlib.Path.cwd())
>>> A:\Programming\Python\generalfile

Python3 unable to find a package module

I have a file called dns_poison.py that needs to call a package called netscanner. When i try and load the icmpscan module from dns_poison.py I get this message:
ModuleNotFoundError: No module named 'icmpscan'
I've done a sys.path and can confirm that the correct path is in place. The files are located at D:\PythonProjects\Networking\tools and D:\PythonProjects appears when I do a sys.path.
Here is my directory structure:
dns_poison.py
netscanner/
__init__.py
icmpscan.py
Code snippets for the files are as follows:
dns_poison.py
import netscanner
netscanner\__init__.py
from icmpscan import ICMPScan
netscanner\icmpscan.py
class ICMPScan:
def __init__(self, target, count=2, timeout=1):
self.target = target
self.count = count
self.timeout = timeout
self.active_hosts = []
# further code below here....
I don't understand why it cannot find the module, as I've used this exact same method on other python projects without any problems. Any help would be much appreciated.

When you run python dns_poison.py, the importer checks the module path then the local directory and eventually finds your netscanner package that has the following available:
netscanner
netscanner.icmpscan
netscanner.icmpscan.ICMPScan
Now I ask you, where is just icmpscan? The importer cannot find because well, it doesnt exist. The PYTHONPATH exists at wherever dns_poison.py resides, and doesn't append itself to include the absolute path of any imported modules because that simply not how it works. So netscanner can be found because its at the same level as dns_poison.py, but the importer has no clue where icmpscan.py exists because you havent told it. So you have two options to alter your __init__.py:
from .icmpscan import ICMPScan which works with Python 3.x
from netscanner.icmpscan import ICMPScan which works with both Python 2.x/3.x
Couple of references for you:
Python Import System
Python Modules recommend you ref section 6.4.2 Intra-package References

The most simple way to think about this is imports should be handled relative to the program entry-point file. Personally I find this the most simple and fool-proof way of handling import paths.
In your example, I would have:
from netscanner.icmpscan import ICMPScan
In the main file, rather than add it to init.py.

Importing Python files that are in a folder

I've been working on a project that creates its own .py files that store handlers for the method, I've been trying to figure out how to store the Python files in folder and open them. Here is the code I'm using to create the files if they don't already exist, then importing the file:
if os.path.isfile("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py") == False:
TestPy = open("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py","w+")
try:
TestPy.write(StrHandler)
except Exception as Error:
print(Error)
TestPy.close()
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
self.IntBtnID += 1
when I change this line:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
to this:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons\\Btn" + str(self.IntBtnID))
the fill can't be found and ends up throwing an error because it can't find the file in the folder.
Do know why it doesn't work I just don't know how to get around the issue:/
My question is how do I open the .py when its stored in a folder?

There are a couple of unidiomatic things in your code that may be the cause of your issue. First of all, it is generally better to use the functions in os.path to manipulate paths to files. From your backslash usage, it appears you're working on Windows, but the os.path module ensures consistent behaviour across all platforms.
Also there is importlib.import_module, which is usually recommended over __import__. Furthermore, in case you want to load the generated module more than once during the lifetime of your program, you have to do that explicitly using imp.reload.
One last tip: I'd factor out the module path to avoid having to change it in more than one place.

You can't reference a path directory when you are importing files. Instead, you want to add the directory to your path and then import the name of the module.
import sys
sys.path.append( "Buttons" )
__import__("Btn"+str(self.IntBtnId))
See this so question for more information.

The first argument to the __import__() function is the name of the module, not a path to it. Therefore I think you need to use:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons.Btn" + str(self.IntBtnID))
You may also need to put an empty __init__.py file in the Buttons folder to indicate it's a package of modules.

Importing python modules from multiple directories

I have a python 2.6 Django app which has a folder structure like this:
/foo/bar/__init__.py
I have another couple directories on the filesystem full of python modules like this:
/modules/__init__.py
/modules/module1/__init__.py
/other_modules/module2/__init__.py
/other_modules/module2/file.py
Each module __init__ has a class. For example module1Class() and module2Class() respectively. In module2, file.py contains a class called myFileClass().
What I would like to do is put some code in /foo/bar/__init__.py so I can import in my Django project like this:
from foo.bar.module1 import module1Class
from foo.bar.module2 import module2Class
from foo.bar.module2.file import myFileClass
The list of directories which have modules is contained in a tuple in a Django config which looks like this:
module_list = ("/modules", "/other_modules",)
I've tried using __import__ and vars() to dynamically generate variables like this:
import os
import sys
for m in module_list:
sys.path.insert(0, m)
for d in os.listdir(m):
if os.path.isdir(d):
vars()[d] = getattr(__import__(m.split("/")[-1], fromlist=[d], d)
But that doesn't seem to work. Is there any way to do this?
Thanks!

I can see at least one problem with your code. The line...
if os.path.isdir(d):
...won't work, because os.listdir() returns relative pathnames, so you'll need to convert them to absolute pathnames, otherwise the os.path.isdir() will return False because the path doesn't exist (relative to the current working directory), rather than raising an exception (which would make more sense, IMO).
The following code works for me...
import sys
import os
# Directories to search for packages
root_path_list = ("/modules", "/other_modules",)
# Make a backup of sys.path
old_sys_path = sys.path[:]
# Add all paths to sys.path first, in case one package imports from another
for root_path in root_path_list:
sys.path.insert(0, root_path)
# Add new packages to current scope
for root_path in root_path_list:
filenames = os.listdir(root_path)
for filename in filenames:
full_path = os.path.join(root_path, filename)
if os.path.isdir(full_path):
locals()[filename] = __import__(filename)
# Restore sys.path
sys.path[:] = old_sys_path
# Clean up locals
del sys, os, root_path_list, old_sys_path, root_path, filenames, filename, full_path
Update
Thinking about it, it might be safer to check for the presence of __init__.py, rather than using os.path.isdir() in case you have subdirectories which don't contain such a file, otherwise the __import__() will fail.
So you could change the lines...
full_path = os.path.join(root_path, filename)
if os.path.isdir(full_path):
locals()[filename] = __import__(filename)
...to...
full_path = os.path.join(root_path, filename, '__init__.py')
if os.path.exists(full_path):
locals()[filename] = __import__(filename)
...but it might be unnecessary.

We wound up biting the bullet and changing how we do things. Now the list of directories to find modules is passed in the Django config and each one is added to sys.path (similar to a comment Aya mentioned and something I did before but wasn't too happy with). Then for each module inside of it, we check for an __init__.py and if it exists, attempt to treat it as a module to use inside of the app without using the foo.bar piece.
This required some adjustment on how we interact with the modules and how developers code their modules (they now need to use relative imports within their module instead of the full path imports they used before) but I think this will be an easier design for developers to use long-term.
We didn't add these to INSTALLED_APPS because we do some exception handling where if we cannot import a module due to dependency issues or bad code our software will continue running just without that module. If they were in INSTALLED_APPS we wouldn't be able to leverage that flexibility on when/how to deal with those exceptions.
Thanks for all of the help!

How to retrieve a module's path?

I want to detect whether module has changed. Now, using inotify is simple, you just need to know the directory you want to get notifications from.
How do I retrieve a module's path in python?

import a_module
print(a_module.__file__)
Will actually give you the path to the .pyc file that was loaded, at least on Mac OS X. So I guess you can do:
import os
path = os.path.abspath(a_module.__file__)
You can also try:
path = os.path.dirname(a_module.__file__)
To get the module's directory.

There is inspect module in python.
Official documentation
The inspect module provides several useful functions to help get
information about live objects such as modules, classes, methods,
functions, tracebacks, frame objects, and code objects. For example,
it can help you examine the contents of a class, retrieve the source
code of a method, extract and format the argument list for a function,
or get all the information you need to display a detailed traceback.
Example:
>>> import os
>>> import inspect
>>> inspect.getfile(os)
'/usr/lib64/python2.7/os.pyc'
>>> inspect.getfile(inspect)
'/usr/lib64/python2.7/inspect.pyc'
>>> os.path.dirname(inspect.getfile(inspect))
'/usr/lib64/python2.7'

As the other answers have said, the best way to do this is with __file__ (demonstrated again below). However, there is an important caveat, which is that __file__ does NOT exist if you are running the module on its own (i.e. as __main__).
For example, say you have two files (both of which are on your PYTHONPATH):
#/path1/foo.py
import bar
print(bar.__file__)
and
#/path2/bar.py
import os
print(os.getcwd())
print(__file__)
Running foo.py will give the output:
/path1 # "import bar" causes the line "print(os.getcwd())" to run
/path2/bar.py # then "print(__file__)" runs
/path2/bar.py # then the import statement finishes and "print(bar.__file__)" runs
HOWEVER if you try to run bar.py on its own, you will get:
/path2 # "print(os.getcwd())" still works fine
Traceback (most recent call last): # but __file__ doesn't exist if bar.py is running as main
File "/path2/bar.py", line 3, in <module>
print(__file__)
NameError: name '__file__' is not defined
Hope this helps. This caveat cost me a lot of time and confusion while testing the other solutions presented.

I will try tackling a few variations on this question as well:
finding the path of the called script
finding the path of the currently executing script
finding the directory of the called script
(Some of these questions have been asked on SO, but have been closed as duplicates and redirected here.)
Caveats of Using __file__
For a module that you have imported:
import something
something.__file__
will return the absolute path of the module. However, given the folowing script foo.py:
#foo.py
print '__file__', __file__
Calling it with 'python foo.py' Will return simply 'foo.py'. If you add a shebang:
#!/usr/bin/python
#foo.py
print '__file__', __file__
and call it using ./foo.py, it will return './foo.py'. Calling it from a different directory, (eg put foo.py in directory bar), then calling either
python bar/foo.py
or adding a shebang and executing the file directly:
bar/foo.py
will return 'bar/foo.py' (the relative path).
Finding the directory
Now going from there to get the directory, os.path.dirname(__file__) can also be tricky. At least on my system, it returns an empty string if you call it from the same directory as the file. ex.
# foo.py
import os
print '__file__ is:', __file__
print 'os.path.dirname(__file__) is:', os.path.dirname(__file__)
will output:
__file__ is: foo.py
os.path.dirname(__file__) is:
In other words, it returns an empty string, so this does not seem reliable if you want to use it for the current file (as opposed to the file of an imported module). To get around this, you can wrap it in a call to abspath:
# foo.py
import os
print 'os.path.abspath(__file__) is:', os.path.abspath(__file__)
print 'os.path.dirname(os.path.abspath(__file__)) is:', os.path.dirname(os.path.abspath(__file__))
which outputs something like:
os.path.abspath(__file__) is: /home/user/bar/foo.py
os.path.dirname(os.path.abspath(__file__)) is: /home/user/bar
Note that abspath() does NOT resolve symlinks. If you want to do this, use realpath() instead. For example, making a symlink file_import_testing_link pointing to file_import_testing.py, with the following content:
import os
print 'abspath(__file__)',os.path.abspath(__file__)
print 'realpath(__file__)',os.path.realpath(__file__)
executing will print absolute paths something like:
abspath(__file__) /home/user/file_test_link
realpath(__file__) /home/user/file_test.py
file_import_testing_link -> file_import_testing.py
Using inspect
#SummerBreeze mentions using the inspect module.
This seems to work well, and is quite concise, for imported modules:
import os
import inspect
print 'inspect.getfile(os) is:', inspect.getfile(os)
obediently returns the absolute path. For finding the path of the currently executing script:
inspect.getfile(inspect.currentframe())
(thanks #jbochi)
inspect.getabsfile(inspect.currentframe())
gives the absolute path of currently executing script (thanks #Sadman_Sakib).

I don't get why no one is talking about this, but to me the simplest solution is using imp.find_module("modulename") (documentation here):
import imp
imp.find_module("os")
It gives a tuple with the path in second position:
(<open file '/usr/lib/python2.7/os.py', mode 'U' at 0x7f44528d7540>,
'/usr/lib/python2.7/os.py',
('.py', 'U', 1))
The advantage of this method over the "inspect" one is that you don't need to import the module to make it work, and you can use a string in input. Useful when checking modules called in another script for example.
EDIT:
In python3, importlib module should do:
Doc of importlib.util.find_spec:
Return the spec for the specified module.
First, sys.modules is checked to see if the module was already imported. If so, then sys.modules[name].spec is returned. If that happens to be
set to None, then ValueError is raised. If the module is not in
sys.modules, then sys.meta_path is searched for a suitable spec with the
value of 'path' given to the finders. None is returned if no spec could
be found.
If the name is for submodule (contains a dot), the parent module is
automatically imported.
The name and package arguments work the same as importlib.import_module().
In other words, relative module names (with leading dots) work.

This was trivial.
Each module has a __file__ variable that shows its relative path from where you are right now.
Therefore, getting a directory for the module to notify it is simple as:
os.path.dirname(__file__)

import os
path = os.path.abspath(__file__)
dir_path = os.path.dirname(path)

import module
print module.__path__
Packages support one more special attribute, __path__. This is
initialized to be a list containing the name of the directory holding
the package’s __init__.py before the code in that file is executed.
This variable can be modified; doing so affects future searches for
modules and subpackages contained in the package.
While this feature is not often needed, it can be used to extend the
set of modules found in a package.
Source

If you want to retrieve the module path without loading it:
import importlib.util
print(importlib.util.find_spec("requests").origin)
Example output:
/usr/lib64/python3.9/site-packages/requests/__init__.py

Command Line Utility
You can tweak it to a command line utility,
python-which <package name>
Create /usr/local/bin/python-which
#!/usr/bin/env python
import importlib
import os
import sys
args = sys.argv[1:]
if len(args) > 0:
module = importlib.import_module(args[0])
print os.path.dirname(module.__file__)
Make it executable
sudo chmod +x /usr/local/bin/python-which

you can just import your module
then hit its name and you'll get its full path
>>> import os
>>> os
<module 'os' from 'C:\\Users\\Hassan Ashraf\\AppData\\Local\\Programs\\Python\\Python36-32\\lib\\os.py'>
>>>

So I spent a fair amount of time trying to do this with py2exe
The problem was to get the base folder of the script whether it was being run as a python script or as a py2exe executable. Also to have it work whether it was being run from the current folder, another folder or (this was the hardest) from the system's path.
Eventually I used this approach, using sys.frozen as an indicator of running in py2exe:
import os,sys
if hasattr(sys,'frozen'): # only when running in py2exe this exists
base = sys.prefix
else: # otherwise this is a regular python script
base = os.path.dirname(os.path.realpath(__file__))

If you want to retrieve the package's root path from any of its modules, the following works (tested on Python 3.6):
from . import __path__ as ROOT_PATH
print(ROOT_PATH)
The main __init__.py path can also be referenced by using __file__ instead.
Hope this helps!

When you import a module, yo have access to plenty of information. Check out dir(a_module). As for the path, there is a dunder for that: a_module.__path__. You can also just print the module itself.
>>> import a_module
>>> print(dir(a_module))
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> print(a_module.__path__)
['/.../.../a_module']
>>> print(a_module)
<module 'a_module' from '/.../.../a_module/__init__.py'>

If you would like to know absolute path from your script you can use Path object:
from pathlib import Path
print(Path().absolute())
print(Path().resolve('.'))
print(Path().cwd())
cwd() method
Return a new path object representing the current directory (as returned by os.getcwd())
resolve() method
Make the path absolute, resolving any symlinks. A new path object is returned:

If you installed it using pip, "pip show" works great ('Location')
$ pip show detectron2
Name: detectron2
Version: 0.1
Summary: Detectron2 is FAIR next-generation research platform for object detection and segmentation.
Home-page: https://github.com/facebookresearch/detectron2
Author: FAIR
Author-email: None
License: UNKNOWN
Location: /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages
Requires: yacs, tabulate, tqdm, pydot, tensorboard, Pillow, termcolor, future, cloudpickle, matplotlib, fvcore
Update:
$ python -m pip show mymodule
(author: wisbucky)

If the only caveat of using __file__ is when current, relative directory is blank (ie, when running as a script from the same directory where the script is), then a trivial solution is:
import os.path
mydir = os.path.dirname(__file__) or '.'
full = os.path.abspath(mydir)
print __file__, mydir, full
And the result:
$ python teste.py
teste.py . /home/user/work/teste
The trick is in or '.' after the dirname() call. It sets the dir as ., which means current directory and is a valid directory for any path-related function.
Thus, using abspath() is not truly needed. But if you use it anyway, the trick is not needed: abspath() accepts blank paths and properly interprets it as the current directory.

I'd like to contribute with one common scenario (in Python 3) and explore a few approaches to it.
The built-in function open() accepts either relative or absolute path as its first argument. The relative path is treated as relative to the current working directory though so it is recommended to pass the absolute path to the file.
Simply said, if you run a script file with the following code, it is not guaranteed that the example.txt file will be created in the same directory where the script file is located:
with open('example.txt', 'w'):
pass
To fix this code we need to get the path to the script and make it absolute. To ensure the path to be absolute we simply use the os.path.realpath() function. To get the path to the script there are several common functions that return various path results:
os.getcwd()
os.path.realpath('example.txt')
sys.argv[0]
__file__
Both functions os.getcwd() and os.path.realpath() return path results based on the current working directory. Generally not what we want. The first element of the sys.argv list is the path of the root script (the script you run) regardless of whether you call the list in the root script itself or in any of its modules. It might come handy in some situations. The __file__ variable contains path of the module from which it has been called.
The following code correctly creates a file example.txt in the same directory where the script is located:
filedir = os.path.dirname(os.path.realpath(__file__))
filepath = os.path.join(filedir, 'example.txt')
with open(filepath, 'w'):
pass

From within modules of a python package I had to refer to a file that resided in the same directory as package. Ex.
some_dir/
maincli.py
top_package/
__init__.py
level_one_a/
__init__.py
my_lib_a.py
level_two/
__init__.py
hello_world.py
level_one_b/
__init__.py
my_lib_b.py
So in above I had to call maincli.py from my_lib_a.py module knowing that top_package and maincli.py are in the same directory. Here's how I get the path to maincli.py:
import sys
import os
import imp
class ConfigurationException(Exception):
pass
# inside of my_lib_a.py
def get_maincli_path():
maincli_path = os.path.abspath(imp.find_module('maincli')[1])
# top_package = __package__.split('.')[0]
# mod = sys.modules.get(top_package)
# modfile = mod.__file__
# pkg_in_dir = os.path.dirname(os.path.dirname(os.path.abspath(modfile)))
# maincli_path = os.path.join(pkg_in_dir, 'maincli.py')
if not os.path.exists(maincli_path):
err_msg = 'This script expects that "maincli.py" be installed to the '\
'same directory: "{0}"'.format(maincli_path)
raise ConfigurationException(err_msg)
return maincli_path
Based on posting by PlasmaBinturong I modified the code.

If you wish to do this dynamically in a "program" try this code:
My point is, you may not know the exact name of the module to "hardcode" it.
It may be selected from a list or may not be currently running to use __file__.
(I know, it will not work in Python 3)
global modpath
modname = 'os' #This can be any module name on the fly
#Create a file called "modname.py"
f=open("modname.py","w")
f.write("import "+modname+"\n")
f.write("modpath = "+modname+"\n")
f.close()
#Call the file with execfile()
execfile('modname.py')
print modpath
<module 'os' from 'C:\Python27\lib\os.pyc'>
I tried to get rid of the "global" issue but found cases where it did not work
I think "execfile()" can be emulated in Python 3
Since this is in a program, it can easily be put in a method or module for reuse.

Here is a quick bash script in case it's useful to anyone. I just want to be able to set an environment variable so that I can pushd to the code.
#!/bin/bash
module=${1:?"I need a module name"}
python << EOI
import $module
import os
print os.path.dirname($module.__file__)
EOI
Shell example:
[root#sri-4625-0004 ~]# export LXML=$(get_python_path.sh lxml)
[root#sri-4625-0004 ~]# echo $LXML
/usr/lib64/python2.7/site-packages/lxml
[root#sri-4625-0004 ~]#

If your import is a site-package (e.g. pandas) I recommend this to get its directory (does not work if import is a module, like e.g. pathlib):
from importlib import resources # part of core Python
import pandas as pd
package_dir = resources.path(package=pd, resource="").__enter__()
In general importlib.resources can be considered when a task is about accessing paths/resources of a site package.

If you used pip, then you can call pip show, but you must call it using the specific version of python that you are using. For example, these could all give different results:
$ python -m pip show numpy
$ python2.7 -m pip show numpy
$ python3 -m pip show numpy
Location: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
Don't simply run $ pip show numpy, because there is no guarantee that it will be the same pip that different python versions are calling.