I'm trying to dynamically import a python-based SQL query module from a sub-folder, and that folder is obtained by using the argparse module.
My project structure :
main_file.py
Data_Projects/
ITA_data/
__init__.py
sqlfile.py
UK_data/
__init__.py
sqlfile.py
Within main_file.py is the argparse module with the argument 'dir' containing the location of the specified directory i.e.
parser.add_argument('--dir', default='Data_Projects/ITA_data/', type=str,
help="set the data directory")
My understanding thus far is that modules should be imported at the top and to import just one sql query I would use:
from Data_Project.ITA_data import sqlfile
I know I can't set the import statement after the args have been defined, so how can I keep the format correct with the imports at the top, and yet retrospectively update this with the arguments that get defined afterwards?
Many thanks.
UPDATE
Thanks to the below answer. I've now tried to assign :
sqlfile = __import__(in_arg.dir + 'sqlfile.py')
However I'm getting the following error:
*** ModuleNotFoundError: No module named 'Data_Projects/ITA_data/sqlfile'
I've tried using things like
os.path.join(Path(__file__).resolve().parents[0], in_arg.dir + 'sqlfile')
If it helps, when I try just :
__import__('Data_Projects') - works fine
__import__('Data_Projects/ITA_data') - doesn't work - ModuleNotFound
And as a check to verify I'm not crazy:
os.path.exists('Data_Projects/ITA_Data/sqlfile.py') >>> True
os.path.exists(in_arg.dir + 'sqlfile.py') >>> True
I don't see anything wrong with
import argparse
parser = ...
parser.add_argument('data', choices=['UK', 'ITA'])
args = parser.parse_args()
if args.dir == 'UK':
import UK_data as data
elif args.dir == 'ITA':
import ITA_data as data
else ...
You could refine this with functions and __name__ etc. But a conditional import is ok, just so long as it occurs before the data module is used.
You can use __import__(filename: str) function instead of import statement. It does the same:
# option 1
import foo as bar
# option 2
bar = __import__('foo')
If you need to import from aside, you need to add your directory to module search paths. There are several ways to achieve that, depending on your version of Python. You can find them all in great post:
How to import a module given the full path?
The issue was resolved by using :
import sys
sys.path.insert(0, os.getcwd() + "/" + in_arg.dir)
This sets the PYTHONPATH variable to include the directory I want (which changes depending on the argument) to use to search for the file.
From there using grapes help it was a case of doing:
sqlfile = __import__('sqlfile')
And from there I could use the variable to perform the relevant sql query.
Related
I am using django and I have a file named models.admin.py and I want to do the following idea in models.py:
from "models.admin" import *
however, I get a syntax error for having double quotes. But if I just do
from models.admin import *
then I get "ImportError: No module named admin"
Is there any way to import from a python file that has a period in its name?
Actually, you can import a module with an invalid name. But you'll need to use imp for that, e.g. assuming file is named models.admin.py, you could do
import imp
with open('models.admin.py', 'rb') as fp:
models_admin = imp.load_module(
'models_admin', fp, 'models.admin.py',
('.py', 'rb', imp.PY_SOURCE)
)
But read the docs on imp.find_module and imp.load_module before you start using it.
If you really want to, you can import a module with an unusual filename (e.g., a filename containing a '.' before the '.py') using the imp module:
>>> import imp
>>> a_b = imp.load_source('a.b', 'a.b.py')
>>> a_b.x
"I was defined in a.b.py!"
However, that's generally a bad idea. It's more likely that you're trying to use packages, in which case you should create a directory named "a", containing a file named "b.py"; and then "import a.b" will load a/b.py.
The file is called models/admin.py. (Source)
That is, it should be called admin.py in a directory called models.
Then you can import using from models.admin import *, assuming that it is in your Python path.
Like below
Assume dir structure is like this:
C:.
│ script.py
│
└───Parent
└───Child
├───1.1
│ main.py
│
└───1.2
**assume you want to import main.py in script.py **
your main.py looks like below
def my_function():
print("Hello from a function")
your script.py looks like below
from os import path
import importlib
from os.path import dirname
import sys
import importlib.util
def getPath():
# your logic to get to the path
return path.join(dirname(__file__),'Parent','Child','1.1','main.py')
file_path = getPath()
module_name = 'main'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
#call functions like this
module.my_function()
Check out this gist
No, you can't import a python file as a module if its name contains a period (or a question mark, or exclamation mark, etc). A python module's name (not including the .py) must be a valid python name (ie can be used as a variable name).
In my case, I am using grafanalib, and the filename has to be xx.dashboard.py based on the doc. However, I do want to import this file to simplify the uploading step.
I got warning when I use import imp:
the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
Here is the simple demo using importlib and pathlib:
foo.bar.py and main.py are in the same foler.
# foo.bar.py
num = 42
# main.py
import importlib.machinery
import pathlib
module = importlib.machinery.SourceFileLoader(
"foo_bar",
pathlib.Path(__file__).parent.joinpath("foo.bar.py").resolve().as_posix(),
).load_module()
print(module.num) # 42
You are not referencing files in the import statement, you are referencing modules and packages.
Please read the docs, they are very clear on that matter.
Anyway, since you are using django, the usual approach won't work. If you want to keep models in separate files, rather than in models.py, you have to take extra steps, outlined, for example, here.
Edit:
Well, I don't really know what the questioneer means when he mentions admin and whether or not it is related to the admin interface of django. My points still stand.
I'm trying run an env_setup script that imports modules used in my main_script. But despite successfully running env_setup.py the modules are not being imported (presumably it is being run in its own environment).
Previously I know I have somehow sucesfully used:
from env_setup import *
However this fails for me now.
I tried a second approach using:
importlib.util.spec_from_file_location(name, location)
But this also fails.
Below is an example of what I am attempting (using the second approach in my main_script.py):
Example env_setup.py script:
import datetime # import module
print("modules imported!!!") # confirm import
Example main_script.py script:
# This first section should import `datetime` using `env_setup.py`
import importlib
spec = importlib.util.spec_from_file_location(
name='setup',
location='/home/solebay/my project/env_setup.py' # path to `set_up` script
)
my_mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(my_mod)
# This returns "modules imported!!!"
# Here we run a basic command to check if `datetime` was imported...
now = datetime.datetime.now()
print(now.strftime('%H:%M:%S on %A, %B the %dth, %Y')) # Should print time/date
# NameError: name 'datetime' is not defined
How do I get python to actually import the required modules into the environment running main_script.py? It creates a __pycache__ folder so I know the path is correct.
After dynamically importing a module you can either access the module directly by using my_mod.function() or import everything (imitate from module import *) like so:
import sys
sys.modules["setup"] = my_mod
from setup import *
del sys.modules["setup"] # Optional
After lots of searching I decided:
from env_setup import *
Should absolutely work.
I moved my most recent scripts to a fresh directory with a simpler tree and everything works.
I'm thinking it was a bug?
Update (Not a bug):
As per the useful suggestion of Bharel I ran..
import os
os.getcwd() # Returned 'wrong' directory
os.listdir() # Returned 'wrong' listing
Visual inspection of the folder tree showed that env_setup.py was present, but this file and others were absent from the true listing returned by os.listdir().
I'm running my code through the IDE "Atom" using the "Hydrogen" module. I opened a new window, added a new project folder and ran the command again and it updated.
I'm assuming I moved a folder and Atom didn't have a chance to update the path.
End result:
from env_setup import *
Works prefectly.
I have different config files which are basically python files that define variables and I want to import them in my main program.
Usually I will have a "config_file1.py" and do something like:
import config_file1 as params
# and then I can access the different parameters as
params.var1
params.var2
Now I want to be able to select which config_file I want to use, so I want to pass a parameter when calling my program that tells which config file to use. Something like:
config_filename = sys.argv[1]
import config_filename as params
However this looks for a file named "config_filename".
I want instead to import the file referenced by the value of config_filename
EDIT:
Basically my program will run a set of experiments, and those experiments need a set of parameters to run.
Eg:
*** config1.py ****
num_iterations = 100
initial_value1 = 10
initial_value2 = 20
So I can run my program loading those variables into memory.
However another config file (config2.py) might have another set of parameters, so I want to be able to select which experiment I want to run by loading the desired config file.
If you really want to do this, you can use the importlib module:
import importlib
params = importlib.import_module(sys.argv[1])
Then you can use the var like this
params.var1
This is in response to your details and not the question.
If you want to load variables such as num_iterations = 100, initial_value1 = 10, initial_value2 = 20 from a file, then I'd really recommend some sort of config instead of abusing imports for global variables.
Json would be the easiest way, where you'd load the file and you'd straight up get a dict:
>>> import json
>>> params = json.loads(config_file1)
{'num_iterations': 100, 'initial_value1': 10, 'initial_value2': 20}
Alternatively you could use ConfigParser, which looks nicer, but I've found it to be quite prone to breaking.
You can do like this:
config_filename = sys.argv[1]
params = __import__(config_filename)
I wouldn't recommend such a risky approach. You relinquish all controls over at the point of the sys.argv and your script can fail if any one of the named attribute doesn't exist within your module.
Instead I would suggest explicitly controlling what are the supported modules being passed in:
config_filename = sys.argv[1].lower()
if config_filename == 'os':
import os as params
elif config_filename == 'shutil':
import shutil as params
else: # unhandled modules
raise ImportError("Unknown module")
Instead of using import statement, you can use __import__ function, like this:
params = __import__('module_name')
Definition:
importlib.__import__(name, globals=None, locals=None, fromlist=(), level=0)
Reference:
https://docs.python.org/3/library/importlib.html#importlib.import
The path is app/base/admin/crud/__init__.py.
I want to import an entire folder as a package like this:
import app.base.admin.crud as cx
But it doesn't work and gives this error:
AttributeError: module 'app.base' has no attribute 'admin'
But when I import it's function like this from app.base.admin.crud import crud, it works.
What's going on here?
See the documentation about packages.
More specifically that part:
[...] when using syntax like import item.subitem.subsubitem, each item except for the last must be a package; the last item can be a module or a package but can’t be a class or function or variable defined in the previous item.
When using the import x.y.z statement alone (without from), you're actually importing a package for usage in your code as x.y.z.something(). Each part of that path must be a proper package (in other words, contain a __init__.py file)
I need to find out the path for a particular library in my script.
I tried using :
os.path.dirname(MODULE.__file__),
but it gives an attribute error:
"type object 'MODULE' has no attribute '__file__'."
Is there any other way to find the path for a particular module? I do not want to use sys.path as it gives the list of all the libraries.
You have space between underscores _ _ in MODULE.__file__. Call it like:
>>> import os, operator
>>> os.path.dirname(operator.__file__) # where MODULE is "operator"
'/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4'
You can use the both dirname() and basename() if you are going to split the passed filename into components without taking into account the current directory. If you want to also consider the current directory, you have to do so explicitly.
To get the dirname of the absolute path, use
os.path.dirname(os.path.abspath(__file__))
Note: Remove the spaces before file. As the response of #paxdiablo here, when a module is loaded in Python, __file__ is set to its name. You can then use that with other functions to find the directory that the file is located in.
It looks like you are not examining a module, but rather a type object. You might try:
os.path.dirname(MODULE.__module__.__file__)
caveat
Usually you can use the __file__ attribute to find out where the module is located, but as stated in the docs (you have to scroll down a bit or ctrl + f and search for __file__) this attribute is not present.
The __file__ attribute is not present for C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file.
You can however use the imp module. It has the find_module function which will perform the same search that is performed when you actually import the module. This would lead to code like this:
import imp
data = imp.find_module("MODULE")
# The path of the module is the second element of the tuple
pathname = data[1]
Use os module
os.path.dirname(<module>.__file__)
An example:
>>> import os, requests
>>>
>>> requests.__file__
'/usr/lib/python2.7/site-packages/requests/__init__.pyc'
>>>
>>> os.path.dirname(requests.__file__)
'/usr/lib/python2.7/site-packages/requests'
>>>
>>> os.path.abspath(requests.__file__)
'/usr/lib/python2.7/site-packages/requests/__init__.pyc'
os.path.dirname - just removes the last segment in the path i.e. __init__.pyc - in short it returns the project root.
Actually my code was something like :
from <module> import blah
I was trying to find the module path using :
os.path.dirname(module.__file__)
It gave an error saying "module" does not exist.
So I did :
import module and then
from <module> import blah
Then it worked !!