I'm new to python, and confused by certain behavior.
I have a directory called d. In this directory, I have two files:
__init__.py:
from d import *
and
d.py:
var = None
def printVar():
global var
print "from d: var=%s" % `var`
From the directory above d, I get this interaction within python:
>>> import d
>>> d.var = 5
>>> d.printVar()
from d: var=None
Why is var not changed from the perspective of d.py?
My real goal is to accomplish the following:
Keep __init__.py small
Be able to change a d.py-global variable
If it makes a difference, I have multiple files in my package directory, and it would be sub-optimal to combine these into a single file.
What is an acceptable way to do that?
When you say:
import d
you're importing the package, not the module.
Just import the module d within the package:
>>> from d import d
>>> d.var = 5
>>> d.printVar()
from d: var=5
I think the actual name of the global var should be d.d.var (because it is in module d in package d)
So you could
1) Just refer to it as d.d.var when you set it
2) Make a setter in d.py
Unfortunately these probably won't work:
a) Copy it into d.var and try and set it there (what you have in your question) -- it's a different variable
a) import it back like from __init__ import var (recursive imports)
from X import *
Copies all of the names from X into the local module. As a result, modifying the local module won't modify the original.
As for what you actually want to do, you could use some python hackery and replace the module object with one of your own which overloads the operation of assigning an attribute. But don't do that. That'll require a lot of code and just make you look odd.
I suggest having a function that the foo's client code can call.
Related
I'm new to Python and programming in general (a couple of weeks at most).
Concerning Python and using modules, I realise that functions can imported using from a import *.
So instead of typing
a.sayHi()
a.sayBye()
I can say
sayHi()
sayBye()
which I find simplifies things a great deal. Now, say I have a bunch of variables that I want to use across modules and I have them all defined in one python module. How can I, using a similar method as mentioned above or an equally simple one, import these variables. I don't want to use import a and then be required to prefix all my variables with a..
The following situation would by ideal:
a.py
name = "Michael"
age = 15
b.py
some_function
if name == "Michael":
if age == 15:
print("Simple!")
Output:
Simple!
You gave the solution yourself: from a import * will work just fine. Python does not differentiate between functions and variables in this respect.
>>> from a import *
>>> if name == "Michael" and age == 15:
... print('Simple!')
...
Simple!
Just for some context, most linters will flag from module import * with a warning, because it's prone to namespace collisions that will cause headaches down the road.
Nobody has noted yet that, as an alternative, you can use the
from a import name, age
form and then use name and age directly (without the a. prefix). The from [module] import [identifiers] form is more future proof because you can easily see when one import will be overriding another.
Also note that "variables" aren't different from functions in Python in terms of how they're addressed -- every identifier like name or sayBye is pointing at some kind of object. The identifier name is pointing at a string object, sayBye is pointing at a function object, and age is pointing at an integer object. When you tell Python:
from a import name, age
you're saying "take those objects pointed at by name and age within module a and point at them in the current scope with the same identifiers".
Similarly, if you want to point at them with different identifiers on import, you can use the
from a import sayBye as bidFarewell
form. The same function object gets pointed at, except in the current scope the identifier pointing at it is bidFarewell whereas in module a the identifier pointing at it is sayBye.
Like others have said,
from module import *
will also import the modules variables.
However, you need to understand that you are not importing variables, just references to objects. Assigning something else to the imported names in the importing module won't affect the other modules.
Example: assume you have a module module.py containing the following code:
a= 1
b= 2
Then you have two other modules, mod1.py and mod2.py which both do the following:
from module import *
In each module, two names, a and b are created, pointing to the objects 1 and 2, respectively.
Now, if somewhere in mod1.py you assign something else to the global name a:
a= 3
the name a in module.py and the name a in mod2.py will still point to the object 1.
So from module import * will work if you want read-only globals, but it won't work if you want read-write globals. If the latter, you're better off just importing import module and then either getting the value (module.a) or setting the value (module.a= …) prefixed by the module.
You didn't say this directly, but I'm assuming you're having trouble with manipulating these global variables.
If you manipulate global variables from inside a function, you must declare them global
a = 10
def x():
global a
a = 15
print a
x()
print a
If you don't do that, then a = 15 will just create a local variable and assign it 15, while the global a stays 10
Let's say I have two modules:
a.py
value = 3
def x()
return value
b.py
from a import x
value = 4
My goal is to use the functionality of a.x in b, but change the value returned by the function. Specifically, value will be looked up with a as the source of global names even when I run b.x(). I am basically trying to create a copy of the function object in b.x that is identical to a.x but uses b to get its globals. Is there a reasonably straightforward way to do that?
Here is an example:
import a, b
print(a.x(), b.x())
The result is currently 3 3, but I want it to be 3 4.
I have come up with two convoluted methods that work, but I am not happy with either one:
Re-define x in module b using copy-and paste. The real function is much more complex than shown, so this doesn't sit right with me.
Define a parameter that can be passed in to x and just use the module's value:
def x(value):
return value
This adds a burden on the user that I want to avoid, and does not really solve the problem.
Is there a way to modify where the function gets its globals somehow?
I've come up with a solution through a mixture of guess-and-check and research. You can do pretty much exactly what I proposed in the question: copy a function object and replace its __globals__ attribute.
I am using Python 3, so here is a modified version of the answer to the question linked above, with an added option to override the globals:
import copy
import types
import functools
def copy_func(f, globals=None, module=None):
"""Based on https://stackoverflow.com/a/13503277/2988730 (#unutbu)"""
if globals is None:
globals = f.__globals__
g = types.FunctionType(f.__code__, globals, name=f.__name__,
argdefs=f.__defaults__, closure=f.__closure__)
g = functools.update_wrapper(g, f)
if module is not None:
g.__module__ = module
g.__kwdefaults__ = copy.copy(f.__kwdefaults__)
return g
b.py
from a import x
value = 4
x = copy_func(x, globals(), __name__)
The __globals__ attribute is read-only, which is why it must be passed to the constructor of FunctionType. The __globals__ reference of an existing function object can not be changed.
Postscript
I've used this enough times now that it's implemented in a utility library I wrote and maintain called haggis. See haggis.objects.copy_func.
So I found a way to (sort of) do this, although I don't think it entirely solves your problems. Using inspect, you can access the global variables of the file calling your function. So if you set up your files like so:
a.py
import inspect
value = 3
def a():
return inspect.stack()[1][0].f_globals['value']
b.py
from a import a
value = 5
print(a())
The output is 5, instead of 3. However, if you imported both of these into a third file, it would look for the globals of the third file. Just wanted to share this snippet however.
I had the same problem. But then I remembered eval was a thing.
Here's a much shorter version(if you don't need arguments):
b.py:
from a import x as xx
# Define globals for the function here
glob = {'value': 4}
def x():
return eval(xx.__code__, glob)
Hopefully after 2 years it'll still be helpful
I have following structure of configuration files:
app
\config
\development
\__init__.py
\settings.py
\app_config.py
\production
\__init__.py
\settings.py
\app_config.py
\testingpy
\settings.py
\app_config.py
\settinngs.py
\app_config.py
Actually app.config.settings just check environment variable RUNTIME_ENV (which could be development|production|testing, equivalent to one of config's subfolders) and load corresponding settings.
I know only about importing with importlib which return to me module as local variable and I forced to write something like that:
SCALA_URL = imported_settings.SCALA_URL
REDIS_URL = imported_settings.REDIS_URL
SOME_SETTINGS_VAR = imported_settings.REDIS_URL
.... tons of duplicated strings here, i.e. variables names are the same ...
Is there way to do something similar to python's expression: from config.${RUNTIME_ENV}.settings import *?
The return value of globals() is mutable. You could do something like this:
imported_foo = importlib.import_module('foo')
globals().update(vars(imported_foo))
Note that this imports underscore-prefixed things into the global namespace. If you want to exclude those, write a dictionary comprehension that only includes the things you want. For example:
globals().update({name: value
for name, value in vars(imported_foo).items()
if not name.startswith('_')})
This does not work with locals(), which returns a read-only value. It is not reasonably possible to do that (import * into a non-global namespace), because Python has to know the names of all local variables at compile time in order to generate the correct LOAD_FOO instructions in the bytecode (along with a variety of other interesting problems such as identifying the variables captured by a closure). You will find that import * is illegal inside a function or class:
>>> def foo():
... from foo import *
...
File "<stdin>", line 1
SyntaxError: import * only allowed at module level
>>>
That's not just a matter of "import * is bad design." It's a fundamental language limitation and can't be worked around with importlib.
I'd like to load a module dynamically, given its string name (from an environment variable). I'm using Python 2.7. I know I can do something like:
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
This is roughly equivalent to
import my_settings
(where SETTINGS_MODULE = 'my_settings'). The problem is, I need something equivalent to
from my_settings import *
since I'd like to be able to access all methods and variables in the module. I've tried
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
from my_module import *
but I get a bunch of errors doing that. Is there a way to import all methods and attributes of a module dynamically in Python 2.7?
If you have your module object, you can mimic the logic import * uses as follows:
module_dict = my_module.__dict__
try:
to_import = my_module.__all__
except AttributeError:
to_import = [name for name in module_dict if not name.startswith('_')]
globals().update({name: module_dict[name] for name in to_import})
However, this is almost certainly a really bad idea. You will unceremoniously stomp on any existing variables with the same names. This is bad enough when you do from blah import * normally, but when you do it dynamically there is even more uncertainty about what names might collide. You are better off just importing my_module and then accessing what you need from it using regular attribute access (e.g., my_module.someAttr), or getattr if you need to access its attributes dynamically.
Not answering precisely the question as worded, but if you wish to have a file as proxy to a dynamic module, you can use the ability to define __getattr__ on the module level.
import importlib
import os
module_name = os.environ.get('CONFIG_MODULE', 'configs.config_local')
mod = importlib.import_module(module_name)
def __getattr__(name):
return getattr(mod, name)
My case was a bit different - wanted to dynamically import the constants.py names in each gameX.__init__.py module (see below), cause statically importing those would leave them in sys.modules forever (see: this excerpt from Beazley I picked from this related question).
Here is my folder structure:
game/
__init__.py
game1/
__init__.py
constants.py
...
game2/
__init__.py
constants.py
...
Each gameX.__init__.py exports an init() method - so I had initially a from .constants import * in all those gameX.__init__.py which I tried to move inside the init() method.
My first attempt in the lines of:
## -275,2 +274,6 ## def init():
# called instead of 'reload'
+ yak = {}
+ yak.update(locals())
+ from .constants import * # fails here
+ yak = {x: y for x,y in locals() if x not in yak}
+ globals().update(yak)
brec.ModReader.recHeader = RecordHeader
Failed with the rather cryptic:
SyntaxError: import * is not allowed in function 'init' because it contains a nested function with free variables
I can assure you there are no nested functions in there. Anyway I hacked and slashed and ended up with:
def init():
# ...
from .. import dynamic_import_hack
dynamic_import_hack(__name__)
Where in game.__init__.py:
def dynamic_import_hack(package_name):
print __name__ # game.init
print package_name # game.gameX.init
import importlib
constants = importlib.import_module('.constants', package=package_name)
import sys
for k in dir(constants):
if k.startswith('_'): continue
setattr(sys.modules[package_name], k, getattr(constants, k))
(for setattr see How can I add attributes to a module at run time? while for getattr How can I import a python module function dynamically? - I prefer to use those than directly access the __dict__)
This works and it's more general than the approach in the accepted answer cause it allows you to have the hack in one place and use it from whatever module. However I am not really sure it's the best way to implement it - was going to ask a question but as it would be a duplicate of this one I am posting it as an answer and hope to get some feedback. My questions would be:
why this "SyntaxError: import * is not allowed in function 'init'" while there are no nested functions ?
dir has a lot of warnings in its doc - in particular it attempts to produce the most relevant, rather than complete, information - this complete worries me a bit
is there no builtin way to do an import * ? even in python 3 ?
I want to import some package depending on which value the user chooses.
The default is file1.py:
from files import file1
If user chooses file2, it should be :
from files import file2
In PHP, I can do this using variable variables:
$file_name = 'file1';
include($$file_name);
$file_name = 'file2';
include($$file_name);
How can I do this in Python?
Python doesn't have a feature that's directly equivalent to PHP's "variable variables". To get a "variable variable"'s value (or the value of any other expression) you can use the eval function.
foo = "Hello World"
print eval("foo")
However, this can't be used in an import statement.
It is possible to use the __import__ function to import using a variable.
package = "os"
name = "path"
imported = getattr(__import__(package, fromlist=[name]), name)
is equivalent to
from os import path as imported
Old thread, but I needed the answer, so someone else still might...
There's a cleaner way to do this in Python 2.7+:
import importlib
my_module = importlib.import_module("package.path.%s" % module_name)
As Fredrik Lundh states:
Anyway, here’s how these statements and functions work:
import X imports the module X, and creates a reference to that module
in the current namespace. Or in other words, after you’ve run this
statement, you can use X.name to refer to things defined in module X.
from X import * imports the module X, and creates references in the
current namespace to all public objects defined by that module (that
is, everything that doesn’t have a name starting with “_”). Or in
other words, after you’ve run this statement, you can simply use a
plain name to refer to things defined in module X. But X itself is not
defined, so X.name doesn’t work. And if name was already defined, it
is replaced by the new version. And if name in X is changed to point
to some other object, your module won’t notice.
from X import a, b, c imports the module X, and creates references in
the current namespace to the given objects. Or in other words, you can
now use a and b and c in your program.
Finally, X = __import__(‘X’) works like import X, with the difference
that you 1) pass the module name as a string, and 2) explicitly assign
it to a variable in your current namespace.
And by the way that's the last one method that you're intrested in.
Simply write (for example):
var = "datetime"
module = __import__(var)
Basing myself on mattjbray's answer:
from importlib import import_module
# lookup in a set is in constant time
safe_names = {"file1.py", "file2.py", "file3.py", ...}
user_input = ...
if user_input in safe_names:
file = import_module(user_input)
else:
print("Nope, not doing this.")
Saves a few lines of code, and allows you to set safe_names programmatically, or load multiple modules and assign them to a dict, for example.
It's probably a very bad idea to let the user choose what to import. Packages can execute code on import, so you're effectively allowing a user to arbitrarily execute code on your system! Much safer to do something like
if user_input == 'file1.py':
from files import file1 as file
elif user_input == 'file2.py':
from files import file2 as file
else:
file = None
print "Sorry, you can't import that file"