I'd like to define a helper function that has the ability to modify a module-level variable (with known name) from surrounding context without explicitly passing it, e.g.
# mod1.py
mod_var = 1
modify_var()
# mod_var modified
print mod_var
The problem is - I can't reference variable by mod1.mod_var, because I want to use helper function across many modules (helper itself will be defined in other module); it should dynamically 'pick' mod_var from surrounding calling context/scope.
Is this possible? How to obtain this?
My use case is to enhance defining URL -> view mapping in Django. Those definitions are spread across many sub-modules that define urlpatterns module-level variable. Helper function should pick this variable from the module that calls it and modify it. Avoiding explicitly passing it as argument would be great.
Edit:
For additional solution - check this answer.
Edit2:
Wrong solution below! (left for references in comments)
Recently I've found another solution (the least magical in my opinion ;))
modify_var() function could be implemented like this:
def modify_var():
calling_module = __import__("__main__")
calling_module.mod_var = 42
Still, potential profits are arguable.
unittest module uses this technique in its main method.
It's a truly bad, horrible, and awful idea, which will lead to future maintenance nightmares. However, Python does offer "enough rope to shoot yourself in the foot", if you truly insist: introspection and metaprogramming tools which are mostly intended for debugging purposes, but can be abused to perform the ill-conceived task you so desperately crave.
For example, in evil.py:
import inspect
def modify_var():
callersframe = inspect.stack()[1][0]
callersglobals = callersframe.f_globals
if 'mod_var' not in callersglobals:
raise ValueError, 'calling module has no "mod_var"!'
callersglobals['mod_var'] += 1
now say you have two modules, a.py:
import evil
mod_var = 23
evil.modify_var()
print 'a mod_var now:', mod_var
and b.py:
import evil
mod_var = 100
evil.modify_var()
print 'b mod_var now:', mod_var
you could do:
$ python -c'import a; import b'
a mod_var now: 24
b mod_var now: 101
However, maintaining this kind of black-magic tricks in the future is going to be a headache, so I'd strongly recommend not doing things this way.
What you want to do sounds like too much magic. Pass in urlpatterns and be done with it. Explicit is better than implicit.
OK, here's the magic, but again, I recommend not using it:
import sys
def modify_var():
"""Mysteriously change `mod_var` in the caller's context."""
f = sys._getframe(1)
f.f_locals['mod_var'] += " (modified)"
mod_var = "Hello"
modify_var()
print mod_var
prints:
Hello (modified)
As a further warning against this technique: _getframe is one of those functions that other implementations of Python don't provide, and the docs include this sentence: "This function should be used for internal and specialized purposes only."
If you really want to do that then you'll need to import mod1 in either the other module or directly in the function, and then modify it off that import. But don't do that; seasoned programmers will point and laugh.
Related
It is recommended to not to use import * in Python.
Can anyone please share the reason for that, so that I can avoid it doing next time?
Because it puts a lot of stuff into your namespace (might shadow some other object from previous import and you won't know about it).
Because you don't know exactly what is imported and can't easily find from which module a certain thing was imported (readability).
Because you can't use cool tools like pyflakes to statically detect errors in your code.
According to the Zen of Python:
Explicit is better than implicit.
... can't argue with that, surely?
You don't pass **locals() to functions, do you?
Since Python lacks an "include" statement, and the self parameter is explicit, and scoping rules are quite simple, it's usually very easy to point a finger at a variable and tell where that object comes from -- without reading other modules and without any kind of IDE (which are limited in the way of introspection anyway, by the fact the language is very dynamic).
The import * breaks all that.
Also, it has a concrete possibility of hiding bugs.
import os, sys, foo, sqlalchemy, mystuff
from bar import *
Now, if the bar module has any of the "os", "mystuff", etc... attributes, they will override the explicitly imported ones, and possibly point to very different things. Defining __all__ in bar is often wise -- this states what will implicitly be imported - but still it's hard to trace where objects come from, without reading and parsing the bar module and following its imports. A network of import * is the first thing I fix when I take ownership of a project.
Don't misunderstand me: if the import * were missing, I would cry to have it. But it has to be used carefully. A good use case is to provide a facade interface over another module.
Likewise, the use of conditional import statements, or imports inside function/class namespaces, requires a bit of discipline.
I think in medium-to-big projects, or small ones with several contributors, a minimum of hygiene is needed in terms of statical analysis -- running at least pyflakes or even better a properly configured pylint -- to catch several kind of bugs before they happen.
Of course since this is python -- feel free to break rules, and to explore -- but be wary of projects that could grow tenfold, if the source code is missing discipline it will be a problem.
That is because you are polluting the namespace. You will import all the functions and classes in your own namespace, which may clash with the functions you define yourself.
Furthermore, I think using a qualified name is more clear for the maintenance task; you see on the code line itself where a function comes from, so you can check out the docs much more easily.
In module foo:
def myFunc():
print 1
In your code:
from foo import *
def doThis():
myFunc() # Which myFunc is called?
def myFunc():
print 2
It is OK to do from ... import * in an interactive session.
Say you have the following code in a module called foo:
import ElementTree as etree
and then in your own module you have:
from lxml import etree
from foo import *
You now have a difficult-to-debug module that looks like it has lxml's etree in it, but really has ElementTree instead.
Understood the valid points people put here. However, I do have one argument that, sometimes, "star import" may not always be a bad practice:
When I want to structure my code in such a way that all the constants go to a module called const.py:
If I do import const, then for every constant, I have to refer it as const.SOMETHING, which is probably not the most convenient way.
If I do from const import SOMETHING_A, SOMETHING_B ..., then obviously it's way too verbose and defeats the purpose of the structuring.
Thus I feel in this case, doing a from const import * may be a better choice.
http://docs.python.org/tutorial/modules.html
Note that in general the practice of importing * from a module or package is frowned upon, since it often causes poorly readable code.
These are all good answers. I'm going to add that when teaching new people to code in Python, dealing with import * is very difficult. Even if you or they didn't write the code, it's still a stumbling block.
I teach children (about 8 years old) to program in Python to manipulate Minecraft. I like to give them a helpful coding environment to work with (Atom Editor) and teach REPL-driven development (via bpython). In Atom I find that the hints/completion works just as effectively as bpython. Luckily, unlike some other statistical analysis tools, Atom is not fooled by import *.
However, lets take this example... In this wrapper they from local_module import * a bunch modules including this list of blocks. Let's ignore the risk of namespace collisions. By doing from mcpi.block import * they make this entire list of obscure types of blocks something that you have to go look at to know what is available. If they had instead used from mcpi import block, then you could type walls = block. and then an autocomplete list would pop up.
It is a very BAD practice for two reasons:
Code Readability
Risk of overriding the variables/functions etc
For point 1:
Let's see an example of this:
from module1 import *
from module2 import *
from module3 import *
a = b + c - d
Here, on seeing the code no one will get idea regarding from which module b, c and d actually belongs.
On the other way, if you do it like:
# v v will know that these are from module1
from module1 import b, c # way 1
import module2 # way 2
a = b + c - module2.d
# ^ will know it is from module2
It is much cleaner for you, and also the new person joining your team will have better idea.
For point 2: Let say both module1 and module2 have variable as b. When I do:
from module1 import *
from module2 import *
print b # will print the value from module2
Here the value from module1 is lost. It will be hard to debug why the code is not working even if b is declared in module1 and I have written the code expecting my code to use module1.b
If you have same variables in different modules, and you do not want to import entire module, you may even do:
from module1 import b as mod1b
from module2 import b as mod2b
As a test, I created a module test.py with 2 functions A and B, which respectively print "A 1" and "B 1". After importing test.py with:
import test
. . . I can run the 2 functions as test.A() and test.B(), and "test" shows up as a module in the namespace, so if I edit test.py I can reload it with:
import importlib
importlib.reload(test)
But if I do the following:
from test import *
there is no reference to "test" in the namespace, so there is no way to reload it after an edit (as far as I can tell), which is a problem in an interactive session. Whereas either of the following:
import test
import test as tt
will add "test" or "tt" (respectively) as module names in the namespace, which will allow re-loading.
If I do:
from test import *
the names "A" and "B" show up in the namespace as functions. If I edit test.py, and repeat the above command, the modified versions of the functions do not get reloaded.
And the following command elicits an error message.
importlib.reload(test) # Error - name 'test' is not defined
If someone knows how to reload a module loaded with "from module import *", please post. Otherwise, this would be another reason to avoid the form:
from module import *
As suggested in the docs, you should (almost) never use import * in production code.
While importing * from a module is bad, importing * from a package is probably even worse.
By default, from package import * imports whatever names are defined by the package's __init__.py, including any submodules of the package that were loaded by previous import statements.
If a package’s __init__.py code defines a list named __all__, it is taken to be the list of submodule names that should be imported when from package import * is encountered.
Now consider this example (assuming there's no __all__ defined in sound/effects/__init__.py):
# anywhere in the code before import *
import sound.effects.echo
import sound.effects.surround
# in your module
from sound.effects import *
The last statement will import the echo and surround modules into the current namespace (possibly overriding previous definitions) because they are defined in the sound.effects package when the import statement is executed.
Example:
$ cat m1.py
a = 1
def f():
print a
$ cat m2.py
from m1 import f
a = 2
f()
I want python m2.py to print 2 when I run it, but it prints 1.
Do I have to make f take a as an argument or is there a better way of achieving this? I'm trying to code DRY and reuse the same function in a different "environment" this way. It would make sense to define a inside f, as well, if I could override it upon importing.
Another way I thought of is:
$ cat m1.py
a = 1
def make_f(a):
def f():
print a
return f
f = make_f(a)
$ cat m2.py
from m1 import make_f
a = 2
f = make_f(a)
f()
This works as needed, but are there more concise ways?
Edit: Thanks for the answers so far; I don't think I can clarify anything by providing a more realistic example, but I'd say that the reason why I'm even asking this is because in my mind there is a distinction between the actual arguments of f (which it would use the same way in both modules) and the "environment" a, which should differ. May be I shouldn't really distinguish (judging by the need to use different values of a in different modules) but the distinction makes sense based on the meaning a bears.
Edit 2: I gave it another thought and concluded that I probably want to use a closure, the reason being that I don't want other functions in each module to have to supply a when calling f. This is, I guess, the observable, non-virtual distinction that is there.
(If I understand you properly) -- Try this:
#m2.py
from m1 import f
import m1
m1.a = 2
f()
However, I should mention that the very fact that you need to do this throws off all sorts of bells and whistles in my head -- This seems like a very bad design.
I'd imagine that your code is more complex than this example, so I'd recommend that you try using a class:
YourClass.py
class YourClass(object):
def __init__(self, a=1):
self.a = a
def f(self):
print self.a
YourOtherFile.py
from YourClass import YourClass
o = YourClass(a=2) # Without explicitly setting `a=2`, `a` defaults to `1`
o.f()
Without much more context, I can't offer any more advice.
As I understand it python has the following outermost namespaces:
Builtin - This namespace is global across the entire interpreter and all scripts running within an interpreter instance.
Globals - This namespace is global across a module, ie across a single file.
I am looking for a namespace in between these two, where I can share a few variables declared within the main script to modules called by it.
For example, script.py:
import Log from Log
import foo from foo
log = Log()
foo()
foo.py:
def foo():
log.Log('test') # I want this to refer to the callers log object
I want to be able to call script.py multiple times and in each case, expose the module level log object to the foo method.
Any ideas if this is possible?
It won't be too painful to pass down the log object, but I am working with a large chunk of code that has been ported from Javascript. I also understand that this places constraints on the caller of foo to expose its log object.
Thanks,
Paul
There is no namespace "between" builtins and globals -- but you can easily create your own namespaces and insert them with a name in sys.modules, so any other module can "import" them (ideally not using the from ... import syntax, which carries a load of problems, and definitely not using tghe import ... from syntax you've invented, which just gives a syntax error). For example, in script.py:
import sys
import types
sys.modules['yay'] = types.ModuleType('yay')
import Log
import foo
yay.log = Log.Log()
foo.foo()
and in foo.py
import yay
def foo():
yay.log.Log('test')
Do not fear qualified names -- they're goodness! Or as the last line of the Zen of Python (AKA import this) puts it:
Namespaces are one honking great idea -- let's do more of those!
You can make and use "more of those" most simply -- just qualify your names (situating them in the proper namespace they belong in!) rather than insisting on barenames where they're just not a good fit. There's a bazillion things that are quite easy with qualified names and anywhere between seriously problematic and well-nigh unfeasible for those who're stuck on barenames!-)
There is no such scope. You will need to either add to the builtins scope, or pass the relevant object.
Actually, I did figure out what I was looking for.
This hack is actually used PLY and that is where is stumbled across.
The library code can raise a runtime exception, which then gives access to the callers stack.
Suppose you have 3 modules, a.py, b.py, and c.py:
a.py:
v1 = 1
v2 = 2
etc.
b.py:
from a import *
c.py:
from a import *
v1 = 0
Will c.py change v1 in a.py and b.py? If not, is there a way to do it?
All that a statement like:
v1 = 0
can do is bind the name v1 to the object 0. It can't affect a different module.
If I'm using unfamiliar terms there, and I guess I probably am, I strongly recommend you read Fredrik Lundh's excellent article Python Objects: Reset your brain.
The from ... import * form is basically intended for handy interactive use at the interpreter prompt: you'd be well advised to never use it in other situations, as it will give you nothing but problems.
In fact, the in-house style guide at my employer goes further, recommending to always import a module, never contents from within a module (a module from within a package is OK and in fact recommended). As a result, in our codebase, references to imported things are always qualified names (themod.thething) and never barenames (which always refer to builtin, globals of this same module, or locals); this makes the code much clearer and more readable and avoids all kinds of subtle anomalies.
Of course, if a module's name is too long, an as clause in the import, to give it a shorter and handier alias for the purposes of the importing module, is fine. But, with your one-letter module names, that won't be needed;-).
So, if you follow the guideline and always import the module (and not things from inside it), c.v1 will always be referring to the same thing as a.v1 and b.v1, both for getting AND setting: here's one potential subtle anomaly avoided right off the bat!-)
Remember the very last bit of the Zen of Python (do import this at the interpreter prompt to see it all):
Namespaces are one honking great idea -- let's do more of those!
Importing the whole module (not bits and pieces from within it) preserves its integrity as a namespace, as does always referring to things inside the imported module by qualified (dotted) names. It's one honking great idea: do more of that!-)
Yes, you just need to access it correctly (and don't use import *, it's evil)
c.py:
import a
print a.v1 # prints 1
a.v1 = 0
print a.v1 # prints 0
from here I got an idea about how using variables from other modules. this all works fine with
import foo as bar
But I don't want to import my modules as "bar" I want to use it without any prefix like
from foo import *
Using this it´s impossible to modify variables from other modules. reading will work! any idea? suggestions?
Short answer: No, it's impossible, and you'll have to use a prefix.
It's important to understand that from foo import x, y is copying x to your namespace. It's equivallent to:
import foo
# COPY TO YOUR NAMESPACE
x = foo.x
y = foo.y
# `from foo import` does NOT leave `foo` in your namespace
def foo
This way, each module will get a local copy of x and y. Changing x won't be seen in other modules, and you won't see changes other modules do :-(
To change the central copy of a variable you must import the module itself: import foo and change foo.x. This way only one copy exists and everybody is accessing it :-)
[The linked questions also mention the possibility to put the shared variable in the module builtin. DON'T! This would eliminate the prefix for reading it, but not for writing, and is extremely bad style.]
A note in defense of Python
If you resent the need to use a foo. prefix here, you'll probably also resent the need for the self. prefix to access object variables. The bottom line is that's how Python works - since you don't declare variables, there is no choice but to use prefixes.
But there is also an upside: when reading Python code, you easily see where each variable lives. IMHO that's very good.
Supporting evidence: in other languages like C++/Java, many people observe conventions like an m_ prefix on all object variable names to achieve a similar effect...
Style remarks
You don't need import foo as bar, just use import foo.
The as form doesn't do anything new, it just renames it, which is just confusing.
It's only useful if "foo" is a very long name; a particularly accepted case is when "foo" lives deep in some package, so you can do import long.package.foo as foo.
The from foo import * is considered very bad style in programs because:
The person reading your code won't know where names came from.
It pollutes your namespace, which can lead to subtle bugs when names from
different modules clash.
The explicit form from foo import x, y is OK, but starts suffering from the same problems if you use many names from the module.
In such cases, it's best to import foo and explicitly write foo.x, foo.y.
Bottom line: when in doubt, a simple import foo is best.
Exception: It is very handy to use import * when experimenting at the interactive interpreter. Note however that it doesn't play well with reload(), so don't use it when debugging changing code. (To debug a module you are writing, it's best to launch a fresh interpreter inside the module's namespace - python -i mymodule.py / F5 in IDLE.)
As far as I know, there is no way to import a value from a module and have it readable and writable by the importing scope. When you just import foo in Python, it creates a module object named foo. Getting and setting attributes on a module object will change them in the module's scope. But when you from foo import something, foo is imported and a module object is created, but is not returned. Instead, Python copies the values you specified out of foo and puts them in the local scope. If what you are importing is an immutable type like int or str, then changing it and having the changes reflect in the foo module is impossible. It's similar to this:
>>> class N(object):
... def __init__(self, value):
... self.value = value
>>> n = N(3)
>>> value = n.value
>>> print value, n.value
3 3
>>> value = 4
>>> print value, n.value
4 3
Excepting crude hacks, if you really want to be able to modify the module's variable, you will need to import the module itself and modify the variable on the module. But generally, having to do this is indicative of bad design. If you are the writer of the foo module in question, you may want to look at some other, more Pythonic ways to solve your problem.
By using import foo from bar you don't import bar as a variable but as a constant.
from foo import * is frowned upon (by me, by Google's style guide, by the OLPC style guide - which you should see, as it has the best explanations of why this is bad - but not by PEP-8, unfortunately). - it makes for unreadable code.
Consider:
from foo import *
from bar import *
from baz import *
dostuff()
If you have an error running dostuff(), where do you look for the problem? It could have come from any of those imports.
For readable, maintainable code, stick with from foo import bar. For readable, modular, maintainable code, don't hack with globals - extend bar (by subclassing, if you can't change the upstream source) to expose methods for modifying the values you need to access.