From How to override python builtins with an import statement I obtained the following code:
from __future__ import print_function
def print(*args, **kwargs):
.
. some intervening logic
.
return __builtins__.print(*args, **kwargs)
This code works fine, but has module scope. That is, if there are print statements within this file they work as expected, going through the print() function as defined. However, importing it (from foo import *) has no effect within the module that imported it.
If I want to override the print function "globally" how is this best done. Ideally:
from MyOverrides import *
.
.
.
class foo():
.
.
def bar( self ):
print( "this is my overridden print statement" )
What am I missing about future, overrides, and print() here?
My environment is 2.6 and forward, but not 3.0.
You can't (and shouldn't) have a global override that turns on a future statement. Since future statements affect the way a Python source file is parsed and compiled, before Python can even tell that MyOverrides defines a print function, all future statements a module uses must be explicit. You can't import a future statement from another module.
The safest way to replace the print function is to explicitly use from __future__ import print_function in all your modules, then import a replacement for print from one of your modules. Don't replace __builtin__.print.
If you want to affect all output to stdout, whether by print statement or print function, anywhere at all, you can replace sys.stdout with a wrapper.
import sys
stdout = sys.stdout
class new_stdout(object):
def write(*args, **kwargs):
# do whatever
stdout.write(*args, **kwargs)
sys.stdout = new_stdout()
However, this is probably a bad idea, and it won't affect print statements or print functions that use an explicit file argument.
Another stackoverflow user provided most of the answer, but then apparently deleted it (?). Here is a working solution. Once again, I recognize this isn't necessarily a best practice, but it can be handy in certain situations.
Module MyOverrides.py:
from __future__ import print_function
import __builtin__
builtin_print = __builtin__.print
def print(*args, **kwargs):
.
.... whatever code you need here
.
return builtin_print(*args, **kwargs)
__builtin__.print = print
Module test.py:
from __future__ import print_function
from MyOverrides import *
.
.
.
As pointed out by another user, future import needs to happen in every module that intends to use future functionality.
Thanks to user #kindall who answered and then apparently withdrew the answer.
Related
I've run into a bit of a wall importing modules in a Python script. I'll do my best to describe the error, why I run into it, and why I'm tying this particular approach to solve my problem (which I will describe in a second):
Let's suppose I have a module in which I've defined some utility functions/classes, which refer to entities defined in the namespace into which this auxiliary module will be imported (let "a" be such an entity):
module1:
def f():
print a
And then I have the main program, where "a" is defined, into which I want to import those utilities:
import module1
a=3
module1.f()
Executing the program will trigger the following error:
Traceback (most recent call last):
File "Z:\Python\main.py", line 10, in <module>
module1.f()
File "Z:\Python\module1.py", line 3, in f
print a
NameError: global name 'a' is not defined
Similar questions have been asked in the past (two days ago, d'uh) and several solutions have been suggested, however I don't really think these fit my requirements. Here's my particular context:
I'm trying to make a Python program which connects to a MySQL database server and displays/modifies data with a GUI. For cleanliness sake, I've defined the bunch of auxiliary/utility MySQL-related functions in a separate file. However they all have a common variable, which I had originally defined inside the utilities module, and which is the cursor object from MySQLdb module.
I later realised that the cursor object (which is used to communicate with the db server) should be defined in the main module, so that both the main module and anything that is imported into it can access that object.
End result would be something like this:
utilities_module.py:
def utility_1(args):
code which references a variable named "cur"
def utility_n(args):
etcetera
And my main module:
program.py:
import MySQLdb, Tkinter
db=MySQLdb.connect(#blahblah) ; cur=db.cursor() #cur is defined!
from utilities_module import *
And then, as soon as I try to call any of the utilities functions, it triggers the aforementioned "global name not defined" error.
A particular suggestion was to have a "from program import cur" statement in the utilities file, such as this:
utilities_module.py:
from program import cur
#rest of function definitions
program.py:
import Tkinter, MySQLdb
db=MySQLdb.connect(#blahblah) ; cur=db.cursor() #cur is defined!
from utilities_module import *
But that's cyclic import or something like that and, bottom line, it crashes too. So my question is:
How in hell can I make the "cur" object, defined in the main module, visible to those auxiliary functions which are imported into it?
Thanks for your time and my deepest apologies if the solution has been posted elsewhere. I just can't find the answer myself and I've got no more tricks in my book.
Globals in Python are global to a module, not across all modules. (Many people are confused by this, because in, say, C, a global is the same across all implementation files unless you explicitly make it static.)
There are different ways to solve this, depending on your actual use case.
Before even going down this path, ask yourself whether this really needs to be global. Maybe you really want a class, with f as an instance method, rather than just a free function? Then you could do something like this:
import module1
thingy1 = module1.Thingy(a=3)
thingy1.f()
If you really do want a global, but it's just there to be used by module1, set it in that module.
import module1
module1.a=3
module1.f()
On the other hand, if a is shared by a whole lot of modules, put it somewhere else, and have everyone import it:
import shared_stuff
import module1
shared_stuff.a = 3
module1.f()
… and, in module1.py:
import shared_stuff
def f():
print shared_stuff.a
Don't use a from import unless the variable is intended to be a constant. from shared_stuff import a would create a new a variable initialized to whatever shared_stuff.a referred to at the time of the import, and this new a variable would not be affected by assignments to shared_stuff.a.
Or, in the rare case that you really do need it to be truly global everywhere, like a builtin, add it to the builtin module. The exact details differ between Python 2.x and 3.x. In 3.x, it works like this:
import builtins
import module1
builtins.a = 3
module1.f()
As a workaround, you could consider setting environment variables in the outer layer, like this.
main.py:
import os
os.environ['MYVAL'] = str(myintvariable)
mymodule.py:
import os
myval = None
if 'MYVAL' in os.environ:
myval = os.environ['MYVAL']
As an extra precaution, handle the case when MYVAL is not defined inside the module.
This post is just an observation for Python behaviour I encountered. Maybe the advices you read above don't work for you if you made the same thing I did below.
Namely, I have a module which contains global/shared variables (as suggested above):
#sharedstuff.py
globaltimes_randomnode=[]
globalist_randomnode=[]
Then I had the main module which imports the shared stuff with:
import sharedstuff as shared
and some other modules that actually populated these arrays. These are called by the main module. When exiting these other modules I can clearly see that the arrays are populated. But when reading them back in the main module, they were empty. This was rather strange for me (well, I am new to Python). However, when I change the way I import the sharedstuff.py in the main module to:
from globals import *
it worked (the arrays were populated).
Just sayin'
A function uses the globals of the module it's defined in. Instead of setting a = 3, for example, you should be setting module1.a = 3. So, if you want cur available as a global in utilities_module, set utilities_module.cur.
A better solution: don't use globals. Pass the variables you need into the functions that need it, or create a class to bundle all the data together, and pass it when initializing the instance.
The easiest solution to this particular problem would have been to add another function within the module that would have stored the cursor in a variable global to the module. Then all the other functions could use it as well.
module1:
cursor = None
def setCursor(cur):
global cursor
cursor = cur
def method(some, args):
global cursor
do_stuff(cursor, some, args)
main program:
import module1
cursor = get_a_cursor()
module1.setCursor(cursor)
module1.method()
Since globals are module specific, you can add the following function to all imported modules, and then use it to:
Add singular variables (in dictionary format) as globals for those
Transfer your main module globals to it
.
addglobals = lambda x: globals().update(x)
Then all you need to pass on current globals is:
import module
module.addglobals(globals())
Since I haven't seen it in the answers above, I thought I would add my simple workaround, which is just to add a global_dict argument to the function requiring the calling module's globals, and then pass the dict into the function when calling; e.g:
# external_module
def imported_function(global_dict=None):
print(global_dict["a"])
# calling_module
a = 12
from external_module import imported_function
imported_function(global_dict=globals())
>>> 12
The OOP way of doing this would be to make your module a class instead of a set of unbound methods. Then you could use __init__ or a setter method to set the variables from the caller for use in the module methods.
Update
To test the theory, I created a module and put it on pypi. It all worked perfectly.
pip install superglobals
Short answer
This works fine in Python 2 or 3:
import inspect
def superglobals():
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals
save as superglobals.py and employ in another module thusly:
from superglobals import *
superglobals()['var'] = value
Extended Answer
You can add some extra functions to make things more attractive.
def superglobals():
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals
def getglobal(key, default=None):
"""
getglobal(key[, default]) -> value
Return the value for key if key is in the global dictionary, else default.
"""
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals.get(key, default)
def setglobal(key, value):
_globals = superglobals()
_globals[key] = value
def defaultglobal(key, value):
"""
defaultglobal(key, value)
Set the value of global variable `key` if it is not otherwise st
"""
_globals = superglobals()
if key not in _globals:
_globals[key] = value
Then use thusly:
from superglobals import *
setglobal('test', 123)
defaultglobal('test', 456)
assert(getglobal('test') == 123)
Justification
The "python purity league" answers that litter this question are perfectly correct, but in some environments (such as IDAPython) which is basically single threaded with a large globally instantiated API, it just doesn't matter as much.
It's still bad form and a bad practice to encourage, but sometimes it's just easier. Especially when the code you are writing isn't going to have a very long life.
Suppose I have a Python module "main.py":
import math # from the standard Python library
import my_own_module
...
foo = math.cos(bar)
And I also need to import the standard math module in "my_own_module.py":
import math
...
baz = math.sin(qux)
In this case I think import math in "main.py" is redundant and can be omitted.
What's best practice in this case:
Omit import math from "main.py" becuase it's redundant? Or,
Keep import math in "main.py" to clarify that the code in that module requires it?
The reference to math.cos in main.py means that import math is required in main.py, regardless of whether my_own_module.py imports it or not. It is not redundant, and it cannot be omitted (and if you try to omit it, you'll get an error).
import math
does something else than simply including the full text of one file into the other.
It introduces a new namespace with the name math, and this math name will be known in your current namespace.
If you omit the
import math
from your main.py file, your command
foo = math.cos(bar)
becomes illegal, as the math symbol will be not (recognized) in the main.py namespace.
This is not like, eg #include in C++. The import is not optional. Importing a module is required to be able to refer to its contents. This is true for every single file that does it.
A good question. The short answer is yes, if you use a math function in a py file then you need to import the module at the top regardless of how many times its imported elsewhere.
It gets interesting when we throw a thrid file into the mix, lets call this "explanation.py"
And lets suppose that your "main.py" becomes "my_functions.py" and contains a function called foo:
#my_functions.py
import math
import my_own_module
def foo(bar):
return math.cos(bar)
and in my_own_module.py:
#my_own_module.py
import math
def bar(foo):
return math.sin(foo)
and finally explanation.py (new main())
#main.py
import my_functions
import my_own_module
bar = my_functions.foo(10)
foo = my_own_module.bar(10)
print(foo)
print(bar)
Notice how you DO NOT need to add math if you call the functions imported from another file. I hope that might add further clarity to your enquiry :)
However it might be worth noting that this would exclude maths from the current namespace, therefore rendering any further calls to the math functions useless.
I want to restrict the import of some modules in Python 3 (but for the scope of this question let's say that I want to restrict any import). This is the way I am trying to do it:
def __import__(self, *args, **kwrgs):
raise ImportError("Imports are not allowed")
import math
print math.sqrt(4)
It is my understanding that this should raise exception, and it is based on that document - https://docs.python.org/3/reference/import.html#importsystem that says
The search operation of the import statement is defined as a call to the __import__() function, with the appropriate arguments.
However, it does not. I would greatly appreciate an explanation as to why and suggestions how to accomplish my goal
The statement import module_name calls the builtin function __import__ with the parameter 'module_name'. It doesn't, and in fact can't, call module_name.__import__() as you expected, because the module itself hasn't been loaded yet.
A nice way to achieve what you want is to check the __name__ global variable in the non importable module. when it is run, it should be equal to '__main__', but when it is imported, it would be some other string constructed from its name. Therefore you can try:
if __name__ != '__main__':
raise ImportError("Imports are not allowed")
You had a misunderstanding: the docs are not suggesting to define your own __import__ function within the module. What they are trying to say is that when you have an import statement like
import mymodule
then the way in which the name mymodule is resolved here is by calling the built-in function __import__ in this manner:
__import__('mymodule')
If you want to prevent a module from being imported successfully, you can just put any syntax error in there, e.g. putting this as first lines in the file:
# my_unimportable_file.py
error error error
I would like to customize the behavior of my module when it is imported.
For example, let say I want my module to print an incremented number each time another file use import my_module. And when from my_module import some_string is used, it should print "some_string".
How could I do that?
I read several questions here and there but this does not seems to work.
# my_module.py
import sys
class MyImporter:
def find_module(self, module_name, package_path):
print(module_name, package_path)
return self
def load_module(self, module_name):
print(module_name)
return self
sys.meta_path.append(MyImporter())
# file.py
import my_module # Nothing happens
What you're asking for is to have Python work not like Python. Whenever it imports a module it parses and executes the 'opened' code only once so it can pick up the definitions, functions, classes, etc. - every subsequent import of the module just references the cached & parsed first import.
That's why even if you put something like vars()["counter"] = vars().get("counter", 0) + 1 at your module's 'root', the counter will never go above 1 indicating that the module was indeed executed only once. You can force module reload using reload() (or importlib.reload() on Python 3.6+) but then you'd lose your counter if you keep it in the module itself.
Of course, you can have an external counter to be called when your module is imported, but that would have to be a contract with the users of your module at which point the question becomes - can't you just contract your users to call a function to increase your counter whenever they import your module instead of having to reload it for you to capture the count? Reloading a module will also make it have a potentially different state in every context it was reloaded which will make Python behave unexpectedly and should be avoided at any cost.
So, a short answer would be - no, you cannot do that and you should not attempt to do it. If you want something that doesn't work like Python - use something that isn't Python.
However... If you have a really, REALLY good reason to do this (and you don't!) and you don't mind hacking how Python fundamentally behaves (and you should mind) then you might attempt to do this by wrapping the built-in import and checking whenever it gets fired for your module. Something like:
your_module.py:
# HERE BE DRAGONS!!!
import sys
try:
import builtins # Python 3.4+
except ImportError:
import __builtin__ as builtins # Python 2.6+
__builtin_import__ = builtins.__import__ # store a reference to the built-in import
def __custom_import__(name, *args, **kwargs):
# execute builtin first so that the import fails if badly requested
ret = __builtin_import__(name, *args, **kwargs)
if ret is sys.modules[__name__]: # we're trying to load this module
if len(args) > 1 and args[2]: # using the `from your_module import whatever` form
if "some_string" in args[2]: # if some_string is amongst requested properties
print("some_string")
else: # using the `import your_module` form...
print_counter() # increase and print the latest count
return ret # return back the actual import result
builtins.__import__ = __custom_import__ # override the built-in import with our method
counter = 0
# a convinience function, you can do all of this through the `__custom_import__` function
def print_counter():
global counter
counter += 1
print(counter)
print_counter() # call it immediately on the first import to print out the counter
some_string = "I'm holding some string value" # since we want to import this
# HAVE I FORGOT TO TELL YOU NOT TO DO THIS? WELL, DON'T!!!
Keep in mind that this will not account for the first import (be it in the pure import your_module or in the from your_module import whatever form) as the import override won't exist until your module is loaded - that's why it calls print_counter() immediately in hope that the first import of the module was in the form of import your_module and not in the from..import form (if not it will wrongly print out the count instead of some_string the first time). To solve the first-import issue, you can move this 'ovverride' to the __init__.py in the same folder so that the override loads before your module starts and then delegate the counter change / some_string print to the module once loaded, just make sure you do your module name check properly in that case (you need to account for the package as well) and make sure it doesn't automatically execute the counter.
You also, technically, don't need the some_string property at all - by moving the execution of the built-in import around you can do your from..import check first, find the position of some_string in args[2] and pop it before calling the builtin import, then return None in the same position once executed. You can also do your printing and counter incrementing from within the overriden import function.
Again, for the love of all things fluffy and the poor soul who might have to rely on your code one day - please don't do this!
Actually, it does look like it's possible to do what you're looking for in python3.5. It's probably a bad idea, and I've carefully written my code to demonstrate the concept without being polished enough to use as-is, because I'd think carefully before doing something like this in a production project.
If you need to look at a more-or-less production example of this, take a look at the SelfWrapper class in the sh module.
Meanwhile, you can override your own entry in sys.modules to be a subclass of Module. Then you can override getattribute and detect accesses to attributes.
As best I can tell:
Every subsiquent import of the module references spec so you could probably count accesses to spec to count total imports
Each from foo import bar accesses bar as an attribute. I don't think you can distinguish between "from foo import bar" and "import foo; foo.bar"
import sys, types
class Wrapper(types.ModuleType):
def __getattribute__(self, attr):
print(attr)
return super().__getattribute__(attr)
test = "test"
sys.modules[__name__].__class__ = Wrapper
Here is how you can dynamically import modules-
from importlib import import_module
def import_from(module, name):
module = import_module(module, name)
return getattr(module, name)
and use it like this-
funcObj = import_from("<file_name>", "<method_name>")
response = funcObj(arg1,arg2)
Which style is preferable?
Style A:
def foo():
import some_module
some_module.something
Style B:
import some_module
def foo():
some_module.something
Assume that some_module is not used elsewhere in the code, only inside this function.
Indeed, as already noted, it's usually best to follow the PEP 8 recommendation and do your imports at the top. There are some exceptions though. The key to understanding them lies in your embedded question in your second paragraph: "at what stage does the import ... happen?"
Import is actually an executable statement. When you import a module, all the executable statements in the module run. "def" is also an executable statement; its execution causes the defined name to be associated with the (already-compiled) code. So if you have:
def f():
import something
return None
in a module that you import, the (compiled) import and return statements get associated with the name "f" at that point. When you run f(), the import statement there runs.
If you defer importing something that is "very big" or "heavy", and then you never run the function (in this case f), the import never happens. This saves time (and some space as well). Of course, once you actually call f(), the import happens (if it has already happened once Python uses the cached result, but it still has to check) so you lose your time advantage.
Hence, as a rule of thumb, "import everything at the top" until after you have done a lot of profiling and discovered that importing "hugething" is wasting a lot of time in 90% of your runs, vs saving a little time in 10% of them.
PEP 8 recommends that all imports happen at the top of the module. All names, including those bound to modules, are searched for in the local, non-local, global, and built-in scopes, in that order.