We Keep Coding

'import' and 'from import' behaves differently

As per my understanding, the difference between 'import' and 'from import' in Python is: 'import' imports the whole library, 'from import' imports a specific member or members of the library, and there should not be any behavioral difference.
As per this, I was expecting both the test1.py and test2.py to show the same result i.e. "Lucky". But it is different for test2.py. Can anyone explain why?
mymodule.py
message = "Happy"
def set_message(msg):
global message
message = msg
test1.py
import mymodule
mymodule.set_message("Lucky")
print(mymodule.message) #output is Lucky
test2.py
from mymodule import *
set_message("Lucky")
print(message) #output is Happy

'import' imports the whole library, 'from import' imports a specific member or members of the library, and there should not be any behavioral difference.
Both versions - assuming the module has not already been imported - cause the top-level code to execute and a module object to be created.
import ... means that the module object is assigned to the corresponding name in the local namespace. (The reason the . syntax works is that the global variables from when the module's code was running, become attributes of that modle object.)
from ... import * means that Python iterates over the attributes of the module object, and assigns each name into the current namespace, similarly.
For subsequent code, the module object itself is the global namespace in which its functions run.
Your test2.py calls a method from mymodule, causing it to do a global lookup of message. That lookup finds the message attribute of the module object, and replaces it. The message global variable in your own code is unchanged, because neither was that name reassigned, nor was the value modified (it was replaced). It is the same as if you do:
# Functions are the easiest way to get an object with mutable attributes
def namespace(): pass
namespace.a = 3
a = namespace.a # "import" the name
namespace.a = 4 # replace the value; `a` does not change

accessing and changing module level variable [duplicate]

I've run into a bit of a wall importing modules in a Python script. I'll do my best to describe the error, why I run into it, and why I'm tying this particular approach to solve my problem (which I will describe in a second):
Let's suppose I have a module in which I've defined some utility functions/classes, which refer to entities defined in the namespace into which this auxiliary module will be imported (let "a" be such an entity):
module1:
def f():
print a
And then I have the main program, where "a" is defined, into which I want to import those utilities:
import module1
a=3
module1.f()
Executing the program will trigger the following error:
Traceback (most recent call last):
File "Z:\Python\main.py", line 10, in <module>
module1.f()
File "Z:\Python\module1.py", line 3, in f
print a
NameError: global name 'a' is not defined
Similar questions have been asked in the past (two days ago, d'uh) and several solutions have been suggested, however I don't really think these fit my requirements. Here's my particular context:
I'm trying to make a Python program which connects to a MySQL database server and displays/modifies data with a GUI. For cleanliness sake, I've defined the bunch of auxiliary/utility MySQL-related functions in a separate file. However they all have a common variable, which I had originally defined inside the utilities module, and which is the cursor object from MySQLdb module.
I later realised that the cursor object (which is used to communicate with the db server) should be defined in the main module, so that both the main module and anything that is imported into it can access that object.
End result would be something like this:
utilities_module.py:
def utility_1(args):
code which references a variable named "cur"
def utility_n(args):
etcetera
And my main module:
program.py:
import MySQLdb, Tkinter
db=MySQLdb.connect(#blahblah) ; cur=db.cursor() #cur is defined!
from utilities_module import *
And then, as soon as I try to call any of the utilities functions, it triggers the aforementioned "global name not defined" error.
A particular suggestion was to have a "from program import cur" statement in the utilities file, such as this:
utilities_module.py:
from program import cur
#rest of function definitions
program.py:
import Tkinter, MySQLdb
db=MySQLdb.connect(#blahblah) ; cur=db.cursor() #cur is defined!
from utilities_module import *
But that's cyclic import or something like that and, bottom line, it crashes too. So my question is:
How in hell can I make the "cur" object, defined in the main module, visible to those auxiliary functions which are imported into it?
Thanks for your time and my deepest apologies if the solution has been posted elsewhere. I just can't find the answer myself and I've got no more tricks in my book.

Globals in Python are global to a module, not across all modules. (Many people are confused by this, because in, say, C, a global is the same across all implementation files unless you explicitly make it static.)
There are different ways to solve this, depending on your actual use case.
Before even going down this path, ask yourself whether this really needs to be global. Maybe you really want a class, with f as an instance method, rather than just a free function? Then you could do something like this:
import module1
thingy1 = module1.Thingy(a=3)
thingy1.f()
If you really do want a global, but it's just there to be used by module1, set it in that module.
import module1
module1.a=3
module1.f()
On the other hand, if a is shared by a whole lot of modules, put it somewhere else, and have everyone import it:
import shared_stuff
import module1
shared_stuff.a = 3
module1.f()
… and, in module1.py:
import shared_stuff
def f():
print shared_stuff.a
Don't use a from import unless the variable is intended to be a constant. from shared_stuff import a would create a new a variable initialized to whatever shared_stuff.a referred to at the time of the import, and this new a variable would not be affected by assignments to shared_stuff.a.
Or, in the rare case that you really do need it to be truly global everywhere, like a builtin, add it to the builtin module. The exact details differ between Python 2.x and 3.x. In 3.x, it works like this:
import builtins
import module1
builtins.a = 3
module1.f()

As a workaround, you could consider setting environment variables in the outer layer, like this.
main.py:
import os
os.environ['MYVAL'] = str(myintvariable)
mymodule.py:
import os
myval = None
if 'MYVAL' in os.environ:
myval = os.environ['MYVAL']
As an extra precaution, handle the case when MYVAL is not defined inside the module.

This post is just an observation for Python behaviour I encountered. Maybe the advices you read above don't work for you if you made the same thing I did below.
Namely, I have a module which contains global/shared variables (as suggested above):
#sharedstuff.py
globaltimes_randomnode=[]
globalist_randomnode=[]
Then I had the main module which imports the shared stuff with:
import sharedstuff as shared
and some other modules that actually populated these arrays. These are called by the main module. When exiting these other modules I can clearly see that the arrays are populated. But when reading them back in the main module, they were empty. This was rather strange for me (well, I am new to Python). However, when I change the way I import the sharedstuff.py in the main module to:
from globals import *
it worked (the arrays were populated).
Just sayin'

A function uses the globals of the module it's defined in. Instead of setting a = 3, for example, you should be setting module1.a = 3. So, if you want cur available as a global in utilities_module, set utilities_module.cur.
A better solution: don't use globals. Pass the variables you need into the functions that need it, or create a class to bundle all the data together, and pass it when initializing the instance.

The easiest solution to this particular problem would have been to add another function within the module that would have stored the cursor in a variable global to the module. Then all the other functions could use it as well.
module1:
cursor = None
def setCursor(cur):
global cursor
cursor = cur
def method(some, args):
global cursor
do_stuff(cursor, some, args)
main program:
import module1
cursor = get_a_cursor()
module1.setCursor(cursor)
module1.method()

Since globals are module specific, you can add the following function to all imported modules, and then use it to:
Add singular variables (in dictionary format) as globals for those
Transfer your main module globals to it
.
addglobals = lambda x: globals().update(x)
Then all you need to pass on current globals is:
import module
module.addglobals(globals())

Since I haven't seen it in the answers above, I thought I would add my simple workaround, which is just to add a global_dict argument to the function requiring the calling module's globals, and then pass the dict into the function when calling; e.g:
# external_module
def imported_function(global_dict=None):
print(global_dict["a"])
# calling_module
a = 12
from external_module import imported_function
imported_function(global_dict=globals())
>>> 12

The OOP way of doing this would be to make your module a class instead of a set of unbound methods. Then you could use __init__ or a setter method to set the variables from the caller for use in the module methods.

Update
To test the theory, I created a module and put it on pypi. It all worked perfectly.
pip install superglobals
Short answer
This works fine in Python 2 or 3:
import inspect
def superglobals():
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals
save as superglobals.py and employ in another module thusly:
from superglobals import *
superglobals()['var'] = value
Extended Answer
You can add some extra functions to make things more attractive.
def superglobals():
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals
def getglobal(key, default=None):
"""
getglobal(key[, default]) -> value
Return the value for key if key is in the global dictionary, else default.
"""
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals.get(key, default)
def setglobal(key, value):
_globals = superglobals()
_globals[key] = value
def defaultglobal(key, value):
"""
defaultglobal(key, value)
Set the value of global variable `key` if it is not otherwise st
"""
_globals = superglobals()
if key not in _globals:
_globals[key] = value
Then use thusly:
from superglobals import *
setglobal('test', 123)
defaultglobal('test', 456)
assert(getglobal('test') == 123)
Justification
The "python purity league" answers that litter this question are perfectly correct, but in some environments (such as IDAPython) which is basically single threaded with a large globally instantiated API, it just doesn't matter as much.
It's still bad form and a bad practice to encourage, but sometimes it's just easier. Especially when the code you are writing isn't going to have a very long life.

Variables set before importing a python module aren't recognized by module

Here's my code:
import numpy as np
import matplotlib.pyplot as plt
import astropy
import matplotlib
%matplotlib ipympl
import scatterplot_with_hist as sc
badx=[]
bady=[]
import badcomp as bc
#things like data5 and list2 are defined in here--I know that code is functional so I'll omit it for brevity
bc.getlist(start = 2000, end = 2200)
The module code is as follows:
def getlist(start, end):
for f in range(1):
for i in range(1238):
for n in range(int(start),int(end)):
if ((data[n]['col1'] - list2[i]) == 0):
badx.append(data[n]['col2'])
bady.append(data[n]['col3'])
If I run this code in the regular space (instead of importing it and running it as a function) it works fine. When I run it as an imported function, it won't recognize variables like data5, list2, and badx and bady.
Why?

Each Python module has it's own global namespace. That means that code in different modules that each try to access global variables will see separate ones. You can access another module's global variables by importing the module and interacting with the attributes of the module object.
In your code, the getlist function in the badcomp module is trying to interact with several global variables, including badx and bady for output, and data and list2 for input. It's not working because you've defined those in the interactive session, which uses the namespace of a module with the special name __main__.
While you could import __main__ from badcomp and interact with the global variables defined there via the module's attributes, that would be a really bad design, since it won't work if the module gets imported in any other way (e.g. by a different module you write later). Instead, the function should probably use variables defined in its own global namespace. The __main__ module is already importing badcomp (as bc), and can access things like badx and bady as bc.badx and bc.bady if the definitions are moved into the module.
Or you might reconsider if global variables are the best way for this function to work. It's often much better to use arguments and return values to pass data in and out of a function, rather than global variables. Maybe badx and bady should be defined within getlist and returned at the end. Meanwhile, data and list2 could be added as arguments to the function.

When a module is imported, it does NOT have access to the global or local namespace of the module that called it. You can get around this by creating a function that creates a variable in the global namespace inside the imported module and run the function from the calling module with each of the variables you need.
Example code (really bad design, but it'll teach you hopefully):
Put THIS in the imported module:
def putVarsInNamespace(variable, variableNameToInject)
exec("global %s" % variableName)
exec("%s = variable" % variableName)
Put THIS in the calling module:
test = 5
from <MODULENAME> import putVarsInNamespace
putVarsInNamespace(test, "test")
How this works: variableNameToInject is the name that you want the injected variable to be called. It then runs global variableNameToInject but it uses the VALUE of variableNameToInject which is the name that the injected variable should be called. This is useful when you want to inject multiple variables without using multiple functions. It then sets the variable name (the value of variableNameToInject) to the value of variable, and just like that it's injected.

Can't dynamically import using exec from function despite success using interpreter

I'm having some trouble importing some stuff using a function, despite being able to do so in the interpreter.
Imagine there is a file, input.py, in a folder A which in turn is in the same directory as my script. In this file, we define variable 'B'.
B = 5
When I go into the interpreter, the following commands give me the correct value of B
>>> import sys
>>> sys.path.append('A')
>>> exec('from inputs import *')
>>> print(B)
Yet if I move that code to a seperate file, say 'test.py':
import sys
def import_stuff(import_dir):
sys.path.append(import_dir)
exec('from inputs import *')
print(B)
Then call it from the interpreter like so:
>>> import test
>>> test.import_stuff('A')
I get a NameError and B is not found. What's going on?

Local variables in functions are treated differently than global variables and object attributes (which both use dictionaries to map names to values).
When a function is defined, the Python compiler examines its code and makes a note of which local variable names are used, and designates a "slot" for each one. When the function is called, the slots refer to part of the memory in the frame object. Local variable assignments and lookups access the slots by number, not by name. This makes local variable lookups notably faster than global variable and attribute lookups (since indexing a slot is much faster than doing a dict lookup).
When you try to use exec to create local variables, it bypasses the slots. The compiler doesn't know what variables will be created in the exec'd code, so there are no slots allocated for them. This is also the reason Python 3 doesn't allow you to use from module import * inside a function: the names to be imported are not known at the function's compile time (only when it is run and the imported module is loaded) so the compiler can't set up slots for the names.
Even if you separately initialized local variables for the names you expect to be assigned to in the exec'd code, it still wouldn't work. The exec'd code doesn't know it's being run from within a function, and always wants to write variables to a dictionary (never to function slots). Function's do have a local namespace dictionary that does catch the assignments (they don't become global variables), but using it via the locals() function is very flaky. The values of all of the local variables stored in slots are copied into the dictionary each time you call locals(), but no copying in the other direction ever happens (modifying the dictionary returned from locals() doesn't effect the values of variables stored in slots and accessed the normal way).
That brings me to what I think is the best way to work around this issue. Rather than having exec modify the current namespace (which occurs in a flaky way if you're in a function), you should explicitly pass in a dictionary for it to use as its namespace.
def import_stuff(import_dir):
sys.path.append(import_dir)
namespace = {} # create a namespace dict
exec('from inputs import *', namespace) # pass it to exec
print(namespace["B"]) # read the results from the namespace

importing with * (asterisk) versus as a namespace in python

I know that its bad form to use import * in python, and I don't plan to make a habit of it. However I recently came across some curious behaviour that I don't understand, and wondered if someone could explain it to me.
Lets say I have three python scripts. The first, first_script.py, comprises:
MESSAGE = 'this is from the first script'
def print_message():
print MESSAGE
if __name__ == '__main__':
print_message()
Obviously running this script gives me the contents of MESSAGE. I have a second script called second_script.py, comprising:
import first_script
first_script.MESSAGE = 'this is from the second script'
if __name__ == '__main__':
first_script.print_message()
The behaviour (prints this is from the second script) makes sense to me. I've imported first_script.py, but overwritten a variable within its namespace, so when I call print_message() I get the new contents of that variable.
However, I also have third_script.py, comprising:
from first_script import *
MESSAGE = 'this is from the third script'
if __name__ == '__main__':
print MESSAGE
print_message()
This first line this produces is understandable, but the second doesn't make sense to me. My intuition was that because I've imported into my main namespace via * in the first line, I have a global variable
called MESSAGES. Then in the second line I overwrite MESSAGES. Why then does the function (imported from the first script) produce the OLD output, especially given the output of second_script.py. Any ideas?

import module, from module import smth and from module import *can have different use cases.
The simpler:
import tools
loads the tools module and adds a reference to it in the local namespace (also named tools). After that you can access any of the tools references by prepending tools to them like for example tools.var1
Variant:
import tools as sloot
Does exactly the same, but you use the alias to access the references from the module (eg: sloot.var1). It is mainly used for module having well known aliases like import numpy as np.
The other way
from tools import foo
directly imports some symbols from the tools module in the current namespace. That means that you can only use the specified symbols by they do not need to be qualified. A nice use case is when you could import a symbol from different modules giving same functionalities. For example
try:
from mod1 import foo
except ImportError:
from mod2 import foo
...
foo() # actually calls foo from mod1 if available else foo from mod2
This is commonly use as a portability trick.
The danger:
from tools import *
It is a common idiom, but may not do what you expect if the module does not document it. In fact, it imports all the public symbols from the module, by default all the symbols that have no initial _ which can contain unwanted things. In addition, a module can declare a special variable __all__ which is assumed to declare the public interface, and in that case only the symbols contained in __all__ will be imported.
Example:
mod.py
__all__ = ['foo', 'bar']
def baz(x):
return x * 2
def foo():
return baz('FOO')
def bar():
return baz('BAR')
You can use (assuming mod.py is accessible)
from mod import *
print(foo()) # should print FOOFOO
# ERROR HERE
x = baz("test") # will choke with NameError: baz is not defined
while
import mod
print(mod.baz("test")) # will display as expected testtest
So you should only use from tools import * if the documentation of the tools module declares it to be safe and lists the actually imported symbols.

This has to do with Scope. For a very excellent description of this, please see Short Description of the Scoping Rules?
For a detailed breakdown with tons of examples, see http://nbviewer.ipython.org/github/rasbt/python_reference/blob/master/tutorials/scope_resolution_legb_rule.ipynb
Here's the details on your specific case:
The print_message function being called from your third test file is being asked to print out some MESSAGE object. This function will use the standard LEGB resolution order to identify which object this refers to. LEGB refers to Local, Enclosing function locals, Global, Builtins.
Local - Here, there is no MESSAGES defined within the print_message function.
Enclosing function locals - There are no functions wrapping this function, so this is skipped.
Global - Any explicitly declared variables in the outer code. It finds MESSAGE defined in the global scope of the first_script module. Resolution then stops, but i'll include the others for completeness.
Built-ins - The list of python built-ins, found here.
So, you can see that resolution of the variable MESSAGE will cease immediately in Global, since there was something defined there.
Another resource that was pointed out to me for this is Lexical scope vs Dynamic scope, which may help you understand scope better.
HTH

Direct assignment changes the reference of an object, but modification does not. For example,
a = []
print(id(a))
a = [0]
print(id(a))
prints two different IDs, but
a = []
print(id(a))
a.append(0)
print(id(a))
prints the same ID.
In second_script.py, the assignment merely modifies first_script, which is why both first_script.py and second_script.py can locate the same attribute MESSAGE of first_script. In third_script.py, the direct assignment changes the reference of MESSAGE; therefore, after the assignment, the variable MESSAGE in third_script.py is a different variable from MESSAGE in first_script.py.
Consider the related example:
first_script.py
MESSAGE = ['this is from the first script']
def print_message():
print(MESSAGE)
if __name__ == '__main__':
print_message()
third_script.py
from first_script import *
MESSAGE.append('this is from the third script')
if __name__ == '__main__':
print(MESSAGE)
print_message()
In this case, third_script.py prints two identical messages, demonstrating that names imported by import * can still be modified.