We Keep Coding

How to make database cursor global? [duplicate]

I want to make a single database object available across many python modules.
For a related example, I create globl.py:
DOCS_ROOT="c:\docs" ## as an example
SOLR_BASE="http://localhost:8636/solr/"
Any other module which needs it can do a
from globl import DOCS_ROOT
Now this example aside, I want to do the same thing with database connection objects, share them across many modules.
import MySQLdb
conn = MySQLdb.connect (host="localhost"...)
cursor = conn.cursor()
I tried this on the interpreter:
from globl import cursor
and it seems to work. But I suspect that this will cause the same module to be executed each time one imports from it. So is this the proper way?

Even if the import doesn't run the code multiple times, this is definitely not the correct way.
You should instead hide the process of obtaining a connection or cursor behind a function. You can then implement this function using either a Singleton or Object Pool design pattern.
So it would be something like this:
db.py:
_connection = None
def get_connection():
global _connection
if not _connection:
_connection = MySQLdb.connect(host="localhost"...)
return _connection
# List of stuff accessible to importers of this module. Just in case
__all__ = [ 'getConnection' ]
## Edit: actually you can still refer to db._connection
## if you know that's the name of the variable.
## It's just left out from enumeration if you inspect the module
someothermodule.py:
import db
conn = db.get_connection() # This will always return the same object
By the way, depending on what you are doing, it may not be so much of a good idea to share
your connection object rather than create a new one every time you need one.
But, that's why you'd want to write a get_connection() method, to abstract from these issues in the rest of your code.

You suspect wrongly. The code will only be executed once - subsequent imports just refer to the module via sys.modules, and don't re-run it.
(Note that this is the case as long as you always use the same path to import the module - if you do from globl import cursor in one place, and from my.fullyqualified.project.global import cursor in another, you probably will find the code is re-executed.)
Edit to add as S.Lott says in the comment, this is a perfectly good way to handle a global object.

I think Daniel already answered the question, while I'd like to add few comments about the cursor object you want to share.
It is generally not a good idea to share the cursor object that way. Certainly it depends on what your program is, but as a general solution I'd recommend you to hide this cursor object behind a "factory" producing cursors. Basically you can create a method cursor() or get_cursor() instead of making the cursor a global variable. The major benefit (but not the only one) - you can hide a more complex logic behind this "factory" - pooling, automatic re-connection in case the connection is dropped, etc. Even if you don't need it right away - it will be very easy to add it later if you start using this approach now, and while for now you can keep this function implementation as simple as return _cursor.
And yes, still, the module itself will be imported once only.

How do I know when I can/should use `with` keyword?

In C#, when an object implements IDisposable, using should be used to guarantee that resources will be cleaned if an exception is thrown. For instance, instead of:
var connection = new SqlConnection(...);
...
connection.Close();
one needs to write:
using (var connection = new SqlConnection(...))
{
...
}
Therefore, just by looking at the signature of the class, I know exactly whether or not I should initialize the object inside a using.
In Python 3, a similar construct is with. Similarly to C#, it ensures that the resources will be cleaned up automatically when exiting the with context, even if a error is raised.
However, I'm not sure how should I determine whether with should be used or not for a specific class. For instance, an example from psycopg doesn't use with, which may mean that:
I shouldn't either, or:
The example is written for Python 2, or:
The authors of the documentation were unaware of with syntax, or:
The authors decided not to handle exceptional cases for the sake of simplicity.
In general, how should I determine whether with should be used when initializing an instance of a specific class (assuming that documentation says nothing on the subject, and that I have access to source code)?

Regarding when you should use it:
No one forces you to use the with statement, it's just syntactic sugar that's there to make your life easier. If you use it or not is totally up to you but, it is generally recommended to do so. (We're forgetful and with ... looks ways better than explicit initialize resource/finalize recourse calls).
When you can use it:
When you can use it boils down to examining if it defines the context manager protocol. This could be as simple as trying to use with and seeing that it fails :-)
If you dynamically need to check if an object is a context manager, you have two options.
First, wait for the stable release of Python 3.6 which defines an ABC for context managers, ContextManager, which can be used in issubclass/isinstance checks:
>>> from typing import ContextManager
>>> class foo:
... def __enter__(self): pass
... def __exit__(self): pass
...
>>> isinstance(foo(), ContextManager)
True
>>> class foo2: pass
...
>>> isinstance(foo2(), ContextManager)
False
Or, create your own little function to check for it:
def iscontext(inst):
cls = type(inst)
return (any("__enter__" in vars(a) for a in cls.__mro__) and
any("__exit__" in vars(a) for a in cls.__mro__))
As a final note, the with statement is present in Python 2 and in 3, the use case you saw probably just wasn't aware of it :-).

with is for use with context managers.
At the code level, a context manager must define two methods:
__enter__(self)
__exit__(self, type, value, traceback).
Be aware that there are class decorators which can turn otherwise simple classes/functions into context managers - see contextlib for some examples

You should use with whenever you need to perform some similar action before and after executing the statement. For example:
Want to execute SQL query? You need to open and close the connections safely.Use with.
Want to perform some action on file? You have to open and close the file safely. Use with
Want to store some data in temporary file to perform some task? You need to create the directory, and clean it up once you are done. Use with, and so on. . .
Everything you want to perform before the query execution, add it to the __enter__() method. And the action to be performed after, add it to the __exit__() method.
One of the nice thing about with is, __exit__ is executed even if the code within with raises any Exception

Meaning of "with" statement without "as" keyword

I'm familiar with using python's with statement as a means of ensuring finalization of an object in the event of an exception being thrown. This usually looks like
with file.open('myfile.txt') as f:
do stuff...
which is short-hand for
f = file.open('myfile.txt'):
try:
do stuff...
finally:
f.close()
or whatever other finalization routine a class may present.
I recently came across a piece of code dealing with OpenGL that presented this:
with self.shader:
(Many OpenGL commands)
Note that absence of any as keyword. Does this indicate that the __enter__ and __exit__ methods of the class are still to be called, but that the object is never explicitly used in the block (i.e., it works through globals or implicit references)? Or is there some other meaning that is eluding me?

The context manager can optionally return an object, to be assigned to the identifier named by as. And it is the object returned by the __enter__ method that is assigned by as, not necessarily the context manager itself.
Using as <identifier> helps when you create a new object, like the open() call does, but not all context managers are created just for the context. They can be reusable and have already been created, for example.
Take a database connection. You create the database connection just once, but many database adapters let you use the connection as a context manager; enter the context and a transaction is started, exit it and the transaction is either committed (on success), or rolled back (when there is an exception):
with db_connection:
# do something to the database
No new objects need to be created here, the context is entered with db_connection.__enter__() and exited again with db_connection.__exit__(), but we already have a reference to the connection object.
Now, it could be that the connection object produces a cursor object when you enter. Now it makes sense to assign that cursor object in a local name:
with db_connection as cursor:
# use cursor to make changes to the database
db_connection still wasn't called here, it already existed before, and we already have a reference to it. But whatever db_connection.__enter__() produced is now assigned to cursor and can be used from there on out.
This is what happens with file objects; open() returns a file object, and fileobject.__enter__() returns the file object itself, so you can use the open() call in a with statement and assign a reference to the newly created object in one step, rather than two. Without that little trick, you'd have to use:
f = open('myfile.txt')
with f:
# use `f` in the block
Applying all this to your shader example; you already have a reference to self.shader. It is quite probable that self.shader.__enter__() returns a reference to self.shader again, but since you already have a perfectly serviceable reference, why create a new local for that?

The above answer is nicely put.
The only thing I kept asking myself while reading it, is where is the confirmation of the following scenario. In the event there is an assignment in the body of the context of the with statement, anything on the right side of the assignment is first "bound" to the context. So, in the following:
with db_connection():
result = select(...)
... select is ~ ref_to_connection.select(...)
I put this here for anyone like me who comes and goes between languages and might benefit by a quick reminder of how to read and track the refs here.

What does the "as" statement mean in python?

I'm just starting to learn the Flask framework and was wondering what the 'as' statement does? It's used in conjunction with an 'with' statement.
Here's the example:
def init_db():
with closing (connect_db()) as db:
with app.open_resource('schema.sql', mode='r') as f:
db.cursor().executescript(f.read())
db.commit

The as keyword is used to add clauses to a few different statements (e.g., import); there is no "as statement".
In the with statement, it means that the value of the with context gets assigned to that variable. The precise explanation is in the docs under The with statement, With Statement Context Managers, and Context Manager Types; PEP 343 gives a more readable explanation (although it's also a little out of date).
In simple cases, where an object acts as its own context manager, as a file does, or decimal.localcontext, the object gets assigned to the variable. Here, f is the file returned by open('spam'):
with open('spam') as f:
In slightly more complex cases, a context manager provides some other object that gets assigned to the variable. In the case of closing(foo), the object is the foo that it was given in the first place. So here, g ends up being the same thing as f, even though closing(f) is not the same thing:
f = open('spam')
with closing(f) as g:
Some context managers don't provide any object at all. In that case, as you'd expect, as f will assign f to None, and you usually won't have nay good reason to use it. So, the as clause is optional. For example, using a threading.Lock:
with my_lock:
If you're building context managers from scratch, the way you provide an object (whether self or otherwise) to bind to the as target is by returning it from the __enter__ method. Or, if you're building them with the #contextmanager decorator around a generator, you do it by yielding the object.

What is a "runtime context"?

(Edited for even more clarity)
I'm reading the Python book (Python Essential Reference by Beazley) and he says:
The with statement allows a series of statements to execute inside a
runtime context that is controlled by an object that serves as a context manager.
Here is an example:
with open("debuglog","a") as f:
f.write("Debugging\n")
statements
f.write("Done\n")
He goes on to say:
The with obj statement accepts an optional as var specifier. If given, the value
returned by obj._ enter _() is placed into var. It is important to emphasize
that obj is not necessarily the value assigned to var.
I understand the mechanics of what a 'with' keyword does: a file-object is returned by open and that object is accessible via f within the body of the block. I also understand that enter() and eventually exit() will be called.
But what exactly is a run-time context? A few low level details would be nice - or, an example in C. Could someone clarify what exactly a "context" is and how it might relate to other languages (C, C++). My understanding of a context was the environment eg: a Bash shell executes ls in the context of all the (env displayed) shell variables.
With the with keyword - yes f is accessible to the body of the block but isn't that just scoping? eg: for x in y: here x is not scoped within the block and retains it's value outside the block - is this what Beazley means when he talks about 'runtime context', that f is scoped only within the block and looses all significance outside the with-block?? Why does he say that the statements "execute inside a runtime context"??? Is this like an "eval"??
I understand that open returns an object that is "not ... assigned to var"??
Why isn't it assigned to var? What does Beazley mean by making a statement like that?

The with statement was introduced in PEP 343. This PEP also introduced a new term, "context manager", and defined what that term means.
Briefly, a "context manager" is an object that has special method functions .__enter__() and .__exit__(). The with statement guarantees that the .__enter__() method will be called to set up the block of code indented under the with statement, and also guarantees that the .__exit__() method function will be called at the time of exit from the block of code (no matter how the block is exited; for example, if the code raises an exception, .__exit__() will still be called).
http://www.python.org/dev/peps/pep-0343/
http://docs.python.org/2/reference/datamodel.html?highlight=context%20manager#with-statement-context-managers
The with statement is now the preferred way to handle any task that has a well-defined setup and teardown. Working with a file, for example:
with open(file_name) as f:
# do something with file
You know the file will be properly closed when you are done.
Another great example is a resource lock:
with acquire_lock(my_lock):
# do something
You know the code won't run until you get the lock, and as soon as the code is done the lock will be released. I don't often do multithreaded coding in Python, but when I did, this statement made sure that the lock was always released, even in the face of an exception.
P.S. I did a Google search online for examples of context managers and I found this nifty one: a context manager that executes a Python block in a specific directory.
http://ralsina.me/weblog/posts/BB963.html
EDIT:
The runtime context is the environment that is set up by the call to .__enter__() and torn down by the call to .__exit__(). In my example of acquiring a lock, the block of code runs in the context of having a lock available. In the example of reading a file, the block of code runs in the context of the file being open.
There isn't any secret magic inside Python for this. There is no special scoping, no internal stack, and nothing special in the parser. You simply write two method functions, .__enter__() and .__exit__() and Python calls them at specific points for your with statement.
Look again at this section from the PEP:
Remember, PEP 310 proposes roughly this syntax (the "VAR =" part is optional):
with VAR = EXPR:
BLOCK
which roughly translates into this:
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
In both examples, BLOCK is a block of code that runs in a specific runtime context that is set up by the call to VAR.__enter__() and torn down by VAR.__exit__().
There are two main benefits to the with statement and the way it is all set up.
The more concrete benefit is that it's "syntactic sugar". I would much rather write a two-line with statement than a six-line sequence of statements; it's easier two write the shorter one, it looks nicer and is easier to understand, and it is easier to get right. Six lines versus two means more chances to screw things up. (And before the with statement, I was usually sloppy about wrapping file I/O in a try block; I only did it sometimes. Now I always use with and always get the exception handling.)
The more abstract benefit is that this gives us a new way to think about designing our programs. Raymond Hettinger, in a talk at PyCon 2013, put it this way: when we are writing programs we look for common parts that we can factor out into functions. If we have code like this:
A
B
C
D
E
F
B
C
D
G
we can easily make a function:
def BCD():
B
C
D
A
BCD()
E
F
BCD()
G
But we have never had a really clean way to do this with setup/teardown. When we have a lot of code like this:
A
BCD()
E
A
XYZ()
E
A
PDQ()
E
Now we can define a context manager and rewrite the above:
with contextA:
BCD()
with contextA:
XYZ()
with contextA:
PDQ()
So now we can think about our programs and look for setup/teardown that can be abstracted into a "context manager". Raymond Hettinger showed several new "context manager" recipes he had invented (and I'm racking my brain trying to remember an example or two for you).
EDIT: Okay, I just remembered one. Raymond Hettinger showed a recipe, that will be built in to Python 3.4, for using a with statement to ignore an exception within a block. See it here: https://stackoverflow.com/a/15566001/166949
P.S. I've done my best to give the sense of what he was saying... if I have made any mistake or misstated anything, it's on me and not on him. (And he posts on StackOverflow sometimes so he might just see this and correct me if I've messed anything up.)
EDIT: You've updated the question with more text. I'll answer it specifically as well.
is this what Beazley means when he talks about 'runtime context', that f is scoped only within the block and looses all significance outside the with-block?? Why does he say that the statements "execute inside a runtime context"??? Is this like an "eval"??
Actually, f is not scoped only within the block. When you bind a name using the as keyword in a with statement, the name remains bound after the block.
The "runtime context" is an informal concept and it means "the state set up by the .__enter__() method function call and torn down by the .__exit__() method function call." Again, I think the best example is the one about getting a lock before the code runs. The block of code runs in the "context" of having the lock.
I understand that open returns an object that is "not ... assigned to var"?? Why isn't it assigned to var? What does Beazley mean by making a statement like that?
Okay, suppose we have an object, let's call it k. k implements a "context manager", which means that it has method functions k.__enter__() and k.__exit__(). Now we do this:
with k as x:
# do something
What David Beazley wants you to know is that x will not necessarily be bound to k. x will be bound to whatever k.__enter__() returns. k.__enter__() is free to return a reference to k itself, but is also free to return something else. In this case:
with open(some_file) as f:
# do something
The call to open() returns an open file object, which works as a context manager, and its .__enter__() method function really does just return a reference to itself.
I think most context managers return a reference to self. Since it's an object it can have any number of member variables, so it can return any number of values in a convenient way. But it isn't required.
For example, there could be a context manager that starts a daemon running in the .__enter__() function, and returns the process ID number of the daemon from the .__enter__() function. Then the .__exit__() function would shut down the daemon. Usage:
with start_daemon("parrot") as pid:
print("Parrot daemon running as PID {}".format(pid))
daemon = lookup_daemon_by_pid(pid)
daemon.send_message("test")
But you could just as well return the context manager object itself with any values you need tucked inside:
with start_daemon("parrot") as daemon:
print("Parrot daemon running as PID {}".format(daemon.pid))
daemon.send_message("test")
If we need the PID of the daemon, we can just put it in a .pid member of the object. And later if we need something else we can just tuck that in there as well.

The with context takes care that on entry, the __enter__ method is called and the given var is set to whatever __enter__ returns.
In most cases, that is the object which is worked on previously - in the file case, it is - but e.g. on a database, not the connection object, but a cursor object is returned.
The file example can be extended like this:
f1 = open("debuglog","a")
with f1 as f2:
print f1 is f2
which will print True as here, the file object is returned by __enter__. (From its point of view, self.)
A database works like
d = connect(...)
with d as c:
print d is c # False
print d, c
Here, d and c are completely different: d is the connection to the database, c is a cursor used for one transaction.
The with clause is terminated by a call to __exit__() which is given the state of execution of the clause - either success or failure. In this case, the __exit__() method can act appropriately.
In the file example, the file is closed no matter if there was an error or not.
In the database example, normally the transaction is committed on success and rolled back on failure.
The context manager is for easy initialisation and cleanup of things like exactly these - files, databases etc.
There is no direct correspondence in C or C++ that I am aware of.
C knows no concept of exception, so none can be caught in a __exit__(). C++ knows exceptions, and there seems to be ways to do soo (look below at the comments).