We Keep Coding

Why is the Python repl in Visual Studio Code telling me my objects are undefined?

Test.py:
def test():
print("Hello World")
test()
When I run this using the interpreter (ctrl+shift+p > Python: Select Interpreter > target interpreter), it works.
If I then try to run the repl (ctrl+shift+p > Python: Start REPL), I see the repl started in the terminal:
PS C:\Development\personal\python\GettingStarted> & c:/Development/personal/python/GettingStarted/.venv/Scripts/python.exe
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
However, if I try to execute the defined method in the repl, I get an undefined error:
>>> test()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'test' is not defined

Correct way #1 to import and use:
>>> import test
Hello World
>>> test.test()
Hello World
Correct way #2 to import and use:
>>> from test import test
Hello World
>>> test()
Hello World
If you get an ImportError for both ways above, then you are running the Python REPL from the wrong directory.
It's a bit confusing that the filename and function name are the same.
Also it's a bit unusual that there is a call to test() at the end of the file (causing the function to be called at import time). Typically it's wrapped like if __name__ == '__main__': test() to avoid the call at import time, but make the call when run as a script from the command-line.
Import test doesn't work, because Python keywords are lowercase and case sensitive.
from test import Test doesn't work, because Python identifiers (e.g. function names) are case sensitive.
import Test may work on Windows (but not on macOS, Linux and many other operating systems), because filenames on Windows are case insensitive.
import test.py doesn't work, because it's not allowed to have the .py extension as part of the import module name.
import test from test doesn't work, because from ... must come before import ....

importing a class from a module starting with number

I need to import a single class (not the whole file) from a python file starting with number.
There was a topic on importing a whole module and it works, but can't find my way around this one.
(In python, how to import filename starts with a number)
Normally it would be:
from uni_class import Student
though the file is called 123_uni_class.
Tried different variations of
importlib.import_module("123_uni_class")
and
uni_class=__import__("123_uni_class")
Error:
from 123_uni_class import Student
^
SyntaxError: invalid decimal literal

importlib.import_module("123_uni_class") returns the module after importing it, you must give it a valid name in order to reuse it:
import importlib
my_uni_class = importlib.import_module("123_uni_class")
Then you can access your module under the name 'my_uni_class'.
This would be equivalent to import 123_uni_class as my_uni_class if 123_uni_class were valid in this context.

It works for me:
Python 3.6.8 (default, Feb 14 2019, 22:09:48)
[GCC 7.4.0] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.import_module('123_a')
<module '123_a' from '/path/to/123_a.py'>
>>> __import__('123_a')
<module '123_a' from '/path/to/123_a.py'>
You would not see an actual syntax error containing the literal text "from 123_uni_class import ..." unless you actually have some source code containing that line.
If you must, you can also bypass the import system entirely by reading the contents of the file and exec()-ing them, possibly into a namespace you provide. For example:
mod = {}
with open('123_uni_class.py') as fobj:
exec(fobj.read(), mod)
Student = mod['Student']
This general technique is used, for example, to read config files written in Python and things like that. I would discourage it though for normal usage and suggest you just use a valid module name.

Python global variable instantiated twice

In my project, I am trying to share a global variable across modules as described in the Python documentation: https://docs.python.org/3/faq/programming.html#how-do-i-share-global-variables-across-modules
I seemed to find, however, that the global variable is instantiated twice and I am not sure why. I want the variable to be a singleton.
I was able to reproduce the issue with this minimal example:
# main.py
class TheClass:
def __init__(self):
print("init TheClass")
def do_work(self):
import worker
worker.do_work()
the_class:TheClass = TheClass()
if __name__ == '__main__':
the_class.do_work()
# worker.py
from main import the_class
def do_work():
print("doing work...")
Output of python main.py:
init TheClass
init TheClass
doing work...
'init TheClass' is logged twice which implies that the class was instantiated twice.
I was unable to reproduce this behaviour in the interactive Python shell:
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import main
init TheClass
>>> main.the_class.do_work()
doing work...
>>>
How do I fix this undesired behaviour?

This happens because you are having a circular import and because of something special that happens with the codefile that you are actually starting.
If you start a file (as the argument to python) it will become the __main__ module independent of its filename. If at a later point in time the same file is imported, it is not recognized as already imported and is reimported under its real name, main in your case. This is why the code is run twice. A third import will not rerun the code.
You can fix it by preventing circular imports (don't put stuff to import into the starter), or by putting the code that should not run twice into the name checking if you already have.

what is the difference between 'import a.b as b' and 'from a import b' in python [duplicate]

This question already has answers here:
from ... import OR import ... as for modules
(6 answers)
Closed 4 years ago.
I've always used from a import b but recently a team at work decided to move a module into a new namespace, and issued a warning notice telling people to replace import b with import a.b as b.
I've never used import as and the only documentation I can find seems to suggest it doesn't support import a.b as b, though clearly it does.
but is there actually a difference, and if so what?

As far as I know, correct me if I am wrong.
First, import a.b must import a module(a file) or a package(a directory contains __init__.py).
For example, you can import tornado.web as web but you cannot import flask.Flask as Flask as Flask is an object in package flask.
Second, import a.b also import the namespace a which from a import b won't. You can check it by globals().
So what's the influence? For example:
import tornado.web as web
Now you have access to namespace tornado, but you cannot access tornado.options even though tornado has this module. But as python's global package management, if you from tornado import options, you will not only get access to options but also add it to namespace tornado. So now you can also access options by tornado.options.

I am going to provide only a partial answer, and I will speculate a bit.
1) Sometimes I have observed that the second way works while the first does not. On my system:
Python 3.6.3 (default, Oct 3 2017, 21:45:48)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorflow import keras # <- this works
>>>
>>> import tensorflow.keras as K # <- this fails
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'tensorflow.keras'
>>>
2) I usually don't see a difference between these two approaches to importing. I haven't investigated WHY there is a difference with TensorFlow. It may have to do with what names are imported to the top level by the various TensorFlow subfolder init.py files (which are completely empty in most cases, but the one in ../dist_packages/tensorflow/python is pretty long).

How to load compiled python modules from memory?

I need to read all modules (pre-compiled) from a zipfile (built by py2exe compressed) into memory and then load them all.
I know this can be done by loading direct from the zipfile but I need to load them from memory.
Any ideas? (I'm using python 2.5.2 on windows)
TIA Steve

It depends on what exactly you have as "the module (pre-compiled)". Let's assume it's exactly the contents of a .pyc file, e.g., ciao.pyc as built by:
$ cat>'ciao.py'
def ciao(): return 'Ciao!'
$ python -c'import ciao; print ciao.ciao()'
Ciao!
IOW, having thus built ciao.pyc, say that you now do:
$ python
Python 2.5.1 (r251:54863, Feb 6 2009, 19:02:12)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> b = open('ciao.pyc', 'rb').read()
>>> len(b)
200
and your goal is to go from that byte string b to an importable module ciao. Here's how:
>>> import marshal
>>> c = marshal.loads(b[8:])
>>> c
<code object <module> at 0x65188, file "ciao.py", line 1>
this is how you get the code object from the .pyc binary contents. Edit: if you're curious, the first 8 bytes are a "magic number" and a timestamp -- not needed here (unless you want to sanity-check them and raise exceptions if warranted, but that seems outside the scope of the question; marshal.loads will raise anyway if it detects a corrupt string).
Then:
>>> import types
>>> m = types.ModuleType('ciao')
>>> import sys
>>> sys.modules['ciao'] = m
>>> exec c in m.__dict__
i.e: make a new module object, install it in sys.modules, populate it by executing the code object in its __dict__. Edit: the order in which you do the sys.modules insertion and exec matters if and only if you may have circular imports -- but, this is the order Python's own import normally uses, so it's better to mimic it (which has no specific downsides).
You can "make a new module object" in several ways (e.g., from functions in standard library modules such as new and imp), but "call the type to get an instance" is the normal Python way these days, and the normal place to obtain the type from (unless it has a built-in name or you otherwise have it already handy) is from the standard library module types, so that's what I recommend.
Now, finally:
>>> import ciao
>>> ciao.ciao()
'Ciao!'
>>>
...you can import the module and use its functions, classes, and so on. Other import (and from) statements will then find the module as sys.modules['ciao'], so you won't need to repeat this sequence of operations (indeed you don't need this last import statement here if all you want is to ensure the module is available for import from elsewhere -- I'm adding it only to show it works;-).
Edit: If you absolutely must import in this way packages and modules therefrom, rather than "plain modules" as I just showed, that's doable, too, but a bit more complicated. As this answer is already pretty long, and I hope you can simplify your life by sticking to plain modules for this purpose, I'm going to shirk that part of the answer;-).
Also note that this may or may not do what you want in cases of "loading the same module from memory multiple times" (this rebuilds the module each time; you might want to check sys.modules and just skip everything if the module's already there) and in particular when such repeated "load from memory" occurs from multiple threads (needing locks -- but, a better architecture is to have a single dedicated thread devoted to performing the task, with other modules communicating with it via a Queue).
Finally, there's no discussion of how to install this functionality as a transparent "import hook" which automagically gets involved in the mechanisms of the import statement internals themselves -- that's feasible, too, but not exactly what you're asking about, so here, too, I hope you can simplify your life by doing things the simple way instead, as this answer outlines.

Compiled Python file consist of
magic number (4 bytes) to determine type and version of Python,
timestamp (4 bytes) to check whether we have newer source,
marshaled code object.
To load module you have to create module object with imp.new_module(), execute unmashaled code in new module's namespace and put it in sys.modules. Below in sample implementation:
import sys, imp, marshal
def load_compiled_from_memory(name, filename, data, ispackage=False):
if data[:4]!=imp.get_magic():
raise ImportError('Bad magic number in %s' % filename)
# Ignore timestamp in data[4:8]
code = marshal.loads(data[8:])
imp.acquire_lock() # Required in threaded applications
try:
mod = imp.new_module(name)
sys.modules[name] = mod # To handle circular and submodule imports
# it should come before exec.
try:
mod.__file__ = filename # Is not so important.
# For package you have to set mod.__path__ here.
# Here I handle simple cases only.
if ispackage:
mod.__path__ = [name.replace('.', '/')]
exec code in mod.__dict__
except:
del sys.modules[name]
raise
finally:
imp.release_lock()
return mod
Update: the code is updated to handle packages properly.
Note that you have to install import hook to handle imports inside loaded modules. One way to do this is adding your finder into sys.meta_path. See PEP302 for more information.