I am writing an application in C# with VisualStudio and am using IronPython to write some Python scripts for my application. However, it does not have the entire standard library support by default. So to import some modules (such as os) I need to point my C# code to where the os module actually is. I also understand that it will still be limited to libraries implemented in pure python.
Ultimately I want to have something that can be installed on another machine. My current workaround is to include a copy of https://github.com/python/cpython/tree/2.7/Lib in the Debug folder where the executable is running and it seems excessive/unnecessary to have to include the entire thing. I tried just placing the files I need (for example os.py) here but obviously it imports other modules, which import other modules, etc... I would have to re-run the code to get the error for which module it couldn't find and add them in 1 by 1 and it was getting too tedious.
I was wondering if there was any sort of resource that specifies the relationships between standard library modules and could tell me exactly what files to copy. Essentially what I'm looking for is the graph of the standard library imports. So if I want to import os in these scripts I know to copy os.py, ntpath.py, ...
Thanks
you probably don't need the imports as a tree, but as a simple list, so you can just copy the needed files. You can get that from sys.modules, after you import everything that your script needs - it will contain all modules needed by those that you imported, e.g.:
import sys # even if you don't use it - it's a built-in module, won't add a file to the list, needed to get sys.modules
import os
import time
#import whatever-else
# this gives a list of tuples (module,file)
m=[(z,x.__file__) for z,x in sys.modules.items() if hasattr(x,"__file__") ]
for x in m:
print x[0],x[1]
Related
I created two new files, random.py and main.py, in the directory. The code is as follows:
# random.py
if __name__ == "__main__":
print("random")
# main.py
import random
if __name__ == "__main__":
print(random.choice([1, 2, 3]))
When I run the main.py file, the program reports an error.
Traceback (most recent call last):
File "main.py", line 8, in <module>
print(random.choice([1, 2, 3]))
AttributeError: module 'random' has no attribute 'choice'
Main.py imports my own defined random module.
However, if I create a new sys.py file and a main.py file in the same directory, the code is as follows:
# sys.py
if __name__ == "__main__":
print("sys")
# main.py
import sys
if __name__ == "__main__":
print(sys.path)
When I run the main.py file, successfully.
main.py imports the built-in modules sys.
Why is there such a clear difference?
The directory relationship of the script file is as follows:
C:.
main.py
random.py
sys.py
Thank you very much for your answer.
Forgive my poor english.
sys is a built-in module, meaning it's compiled directly into the Python executable itself. Built-in modules outprioritize external files when Python is looking for modules. The standard random module isn't built-in, so it doesn't get that treatment.
Quoting the docs:
When the named module is not found in sys.modules, Python next searches sys.meta_path, which contains a list of meta path finder objects. These finders are queried in order to see if they know how to handle the named module...
Python’s default sys.meta_path has three meta path finders, one that knows how to import built-in modules, one that knows how to import frozen modules, and one that knows how to import modules from an import path (i.e. the path based finder).
Since the finder for built-in modules comes before the finder that searches the import path, built-in modules will be found before anything on the import path.
You can see a tuple of the names of all modules your Python has built-in in sys.builtin_module_names.
That said, while any built-in module would outprioritize a module loaded from a file, sys has its own special handling. sys is one of the foundational building blocks of Python, and much of the sys module's setup needs to happen before the import system is functional enough for the normal import process to work. sys gets explicitly created during interpreter setup in a way that bypasses the normal import system, and then future imports for sys find it in sys.modules without hitting any meta path finders.
How and where sys is created is an implementation detail that varies from Python version to Python version (and is wildly different in different Python implementations), but in the CPython 3.7.4 code, you can see it beginning on line 755 in Python/pylifecycle.c.
tl;dr Caching
sys is somewhat of a special case among other python modules because it gets loaded at program start, unconditionally (presumably because a lot of the constants, functions, and data within - such as the streams stdout and stderr - are used by the python interpreter). As #user2357112 noted in the other answer, this is partly because it's built-in to the python executable, but also because it's necessary for running a substantial amount of python's core functionality (see below how it needs to be loaded for imports to work). random is part of the standard library, but it doesn't get loaded automatically when you execute, which is the primary relevant difference between it and sys, for our purposes
Looking at python's documentation on the subject clarifies how python resolves imports:
The first place checked during import search is sys.modules. This mapping serves as a cache of all modules that have been previously imported, including the intermediate paths.
...
During import, the module name is looked up in sys.modules and if present, the associated value is the module satisfying the import, and the process completes. However, if the value is None, then a ModuleNotFoundError is raised. If the module name is missing, Python will continue searching for the module.
As for where it looks for the module, you can see in your observed behavior that it looks in the local directory first. That is, it searches the local directory first and then the "usual places" afterwards.
The reason for the discrepancy between how sys is handled and how random is handled is caching - sys is cached (so python doesn't even check the path to import), whereas random is not cached (so python does check the path to import it, and imports locally).
There are a few ways you can change this behavior.
First, if you must have a local module called sys, you can use importlib to import it in relative or absolute terms, without running into the ambiguity with the sys that's already cached. I have no idea how this would affect other modules that independently try to import sys, and you really shouldn't be naming your files the same as standard library modules anyway.
Alternatively, if you want the code to check python's built-in modules before checking the local directory, then you should be able to do that by modifying sys.path, which shows the order in which paths are searched for input (the same as the $PATH environment variable, or any other similar language-specific one). The first element of sys.path is usually going to be an empty string '', that would result in searching the current working directory. So you can simply move that to the back of sys.path, to have it searched last instead of first:
sys.path.append(sys.path.pop(0))
I have been coding in python for about 2 months, but I'm only familiar with basic object-oriented programming, so I do not really understand things like how searching for modules is implemented. (Basically I'm a noob.)
I pip installed a package called Opentrons Opentrons 2.5.2 and all its dependencies into the samefolder as a python script I'm currently writing. However when I tried to import the module below[1], I get an error saying that "Opentrons is not a module". Then, I tried shifting it into the python library because I found out the search path using the pprint module and it seems to work. I was wondering if I can specify the search path from the .py file itself instead of manually printing the search path and putting the file into the library that the script searches for. (Willing to put in images of the directories I put the opentrons package in if it helps.)
[1]
import sys
import pprint
pprint.pprint(search.path)
from opentrons import robot, containers, instruments
Edit: I realise that the fact that I am running all my scripts in a Spyder console located in a python 3.6 environment might be important.
You can try using the __import__ function, or importlib. This should allow you to specify the path.
I recently installed Python 3 on my Mac OSX 10.6.8 and I haven't had any problems with modules or imports until now. I'm writing a function that tests whether or not a triangle is right angled based on the length of the sides and the guide that the exercise was in has a bunch of equalities to test so I can see if it works:
testEqual(is_rightangled(1.5,2.0,2.5), True)
testEqual(is_rightangled(4.0,8.0,16.0), False)
testEqual(is_rightangled(4.1,8.2,9.1678787077), True)
testEqual(is_rightangled(4.1,8.2,9.16787), True)
testEqual(is_rightangled(4.1,8.2,9.168), False)
testEqual(is_rightangled(0.5,0.4,0.64031), True)
I should apparently import a function called testEqual(a,b,c) from a module called test, since the example programme in the guide starts with from test import testEqual, but when I typed that into my file I got this message:
from test import testEqual
ImportError: cannot import name testEqual
I suppose I should specify the path to the test module, but I can't find it my Python 3 library anywhere in my computer – just the 2.x ones that came installed with the computer, which are in /Library/Python. import turtle and import math worked, so it must be somewhere.
The test module in the Python stdlib doesn't contain a function called testEqual(). Its documentation starts with
Note: The test package is meant for internal use by Python only. It is
documented for the benefit of the core developers of Python. Any use
of this package outside of Python’s standard library is discouraged as
code mentioned here can change or be removed without notice between
releases of Python.
Are you sure that this guide you're following doesn't have its own test.py program that you're supposed to use instead?
When you write your testEqual() function make note of the directory you are working in. For instance on my mac I created a directory (folder) in documents so my path looks like: /Users/myName/Documents/python. Save your function (module) as testEqual.py and when you write you test.py script import testEqual after the shebang line. Once you have your scripts debugged your modules will be in a folder that python creates titled pycache don't remove that as it is compiled code. Now, as long as you are working in the same dir as your module you should not need to do anything other than use the import statement.
First of all let me tell you that I'm a new user and I'm just starting to learn Python in College so my apologies if this question is answered in other topic, but I searched and I can't seem to find it.
I received a file work.pyc from my teacher and he says I have to import it in my Wing IDE using the command from work import *, the question is I don't know where to put the file to import it.
It just says ImportError: No module named work.
Thank you
There are several options for this.
The most straightforward is to just place it in the same folder as the py file that wants to import it.
You may also want to have a look at this
if you're using the python interpreter (the one that lets you directly input python code into it and executes) you'll have to do this:
sys.path.append('newpath')
from work import *
where newpath is the path on your filesystem containing your work.pyc file
If you're working on a script called main.py in the folder project, one option is to place it at project/work.pyc
This will make the module importable because it's in the same working directory as your code.
The way Python resolves import statements works like this (simplified):
The Python interpreter you're using (/usr/bin/python2.6 for example, there can be several on your system) has a list of search paths where it looks for importable code. This list is in sys.path and you can look at it by firing up your interpreter and printing it out like this:
>>> import sys
>>> from pprint import pprint
>>> pprint(sys.path)
sys.path usually contains the path to modules from the standard library, additional installed packages (usually in site-packages) and possibly other 3rd party modules.
When you do something like import foo, Python will first look if there is a module called foo.py in the directory your script lives. If not, it will search sys.path and try to import it from there.
As I said, this explanation is a bit simplified. The details are explained in the section about the module search path.
Note 1:
The *.pyc you got handed is compiled Python bytecode. That means it's contents are binary, it contains instructions to be executed by a Python virtual machine as opposed to source code in *.py that you will normally deal with.
Note 2:
The advice your teacher gave you to do from work import * is rather bad advice. It might be ok to do this for testing purposes in the interactive interpreter, but your should never do that in actual code. Instead you should do something like from work import chop, hack
Main reasons:
Namespace pollution. You're likely to import things you don't need but still pollute your global namespace.
Readability. If you ever read someone elses code and wonder where foo came from, just scroll up and look at the imports, and you'll see exactly where it's being imported from. If that person used import *, you can't do that.
How do I set up module imports so that each module can access the objects of all the others?
I have a medium size Python application with modules files in various subdirectories. I have created modules that append these subdirectories to sys.path and imports a group of modules, using import thisModule as tm. Module objects are referred to with that qualification. I then import that module into the others with from moduleImports import *. The code is sloppy right now and has several of these things, which are often duplicative.
First, the application is failing because some module references aren't assigned. This same code does run when unit tested.
Second, I'm worried that I'm causing a problem with recursive module imports. Importing moduleImports imports thisModule, which imports moduleImports . . . .
What is the right way to do this?
"I have a medium size Python application with modules files in various subdirectories."
Good. Make absolutely sure that each directory include a __init__.py file, so that it's a package.
"I have created modules that append these subdirectories to sys.path"
Bad. Use PYTHONPATH or install the whole structure Lib/site-packages. Don't update sys.path dynamically. It's a bad thing. Hard to manage and maintain.
"imports a group of modules, using import thisModule as tm."
Doesn't make sense. Perhaps you have one import thisModule as tm for each module in your structure. This is typical, standard practice: import just the modules you need, no others.
"I then import that module into the others with from moduleImports import *"
Bad. Don't blanket import a bunch of random stuff.
Each module should have a longish list of the specific things it needs.
import this
import that
import package.module
Explicit list. No magic. No dynamic change to sys.path.
My current project has 100's of modules, a dozen or so packages. Each module imports just what it needs. No magic.
Few pointers
You may have already split
functionality in various module. If
correctly done most of the time you
will not fall into circular import
problems (e.g. if module a depends
on b and b on a you can make a third
module c to remove such circular
dependency). As last resort, in a
import b but in b import a at the
point where a is needed e.g. inside
function.
Once functionality is properly in
modules group them in packages under
a subdir and add a __init__.py file
to it so that you can import the
package. Keep such pakages in a
folder e.g. lib and then either add
to sys.path or set PYTHONPATH env
variable
from module import * may not
be good idea. Instead, import whatever
is needed. It may be fully qualified. It
doesn't hurt to be verbose. e.g.
from pakageA.moduleB import
CoolClass.
The way to do this is to avoid magic. In other words, if your module requires something from another module, it should import it explicitly. You shouldn't rely on things being imported automatically.
As the Zen of Python (import this) has it, explicit is better than implicit.
You won't get recursion on imports because Python caches each module and won't reload one it already has.