Python Snakefood module import generalized failure - python

I want to run snakefood (an AST-based dependency graph analyser; source code can be found here). My project has a structureinvolving several levels of Python packages, like this:
myproject
|code
|Utils
|AdaptedConfigParser
Configs_Parser.py
...
...
main.py
However, when I start running snakefood on the root directory of my project it claims that it can't find the modules from my Python package being imported:
$ sfood --internal --follow --ignore-unused ./PycharmProjects/myproject/ > ~/static_analysis.txt
WARNING : Line 9: Could not import module 'myproject.Utils.AdaptedConfigParser.Configs_parser'
I tried to get around it by adding a .pth file with the project root to the lib/python2.7/site-packages
Now when I call python with that virtual environment activated from anywhere, I can do the following:
$ python
Python 2.7.6rc1 (default, Jan 19 2014, 18:57:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import myproject.Utils.AdaptedConfigParser.Configs_parser
>>>
And it works just fine.
However, when snakefood is launched with that virtual environement, it still returns the same error.
$ sfood --internal --follow --ignore-unused ./PycharmProjects/myproject/ > ~/static_analysis.txt
WARNING : Line 9: Could not import module 'myproject.Utils.AdaptedConfigParser.Configs_parser'
At this point I don't even understand where the problem with imports could come from.
In addition to that, when --internal flag is taken away, it fails imports even of the python builtin module, which doesn't make any sense to me:
WARNING : Line 80: Could not import module 'builtins'
WARNING : Line 190: Could not import module 'pyamg'
Have anyone encountered such a problem previously? If yes, is there a way of getting around it?

I had similar issues, turns out the warning in my case was generated because of __all__ declarations.
e.g.
__all__ = ['abc',
'aaa',
...
]
Above code gives out the warnings:
WARNING : Line xx: Could not import module 'abc'
WARNING : Line xx: Could not import module 'aaa'
I modified the warning constant ERROR_IMPORT within snakefood\lib\python\find.py to add the name of the file where the error occurs along with the line number and module name. That way you can target the specific file and line number pretty much figure out the problem.
Hope this helps!

Related

Why is my script's directory not in the Python sys.path?

Python 3.6.5
I am aware of this one: Why does my python not add current working directory to the path?
But the problem there is that he's doing something more complicated (referring to a sub-folder but executing from a main folder). The answers there are to either simplify things or to add package definitions.
And the selected answer even says: "It is the script's directory that is added"
However, my problem is really more simple: My script's directory ISN'T added.
Basically, all the tutorials on the internet say: import mymodule
When I do that, I get a name error...
My folder structure:
C:/Projects/interner
interner.py # this is the main program body
aux.py # this is the auxiliary file I would like to import into the above
I've tried both coding 'import aux' inside interner.py, and also using the interactive console:
cd c:/Projects/interner
python
import aux
To no avail (ModuleNotFoundError: No module named 'aux')
My sys.path:
['C:\\Tools\\Python\\python365\\python36.zip', 'C:\\Tools\\Python\\python365']
(both from inside the script and from interactive console)
Could you please tell me why I can't import local scripts? Is it because my sys.path is missing the PWD? If so, why is it missing it?
Edit: Doing this to help investigation:
>>> import os; print(os.listdir("."))
['aux.py', 'hw.py', 'interner.py', 'my_funcs.py']
I believe this is a Python bug, specific to the embeddable (ZIP file without an installer) Windows distribution. I’ve filed https://bugs.python.org/issue34841.
Removing the python37._pth file (presumably python36._pth in your case) from the distribution fixed it for me.
I don't know why but it seems that "" is missing from your sys.path variable, and that prevents from importing modules from current directory all right!
I can somehow reproduce your issue (eatcpu.py is in my current dir):
$ python.exe
Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\Windows\\system32\\python27.zip', 'D:\\AppX64\\Python27\\DLLs', ... etc...]
>>> import eatcpu
works. Now in another python session:
$ python.exe
Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.remove("")
>>> import eatcpu
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named eatcpu
>>>
So the quickfix for you is to do:
import sys
sys.path.append("")
It looks like you are using the embeddable distribution of CPython rather than one of the regular installers. As described on the documentation page:
The embedded distribution is a ZIP file containing a minimal Python environment. It is intended for acting as part of another application, rather than being directly accessed by end-users.
Since you seem to be directly accessing Python rather than embedding it, you should consider using the regular (or Microsoft Store) installer (also described on the page I linked above).
Try making it explicit:
from . import aux

Pycharm - using pwntools with remote interpreter on WSL

I am using remote interpreter on pycharm on WSL (configured it with this tutorial: https://www.jetbrains.com/help/pycharm/using-wsl-as-a-remote-interpreter.html)
I was able to run everything I needed successfully, but when I tried to use pwntools (https://github.com/Gallopsled/pwntools) I was able to import it successfully on the WSL bash python interpreter, but not on Pycharm.
This is what I ran:
from pwn import *
On Pycharm it was stucked and I interrupted it, this is the trace of the Exception (where it stucks):
ssh://shahar#localhost:22/usr/bin/python -u /tmp/pycharm_project_271/pwnablekr/fd.py
Traceback (most recent call last):
File "/tmp/pycharm_project_271/pwnablekr/fd.py", line 1, in <module>
from pwn import *
File "/home/shahar/.local/lib/python2.7/site-packages/pwn/__init__.py", line 6, in <module>
pwnlib.args.initialize()
File "/home/shahar/.local/lib/python2.7/site-packages/pwnlib/args.py", line 208, in initialize
term.init()
File "/home/shahar/.local/lib/python2.7/site-packages/pwnlib/term/__init__.py", line 74, in init
term.init()
File "/home/shahar/.local/lib/python2.7/site-packages/pwnlib/term/term.py", line 109, in init
c = os.read(fd.fileno(), 1)
KeyboardInterrupt
Process finished with exit code 1
enter code here
On my WSL bash it ran just fine:
shahar#MYCOMPUTERNAME:/mnt/c/Users/shahar$ python
Python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pwn import *
>>>
When I looked at the piece of code where it stuck (from the trace of the Exception):
while True:
c = os.read(fd.fileno(), 1)
s += c
if c == 'R':
break
at the beginning of the script as a global variable:
fd = sys.stdout
I understood from the internet that this function (which this loop is part of it) is related to take over the terminal. Maybe it is related to the fact I am not running from terminal?
Had anyone seen this kind of problem before? has some helpful tips?
Thank you very much!
I have a potential fix as well, and it's adding a PWNLIB_NOTERM to the environment.
import os
os.environ['PWNLIB_NOTERM'] = 'True' # Configuration patch to allow pwntools to be run inside of an IDE
import pwn
Screenshot showing it runs and we get an Encoder object instance
There is another way to solve it.
If you use Pycharm , you can tick the box Run with Python console in Run configurations.
It will work in Pycharm 2020.3 with IPython.(I think it also works without IPython)
screenshot
There is no effective way, I debug it, the problem is term initialization.it may also be related to the environment variables of the TERM and TERMINFO.My solution is to modify the last line of /usr/local/lib/python2.7/dist-packages/pwnlib/args.py,delete term.init(), replace it with anything else to bypass the initialization of pwnlib.
replace this line:
debug pwntools:

What I cannot import a verified code in the current directory?

So I have the following script, test.py:
>cat test.py
def foo(x):
y=x*2
return y
print foo(100)
x = "aaa100"
print x
I can run it well:
>python test.py
200
aaa100
But inside the interactive interpreter, I cannot import it:
>python
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "test.py", line 6, in <module>
x + 10
TypeError: cannot concatenate 'str' and 'int' objects
Do you see what is wrong?
There is already a test module (https://docs.python.org/3/library/test.html or https://docs.python.org/2/library/test.html), which is loaded before your local test.py on the PYTHONPATH because you used import.
In the interpreter, you can see where it is
> import test
> test
<module 'test' from 'C:\Python27\lib\test\__init__.pyc'>
Yours may not be the test module. It tends to successfully import. You must have another test.py on your PYTHONPATH.
Your command line does not have an import, so it is free to use the local filename when you pass it directly to the python executable as an argument.
A simple way to avoid the name conflict, name your file "test_foo.py" and import test_foo. (unless you use test_foo.py in multiple places)
Additionally, consider a run function, in combination with
if __name__ == __main__:
because right now it will execute those prints as part of the import. You will want to do that also in ... whatever other "test.py" you have.
It's either these two cases:
1) Python built-in module "test" (https://docs.python.org/2.7/library/test.html) is being loaded and not your local test.py, however, you shouldn't have any problem importing your local test.py since its a default module, its strange that you are getting an error when you try importing it (unless you have modified it). You can try this to see the imported module cache and check for the test module path
import sys
print sys.modules
If this is the case then you simply have to open a new python interpreter to make sure no test module has been cached. If its still cached, then change the name of your test.py to something_test.py
2) The code you are representing above is not complete and you have a line "x + 10" and since you are assigning x as a string and trying to add integer, your import isn't working. I doubt this being true as I am sure if this was the case you would be aware of it :-)

Pycharm 4 no module named ROOT, when importing a locally installed module (PyROOT), that is working fine from shell on Mac OS X Yosemite

I am trying to set up PyROOT to work with Pycharm 4 on Mac OS X Yosemite.
I have installed ROOT (locally), with the python option enabled, and set up all of the necessary environment paths.
echo $PYTHONPATH
/Users/natalia/Software/root/lib:/Users/natalia/Software/root/bin:/Users/natalia/Software/root
It works just fine from the shell interpreter:
python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ROOT
>>> ROOT.__file__
'/Users/natalia/Software/root/lib/ROOT.pyc'
>>>
In Pycharm I have tried to add these paths to the interpreter using Preferences->Project Interpreter->More->Show paths...
The paths that show there are the following:
file:///Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg
file:///Users/natalia/Software/root/lib
file:///System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
file:///System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
file:///System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
file:///System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
file:///System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
file:///System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
file:///System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
file:///System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
file:///Library/Python/2.7/site-packages
file:///Users/natalia/Software/root/bin
file:///Users/natalia/Software/root
I created the following file in Pycharm:
import os
os.system("echo $PYTHONPATH")
import ROOT
When run, it returns this:
Traceback (most recent call last):
/Users/natalia/Software:/Users/natalia/Software/root:/Users/natalia/Work/Projects/untitled
File "/Users/natalia/Work/Projects/untitled/l.py", line 3, in <module>
import ROOT
ImportError: No module named ROOT
Notice how this PYTHONPATH that's printed from python in Pycharm doesn't include (for a reason unknown to me) the path that actually includes the ROOT.pyc file, that is: '/Users/natalia/Software/root/lib'
I have also tried doing the dirty trick of
os.system("export PYTHONPATH=$PYTHONPATH:/Users/natalia/Software/root/lib")
but I found it doesn't actually change the path if I print it afterwards.
I am absolutely confused as to where Pycharm gets the paths from.
Any possible solutions would be welcome and greatly appreciated.
You may have figured this out already, but just in case...
import sys
sys.path.append('/Applications/Misc/root/lib')
import ROOT
print ROOT.TTimeStamp().AsString()
will add that search path to Python (& PyCharm by extension); the snippet gives me the following output in PyCharm:
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/Users/rrios/PycharmProjects/untitled/dummy.py
Thu, 02 Jul 2015 01:07:57 UTC +628998000 nsec
Process finished with exit code 0

Bizarre python ImportError

Here's my setup: a Mac, running OS X Tiger. Windows XP running in a virtual machine (Parallels). Windows XP has my Mac home directory mapped as a network drive.
I have two files in a directory of my Mac home directory:
foo.py
pass
test.py
import foo
If I run test.py from within my virtual machine by typing 'python test.py', I get this:
Traceback (most recent call last):
File "test.py", line 1, in <module>
import foo
ImportError: No module named foo
If I try to import foo from the console (running python under Windows from the same directory), all is well:
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
>>>
If I run test.py with Mac python, all is well.
If I copy test.py and foo.py to a different directory, I can run test.py under Windows without problems.
There is an init.py in the original directory, but it is empty. Furthermore, copying it with the other files doesn't break anything in the previous paragraph.
There are no python-related environment variables set.
Any ideas?
Add import sys; print sys.path to the start of test.py. See what it prints out in the failing case. If "." isn't on the list, that may be your problem.
As a random guess: are the permissions on foo.py accessable from the windows client? (eg try opening with notepad from the virtual machine).
If that's OK, try running:
python -v -v test.py
and looking at the output (alternatively, set PYTHONVERBOSE=2). This should list all the places it tries to import foo from. Comparing it with a similar trace on the working machine may give some further clues.

Categories