My initial goal is to open a dll file on Cygwin using ctypes. However I found some issues with it. I dug up to sys.dl which returns an unknown Permission denied only on IPython.
With python everything looks fine:
$ ls
my.dll
$ python
Python 2.7.8 (default, Jul 28 2014, 01:34:03)
[GCC 4.8.3] on cygwin
>>> import dl
>>> dl.open('my.dll')
<dl.dl object at 0xfffaa0c0>
With ipython I get the error:
$ ipython
Python 2.7.8 (default, Jul 28 2014, 01:34:03)
In [1]: import dl
In [2]: dl.open('my.dll')
---------------------------------------------------------------------------
error Traceback (most recent call last)
<ipython-input-2-c681630fa713> in <module>()
----> 1 dl.open('my.dll')
error: Permission denied
I investigated on this using strace. The output log for `IPython is huge, more than 4MB. Fortunately, I identified some weird things:
symlink.check(C:\Users\user\Home\projects\foo\my.dll, 0x28AB88) (0x4022)
35 2705178 [main] python2.7 16924 path_conv::check: this->path(C:\Users\user\Home\projects\foo\my.dll), has_acls(1)
37 2705215 [main] python2.7 16924 cwdstuff::get: posix /cygdrive/c/Users/user/Home/projects/foo
32 2705247 [main] python2.7 16924 cwdstuff::get: (C:\Users\user\Home\projects\foo) = cwdstuff::get (0x8006ECF0, 32768, 0, 0), errno 11
--- Process 14376, exception c0000138 at 7726163E
3286 2708533 [main] python2.7 16924 seterrno_from_win_error: /home/corinna/src/cygwin/cygwin-1.7.35/cygwin-1.7.35-1.i686/src/src/winsup/cygwin/dlfcn.cc:174 windows error 182
42 2708575 [main] python2.7 16924 geterrno_from_win_error: unknown windows error 182, setting errno to 13
36 2708611 [main] python2.7 16924 dlopen: ret 0x0
Who is /home/corinna? I have no corinna user in my installation, neither on my Windows. Corinna does not come from my installation. Is it some hard-coded stuff?
Now, here is what I get from strace for python:
symlink.check(C:\Users\user\Home\projects\foo\my.dll, 0x28B728) (0x4022)
26 10440048 [main] python 12604 path_conv::check: this->path(C:\Users\user\Home\projects\foo\my.dll), has_acls(1)
23 10440071 [main] python 12604 cwdstuff::get: posix /cygdrive/c/Users/user/Home/projects/foo
25 10440096 [main] python 12604 cwdstuff::get: (C:\Users\user\Home\projects\foo) = cwdstuff::get (0x8006ECF0, 32768, 0, 0), errno 0
3405 10443501 [main] python 12604 dlopen: ret 0x5B9C0000
dlopen is returning 0x0 in IPython while it is returning 0x5B9C0000 for python. I notice that cwdstuff::get is raising an error before dlopen is called.
EDIT
I sent a message to Cygwin's mailing list and the answer of Corinna regarding this issue is:
This is not Cygwin's fault, AFAICS. Cygwin never loads functions by
ordinal. This is also a bit on the lean side as far as information is
concerned. One can't see how the process calls dlopen, for instance.
Corinna
How to solve this issue?
My earlier tests using ctypes
Initially when I asked my question I was just playing with ctypes.
I am working on Cygwin 32-bit and Windows 7.
With IPython I got an OSError when I tried to load a dll using cdll.LoadLibrary.
Two ideas:
1) in the next cell, type %pdb, and then interactively "print self._name" to see what it is.
2) Use a full path to cdll.LoadLibrary("foo.dll") to see if that works.
Once you know what the issue is, then you can decide whose bug it is, and report it (could be a ctypes issue, but probably ipython)
More ideas:
Check that the user accessing the DLL is the same. You do that like this:
import getpass
print(getpass.getuser())
Check what's the current process actually doing. I haven't used cygwin but in the linux shell the executable strace should show you this.
Usage: get the PID of your current process: import os; os.getpid()
After this, you can use (from outside the python/ipython console the command strace -p <the pid> -e file. After this setup, you can try to load your DLL.
Remarks: the -e file flag should be written exactly like that. The word file tells strace to report all file operations that the process makes. If no differences occur when running on python/ ipython, you can try dropping the -e file flag. Then you will see all system calls that the process makes. I haven't worked on windows like that, so this might not work at all there, but on linux at least, this should report to you everything that the process did. You could see there at least all the files that got opened, but more interesting things could be found there as well. If the outputs are identical, then the problem can be debugged further in python/ ipython. This would require basically what #Doug Blank suggested, but I'd also recommend investigating every name (variable) that gets touched. The self, _dlopen and mode names also sound like they might contain useful information.
Otherwise, do a dir(self) and dir(_dlopen) to see what other properties you could find there that might have gotten modified by IPYthon.
Try these first, and after that we can help you dig further.
Perhaps Python executable and IPython kernel use different manifest files which define loading policy?
Try appending the DLL path to sys.path in both cases.
Check admin rights (UAC) in both cases.
Use dependency walker to figure out dependencies of this DLL. Maybe the problem comes from dependencies?
Possibly your machine has multiple copies of this DLL?
Finally you can use Process Explorer to see the list of DLLs loaded in both cases and see any differences.
I'm working on a very similar issue:
ipython notebook & script difference - loading DLLs
Related
I've tweaked a copy of one of the Nsight Systems report scripts (gpukernsum), and I now want to run it myself. So, I write:
./gpukernsum.py report.sqlite
This doesn't work; I get:
ERROR: Script 'gpukernsum.py' encountered an internal error.
$ ./gpukernsum.py report.sqlite
File "./gpukernsum.py", line 40
"""
^
SyntaxError: invalid syntax
I know this is because f"""whatever""" is Python-3 syntax, so I change the script's hash-bang line from:
#!/usr/bin/env python
to:
#!/usr/bin/env python3
and now I get:
$ ./gpukernsum.py report.sqlite
Traceback (most recent call last):
File "/path/to/./gpukernsum.py", line 7, in <module>
import nsysstats
ModuleNotFoundError: No module named 'nsysstats'
So I added the relevant directory to the lookup path:
export PYTHONPATH="$PYTHONPATH:/opt/nvidia/nsight-systems/2022.1.1/host-linux-x64/python/lib"
and now I get:
$ ./gpukernsum.py report.sqlite
near "WITH": syntax error
... and I'm stuck. The relevant area of the code is:
and not a percentage of the application wall or CPU execution time.
"""
query_stub = """
WITH
summary AS (
SELECT
coalesce({NAME_COL_NAME}, demangledName) AS nameId,
i.e. the "WITH" is part of a string literal which is an SQL query. So, what's the problem? Is Python complaining? Is sqlite complaining?
Note:
Nsight Systems 2022.1.1
CentOS 7
I'm using the original gpukernsum.py code - I have not made any changes to it (other than as described above).
My system has Python 3.9.1 for python3.
A workaround answer:
Nsight Systems bundles its own version of Python, with lib and bin directories.
If you run your script with this specific version, having set PYTHONPATH as described in your question - then the script will work. It's what Nsight itself does, after all.
I am trying to run a python script, and seems when it try to import a library called prctl.so, some error happens:
def update_cmd_title():
"""Remove the secure informations in the command title"""
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + "/script/lb")
import prctl
prctl.setprocname(" ".join(sys.argv))
but I can find this file prctl.so in directory /home/dminstalluser/script/lib/,
-rwxr-xr-x 1 dminstalluser dm_group 10344 Aug 1 03:55 prctl.so
I search from google, and can find somebody had the same problem, but after i followed their solutions, they all failed, like:
export LD_LIBRARY_PATH=/home/dminstalluser/script/lib
or run:
ldconfig
I don't know what's the problem for this error for my case,
On a 64-bit system, the error is most likely caused by a mismatch between the Python you are running, and prctl.so that you've installed.
For example, trying to load 64-bit prctl.so into a 32-bit Python, or vice versa, will produce the error you've observed.
Run file $(which python) /home/dminstalluser/script/lib/prctl.so. If one of them says ELF 64-bit ..., and the other ELF 32-bit ..., then that's exactly your problem.
The fix is to install prctl.so matching your python.
I'm trying to import ocrmypdf on my company's client's Windows Server 2016 Build 14393 computer using Python 37-32. When I import the library, in a Jupyter Notebook, it is unable to locate leptonica by using ctypes.utility.find_library().
Ocrmypdf is a Linux-developed Python 3 package. Per the documentation (https://ocrmypdf.readthedocs.io/en/latest/introduction.html) it does not support Windows. The suggested workarounds are a docker container and Windows Subsystem for Linux.
I would rather not use a docker container as neither I nor my coworkers are very experienced with it. I am unable to use wsl as it is not available for my build of Windows Server 2016 (see the troubleshoot subsection: https://learn.microsoft.com/en-us/windows/wsl/install-on-server)
This discussion (find_library() in ctypes) states that you can point ctypes.utility.find_library to the needed library file by editing the environment Path variable to be a folder which includes it. Conveniently, Tesseract OCR's windows download includes liblept. Would editing the Path variable to point toward that folder be a dangerous thing to do?
Edit: I tried adding the path to Tesseract-OCR's folder on my laptop's environment Path and restarted Anaconda, etc. ocrmypdf still gave the same error.
A closer read of that discussion brought up the point that find_library operates differently on Windows. A read of the documentation (https://docs.python.org/2.5/lib/ctypes-finding-shared-libraries.html) states that "On Windows, find_library searches along the system search path, and returns the full pathname, but since there is no predefined naming scheme a call like find_library("c") will fail and return None." Does this mean I have to hardcode in a name to use in order to find the library?
This issue has been replicated, albeit on a different machine, here: https://github.com/jbarlow83/OCRmyPDF/issues/341. You can reproduce the issue by running the below code on a Windows machine.
!pip install ocrmypdf
import ocrmypdf
The expected result of the above code is that ocrmypdf is successfully imported in a usable form.
The result of the above code is:
OSError Traceback (most recent call last)
<ipython-input-2-a81f3474d7ad> in <module>
----> 1 import ocrmypdf
~\AppData\Local\Continuum\anaconda3\lib\site-packages\ocrmypdf\__init__.py in <module>
16 # along with OCRmyPDF. If not, see <http://www.gnu.org/licenses/>.
17
---> 18 from . import helpers, hocrtransform, leptonica, pdfa, pdfinfo
19 from ._version import PROGRAM_NAME, __version__
20 from .api import Verbosity, configure_logging, ocr
~\AppData\Local\Continuum\anaconda3\lib\site-packages\ocrmypdf\leptonica.py in <module>
40 logger = logging.getLogger(__name__)
41
---> 42 lept = ffi.dlopen(find_library('lept'))
43 lept.setMsgSeverity(lept.L_SEVERITY_WARNING)
44
OSError: cannot load library '<None>': error 0x57
I have been able to get this working Windows 10 by updating the path and it works fine. I used msys2 to install it, hence, the path name. Update to point where your liblept-5.dll is located.
if os.name == 'nt':
os.environ['PATH'] = os.environ.get("PATH", "") + ';C:\\msys64\\mingw64\\bin'
I'm using PyDev installed as an Eclipse extension (through the update site) on Windows 7 and WinPython 3.6.0.1. I can run code just fine even though the syntax checker is broken, i.e., marks as error "Undefined variable: print" lines as simple as
print("hello")
I think the issue is connected with this error I get in the error log as soon as I start Eclipse:
Error creating python process - exited before creating sockets - exitValue = (1).
This is the Exception stack trace
java.lang.RuntimeException: Error creating python process - exited before creating sockets - exitValue = (1).
ProcessInfo:
- Executed: C:\tools\winpython\3.6.0.1\python-3.6.0.amd64\python.exe -u C:\Portables\Eclipse\plugins\org.python.pydev_5.9.2.201708151115\pysrc\pycompletionserver.py 55270
- Environment:
ALLUSERSPROFILE=C:\ProgramData
[... list of all the environment variables ...]
- Working Dir:
C:\Portables\Eclipse\plugins\org.python.pydev_5.9.2.201708151115\pysrc
- OS:
Windows 7
- Std output:
- Err output:
Traceback (most recent call last):
File "C:\Portables\Eclipse\plugins\org.python.pydev_5.9.2.201708151115\pysrc\pycompletionserver.py", line 13, in <module>
from _pydevd_bundle.pydevd_constants import IS_JYTHON
ModuleNotFoundError: No module named '_pydevd_bundle'
at org.python.pydev.core.log.Log.log(Log.java:54)
at org.python.pydev.editor.codecompletion.shell.AbstractShell.startIt(AbstractShell.java:290)
at org.python.pydev.editor.codecompletion.shell.ShellsContainer.getServerShell(ShellsContainer.java:232)
at org.python.pydev.editor.codecompletion.shell.ShellsContainer.getServerShell(ShellsContainer.java:165)
at org.python.pydev.editor.codecompletion.shell.AbstractShell.getServerShell(AbstractShell.java:222)
at org.python.pydev.editor.codecompletion.revisited.modules.CompiledModule.createTokensFromServer(CompiledModule.java:372)
at org.python.pydev.editor.codecompletion.revisited.modules.CompiledModule.<init>(CompiledModule.java:176)
at org.python.pydev.editor.codecompletion.revisited.SystemModulesManager.getBuiltinModule(SystemModulesManager.java:332)
at org.python.pydev.editor.codecompletion.revisited.SystemModulesManager.getBuiltinModule(SystemModulesManager.java:1)
at org.python.pydev.ui.interpreters.AbstractInterpreterManager.getBuiltinMod(AbstractInterpreterManager.java:146)
at org.python.pydev.ui.interpreters.AbstractInterpreterManager.getBuiltinCompletions(AbstractInterpreterManager.java:118)
at org.python.pydev.plugin.nature.PythonNature.getBuiltinCompletions(PythonNature.java:1126)
at org.python.pydev.editor.codecompletion.revisited.AbstractASTManager.getBuiltinComps(AbstractASTManager.java:1801)
at org.python.pydev.editor.codecompletion.revisited.AbstractASTManager.getBuiltinCompletions(AbstractASTManager.java:1787)
at com.python.pydev.analysis.scopeanalysis.AbstractScopeAnalyzerVisitor.<init>(AbstractScopeAnalyzerVisitor.java:151)
at com.python.pydev.analysis.visitors.OccurrencesVisitor.<init>(OccurrencesVisitor.java:105)
at com.python.pydev.analysis.OccurrencesAnalyzer.analyzeDocument(OccurrencesAnalyzer.java:50)
at com.python.pydev.analysis.builder.AnalysisBuilderRunnable.doAnalysis(AnalysisBuilderRunnable.java:271)
at com.python.pydev.analysis.builder.AbstractAnalysisBuilderRunnable.run(AbstractAnalysisBuilderRunnable.java:162)
at org.python.pydev.core.concurrency.RunnableAsJobsPoolThread$1.run(RunnableAsJobsPoolThread.java:140)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56)
So, apparently the error is this
ModuleNotFoundError: No module named '_pydevd_bundle'
I tried reinstalling PyDev (it looks like the installation might be broken), but to no avail. Ideas?
I might have actually found the issue. I discovered the WinPython distribution I installed was "WinPython Zero", which comes with no additional packages. I uninstalled it and installed the "default" WinPython; now the debugger is working as expected. I suppose then there's some library needed to enable debugging. Maybe a more informative error message would help here, as this one I got really doesn't say much.
Funny thing, that's exactly what happens in PyCharm too: https://youtrack.jetbrains.com/issue/PY-24198
I'm using the trick "python -c 'import myscript.py'" to perform a syntax check on a script which uses 'import gtk'.
I get the following error when performing the syntax check, which implies that the gtk module is executing a check for the X display, even though all that's being done at this point is to import the module.
Traceback (most recent call last):
File "<stdin>", line 15, in ?
File "myscript.py", line 21, in ?
import gtk
File "/usr/src/build/463937-i386/install/usr/lib/python2.3/site-packages/gtk-2.0/gtk/__init__.py", line 37, in ?
RuntimeError: could not open display
Is there a way to avoid this error when performing the syntax check?
Before you ask - I'm not able to set $DISPLAY before the syntax check is run. The check is being run on remote servers as part of a distributed build system. These servers do not have an X display available.
Importing modules in Python executes their code!
Well-behaved modules use the if __name__ == '__main__' trick to avoid side effects, but they can still fail - as happened to you.
[BTW, getting to ImportError means the whole file already has correct syntax.]
If you just want to check syntax, without running at all:
python -m py_compile my_script.py
will check one file (and produce a .pyc as a side effect).
python -m compileall ./
will check a whole dir recursively.
python -c 'compile(open("myscript.py").read(), "myscript.py", "exec")'
avoids creating a .pyc.
But note that merely checking the syntax in Python catches very few bugs! Importing does catch more, e.g. mispelled names. For even better checks, use tools like Pychecker / Pyflakes.
What exactly do you mean by 'syntax checking'?
Can't you use a tool like pylint to check for syntax errors?
Otherwise: a very ugly (but probably possible hack):
In your python script detect whether X is present.
If it's not => use GTK on DirectFramebuffer (no X needed then). You'll need to compile GTK on DirectFB (and/or pygtk) from source (some pointers here).
If the remote machine has vncserver installed, you can have a dummy server running and connect to that. Sample instructions:
remotemachine $ vncserver -depth 16 -geometry 800x600 :7
New 'X' desktop is remotemachine:7
Starting applications specified in /home/user/.vnc/xstartup
Log file is /home/user/.vnc/userve:7.log
remotemachine $ DISPLAY=:7 python -c 'import myscript.py'
…
remotemachine $ vncserver -kill :7
Killing Xtightvnc process ID 32058
In your myscript.py, you could do like this
if __name__=="__main__":
import gtk
That will not execute gtk's __init__.py when you do "python -c 'import myscript.py'"
If you are editing with IDLE, Alt+X will check syntax of current file without running it.