I have a problem with a segfault in pyodbc and would like to debug it in Windows XP x86. However, the information online seems primarily Linux-centric. What is the best way to go about this?
So I was able to successfully resolve my issue by using Visual Studio 2008. I loosely followed the steps listed here -
http://www.velocityreviews.com/forums/t329214-debugging-python-extensions.html
And some tips on workarounds here -
Compiling python modules whith DEBUG defined on MSVC
Here is my version of the steps for anyone else who may encounter this problem.
In case you haven't already, be sure to setup the Python header and libs directories in VS
a. Go to Tools > Options > Projects and Solutions > VC++ Directories. Be sure to add your include and libs path to the Include and Library files' path, respectively. (e.g. C:\Python27\include, C:\Python27\libs)
Go to your Python include folder (once again, e.g. C:\Python27\include) and edit pyconfig.h. Comment out the line # define Py_DEBUG and save. Go to your libs folder (e.g. C:\Python27\libs) and make a copy of python27.lib. Name the copy python27_d.lib.
Create a new project. Choose Win32 Project and name it the module name (in my case pyodbc. Click Next then choose DLL for Application type and check Empty Project.
In the Solution Explorer, right-click on Header Files and choose Add > Existing Item. Select all of the header files that you need. Do the same for Source Files.
Go to Project > Properties, then under Configuration Properties -
a. General - ensure that you are using the correct Character Set. For me it was Use Multi-Byte Character Set. Python 3 probably needs Use Unicode Character Set.
b. Debugging - enter the path to Python in the Command field. (e.g. C:\Python27\python.exe). Then set Attach to Yes.
c. Linker > General - change the Output File to end in .pyd instead of .dll.
Ensure that your configuration is set to Debug. Go to Build > Build Solution.
Open a cmd and cd into the directory where your pyd file was compiled. Start python from the cmd window. To attach debugger on this running python process, go back to Visual Studio and click the green play button to start debugging. You can also use Debugging -> Attach to Process... Now go back to Python and import your module. Play, test, and try to break it!
Debugging workflow with WinDbg
This workflow will create debugging information for a Release build, so you
don't have to mess with the original include and library files of Python.
Download and install Debugging Tools for Windows
Get the symbol files for your Python version and extract them. For Python
2.7.3 this would be http://www.python.org/ftp/python/2.7.3/python-2.7.3-pdb.zip.
Modify setup.py to generate debugging files. You have to add '/Zi' to
extra_compile_args and '/DEBUG' to extra_link_args. Example:
ext_modules = [Extension('pyuv', sources=['src/pyuv.c'],
extra_compile_args=['/Zi'],
extra_link_args=['/DEBUG'])
]
Build the extension as always (python setup.py ...).
Start WinDbg and specify the Symbol Search Path (Ctrl + S).
C:\Path\To\Extension_pdb
C:\Path\To\Extracted\python-2.7.3-pdb
srv*;SRV*c:\tmp*http://msdl.microsoft.com/download/symbols
The last line will download and cache required symbols for Windows modules.
Start the Python executable (Ctrl + E). You can directly execute a
script or run in interactive mode.
Skip the initial breakpoint with "Go" (F5).
If there is a Segmentation fault the execution will break and you will
see something like Access violation - code c0000005 (first chance)
in the WinDbg console.
You can get detailed exception information by typing !analyze -v
in the WinDbg console and the current stack trace with kb.
Here is an
example of such an output.
You should be able to combine this approach with pyrospade's
answer to debug with
Visual Studio if you omit his second step and build the project with
Release configuration.
A further tutorial for WinDbg could be found here.
Segfaults are especially mysterious, as there is no way to trap for them from your Python code, or even to get much stacktrace information on the C side of things. One thing that can give you at least a little more info is to use the Google breakpad C library to report a C stack trace when the segfault occurs.
You may want to try David Malcolm's tool CPyChecker which statically analyses C extensions for memory leaks and other bugs. The tool is documented here.
Related
I have a problem with Intellisense. Even though Python, Pylance and even Visual Studio IntelliCode extensions are installed, VS code is still unable to suggest object attributes.
Could you please advise any settings.json parameters which will trigger python attribute suggestion globally ?
Sometime it's tedious to remember all object attributes without documentation being opened.
You can use ctrl + space for windows for activating intelliSense in VsCode.
However it also depends on processing speed of computer.
Intellisense was provided by the Language Server. The search paths depend on the python interpreter you have selected.
If you modify the PYTHONPATH manually to import the package, you need to add these locations to the Language Server too(settings.json), like this:
"python.autoComplete.extraPaths": [
"C:/Program Files (x86)/Google/google_appengine",
"C:/Program Files (x86)/Google/google_appengine/lib/flask-0.12"]
You can refer to here.
Or, there are some problems of your Language Server, you can have a look at the
OUTPUT panel --> Python Language Server channel. And attach the error message on the question.
Open the Command Palette (Command+Shift+P on macOS and Ctrl+Shift+P on Windows/Linux) and type:
Python: Select Interpreter
then choose Interpreter.
I've downloaded the python 3.6.6 source from here...
https://www.python.org/downloads/release/python-366/
...and followed the instruction on how to build on Windows (run ../PCbuild/build.bat). Python compiles and seems to be working (funny and scary: while fetching externals, it actually downloads python-3.7.0 as a dependency... :/ ). However, it looks like the build is somehow 'in place', and the binaries end up in some sub-folder of the source (../PCbuild/amd64/python.exe). This means I'm left with source and compiled code mixed up instead of some clean/lean and deployable package.
can I somehow provide '--prefix=/target/build/path' to define a target location to build to, like I would on linux?
is there a way of removing all src files/folders and leave only the required files/folders (../lib, ../include, etc...).
Or in general, is there a way of making the build process more behave like on linux?
Thanks for your help,
Max
The build.bat from PCBuild is intended for developers, that is, for testing purposes. What you want is under \Tools\msi\buildrelease.bat. This creates a subdirectory under \PCBuild\ that places all msi, cab and exe files ready for later installation. According to the readme, there doesn't seem to be an option to pack all those files in a single .exe file, like all installers eventually do, but another option is under \Tools\msi\build.bat which does have an option for packing (namely build.bat --pack). "But", the readme does state that the buildrelease.bat should be used for an official release. The advantage of doing so is that Pyhton would be optimized using PGO to your own hardware. I am also trying to compile from source using this method but I am having an issue with a recurring error (and other ones):
PGO run did not succeed (no python36!*.pgc files) and there is no data to merge [E:\RepoGiT\3.6\PCbuild\pythoncore.vcxproj]
so, if you do go this route, and find this, or other errors, please send the bug report to python's bug tracker webpage. And better yet, if you find errors and their solution, please report back here!
In order to develop complex gem5 python configuration scripts with more convenient IDE the gem5 lib has to be added
to the project. However, for those who are not experts in python and Pycharm (Such as myself)
there are some difficulties.
For those who are unfamiliar with gem5, this is
a short explanation:
gem5 is an open source simulator that inspects hardware architecture.
It can be downloaded from github: link to github.
The installation process is described at the following link.
<gem5_installation_dir>/gem5/configs/learning_gem5/part1
Inside the above path, there is a basic python script file: simple.py
This file contains some imports.
Editing this file with Pycharm requires some dependencies that are located in the gem5 installation directory.
The questions:
1) How to add those dependencies of gem5 into Pycharm?
2) How to config the Pycharm with gem5 execution command?
Any help would be appreciated.
A few pointers from what I see at gem5 d9cb548d83fa81858599807f54b52e5be35a6b03 (May 2020) under gem5/configs/learning_gem5/part1/two_level.py:
from common: common is at configs/common which gets found because of the above m5.util.addToPath('../../') call, so add configs/ to the PYTHONPATH as shown at: PyCharm and PYTHONPATH
import m5 comes from src/python/m5 so add src/python to the PYTHONPATH as above
from caches import * comes from the sibling learning_gem5/part1/caches.py, so likely this will be found automatically by PyCharm. Otherwise add that directory to the PYTHONPATH.
from m5.objects import *: this is likely the one you are really interested in as it contains all the interesting objects, but unfortunately PyCharm simply cannot handle it I believe since the SimObjects are added dynamically to that namespace at startup in a very convoluted way via PyBind11 native modules + code generation.
A description of how this works in more detail can be found here, but basically every SimObject class goes through some heavy code autogeneration to make this work, say e.g. src/cpu/simple/AtomicSimpleCPU.py due to SimObject('AtomicSimpleCPU.py') in src/cpu/simple/SConscript.
As of 2017, PyCharm said they did not have plans for a proper native C/C++ extension setup: https://intellij-support.jetbrains.com/hc/en-us/community/posts/206018984-Developing-Python-extension-in-C-using-PyCharm
With that said, I've found that it is not worth to use an IDE for the Python part of gem5. This is because the Python tends to be very simple to understand with IPDB (or impossible to setup an IDE for), and if you just grep stuff you tend to quickly guess what is going on. For C++ though I do recommend setting up Eclipse: How to setup Eclipse IDE for gem5 development?
Related: Add custom modules to PyCharm Linter
I have a project which compiles several executables for embedded systems. Each executable is generated by making an appropriate target:
firmware1.bin
firmware2.bin
...
Now I want to automate the firmware flashing process. I have a python script that can flash firmware to the embedded system flash. To call this script, I have to create additional targets:
firmware1.bin
firmware1.bin_upload
firmware1.bin
firmware1.bin_upload
...
This is not very convenient, so I want to create CLion external tool that calls the python script:
The problem is that I cannot find a way to pass the current target executable to the external tool in CLion. There is no macro for generated firmwares.
Is there any way to pass an external tool path to firmware? Or maybe other way to automate the uploading process except creating additional targets?
I do not think this is possible at the time of writing;
I previewed all macros and I did not see the name of the executable/target as registered in CMake.
Here's a list of things you may be able to do instead:
Parse your CmakeLists.txt and determine the executables being created from your python script, to which the source directory is passed.
Try to solve it completely in CMake; which may very well be possible
Register a feature request with CLion team, and inform me, as I will immediately +1 it.
Write a CLion plugin (provided it can gather the info required)
I hope you managed to resolve your workflow by now.
In Visual Studio with PTVS I have two separate Python projects, one contains a Python source file named lib.py for use as a library of functions and the other is a main that uses the functions in the library. I am using an import statement in the main to reference the functions in the library project but get the following error:
No module named lib
I primarily program in F# using Visual Studio so my mindset is adding references to other .NET projects.
How do I think in the Pythonic way to accomplish this?
Python does not use references like .NET does but uses a path which is searched. The search path needs to be modified to include the directory containing the source file. See: The Module Search Path
Looking at the project in Visual Studio with Solution Explorer shows Search Paths for each project.
To modify the search path:
Get the directory for the Python file containing the source code to import.
e.g. lib.py
In Solution Explorer right click on lib.py and select Copy Path
Now for the project that will import the module
e.g. ConsoleDriver_Python
Right click Search Paths and select Add Folder to Search Path...
which displays a select folder dialog
Right click and paste in the path from the clipboard. Also change it to a directory by removing the file name.
Click Select Folder
Now check the project to make sure Search Path was updated.
The import error should now be cleared.
I just wanted to add the below in addition to the verified answer, for a very specific scenario.
I was recently asked to fix the same problem that the OP was experiencing for a work machine, which had recently had the user accounts migrated over to a new domain.
Setup:
Visual Studio 2013
PTVS 2.2.30718
Anaconda 3.5
Basically, Anaconda was installed for localmachine/UserA.
Once the users were migrated over to the new domain (newdomain/UserA), the Python environment had to be updated from within VS2013, by clicking View > Other Windows > Python Environments.
Once that was setup, the python scripts would run as expected, although none of the Search Folder references would work. They were then removed and re-added but to no avail.
Various other things were tried, including setting up totally fresh projects, and linking them using the Search Paths, but to no avail.
The only thing that fixed the problem was to reinstall the Python Environment (in my case Anaconda3) outside of a user account (by clicking the "for all users, using administrator privileges" option during the install).
Then I restarted, removed and re-added the search folders, and the python worked as expected, including all the search paths.
I hope that helps someone, as I just wasted hours solving it...
D :)
Or you can do this in code with the following:
sys.path.append("search path")
So that the code can be run outside the IDE.