I am trying to made multiple platform script for checking if binary has a debug symbols. I want a whole script be written in python. I am wondering to evoke commands like gdb or objdump. for this purpose. What are your suggestions for the efficient use of this tools?
For example objdump -g should return if there is any debug symbols, however it doesn't. Maybe someone know any API for python for checking debug symbols.
Related
I am looking for help to be able to execute a python script on oracle data integrator (ODI)
I have not found any documentation for this process
I would appreciate if someone can help me with this process
I don't know where in ODI I could do this type of execution
Essentially ODI doesn't support Python directly but there are a couple of things you can do. The things to consider are:
where you need to run the code
what you want the code to do
how integrated into ODI do you need it to be
Jython
ODI does support Jython which is a Java implementation of Python. This can be embedded within procedures and Knowledge Modules which allows you to (relatively) easily make use of the ODI metadata. It isn't particularly friendly to code or debug but is functional and powerful, you get access to ODI logging etc.
Whilst this is possible I would look to do this is Groovy rather than Jython as it is much cleaner and simpler
Shell Script
If your python script is already there and is completely stand alone you can use an OdiOsCommand inside of a package. You would need an agent installed on the box on which you want to run the script and you can just do something like
python /path/mypythonscript.py
just as you would from the command line.
This is simple enough but the integration into ODI is very limited. It will handle errors just like a shell script (so handled exceptions will be swallowed and lost) and any parameters you want to pass will need to be via the command line.
context: I have a program which runs on server, which segfaults several times a month. The program is a python program which uses some library implemented in C++ and exposed by pybinder.
I am able to capture the corefile on server and I have the source code (both C++ and python part). I want to know how I can get the segfault stacktrace?
Several things I have tried to
build the source code (C++ part) with -g3 option. From my understand, it should have the same binary and address as the one running on server. The only difference should be symbol table (and possibly several other sections in ELF).
I tried to gdb -ex r bazel-bin/username/coredump/capture_corefile /tmp/test_coredump/corefile.python.3861066.
bazel-bin/username/coredump/capture_corefile is the python script in C++ with symbol table.
/tmp/test_coredump/corefile.python.3861066 is the corefile I have collected.
But it shows
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f58ca51332b in ?? ()
Starting program:
No executable file specified.
I tried to directly get the line of code by llvm-symbolizer.
For python script as the object, it fails directly.
desktop$ llvm-symbolizer --obj=bazel-bin/username/coredump/capture_corefile 0x00007f58ca51332b
LLVMSymbolizer: error reading file: The file was not recognized as a valid object file
??
??:0:0
For shared object, it also fails:
desktop$ llvm-symbolizer --obj=bazel-bin/username/coredump/coredump_pybind.so 0x00007f58ca51332b
_fini
??:0:0
I confirm the symbol table is not stripped:
file bazel-bin/username/coredump/coredump_pybind.so
bazel-bin/experimental/hjiang/coredump/coredump_pybind.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[md5/uuid]=baf3b4d9a8f7955b5db6b977843e2eb0, not stripped
Does someone know how to get the stacktrace with everything I have?
build the source code (C++ part) with -g3 option. From my understand, it should have the same binary and address as the one running on server.
This is by no means guaranteed, and actually pretty hard to achieve with GCC.
You didn't mention the compiler you use, but if it is Clang (implied by your later use of llvm-symbolizer), then note that Clang currently doesn't produce the same code with and without -g.
In addition, to make this work, you need to keep all the flags originally used (including all optimization flags) -- it's no good to replace -O2 with -g3 -- the binary will be vastly different.
You can check whether your rebuilt library is any good by running nm original.so, nm replacement.so, and comparing the addresses of any symbols which appear in both outputs. The replacement.so is usable IFF all common symbol's addresses match.
The best practice here is to build the .so with optimization and debug info (e.g. gcc ... -g3 -O2 ...), keep that binary for future debugging, but send a striped binary to the server. That way you are guaranteed to have the exact binary you need if/when the stripped binary crashes.
gdb -ex r bazel.../corefile
The above command asks gdb to run a core file, which makes no sense.
Whatever you tried to achieve here, that isn't the right way to do it.
Also, GDB (in general) can't help if you give it only the core -- for most tasks you also need the binary which produced that core.
Your first step should be to get a crash stack trace, as described e.g. here. Once you have something that looks like a reasonable stack trace, you could try swapping full-debug version of .so.
I am trying to debug Cython code that wraps a c++ class, and the error I am hunting is somewhere in the C++ code.
It would be awfully convenient if I could somehow debug as if it were written in one language, i.e. if there's an error in the C++ part, it would show me the source code line there, and if the error is in the Python part it would do the same.
Right now I always have to try and replicate the Python code using the class in C++, and right now I have an error that only occurs when running through Python ... I hope somebody can help me out :)
It's been a while for me and I forgot how I exactly did it, but when I was writing my own C/C++ library and interfaced it with swig into python, I was able to debug the C code with DDD. It was important to compile with debug options. It wasn't great, but it worked for me. I think you had to run ddd python and within the python terminal run my faulty C code. You would have to make sure all linked libraries including yours is loaded with the source code so that you could set breakpoints.
I have created a Custom Extension Command in Python. I installed it, but as expected I am getting errors (quote from SPSS log output - the only way I know for debugging Python programs in SPSS):
Extension command TEST_EXTENSION could not be loaded. The module or a module that it requires may be missing, or there may be syntax errors in it.
The error is probably from the .xmlor from the Run(args) function. The CustomFunction() I am implementing was tested thoroughly.
What would be a good practice for debugging this, and the other potential errors ? The official IBM-SPSS-Statistics-Extension-Command says to
set the
SPSS_EXTENSIONS_RAISE
environment variable to "true"
but I don't know how to do that, nor of this will work regardless of the source of the error.
#horace
You set the environment variable on Windows via the Control Panel > System > Advanced system settings > Environment Variables. The exact wording varies with different Windows versions. I usually choose System variables, although either will usually work. You need to restart Statistics after that. Once you have set this variable, errors in the Python code will produce a traceback. The traceback is ordinarily suppressed as it is of no use to users, but it is very helpful for developers.
The traceback only appears for errors in the Python code. The "could not be loaded" error you reported happens before Python gets control, so no traceback would be produced. There are two common causes for this error. The first is that the xml file defining the extension command or the corresponding Python module was not found by Statistics. The extension command definitions are loaded at Statistics startup or by running the EXTENSION command. Execute SHOW EXT. from the Syntax Editor to see the places where Statistics looks for extension files.
The second cause is a syntax error in the Python code. Run
begin program.
import yourmodule
end program.
to see if any errors are reported.
More generally, there are two useful strategies for debugging. The first is to run the code in external mode, where you run the code from Python. That way you can step through the code using your IDE or the plain Python debugger. See the programmability documentation for details. There are some limitations on what can be done in external mode, but it is often a good solution.
The second is to use an IDE that supports remote debugging. I use Wing IDE, but there are other IDEs that can do this. That lets me jump into the debugger from within Statistics, step through the Python code, and do all the other things you want in a debugger.
HTh
I have a project which involves some (fairly simple) C-code generation as part of a build system. In essence, I have some version information associated with a project (embedded C) which I want to expose in my binary, so that I can easily determine what firmware version was programmed to a particular device for debugging purposes.
I'm writing some simplistic python tools to do this, and I want to make sure they're thoroughly tested. In general, this has been fairly straightforward, but I'm unsure what the best strategy is for the code-generation portion. Essentially, I want to make sure that the generated files both:
Are syntactically correct
Contain the necessary information
The second, I can (I believe) achieve to a reasonable degree with regex matching. The first, however, is something of a bigger task. I could probably use something like pycparser and examine the resulting AST to accomplish both goals, but that seems like an unnecessarily heavyweight solution.
Edit: A dataflow diagram of my build hierarchy
Thanks for the diagram! Since you are not testing for coverage, if it were me, I would just compile the generated C code and see if it worked :) . You didn't mention your toolchain, but in a Unix-like environment, gcc <whatever build flags> -c generated-file.c || echo 'Oops!' should be sufficient.
Now, it may be that the generated code isn't a freestanding compilation unit. No problem there: write a shim. Example shim.c:
#include <stdio.h>
#include "generated-file.c"
main() {
printf("%s\n", GENERATED_VERSION); //or whatever is in generated-file.c
}
Then gcc -o shim shim.c && diff <(./shim) "name of a file holding the expected output" || echo 'Oops!' should give you a basic test. (The <() is bash process substitution.) The file holding the expected results may already be in your git repo, or you might be able to use your Python routine to write it to disk somewhere.
Edit 2 This approach can work even if your actual toolchain isn't amenable to automation. To test syntactic validity of your code, you can use gcc even if you are using a different compiler for your target processor. For example, compiling with gcc -ansi will disable a number of GNU extensions, which means code that compiles with gcc -ansi is more likely to compile on another compiler than is code that compiles with full-on, GNU-extended gcc. See the gcc page on "C Dialect Options" for all the different flavors you can use (ditto C++).
Edit Incidentally, this is the same approach GNU autoconf uses: write a small test program to disk (autoconf calls it conftest.c), compile it, and see if the compilation succeeded. The test program is (preferably) the bare minimum necessary to test if everything is OK. Depending on how complicated your Python is, you might want to test several different aspects of your generated code with respective, different shims.