I have a python script compiled to an EXE without the knowledge of the compiler that was used.
I've been searching for an answer for a while but I can't seem to find anything about this. Most, if not all only show how to decompile Pyinstaller or py2exe using scripts that are on Github and some are just outdated. However, I have attempted something different. Like code execution through the application but from what I have tried, none have worked.
I've attempted PyMem, and Pynject along with some others.
pymem:
import pymem
import subprocess
try:
mem = pymem.Pymem("test.exe")
except:
subprocess.Popen("test.exe")
mem = pymem.Pymem("test.exe")
mem.inject_python_interpreter()
code = """
import os
os.system("cls")
print("some text")
# or anything really, I've tried to write functions or even print/write __file__ then nothing happens
"""
mem.inject_python_shellcode(code)
Any help would be appreciated.
Related
Short version: How do I debug frozen libs? Can I tell Python not to use them (and use sourcefiles) or re-freeze them some how?
I'm trying to learn more about the inner functionality of pythons import mechanisms. And by going down that rabbit hole, I want to debug /usr/lib/python37/importlib/_bootstrap_external.py.
Using python -m trace --trace myscript.py gave me indications of where in _bootstrap_external.py I am landing, but not values. So I turned to pdb and it gave me a bit of information but appears to be skipping _frozen objects. (?).
So I added a few print('moo') where I saw my script ending up in _boostrap_external.py but no prints are ever made, because Python internally uses <class '_frozen_importlib_external.PathFinder'>
That lead me in a desperate attempt to try and renaming/removing /usr/lib/python37/importlib/__pycache__ in a hope that Python would re-compile my changes. But no such luck, Python still uses a frozen version.
So I modified /usr/lib/python37/__init__.py where it imports the _bootstrap module, so I changed this:
try:
import _frozen_importlib_external as _bootstrap_external
except ImportError:
from . import _bootstrap_external
_bootstrap_external._setup(_bootstrap)
_bootstrap._bootstrap_external = _bootstrap_external
else:
_bootstrap_external.__name__ = 'importlib._bootstrap_external'
_bootstrap_external.__package__ = 'importlib'
try:
_bootstrap_external.__file__ = __file__.replace('__init__.py', '_bootstrap_external.py')
except NameError:
# __file__ is not guaranteed to be defined, e.g. if this code gets
# frozen by a tool like cx_Freeze.
pass
sys.modules['importlib._bootstrap_external'] = _bootstrap_external
to this:
from . import _bootstrap_external
_bootstrap_external._setup(_bootstrap)
_bootstrap._bootstrap_external = _bootstrap_external
_bootstrap_external.__name__ = 'importlib._bootstrap_external'
_bootstrap_external.__package__ = 'importlib'
try:
_bootstrap_external.__file__ = __file__.replace('__init__.py', '_bootstrap_external.py')
except NameError:
# __file__ is not guaranteed to be defined, e.g. if this code gets
# frozen by a tool like cx_Freeze.
pass
sys.modules['importlib._bootstrap_external'] = _bootstrap_external
And that worked, ish. Obviously the problem will become more and more complex, and there has to be a more generic solution. But I continued my journey and found that _bootstrap.py is indeed loaded with my print('moo') changes. But when /usr/lib/python37/importlib/_bootstrap.py later calls the function _find_spec, and it does:
for finder in meta_path:
with _ImportLockContext():
find_spec = finder.find_spec
spec = find_spec(name, path, target)
find_spec will refer to the frozen version again. (Code snippet shortened to the relevant code in _bootstrap.py)
So my question finally ends up in, how do I debug these frozen files? or how do I make it so Python ignores frozen libraries (I can take the performance impact when debugging). Every attempt to find information on this (searching, docs, irc, etc) ends up on "why?" or "don't do this" or it points me to py2exe and how to freeze my libraries. I just want to be able to debug and understand more of how the inner mechanics work by trying things and looking at the variables.
I want to have an object, generated by an existing MATLAB script, modelled in FreeCAD. See bolded titles for important content of Intro, Matlab code, python code, Testing via terminal, Matlab error, Notes (OS, versions)
Intro
I have each side of the process worked out, but I am having issues with the calling of the python script through MATLAB.
I have both object constructor, and a script that works in building things in FreeCAD; so essentially, I have either end. What I need now is to interface them.
My test is basic, but should have been enough, or so I assumed.
Matlab:
pyfile = '~/Desktop/FreeCADworkspace/testvec2.py';
filename = 'atestname';
call = "python " + pyfile + " " + filename;
[status,result] = system("sh ~/Desktop/FreeCADworkspace/freecad.sh")
Python:
# Import python roots
import sys
import os
sys.path.append('/usr/lib/freecad-python2/lib')
sys.path.append('/usr/lib/freecad/lib')
# Other imports
import numpy as np
import math
# Import FreeCAD and parts
import FreeCAD
from FreeCAD import Base
import Part, Sketcher, Draft
try:
filename = sys.argv[1]
except: # this should never be thrown, exists for testing and other integration
print('No arguments called into script.')
print('Please use format: python thisScript.py arg_filename')
quit() # exit script
print(filename) # checkmeplz
So, essentially what should happen is that I should get a print-out of the definition of filename in MATLAB. In this case, I should see 'atestname'
Testing
If I call this through terminal:
python ~/Desktop/FreeCADworkspace/testvec2.py atestname
I get:
FreeCAD 0.18.1, Libs: 0.18.1R
atestname
Which is exactly as expected. The same cannot be said of my MATLAB
MATLAB ERROR
Traceback (most recent call last):
File "/home/ashaiden/Desktop/FreeCADworkspace/testvec2.py", line 11, in <module>
import FreeCAD # no FreeCADGui??
ImportError: /usr/lib/freecad-python2/lib/libFreeCADBase.so: undefined symbol: _ZN11xercesc_3_111InputSource11setEncodingEPKt
I have also tried to execute the python script from MATLAB via a bash script. Same error.
What is confusing me: why would some command that executes perfectly through the terminal, be failing when calling via MATLAB?
I feel like I am misunderstanding how MATLAB system calls are performed. I assumed that the call was sent to, and processed by, the OS itself. But where that the case, it would not make sense to be getting an error. Does MATLAB do some interpreting of files that it is handling?
Notes
Operating system: Ubuntu 16.04
MATLAB 2018b
Python 2.7 -> is this potentially the issue? Python 2 seemed to be the default language for my particular FreeCAD install.
FreeCAD 0.18.1
Edit
After much searching by myself and my supervisor, it appears that this might be able to be put down to a conflict between binary files.
MATLAB has libxerces-c.so files defined inside /bin/glnxa64/ and another toolbox folder.
These may be conflicting with the linux binary /usr/lib/x86_64-linux-gnu/libxerces-c.so such that when the terminal is called via matlab, it uses the matlab binary instead of the system binary.
I will continue to investigate further.
Answering my own questions!
Edit 30/7 : This does not actually adequately solve the problem for linux, given that paths may have some slight difference between machines. It also does not solve the problem for mac or windows (the issue does occur on mac, but I have no data if it occurs on windows.
After digging and searching, I edited the question:
MATLAB has libxerces-c.so files defined inside /bin/glnxa64/ and
another toolbox folder.
These may be conflicting with the linux binary
/usr/lib/x86_64-linux-gnu/libxerces-c.so such that when terminal is
called via matlab, it uses the matlab binary instead of the system
binary.
I was correct that this was due to a conflict between MATLAB's bin libxerces-c.so and the linux native libxerces-c.so. I have found at least a partial fix for this. I do not know of its robustness. At worst, the path will be edited and then returned to original format each time FreeCAD will be called within my code (gross).
SOLUTION:
Important first step: oldpath = getenv("LD_LIBRARY_PATH") to get and then SAVE some record of the current path. This, for me, is in case some
issues or conflicts arise later.
I then removed the section of the path '/usr/local/MATLAB/R2018b/bin/glnxa64/' and saved the string to newpath
I then made this the new path: setenv("LD_LIBRARY_PATH", newpath),
where newpath no longer has the reference to the conflicting
folder
And now I am getting the expected result in my MATLAB command window:
FreeCAD 0.18.1, Libs: 0.18.1R
atestname
The comparative paths:
Newpath = '/usr/local/MATLAB/R2018b/sys/opengl/lib/glnxa64:/usr/local/MATLAB/R2018b/sys/os/glnxa64:/usr/local/MATLAB/R2018b/extern/lib/glnxa64:/usr/local/MATLAB/R2018b/runtime/glnxa64:/usr/local/MATLAB/R2018b/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2018b/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/lib/x86_64-linux-gnu/';
Oldpath = '/usr/local/MATLAB/R2018b/sys/opengl/lib/glnxa64:/usr/local/MATLAB/R2018b/sys/os/glnxa64:/usr/local/MATLAB/R2018b/bin/glnxa64:/usr/local/MATLAB/R2018b/extern/lib/glnxa64:/usr/local/MATLAB/R2018b/runtime/glnxa64:/usr/local/MATLAB/R2018b/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2018b/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/lib/x86_64-linux-gnu/';
Whenever I try to display symbolic math in Spyder via the IPython console, several black console windows pop up and then disappear in quick succession. It prints the expression, but I'd like to know if there is a way to get rid of these windows. The windows have the title "C:\Program Files\MikTex 2.9..." if that helps.
It looks like someone already figured it out and posted a solution on GitHub. This is the link: https://github.com/sympy/sympy/issues/11882
It took me (as a novice) some time to figure out exactly what he did, so the following is just a more detailed explanation:
You first need to find the compatibility module in the sympy package. For me, it was located at "C:\Users\Lucas\Anaconda3\Lib\site-packages\sympy\core\compatibility.py". Next, you need to search (in the source code of that module) for the check_output function. The surrounding code should look something like:
# check_output() is new in Python 2.7
import os
try:
try:
from subprocess import check_output
Finally, you need to get rid of the last line, and replace it with the code found in the GitHub link. The resulting block should look like:
# check_output() is new in Python 2.7
import os
try:
try:
from subprocess import check_output as subprocess_check_output
def check_output(*args, **kwargs):
return subprocess_check_output(*args, **kwargs, creationflags=0x08000000) # CREATE_NO_WINDOW
It appears to me that he defines a function which takes the place of check_output, except that the argument to suppress the output windows is always fed in. Hope this helps anyone else having this problem, and I appreciate the fix from Adam on GitHub.
I submitted a pull request to fix this for good:
https://github.com/sympy/sympy/pull/12391
I'm trying to run an R Script through Python using subprocess but unfortunately, I'm continually getting the following error:
WindowsError: [Error 2] The system cannot find the file specified
This is extremely frustrating as I've checked the path over everything multiple times (done it without C:/, moved to different directories, changed the R script I'm trying to run, etc). I've also checked the CompSec Environment Variable, and it is also correct (through Windows Systems, although I never checked it directly in my IDE, Spyder). I've tried doing it in Python 2.7 and Python 3.5, and neither version works.
The code goes as follows:
import subprocess
def sort_ByInputColumn(inputPath, inputFileTermination, sortColumn, outputPath, outputFileTermination):
scriptPath = "C:/Users/Kyle/Documents/Rscript_SortByInputColumn.R"
subprocess.call(["Rscript", scriptPath, inputPath, inputFileTermination, sortColumn, outputPath, outputFileTermination])
fileName = 'Alabama'
outputPath = "C:/Users/Kyle/Documents/HillData/Data/Output/Module2/"
sortColumn = str(16)
inputTermination = fileName + 'Module2NN_WorkCounty_Work.csv'
outputFileTermination = fileName + 'Module2NN_SortedWorkCounty.csv'
sort_ByInputColumn(outputPath, inputTermination, sortColumn, outputPath, outputFileTermination)
The fact that I get this error no matter what code I try to run (even blatantly copy-pasting this tutorial to try and make it work) makes me feel like something deeper (or something extremely obvious) is going on that I'm not seeing.
Would appreciate any feedback on the matter.
To solve this most annoying and horrendous problem, I reinstalled R into a directory with no spaces and called Rscript with its full pathing (that is "C:/R/R-3.3.1/bin/Rscript.exe" from C:/Program Files/R/R-3.3.1/bin/Rscript.exe" because Program Files has a space and this kills subprocess because it's command line based, I guess). This time, it worked.
See this similar question for a hint as to where I got the inspiration for this.
I'm having an interesting problem with threads and the tempfile module in Python. Something doesn't appear to be getting cleaned up until the threads exit, and I'm running against an open file limit. (This is on OS X 10.5.8, Python 2.5.1.)
Yet if I sort of replicate what the tempfile module is doing (not all the security checks, but just generating a file descriptor and then using os.fdopen to produce a file object) I have no problems.
Before filing this as a bug with Python, I figured I'd check here, as it's much more likely that I'm doing something subtly wrong. But if I am, a day of trying to figure it out hasn't gotten me anywhere.
#!/usr/bin/python
import threading
import thread
import tempfile
import os
import time
import sys
NUM_THREADS = 10000
def worker_tempfile():
tempfd, tempfn = tempfile.mkstemp()
tempobj = os.fdopen(tempfd, 'wb')
tempobj.write('hello, world')
tempobj.close()
os.remove(tempfn)
time.sleep(10)
def worker_notempfile(index):
tempfn = str(index) + '.txt'
# The values I'm passing os.open may be different than tempfile.mkstemp
# uses, but it works this way as does using the open() function to create
# a file object directly.
tempfd = os.open(tempfn,
os.O_EXCL | os.O_CREAT | os.O_TRUNC | os.O_RDWR)
tempobj = os.fdopen(tempfd, 'wb')
tempobj.write('hello, world')
tempobj.close()
os.remove(tempfn)
time.sleep(10)
def main():
for count in range(NUM_THREADS):
if count % 100 == 0:
print('Opening thread %s' % count)
wthread = threading.Thread(target=worker_tempfile)
#wthread = threading.Thread(target=worker_notempfile, args=(count,))
started = False
while not started:
try:
wthread.start()
started = True
except thread.error:
print('failed starting thread %s; sleeping' % count)
time.sleep(3)
if __name__ == '__main__':
main()
If I run it with the worker_notempfile line active and the worker_tempfile line commented-out, it runs to completion.
The other way around (using worker_tempfile) I get the following error:
$ python threadtempfiletest.py
Opening thread 0
Opening thread 100
Opening thread 200
Opening thread 300
Exception in thread Thread-301:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 460, in __bootstrap
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 440, in run
File "threadtempfiletest.py", line 17, in worker_tempfile
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tempfile.py", line 302, in mkstemp
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tempfile.py", line 236, in _mkstemp_inner
OSError: [Errno 24] Too many open files: '/var/folders/4L/4LtD6bCvEoipksvnAcJ2Ok+++Tk/-Tmp-/tmpJ6wjV0'
Any ideas what I'm doing wrong? Is this a bug in Python, or am I being bone-headed?
UPDATE 2009-12-14:
I think I've found the answer, but I don't like it. Since nobody was able to replicate the problem, I went hunting around our office for machines. It passed on everything except my machine. I tested on a Mac with the same software versions I was using. I even went hunting for a Desktop G5 with the EXACT same hardware and software config I had -- same result. Both tests (with tempfile and without tempfile) succeeded on everything.
For kicks, I downloaded Python 2.6.4, and tried it on my desktop, and same pattern on my system as Python 2.5.1: tempfile failed, and notempfile succeeded.
This is leading me to the conclusion that something's hosed on my Mac, but I sure can't figure out what. Any suggestions are welcome.
I am unable to reproduce the problem with (Apple's own build of) Python 2.5.1 on Mac OS X 10.5.9 -- runs to completion just fine!
I've tried both on a Macbook Pro, i.e., an Intel processor, and an old PowerMac, i.e., a PPC processor.
So I can only imagine there must have been a bug in 10.5.8 which I never noticed (don't have any 10.5.8 around to test, as I always upgrade promptly whenever software update offers it). All I can suggest is that you try upgrading to 10.5.9 and see if the bug disappears -- if it doesn't, I have no idea how this behavior difference between my machines and yours is possible.
I think your answer can be found here. You have to explicitly os.close() the file descriptor given as the first part of the tuple that mkstemp gives you.
Edit: no, the OP is already doing what is supposed to be done. I'm leaving the answer up for the nice link.
I just tested your code on my Ubuntu Linux computer here, and it worked perfectly for me.
I have one suggestion for you to try. I don't know that it will help but it can't hurt. Rewrite your code to use with:
from __future__ import with_statement
def worker_tempfile():
tempfd, tempfn = tempfile.mkstemp()
with os.fdopen(tempfd, 'wb') as tempobj:
tempobj.write('hello, world')
os.remove(tempfn)
time.sleep(10)
The with statement is supposed to make sure that the file object gets closed no matter what. Perhaps it might help?
Good luck. Great job on the question, by the way.
Why do you think the error is not genuine? You are launching 10000 threads, each opening a file, while the maximum number of open files is typically 1024 under Unix systems.
First try to keep manually track of the number of files currently open and check whether it bumps past the OS limit.
Since nobody was able to replicate the problem, I went hunting around our office for machines. It passed on everything except my machine. I tested on a Mac with the same software versions I was using. I even went hunting for a Desktop G5 with the EXACT same hardware and software config I had -- same result. Both tests (with tempfile and without tempfile) succeeded on everything.
For kicks, I downloaded Python 2.6.4, and tried it on my desktop, and same pattern on my system as Python 2.5.1: tempfile failed, and notempfile succeeded.
This is leading me to the conclusion that something's hosed on my Mac, so this isn't likely to be a problem that anyone else runs into ever.
Thanks VERY much to everyone (especially Alex Martelli) who helped on this!