We Keep Coding

Pycharm - using pwntools with remote interpreter on WSL

I am using remote interpreter on pycharm on WSL (configured it with this tutorial: https://www.jetbrains.com/help/pycharm/using-wsl-as-a-remote-interpreter.html)
I was able to run everything I needed successfully, but when I tried to use pwntools (https://github.com/Gallopsled/pwntools) I was able to import it successfully on the WSL bash python interpreter, but not on Pycharm.
This is what I ran:
from pwn import *
On Pycharm it was stucked and I interrupted it, this is the trace of the Exception (where it stucks):
ssh://shahar#localhost:22/usr/bin/python -u /tmp/pycharm_project_271/pwnablekr/fd.py
Traceback (most recent call last):
File "/tmp/pycharm_project_271/pwnablekr/fd.py", line 1, in <module>
from pwn import *
File "/home/shahar/.local/lib/python2.7/site-packages/pwn/__init__.py", line 6, in <module>
pwnlib.args.initialize()
File "/home/shahar/.local/lib/python2.7/site-packages/pwnlib/args.py", line 208, in initialize
term.init()
File "/home/shahar/.local/lib/python2.7/site-packages/pwnlib/term/__init__.py", line 74, in init
term.init()
File "/home/shahar/.local/lib/python2.7/site-packages/pwnlib/term/term.py", line 109, in init
c = os.read(fd.fileno(), 1)
KeyboardInterrupt
Process finished with exit code 1
enter code here
On my WSL bash it ran just fine:
shahar#MYCOMPUTERNAME:/mnt/c/Users/shahar$ python
Python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pwn import *
>>>
When I looked at the piece of code where it stuck (from the trace of the Exception):
while True:
c = os.read(fd.fileno(), 1)
s += c
if c == 'R':
break
at the beginning of the script as a global variable:
fd = sys.stdout
I understood from the internet that this function (which this loop is part of it) is related to take over the terminal. Maybe it is related to the fact I am not running from terminal?
Had anyone seen this kind of problem before? has some helpful tips?
Thank you very much!

I have a potential fix as well, and it's adding a PWNLIB_NOTERM to the environment.
import os
os.environ['PWNLIB_NOTERM'] = 'True' # Configuration patch to allow pwntools to be run inside of an IDE
import pwn
Screenshot showing it runs and we get an Encoder object instance

There is another way to solve it.
If you use Pycharm , you can tick the box Run with Python console in Run configurations.
It will work in Pycharm 2020.3 with IPython.(I think it also works without IPython)
screenshot

There is no effective way, I debug it, the problem is term initialization.it may also be related to the environment variables of the TERM and TERMINFO.My solution is to modify the last line of /usr/local/lib/python2.7/dist-packages/pwnlib/args.py,delete term.init(), replace it with anything else to bypass the initialization of pwnlib.
replace this line:
debug pwntools:

KeyError: 'PYTHONPATH', how can I fix PYTHONPATH? (Python 3.5.2)

I have seen this post asked before in stackoverflow but it was 4 years ago (How to fix the python path) so I am not sure wether this is the right solution because I am using a newer version of python (3.5.2). This is what I see in a Python Shell:
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import os
>>> os.environ['PYTHONPATH'].split(os.pathsep)
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
os.environ['PYTHONPATH'].split(os.pathsep)
File "C:\Users\John\AppData\Local\Programs\Python\Python35\lib\os.py", line 725, in __getitem__
raise KeyError(key) from None
KeyError: 'PYTHONPATH'
I want to find the pythonpath.So, how can I fix this error?

There is no PYTHONPATH variable in your OS Environemental variables. Hence the error.
It is not created by Python installation (atleast in windows). You have to create one variable.
To check if there is one such environmental variable, type below command:
SET PYTHONPATH
You can also create and set it using below command
SETX PYTHONPATH <your desired path>

You may want to check sys.path
A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default.

First of all make sure if you have any environment variable with this name. I also got an error like this. I used PATH instead of PYHONPATH and this worked for me. I am working on Ubuntu btw.

Matlab Engine Python - OSx Anaconda Segfault or DYLD_LIBRARY_PATH error with iPython

After installing the matlab python package in terminal using:
cd "matlabroot\extern\engines\python"
python setup.py install
And trying to run it, I get a segfault:
:~$ python
Python 2.7.10 |Anaconda 2.3.0 (x86_64)| (default, May 28 2015, 17:04:42)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import matlab.engine
Segmentation fault: 11
However, I can get around this by setting DYLD_LIBRARY_PATH after which matlab.engine works:
:~$ export DYLD_LIBRARY_PATH=/System/Library/Frameworks/Python.framework/Versions/Current/lib:$DYLD_LIBRARY_PATH
:~$ python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import matlab.engine
>>> eng = matlab.engine.start_matlab()
>>> exit()
However, when I try to launch iPython afterwards I get this error:
Traceback (most recent call last):
File "//anaconda/bin/ipython", line 4, in <module>
from IPython import start_ipython
File "//anaconda/lib/python2.7/site-packages/IPython/__init__.py", line 45, in <module>
from .config.loader import Config
File "//anaconda/lib/python2.7/site-packages/IPython/config/__init__.py", line 6, in <module>
from .application import *
File "//anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 19, in <module>
from IPython.config.configurable import SingletonConfigurable
File "//anaconda/lib/python2.7/site-packages/IPython/config/configurable.py", line 12, in <module>
from .loader import Config, LazyConfigValue
File "//anaconda/lib/python2.7/site-packages/IPython/config/loader.py", line 16, in <module>
from IPython.utils.path import filefind, get_ipython_dir
File "//anaconda/lib/python2.7/site-packages/IPython/utils/path.py", line 14, in <module>
import tempfile
File "//anaconda/lib/python2.7/tempfile.py", line 32, in <module>
import io as _io
File "//anaconda/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(//anaconda/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyErr_ReplaceException
Referenced from: //anaconda/lib/python2.7/lib-dynload/_io.so
Expected in: dynamic lookup
As you can see the python versions are different. I think this is a conflict between my system Python and Anaconda but I'm not sure how to fix it, any help much appreciated.
Thanks.

You need to install libraries using the conda command line install utility, not python. conda help search and conda help install should get you going.

This is not a direct solution, but I was stuck with the same problem and used this workaround. And I should say that I did finally make matlab.engine run in my Python environment, but that it just plain sucked. It couldn't run scripts with embedded scripts in other folders, and I also experienced that it couldn't find some build-in functions. The following is implemented for a Unix machine, but it would not take a lot of modification to make it work in Windows, I believe.
Here's what I did:
Wrote a main Matlab script that could do everything I needed from Matlab.
Ran that script through a subprocess in Python.
In my case I needed do a series of matrix operations for matrix X and return matrix S. The matrix operations required the use of a particular Matlab function. My first idea was to open a Matlab-session with matlab.engine and then manage the matrix operations in Python only calling the Matlab function when needed. Instead (as the bullets state) I wrote a Matlab function ComputeSimilarityMat that takes X, does all necessary operations, and returns S. Then I basically just ran that Matlab function from Python using a subprocess.
This is what the Python script, that manages the Matlab script through a subprocess, looks like:
import subprocess
import numpy as np
def run_matlab(X):
"""Produce trait-similarity matrix S from a trait-space matrix X
Parameters
----------
X : numpy.ndarray
Returns
-------
S : numpy.ndarray
"""
# Dump input in .csv, so Matlab can load it
np.savetxt('X.csv', X, delimiter=",")
# Code executed in Matlab. Important to end with quit;
matlab_code = "X=csvread('X.csv');" \
"S=ComputeSimilarityMat(X);" \
"csvwrite('S.csv',S);" \
"quit;"
# -nosplash suppresses the splash window on startup
matlab_call = ["matlab", "-nodesktop", "-nosplash", "-r", matlab_code]
subprocess.call(matlab_call)
# Load the .csv file that Matlab dumps after finishing
S = np.genfromtxt('X.csv', delimiter=",")
return S
I have to say, I'm sure there's a nicer way to pass an object to Matlab then to have to save and load like this, but it worked for me. Still hoping for an improvement of matlab.engine though, but until then I'm going with this approach.
Note: To run Matlab from command-line/subprocess you need to add the Matlab bin folder to your PATH variable like this:
export PATH="/Applications/MATLAB_R2015a.app/bin:$PATH"
I just put this in my .profile file.

In my case adding things to LD_LIBRARY_PATH (Ubuntu's version of DYLD_LIBRARY_PATH) only made things worse. Instead I had to make sure it did not refer to the Python installation, and had to add a symbolic link from /usr/lib instead. See https://stackoverflow.com/a/45161288/2524427

Matplotlib won't plot: dbus issue?

I have a rather strange scenario. I'm running on a very old installation, so I've compiled most of the tools I use.
First, this is a server. It is exclusively used by SSH-ing into it. But whoever set this server up made some strange choices. Instead of creating separate users, there is one user on login called fphyiscs, with a home directory of /home/fphysics. (Access is controlled strictly via public key/private key authentication. So everybody logs in as "fphysics", but there are no passwords.) Then there are "profile" directories in this one. For instance, mine is /home/fphysics/joelfred. There is a script that is run by fphysics on login that looks through the directories in /home/fphysics and interprets these as "profile". You can then choose which "profile" you want to use. The script then sources a file called "ENVS" in this profile directory.
I thought this was weird. But I know better than to argue with a sysadmin. So I work within this system. I set up my ENVS to source a more conventional .bash_profile, instead of just setting up environment variables as they probably intended. I set my HOME environment variable in ENVS: export HOME=/home/fphysics/joelfred. The software on this system is so outdated that I've compiled tons of my own stuff, using a prefix of $HOME/opt. So my path now includes $HOME/opt/bin. I sym-linked /home/fphysics/.Xauthority to /home/fphyiscs/joelfred/.Xauthority, so x sessions work over SSH. I honestly don't remember all the hacky things I've done to make this work more normally, and so I can have access to more current software. (For instance, GCC is v4.1.2, copyright 2006.)
I've since tried to compile and install Matplotlib, with the GTK3Agg backend. This means, of course, that I had to compile a lot of x libraries, GTK, pygobject, etc., etc. At some point along the line, I had to compile DBUS. Now, I have some interesting/annoying Matplotlib behavior. This, for instance, works as expected:
fphysics#facet-srv01 ~$ ipython
Python 2.7.3 (default, Apr 30 2013, 00:58:45)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import matplotlib.pyplot as plt
** (ipython:1648): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-oWyQu0DxxF: Connection refused
In [2]: import numpy as np
In [3]: x=np.linspace(-10,10,100)
In [4]: y=x*x
In [5]: plt.plot(x,y)
Out[5]: [<matplotlib.lines.Line2D at 0xacb2eec>]
In [6]: plt.show()
This works as expected - I get a plot of a parabola. I get a window that blocks my ipython terminal. The window, as far as I can tell, has full functionality. When I close it, I get returned to my ipython terminal. Note the strange dbus error though - I have NO IDEA where it gets the /tmp/dbus-oWyQu0DxxF idea. I've looked and I don't get it.
Okay. So now I want to turn on interactive mode:
fphysics#facet-srv01 ~$ ipython
Python 2.7.3 (default, Apr 30 2013, 00:58:45)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import matplotlib.pyplot as plt
** (ipython:4181): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-oWyQu0DxxF: Connection refused
In [2]: import numpy as np
In [3]: plt.ion()
In [4]: x=np.linspace(-10,10,100)
In [5]: y=x*x
In [6]: plt.plot(x,y)
Out[6]: [<matplotlib.lines.Line2D at 0xb0f1fcc>]
I get nothing. Weird. I try a show:
In [7]: plt.show()
Nothing. I try a draw:
In [8]: plt.draw()
I get a window with a title of "Figure 1", but it's gray. I try another draw:
In [9]: plt.draw()
Now I get a window with my parabola. But no buttons work. I can't even close the window. So I try a close:
In [10]: plt.close()
Nothing happens. Okay. I'll exit my ipython terminal:
In [11]: exit
Window goes away, no errors. But what happens if I don't do a show or a draw or just exit? I get weird errors:
fphysics#facet-srv01 ~$ ipython
Python 2.7.3 (default, Apr 30 2013, 00:58:45)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import matplotlib.pyplot as plt
** (ipython:7262): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-oWyQu0DxxF: Connection refused
In [2]: import numpy as np
In [3]: plt.ion()
In [4]: x=np.linspace(-10,10,100)
In [5]: y=x*x
In [6]: plt.plot(x,y)
Out[6]: [<matplotlib.lines.Line2D at 0xb636bf6c>]
In [7]: exit
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/fphysics/joelfred/opt/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/fphysics/joelfred/opt/lib/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 87, in destroy_all
manager.destroy()
File "/home/fphysics/joelfred/opt/lib/python2.7/site-packages/matplotlib/backends/backend_gtk3.py", line 415, in destroy
self.canvas.destroy()
AttributeError: FigureManagerGTK3Agg instance has no attribute 'canvas'
Error in sys.exitfunc:
Traceback (most recent call last):
File "/home/fphysics/joelfred/opt/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/fphysics/joelfred/opt/lib/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 87, in destroy_all
manager.destroy()
File "/home/fphysics/joelfred/opt/lib/python2.7/site-packages/matplotlib/backends/backend_gtk3.py", line 415, in destroy
self.canvas.destroy()
AttributeError: FigureManagerGTK3Agg instance has no attribute 'canvas'
So I dug into dbus - the only error I can see that may be affecting things. I have two installations apparently:
fphysics#facet-srv01 ~$ $HOME/opt/bin/dbus-daemon --version
D-Bus Message Bus Daemon 1.7.2
Copyright (C) 2002, 2003 Red Hat, Inc., CodeFactory AB, and others
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
fphysics#facet-srv01 ~$ /bin/dbus-daemon --version
D-Bus Message Bus Daemon 1.1.2
Copyright (C) 2002, 2003 Red Hat, Inc., CodeFactory AB, and others
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Maybe they're competing? I don't know. I've symlinked /home/fphysics/.dbus to /home/fphysics/joelfred/.dbus. That doesn't seem to help. I'm at a loss. I have no idea what's going on. How do I diagnose this? How do I get dbus to work in my scenario? How do I test it on a more basic level? What in the world is going on??
BTW, Ipython makes the testing easier, but the behavior is the same if I run from a python prompt.

Django + MySQL on Mac OS 10.6.2 Snow Leopard

There were some excellent answers to this question already, however, they are now outdated.
I've been able to get the module installed, but "python manage.py runserver" fails with
iMac:myproject drhoden$ python manage.py runserver
Validating models...
Unhandled exception in thread started by <function inner_run at 0x10496f0>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/commands/runserver.py", line 48, in inner_run
self.validate(display_num_errors=True)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 249, in validate
num_errors = get_validation_errors(s, app)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/validation.py", line 22, in get_validation_errors
from django.db import models, connection
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/__init__.py", line 41, in <module>
backend = load_backend(settings.DATABASE_ENGINE)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/__init__.py", line 17, in load_backend
return import_module('.base', 'django.db.backends.%s' % backend_name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 13, in <module>
raise ImproperlyConfigured("Error loading MySQLdb module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: dynamic module does not define init function (init_mysql)
^CiMac:segisys drhoden$
Likewise, from the python shell:
iMac:myproject drhoden$ python
Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.3-fat/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.3-fat/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.3-fat/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dynamic module does not define init function (init_mysql)
>>>
Using MySQL-python-1.2.3c1 with setuptools-0.6c11-py2.6.egg
Any help would be appreciated.

I have ultimately solved my own problem, with of course, the subconscious and conscious help from the many posts, blogs, and mail logs I've read. I would give links if I could remember.
In a nutshell, I reinstalled EVERYTHING using MacPorts.
After editing ~/.bash_profile and commenting out all the previous modifications to ${PATH}, I downloaded the dmg for Snow Leopard and ran through its installation.
Then opened the terminal and ran the self update.
sudo port selfupdate
sudo port install python26
That second part, installing Python 2.6, took forever. But when it completed it prompted me with the following:
To fully complete your installation and make python 2.6 the default, please run
sudo port install python_select
sudo python_select python26
I did both and they went quick.
I forgot to mention how handy 'port search ' command is. I searched for 'mysql' and similar to find the thing to type after 'install'. But I proceeded with reinstalling both the client and server for MySQL. Perhaps I did this in reverse order, but the end result worked fine.
sudo port install mysql5
...
---> Installing mysql5 #5.1.41_0
The MySQL client has been installed.
If you also want a MySQL server, install the mysql5-server port.
So naturally:
sudo port install mysql5-server
I love how the so many of the macports installations give you feedback as to what to do next. At the end of the server installation, it said the following:
******************************************************
* In order to setup the database, you might want to run
* sudo -u _mysql mysql_install_db5
* if this is a new install
******************************************************
It was a new install for me (didn't have any local schemas). For completeness, and for my own reference, here is the output of running that command:
Installing MySQL system tables...
OK
Filling help tables...
OK
To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/opt/local/lib/mysql5/bin/mysqladmin -u root password 'new-password'
/opt/local/lib/mysql5/bin/mysqladmin -u root -h iMac.local password 'new-password'
Alternatively you can run:
/opt/local/lib/mysql5/bin/mysql_secure_installation
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the manual for more instructions.
You can start the MySQL daemon with:
cd /opt/local ; /opt/local/lib/mysql5/bin/mysqld_safe &
You can test the MySQL daemon with mysql-test-run.pl
cd /opt/local/mysql-test ; perl mysql-test-run.pl
Please report any problems with the /opt/local/lib/mysql5/bin/mysqlbug script!
The latest information about MySQL is available at http://www.mysql.com/
Support MySQL by buying support/licenses from http://shop.mysql.com/
Almost done. Earlier in my 'port search'ing I came across this interesting port:
py26-mysql #1.2.2 (python, devel, databases)
Python interface to mysql
With much, much hope that this would provide me with MySQLdb package, I installed it (and it did).
sudo port install py26-mysql
Afterwords I cranked up the python interpreter attempted to import MySQLdb, the very thing in my way all this time.
iMac:~ drhoden$ python
Python 2.6.4 (r264:75706, Dec 15 2009, 18:00:14)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated from sets import ImmutableSet
>>>
A warning, but It worked!!
Just one more thing:
sudo port install py26-django
After all of this I was finally able to crank up my Django project and remotely connect to my company's MySQL server!! It may not have been necessary to reinstall Django using MacPorts, but I wasn't going to risk complications.

I wrote a blog post a few months ago following my successful installation of MySQL on Snow Leopard:
http://jboxer.com/2009/09/installing-mysql-on-snow-leopard/
If you follow those steps, it should (theoretically) fix your problem (which sounds like it's caused by a mix of 32-bit and 64-bit software).
By the way, I'm not trying to self-promote here; the text in the blog post is basically what I would've posted here, and I'm trying to apply DRY to more areas of my life :)

This happens when you have mixed 32 and 64bit software.
Basically, for Snow Leopard, you need to install MySQL 64bit package (which still is listed as 10.5, but that is no problem) , after that do an easy install of python-mysql again. All will work.