We Keep Coding

psutil.AccessDenied when using StanfordCoreNLP in Pycharm? [duplicate]

This question already has answers here:
psutil.AccessDenied Error while trying to load StanfordCoreNLP
(3 answers)
Closed 2 years ago.
# coding=utf-8
from stanfordcorenlp import StanfordCoreNLP
nlp = StanfordCoreNLP(r'/Users/silas/stanford-corenlp/', lang='zh')
sentence = '清华大学位于北京。'
print nlp.word_tokenize(sentence)
print nlp.pos_tag(sentence)
print nlp.ner(sentence)
print nlp.parse(sentence)
print nlp.dependency_parse(sentence)
nlp.close()
I'm using Mac. Java, NLKT, and Stanforcorenlp toolkit are all ready. When I'm testing the project, the error came out.
Traceback (most recent call last):
line 5, in <module>
nlp = StanfordCoreNLP(r'/Users/silas/stanford-corenlp/', lang='zh')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/stanfordcorenlp/corenlp.py", line 79, in __init__
if port_candidate not in [conn.laddr[1] for conn in psutil.net_connections()]:
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/psutil/__init__.py", line 2120, in net_connections
return _psplatform.net_connections(kind)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/psutil/_psosx.py", line 255, in net_connections
cons = Process(pid).connections(kind)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/psutil/_psosx.py", line 347, in wrapper
raise AccessDenied(self.pid, self._name)
psutil._exceptions.AccessDenied: psutil.AccessDenied (pid=25422)
I guess it is because the Pycharm project isn't running under a root user. But how can I configure the IDE to fix the problem?

Unfortunately, if you look at the psutil project in _psosx.py, under net_connections, line 243 says..
Note: on macOS this will fail with AccessDenied unless the process is owned by root.
That means that you'll need to run as root by doing something like sudo pycharm.sh.
If you don't want to run your entire IDE as root, there's a few examples on SO on how you can run a specific script with super-user privileges. For instance see Debugging in pyCharm with sudo privileges.

This problem seems to be specific to Mac OS X which would not allow Python to check the current port.
Comment this portion of code of corenlp.py file:
if self.port is None:
for port_candidate in range(9000, 65535):
if port_candidate not in [conn.laddr[1] for conn in psutil.net_connections()]:
self.port = port_candidate
break
if self.port in [conn.laddr[1] for conn in psutil.net_connections()]:
raise IOError('Port ' + str(self.port) + ' is already in use.')
Replace by this line:
self.port = 9999
Source: https://github.com/Lynten/stanford-corenlp/issues/26#issuecomment-445507811
Another solution is to run StanfordCoreNLP with a sudo command line.

How to install packages with python scripts

I want to install a package with a python script. I have read the documentation about PackageManager API (http://doc.aldebaran.com/2-4/naoqi/core/packagemanager-api.html):
So I have packaged the app with choregraphe as it is described in http://doc.aldebaran.com/2-4/naoqi/core/packagemanager.html and I have tried to install it with a python script that looks like:
import qi
import sys
if __name__ == '__main__':
ip = "11.1.11.111"
port = 9559
session = qi.Session()
try:
session.connect("tcp://" + ip + ":" + str(port))
except RuntimeError:
print ("Can't connect to Naoqi at ip \"" + ip + "\" on port " + str(port))
sys.exit(1)
service = session.service("PackageManager")
package = "C:\\test_package_handlers_01-835a92-1.0.0.pkg"
# this is to see if the problem is that python can not locate the file
with open(package) as f:
print f
service.install(package)
And here is what I receive as an error:
# provided package could be opened
<open file 'C:\\test_package_handlers_01-835a92-1.0.0.pkg', mode 'r' at 0x02886288>
Traceback (most recent call last):
File "C:/test.py", line 24, in <module>
service.install(package)
RuntimeError: C:\test_package_handlers_01-835a92-1.0.0.pkg: no such file
I guess this is because the package must be uploaded on the robot and the package file path must be the one that is on the robot.
EDITED
I have added the package to a choreographe blank project and run this blank project on the robot. This way the package was saved to the robot with path /home/nao/.local/share/PackageManager/apps/.lastUploadedChoregrapheBehavior/test_package_handlers_01-835a92-1.0.0.pkg and when I have changed the path in my script ("C:\\test_package_handlers_01-835a92-1.0.0.pkg" with "/home/nao/.local/share/PackageManager/apps/.lastUploadedChoregrapheBehavior/test_package_handlers_01-835a92-1.0.0.pkg") the script worked as it was intended and the package was installed on the robot.
So is there a way to install packages from my PC without uploading them to the robot, because otherwise it is better to use Choregraphe to upload projects.
Maybe it is good to give the following explanation of what I want to achieve:
I have a folder on my PC with 20 packages for example
I want to install all those 20 packages with one python script
There is a python script that installs all the packages from the folder when it is invoked like this:
python package_installer.py path_to_packages_folder
EDITED_2
import qi
import ftplib
import os
ROBOT_URL = "10.80.129.90"
print "Uploading PKG"
pkg_file = "my-application-0.0.1.pkg"
pkg_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), pkg_file)
ftp = ftplib.FTP(ROBOT_URL)
ftp.login("nao", "nao")
with open(pkg_path) as pkg:
ftp.storbinary("STOR "+pkg_file, pkg)
print "Connecting NAOqi session"
app = qi.Application(url='tcp://'+ROBOT_URL+':9559')
app.start()
session = app.session
print "Installing app"
packagemgr = session.service("PackageManager")
packagemgr.install("/home/nao/"+pkg_file)
print "Cleaning robot"
ftp.delete(pkg_file)
ftp.quit()
print "End"
app.stop()
This piece of code ftp = ftplib.FTP(ROBOT_URL) throws the following exception:
Traceback (most recent call last):
File "C:/Stefan/DSK_PEPPER_clode_2/PythonScripts/_local_testing/uploading_and_installing_package.py", line 11, in <module>
ftp = ftplib.FTP(ROBOT_URL)
File "C:\Python27\lib\ftplib.py", line 120, in __init__
self.connect(host)
File "C:\Python27\lib\ftplib.py", line 135, in connect
self.sock = socket.create_connection((self.host, self.port), self.timeout)
File "C:\Python27\lib\socket.py", line 575, in create_connection
raise err
socket.error: [Errno 10061] No connection could be made because the target machine actively refused it
Also when I connect to the robot with username 'nao' and pass 'nao' as described in http://doc.aldebaran.com/2-5/dev/tools/opennao.html and then try to create a folder in /home/nao/.local/share/PackageManager/apps/ with sudo mkdir it informs me that: Sorry, user nao is not allowed to execute '/bin/mkdir dasdas' as root on Pepper.. If I use only mkdir here is what it tells me: mkdir: cannot create directory 'new_folder': Permission denied

Using qibuild, you can also directly install using:
qipkg deploy-package /path/to/my-package.pkg --url nao#10.10.23.45

You indeed need to upload the file before. You can use scp or sftp to do this. Once the .pkg is on the robot then you can use PackageManager.install.
Imagine something like:
import qi
import paramiko
import os
ROBOT_URL = "10.80.129.90"
print "Uploading PKG"
pkg_file = "my-application-0.0.1.pkg"
pkg_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), pkg_file)
transport = paramiko.Transport((ROBOT_URL, 22))
transport.connect(username="nao", password="nao")
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.put(pkg_path, pkg_file)
print "Connecting NAOqi session"
app = qi.Application(url='tcp://'+ROBOT_URL+':9559')
app.start()
session = app.session
print "Installing app"
packagemgr = session.service("PackageManager")
packagemgr.install("/home/nao/"+pkg_file)
print "Cleaning robot"
sftp.remove(pkg_file)
sftp.close()
transport.close()
print "End"
app.stop()

Can access serial port with Python 2.6, but not with 2.7 or 3.5 using same code and port

I am able to run scripts with Python 2.6.6 that open the serial port and exchange data bidirectionally. After installing both Python 2.7 and 3.5, needed for objects that cannot be executed in 2.6, I receive the same error (IOError: [Errno 110] Connection timed out).
I installed both 2.7 and 3.5 using (make altinstall) followed by (easy_install-2.7 -U pyserial) and (easy_install-3.5 -U pyserial).
Does anyone have a good guess as to why 2.6 will open the port and pass data, but both 2.7 and 3.5 fail?
I am running on CentOS 6.8, Kernel Linux 2.6.32-642.11.1.el6.x86_64 and using a USB to DB9 serial cable without any issues in 2.6.
python stest.py
Hello World from serial port running under Python 2.6.6.
python2.7 stest.py
Traceback (most recent call last):
File "stest.py", line 3, in <module>
ser = serial.Serial(port='/dev/ttyUSB0',timeout=0,xonxoff=False,rtscts=False,dsrdtr=False)
File "build/bdist.linux-x86_64/egg/serial/serialutil.py", line 182, in __init__
File "build/bdist.linux-x86_64/egg/serial/serialposix.py", line 267, in open
File "build/bdist.linux-x86_64/egg/serial/serialposix.py", line 588, in _update_rts_state
IOError: [Errno 110] Connection timed out
python3.5 stest.py
Traceback (most recent call last):
File "stest.py", line 3, in <module>
ser = serial.Serial(port='/dev/ttyUSB0',timeout=0,xonxoff=False,rtscts=False,dsrdtr=False)
File "/usr/local/lib/python3.5/site-packages/pyserial-3.2.1-py3.5.egg/serial/serialutil.py", line 236, in __init__
File "/usr/local/lib/python3.5/site-packages/pyserial-3.2.1-py3.5.egg/serial/serialposix.py", line 288, in open
File "/usr/local/lib/python3.5/site-packages/pyserial-3.2.1-py3.5.egg/serial/serialposix.py", line 605, in _update_rts_state
TimeoutError: [Errno 110] Connection timed out
The stest.py script:
import serial
ser = serial.Serial(port='/dev/ttyUSB0',timeout=0,xonxoff=False,rtscts=False,dsrdtr=False)
ser.write("Hello from CentOS box.");
while True:
chunk = ser.read(128)
if(chunk != ''):
print(chunk + '\n')
ser.close()
As suggested earlier, I did try to open the port several times. I had read that may be an issue from other posts. I also increased the timeout from 0 to 5. Still no joy. Here is that version (assuming I understood the way to make the multiple attempts):
import serial
ser = 0
for i in range (1, 20):
print("Attempt" + str(i))
try:
ser = serial.Serial(port='/dev/ttyUSB0',timeout=5,xonxoff=False,rtscts=False,dsrdtr=False)
if(ser):
break
except:
print("Failed")
ser.write("Hello from CentOS");
while True:
chunk = ser.read(24)
if(chunk != ''):
print(chunk + '\n')
ser.close()
I think this may be kernel related. I was able to exchange data by changing the rtscts value from False to 1. Unfortunately, this does not help my end goal since the remote device is 3-wire and does not support handshaking (RTS/CTS). So, I am still without a solution. The question now is:
Why does the (rtscts value of False) fail in 2.7 and 3.5, but work in 2.6.6?
The working code (but not usable to me because my equip does not handshake):
#!/usr/bin/python3.5
import serial
ser = serial.Serial(port='/dev/ttyUSB0',timeout=0,xonxoff=False,rtscts=1,dsrdtr=False)
ser.write(b"Hello from CentOS");
while True:
chunk = ser.read(24)
if(chunk):
print(str(chunk) + '\n')
ser.close()
Adding to the evolution of my discovery, it is indeed an issue in the compatibility of CentOS 6.8 and pyserial's serialutil.py and serialposix.py. I can open a serial port and read/write data just fine with rtscts = 1. If I reconfigure the port during run time and set rtscts to False, 2.6.6 handles it well. Pythons 2.7 and 3.5 immediately issue errors and abort displaying the same ([Errno 110]) error shown at the beginning of my post. Here is the code if others would care to replicate. Preferably CentOS users since this is likely OS related.
import serial
ser = serial.Serial(port='/dev/ttyUSB0',timeout=0,xonxoff=False,rtscts=1,dsrdtr=False)
# Python 2.6.6, 2.7, and 3.5 will send this following string.
ser.write(b"Hello from CentOS");
# Python 2.7 and 3.5 error-out after this change, but system Python (2.6.6) is okay.
ser.rts = False
while True:
chunk = ser.read(24)
if(chunk):
print(str(chunk) + '\n')
ser.close()
Any suggestions on next steps? Am I just out of luck?

pyserial error - cannot open port

I have seen simple code in stackoverflow using pyserial in USB ports with Python 3.3 but I can't get this to work on my new installation of pyserial 2.7 [in Windows 7, 64 bit, with 3 USB ports]. Installation of pyserial went smoothly, I can import without error and methods are recognized in the Pyscripter IDE which boosts confidence in a good installation, however:
The code stripped down to its error producing essentials is:
import serial
def main():
ser = serial.Serial(port='COM2')
ser.close()
if __name__ == '__main__':
main
From this I receive a dialog box with the error "SerialException: could not open port 'COM2': FileNotFoundError(2,'The system cannot find the file specified.',None,2)"
The Traceback states:
*** Remote Interpreter Reinitialized ***
>>>
Traceback (most recent call last):
File "<string>", line 420, in run_nodebug
File "C:\Python33\Lib\site-packages\scanport2.py", line 19, in <module>
main()
File "C:\Python33\Lib\site-packages\scanport2.py", line 15, in main
ser = serial.Serial(port='COM2')
File "C:\Python33\Lib\site-packages\serial\serialwin32.py", line 38, in __init__
SerialBase.__init__(self, *args, **kwargs)
File "C:\Python33\Lib\site-packages\serial\serialutil.py", line 282, in __init__
self.open()
File "C:\Python33\Lib\site-packages\serial\serialwin32.py", line 66, in open
raise SerialException("could not open port %r: %r" % (self.portstr, ctypes.WinError()))
serial.serialutil.SerialException: could not open port 'COM2': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2)
And the code segment in the imported module which raises the SerialException is:
# the "\\.\COMx" format is required for devices other than COM1-COM8
# not all versions of windows seem to support this properly
# so that the first few ports are used with the DOS device name
port = self.portstr
try:
if port.upper().startswith('COM') and int(port[3:]) > 8:
port = '\\\\.\\' + port
except ValueError:
# for like COMnotanumber
pass
self.hComPort = win32.CreateFile(port,
win32.GENERIC_READ | win32.GENERIC_WRITE,
0, # exclusive access
None, # no security
win32.OPEN_EXISTING,
win32.FILE_ATTRIBUTE_NORMAL | win32.FILE_FLAG_OVERLAPPED,
0)
if self.hComPort == win32.INVALID_HANDLE_VALUE:
self.hComPort = None # 'cause __del__ is called anyway
raise SerialException("could not open port %r: %r" % (self.portstr, ctypes.WinError()))
I do have an active device connected to COM2 as identified in the Windows device manager. I also have tried scanning all the ports, but the code stops on the first use of serial.Serial
This appears that something may be going on with win32?
I am a newbie for interfacing Python with hardware.

I would try the following:
Unplug and replug the device.
Reboot.
Run WinObj and look in the GLOBAL?? folder; you should see COM2 there as a symbolic link to something more driver-specific.
What type of device do you have connected to COM2? If it uses usbser.sys, you might have better luck substituting \\.\USBSER000 for COM2 in your code, but remember to escape those backslashes properly.
On some machines there are strange problems with low COM port numbers that I can't explain. Try reassigning the device to COM6 in the Device Manager.

It looks like the pyserial download page only contains links for 32 bit python? This unofficial page seems to have links for 64 bit installations, however be cautious installing from unknown sources.
This answer also suggests installing it using pip: https://stackoverflow.com/a/8491164/66349

web.py + lighttpd + matplotlib not working

I'm trying to deploy my web.py app with lighttpd. It doesn't work if import matplotlib.
This works...
hello.py:
#!/usr/bin/python
import web
# Say hello.
class Index:
def GET(self): return 'hello web.py'
if __name__ == "__main__":
app = web.application(('/*', 'Index'), globals())
app.run()
/etc/init.d/lighttpd restart
I go to my site and see "hello web.py".
But if I add import matplotlib to hello.py and restart the server, then when I go to the site I get a 500 - Internal Server Error.
Here's /var/log/lighttpd/error.log:
2010-12-24 00:17:31: (log.c.166) server started
2010-12-24 00:17:42: (mod_fastcgi.c.1734) connect failed: Connection refused on
unix:/tmp/fastcgi.socket-0
2010-12-24 00:17:42: (mod_fastcgi.c.3037) backend died; we'll disable it for 1 s
econds and send the request to another backend instead: reconnects: 0 load: 1
2010-12-24 00:17:43: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fa
stcgi process died): pid: 4074 socket: unix:/tmp/fastcgi.socket-0
2010-12-24 00:17:43: (mod_fastcgi.c.3320) child exited, pid: 4074 status: 1
2010-12-24 00:17:43: (mod_fastcgi.c.3367) response not received, request sent: 9
53 on socket: unix:/tmp/fastcgi.socket-0 for /hello.py?, closing connection
2010-12-24 00:20:30: (server.c.1503) server stopped by UID = 0 PID = 4095
2010-12-24 00:20:30: (log.c.166) server started
-- Edit --
Here is my lighttpd.conf: http://pastebin.com/n6sG5z9K
Pretty sure it's just the default (except I set server.document-root = "/var/www/hello/")
Here is my fastcgi.conf:
server.modules += ( "mod_fastcgi" )
server.modules += ( "mod_rewrite" )
fastcgi.server = ( "/hello.py" =>
(( "socket" => "/tmp/fastcgi.socket",
"bin-path" => "/usr/bin/python /var/www/hello/hello.py",
"max-procs" => 1,
"bin-environment" => (
"REAL_SCRIPT_NAME" => ""
),
"check-local" => "disable"
))
)
url.rewrite-once = (
"^/favicon.ico$" => "/static/favicon.ico",
"^/static/(.*)$" => "/static/$1",
"^/(.*)$" => "/hello.py/$1",
)
Any suggestions?

Stumbled into this today (with Apache, but it's likely to be exactly the same issue). I redirected stdout and stderr from the script to see what was happening, and the issue is that matplotlib is trying to create a file:
Traceback (most recent call last):
File "/home/ec2-user/dlea/src/dla.py", line 24, in <module>
import dbm
File "/home/ec2-user/dlea/src/dbm.py", line 7, in <module>
import matplotlib
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 709, in <module>
rcParams = rc_params()
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 627, in rc_params
fname = matplotlib_fname()
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 565, in matplotlib_fname
fname = os.path.join(get_configdir(), 'matplotlibrc')
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 240, in wrapper
ret = func(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/matplotlib/__init__.py", line 439, in _get_configdir
raise RuntimeError("Failed to create %s/.matplotlib; consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data"%h)
RuntimeError: Failed to create /var/www/.matplotlib; consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data
Since it's being run as user httpd (Apache), it tries to create the file in /var/www/, which is root-owned, and not writeable by the Apache user.
One valid solution is as simple as setting the MPLCONFIGDIR to a temporary directory before importing matplotlib:
import os
import tempfile
os.environ['MPLCONFIGDIR'] = tempfile.mkdtemp()
import matplotlib
To track the issue, this is how I redirected stdout and stderr to some log file to see what was happening:
sys.stdout = open("/var/log/dla_stdout.txt", 'a')
sys.stderr = open("/var/log/dla_stderr.txt", 'a')
I actually got the solution from this other StackOverflow question: Setting Matplotlib MPLCONFIGDIR: consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data

I was following this recipe: http://webpy.org/cookbook/fastcgi-lighttpd
I overlooked a link at the top to this thread: http://www.mail-archive.com/webpy#googlegroups.com/msg02800.html
That thread had the solution. I run the python process like so:
/var/www/hello.py fastcgi 9080
and then set my fastcgi.conf like so:
fastcgi.server = ( "/hello.py" =>
((
"host" => "127.0.0.1",
"port" => 9080,
"check-local" => "disable"
))
)
Then it works. (Still not sure I've got everything configured properly, but things seem to be working.)

I fix the issue by:
pip install flup
don't need
/var/www/hello.py fastcgi 9080
my system is: amazon ec2, ubuntu 10.04
lighttpd: 1.4.26

My first guess is that you're getting an ImportError because matplotlib wasn't installed properly or isn't on the PYTHONPATH or some other crazy thing. The only way to know for sure is to look at the traceback. It shows you're running fastcgi, which means that the python code is being executed in another process. Therefore, you can't find the traceback in the lighttpd logs.
How are you running the fastcgi process? The traceback would have been written to its stderr. You might also consider using supervisord. It has support for redirecting stderr to a log file and various other things that make creating daemon processes easier.