I'm trying to use dask from Pycharm using a remote (SSH) interpreter.
Here's some code:
from dask.distributed import Client
client = Client(processes=True)
On Python 2 this seems to work fine, but on Python 3 this fails with:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/forkserver.py", line 278, in main
code = _serve_one(child_r, fds,
File "/usr/lib/python3.8/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r, parent_sentinel)
File "/usr/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/usr/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/usr/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/usr/lib/python3.8/runpy.py", line 261, in run_path
code, fname = _get_code_from_file(run_name, path_name)
File "/usr/lib/python3.8/runpy.py", line 231, in _get_code_from_file
with open(fname, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/<input>'
distributed.nanny - WARNING - Restarting worker
/home/ubuntu exists and is writable- I'm not sure what /home/ubuntu/input is though!
Does anyone have an idea as to what is going wrong here?
EDIT
so after a bit of tracing back I can see that in multiprocess/spawn.py on line 226, there is a prepare function which takes a map of properties describing the processed to be spawned. For some reason the 'init_main_from_path' in that map is being set to /home/ubuntu/<input> which is a directory that doesn't exist anywhere (I'm not even usre this is a valid directory name!). It looks like the PyCharm console is setting this to help itself with the remote execution, but it's messing up the ability of multiprocess to correctly spawn a process. If I hack multiprocess/spawn.py to remove the rogue 'init_main_from_path' key then I can start the client.
Related
Im trying to run a python Server using CherryPy for a WebSite but when I run it this error pops up.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cprequest.py", line 638, in respond
self._do_respond(path_info)
File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cprequest.py", line 694, in _do_respond
self.hooks.run('before_handler')
File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cprequest.py", line 95, in run
self.run_hooks(iter(sorted(self[point])))
File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cprequest.py", line 117, in run_hooks
hook()
File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cprequest.py", line 65, in __call__
return self.callback(**self.kwargs)
File "/usr/local/lib/python3.10/dist-packages/cherrypy/_cptools.py", line 280, in _lock_session
cherrypy.serving.session.acquire_lock()
File "/usr/local/lib/python3.10/dist-packages/cherrypy/lib/sessions.py", line 550, in acquire_lock
self.lock = zc.lockfile.LockFile(path)
File "/usr/local/lib/python3.10/dist-packages/zc/lockfile/__init__.py", line 117, in __init__
super(LockFile, self).__init__(path)
File "/usr/local/lib/python3.10/dist-packages/zc/lockfile/__init__.py", line 87, in __init__
fp = open(path, 'a+')
FileNotFoundError: [Errno 2] No such file or directory: '/var/www/html/cncsessions\\/session-73ab2ecbe9bd50153b4f20828fcc08bff6e9cd6e.lock'
It's my first time using this module and I don't know what's wrong.
I'm using Ubuntu 22, Python 3.10.6
Hard to say without seeing your exact code that this is calling.
Judging by the Error you are trying to use Sessions.
The sessions are looking for
/var/www/html/cncsessions
To place the session files in
But it gives an error. It looks like the path might be wrong. There's a double backslash at the end there and a forward slash.
\\/
If you haven't given up on this/figured it out already I would try changing this path to just this
/var/www/html/cncsessions
Also be sure you do not store your session data in your web root. Looks like from that path you might be doing that! Anything in a webroot will be served via public webserver. There's little to no chance anyone would guess the file names though.
I am trying to use pyinstaller to create a .exe for someone else to run. The program uses mysql to ping a sql database and return information from it. When I run the program in PyCharm and with pyinstaller --onedir, everything works fine. However, when I bundle the program with --onefile, I get this error and traceback when running:
Traceback (most recent call last):
File "main.py", line 266, in <module>
File "main.py", line 88, in main
File "main.py", line 108, in grabData
File "sql.py", line 12, in SQLconnect
File "mysql\connector\pooling.py", line 286, in connect
File "mysql\connector\connection_cext.py", line 101, in __init__
File "mysql\connector\abstracts.py", line 1095, in connect
File "mysql\connector\connection_cext.py", line 199, in _open_connection
TypeError: argument 6 must be str, not None
For reference, here is the relevant line in the _open_connection function call in connection_cext.py:
self._cmysql = _mysql_connector.MySQL(
buffered=self._buffered,
raw=self._raw,
charset_name=charset_name,
connection_timeout=(self._connection_timeout or 0),
use_unicode=self._use_unicode,
auth_plugin=self._auth_plugin,
plugin_dir=self._plugin_dir,
)
The __init__ of this file has this code snippet:
self._plugin_dir = os.path.join(
os.path.dirname(os.path.abspath(_mysql_connector.__file__)),
"mysql",
"vendor",
"plugin",
)
I believe this plugin directory is the reason my code is failing, but I do not know how to bypass this or set it so that it references the right file at runtime. I know pyinstaller creates a temp file at runtime, but I don't know how to get the module to use it as the plugin directory.
Any help would be greatly appreciated! I really really wanna keep it in --onefile mode for simplicity for the user, so any solutions that maintain that would be ideal. Thanks!
I'm using AlphaPose from GitHub and I'd like to run the script script/demo_inference.py from another script I created in AlphaPose root called run.py. In run.py I imported demo_inference.py as ap using this script:
def import_module_by_path(path):
name = os.path.splitext(os.path.basename(path))[0] spec =
importlib.util.spec_from_file_location(name, path) mod =
importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod
and
ap = import_module_by_path('./scripts/demo_inference.py')
Then, in demo_inference.py I substituted
if __name__ == "__main__":
with
def startAlphapose():
and in run.py I wrote
ap.StartAlphapose().
Now I got this error:
Load SE Resnet...
Loading YOLO model..
Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/vislab/guerri/alphagastnet/insieme/alphapose/utils/detector.py", line 251, in image_postprocess
(orig_img, im_name, boxes, scores, ids, inps, cropped_boxes) = self.wait_and_get(self.det_queue)
File "/home/vislab/guerri/alphagastnet/insieme/alphapose/utils/detector.py", line 121, in wait_and_get
return queue.get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "/home/vislab/guerri/alphagastnet/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 284, in rebuild_storage_fd
fd = df.detach()
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 487, in Client
c = SocketClient(address)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory
What does it mean?
We were running into this same problem in our cluster.
When using multiprocessing in PyTorch (typically to run multiple DataLoader workers), the subprocesses create sockets in the /tmp directory to communitcate with each other. These sockets all saved in folders named pymp-###### and look like 0-byte files. Deleting these files or folders while your PyTorch scripts are still running will cause the above error.
In our case, the problem was a buggy maintenance script that was erasing files out of the /tmp folder while they were still needed. It's possible there are other ways to trigger this error. But you should start by looking for those sockets and making sure they aren't getting erased by accident.
If that doesn't solve it, take a look at your /var/log/syslog file at the exact time when the error occurred. You'll very likely find the cause of it there.
I'm attempting to download all the files in an SFTP directory to a local folder using the pysftp library. My code looks like this:
import pysftp
sftp = pysftp.Connection('server', username = 'name', password = 'password')
sftp.get_d('Daily_Reports', '/home/jchrysostom/Documents/SupplyChain/Daily_Reports/')
Daily_Reports is a folder that exists on the SFTP server - I have verified this. I have also verified that /home/jchrysostom/Documents/SupplyChain/Daily_Reports/ exists. I can cd to it in terminal with no problems.
However, when I run this python script, I get the following error: IOError: [Errno 2] File not found.
Any ideas what may be causing this?
UPDATE: A little investigation shows that the files actually have downloaded. In fact, all of them have downloaded just fine. However, I'm unable to run the rest of the script, because it's erroring out for some reason. Is this just a bug in the library?
UPDATE 2 - Full Traceback, as requested:
Traceback (most recent call last):
File "supplychain.py", line 20, in <module>
sftp.get_d('Daily_Reports','/home/jchrysostom/Documents/SupplyChain/Daily_Reports/')
File "/usr/local/lib/python2.7/dist-packages/pysftp.py", line 255, in get_d
preserve_mtime=preserve_mtime)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/pysftp.py", line 497, in cd
self.cwd(original_path)
File "/usr/local/lib/python2.7/dist-packages/pysftp.py", line 510, in chdir
self._sftp.chdir(remotepath)
File "/usr/local/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 580, in chdir
if not stat.S_ISDIR(self.stat(path).st_mode):
File "/usr/local/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 413, in stat
t, msg = self._request(CMD_STAT, path)
File "/usr/local/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 729, in _request
return self._read_response(num)
File "/usr/local/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 776, in _read_response
self._convert_status(msg)
File "/usr/local/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 802, in _convert_status
raise IOError(errno.ENOENT, text)
IOError: [Errno 2] File not found
As best as I can tell, this is a bug in pysftp. The files are being copied successfully, but (at least according to the traceback here) the library is blowing up when it tries to change back to the original remote working directory on the FTP server.
Workaround is to iterate over the files in the directory and get() each individually...
for filename in sftp.listdir('Daily_Reports'):
sftp.get('Daily_Reports/' + filename, localpath = '/home/jchrysostom/Documents/SupplyChain/Daily_Reports/' + filename)
I am relatively new to Python, and at the same time, am attempting to install mod_dav_svn into my Apache web server. I am looking to get some idea of the scope of the error I'm receiving.
At the command line, I type in 'sudo yum install mod_dav_svn' and receive this output:
Loaded plugins: fastestmirror
Determining fastest mirrors
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in ?
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 229, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 104, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 339, in doCommands
self._getTs(needTsRemove)
File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 101, in _getTs
self._getTsInfo(remove_only)
File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 112, in _getTsInfo
pkgSack = self.pkgSack
File "/usr/lib/python2.4/site-packages/yum/init.py", line 591, in
pkgSack = property(fget=lambda self: self._getSacks(),
File "/usr/lib/python2.4/site-packages/yum/init.py", line 434, in _getSacks
self.repos.populateSack(which=repos)
File "/usr/lib/python2.4/site-packages/yum/repos.py", line 223, in populateSack
self.doSetup()
File "/usr/lib/python2.4/site-packages/yum/repos.py", line 71, in doSetup
self.ayum.plugins.run('postreposetup')
File "/usr/lib/python2.4/site-packages/yum/plugins.py", line 176, in run
func(conduitcls(self, self.base, conf, **kwargs))
File "/usr/lib/yum-plugins/fastestmirror.py", line 181, in postreposetup_hook
all_urls = FastestMirror(all_urls).get_mirrorlist()
File "/usr/lib/yum-plugins/fastestmirror.py", line 333, in get_mirrorlist
self._poll_mirrors()
File "/usr/lib/yum-plugins/fastestmirror.py", line 376, in _poll_mirrors
pollThread.start()
File "/usr/lib/python2.4/threading.py", line 416, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
The only other question I could find with a similar error was this one: https://stackoverflow.com/search?q=can%27t+start+new+thread+python, but I'm not sure it has to do with running too many threads since I am the only one using the server, and this is one of the first python commands I have used. Could someone point me in the right direction, or towards some material that may help me troubleshoot the issue? Thanks!
Disable yum-fastermirror and you should be able to complete the install.
yum --disableplugin=fastestmirror update
Are you on a virtual machine?