Python 3.6.2
The problem with the code below is that being run, it raises an exception.
But being stepped in debugger, it works perfectly. Where I stop in the debugger is marked as breakpoint in the comments.
I tried the command both in IDE and in the shell. Exception raises. So, this problem is not related to the IDE.
This situation shook me a bit.
I made a video of it: https://www.youtube.com/watch?v=OUcMpEzooDk
Could you give me a kick here? How can it be?
Comment on the code below (not related to the problem, but just for the most curious).
This is an utility to use with Django web framework.
Users upload files, they are put to the media directory.
Of course, Django knows where the media directory is sutuated.
And then Django keeps in the database paths relative to media. Something like this:
it_1/705fad82-2f68-4f3c-90c2-116da3ad9a40.txt'
e5474da0-0fd3-4fa4-a85f-15c767ac32d4.djvu
I want to know exactly that files kept in media correspond to paths in the database. No extra files, no shortage.
Code:
from pathlib import Path
class <Something>():
def _reveal_lack_extra_files(self):
path = os.path.join(settings.BASE_DIR, '../media/')
image_files = Image.objects.values_list("file", flat=True)
image_files = [Path(os.path.join(path, file)) for file in image_files]
item_files = ItemFile.objects.values_list("file", flat=True)
item_files = [Path(os.path.join(path, file)) for file in item_files]
sheet_files = SheetFile.objects.values_list("file", flat=True)
sheet_files = [Path(os.path.join(path, file)) for file in sheet_files]
expected_files = set().union(image_files, item_files, sheet_files)
real_files = set()
glob_generator = list(Path(path).glob("**/*"))
for posix_path in glob_generator:
if os.path.isfile(posix_path._str): # Breakpoint
real_files.add(posix_path)
lack = expected_files.difference(real_files)
extra = real_files.difference(expected_files)
assert bool(lack) == False, "Lack of files: {}".format(lack)
assert bool(extra) == False, "Extra files: {}".format(extra)
Traceback:
/home/michael/PycharmProjects/venv/photoarchive_4/bin/python /home/michael/Documents/pycharm-community-2017.1.5/helpers/pydev/pydevd.py --multiproc --qt-support --client 127.0.0.1 --port 43849 --file /home/michael/PycharmProjects/photoarchive_4/manage.py checkfiles
warning: Debugger speedups using cython not found. Run '"/home/michael/PycharmProjects/venv/photoarchive_4/bin/python" "/home/michael/Documents/pycharm-community-2017.1.5/helpers/pydev/setup_cython.py" build_ext --inplace' to build.
pydev debugger: process 3840 is connecting
Connected to pydev debugger (build 171.4694.67)
Traceback (most recent call last):
File "/home/michael/Documents/pycharm-community-2017.1.5/helpers/pydev/pydevd.py", line 1591, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/michael/Documents/pycharm-community-2017.1.5/helpers/pydev/pydevd.py", line 1018, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/michael/Documents/pycharm-community-2017.1.5/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/michael/PycharmProjects/photoarchive_4/manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/home/michael/PycharmProjects/venv/photoarchive_4/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/home/michael/PycharmProjects/venv/photoarchive_4/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/michael/PycharmProjects/venv/photoarchive_4/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/michael/PycharmProjects/venv/photoarchive_4/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/home/michael/PycharmProjects/photoarchive_4/general/management/commands/checkfiles.py", line 59, in handle
self._reveal_lack_extra_files()
File "/home/michael/PycharmProjects/photoarchive_4/general/management/commands/checkfiles.py", line 39, in _reveal_lack_extra_files
if os.path.isfile(posix_path._str):
AttributeError: _str
Process finished with exit code 1
You're using the _str attribute on paths, which is undocumented and not guaranteed to be set. In general, an underscore prefix indicates that this is a private attribute that should not be used by user code. If you want to convert a path to a string, just use str(the_path) instead.
But in this case, you don't need to do so: Path objects have an is_file method which you can call instead. Another possibility is to pass the Path object itself to the os.path.isfile function, which is supported on Python 3.6.
Related
I am trying to use pyinstaller to create a .exe for someone else to run. The program uses mysql to ping a sql database and return information from it. When I run the program in PyCharm and with pyinstaller --onedir, everything works fine. However, when I bundle the program with --onefile, I get this error and traceback when running:
Traceback (most recent call last):
File "main.py", line 266, in <module>
File "main.py", line 88, in main
File "main.py", line 108, in grabData
File "sql.py", line 12, in SQLconnect
File "mysql\connector\pooling.py", line 286, in connect
File "mysql\connector\connection_cext.py", line 101, in __init__
File "mysql\connector\abstracts.py", line 1095, in connect
File "mysql\connector\connection_cext.py", line 199, in _open_connection
TypeError: argument 6 must be str, not None
For reference, here is the relevant line in the _open_connection function call in connection_cext.py:
self._cmysql = _mysql_connector.MySQL(
buffered=self._buffered,
raw=self._raw,
charset_name=charset_name,
connection_timeout=(self._connection_timeout or 0),
use_unicode=self._use_unicode,
auth_plugin=self._auth_plugin,
plugin_dir=self._plugin_dir,
)
The __init__ of this file has this code snippet:
self._plugin_dir = os.path.join(
os.path.dirname(os.path.abspath(_mysql_connector.__file__)),
"mysql",
"vendor",
"plugin",
)
I believe this plugin directory is the reason my code is failing, but I do not know how to bypass this or set it so that it references the right file at runtime. I know pyinstaller creates a temp file at runtime, but I don't know how to get the module to use it as the plugin directory.
Any help would be greatly appreciated! I really really wanna keep it in --onefile mode for simplicity for the user, so any solutions that maintain that would be ideal. Thanks!
I'm using AlphaPose from GitHub and I'd like to run the script script/demo_inference.py from another script I created in AlphaPose root called run.py. In run.py I imported demo_inference.py as ap using this script:
def import_module_by_path(path):
name = os.path.splitext(os.path.basename(path))[0] spec =
importlib.util.spec_from_file_location(name, path) mod =
importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod
and
ap = import_module_by_path('./scripts/demo_inference.py')
Then, in demo_inference.py I substituted
if __name__ == "__main__":
with
def startAlphapose():
and in run.py I wrote
ap.StartAlphapose().
Now I got this error:
Load SE Resnet...
Loading YOLO model..
Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/vislab/guerri/alphagastnet/insieme/alphapose/utils/detector.py", line 251, in image_postprocess
(orig_img, im_name, boxes, scores, ids, inps, cropped_boxes) = self.wait_and_get(self.det_queue)
File "/home/vislab/guerri/alphagastnet/insieme/alphapose/utils/detector.py", line 121, in wait_and_get
return queue.get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "/home/vislab/guerri/alphagastnet/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 284, in rebuild_storage_fd
fd = df.detach()
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 487, in Client
c = SocketClient(address)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory
What does it mean?
We were running into this same problem in our cluster.
When using multiprocessing in PyTorch (typically to run multiple DataLoader workers), the subprocesses create sockets in the /tmp directory to communitcate with each other. These sockets all saved in folders named pymp-###### and look like 0-byte files. Deleting these files or folders while your PyTorch scripts are still running will cause the above error.
In our case, the problem was a buggy maintenance script that was erasing files out of the /tmp folder while they were still needed. It's possible there are other ways to trigger this error. But you should start by looking for those sockets and making sure they aren't getting erased by accident.
If that doesn't solve it, take a look at your /var/log/syslog file at the exact time when the error occurred. You'll very likely find the cause of it there.
I'm running some custom Python code in Sphinx and need to get the path to the caller module. (Essentially this is the caller's __file__ object; I need to interpret a filename relative to this location.)
I can get the filename from inspect.stack() as per How to use inspect to get the caller's info from callee in Python?, but apparently I need to interpret this filename in the context of the Python startup directory. (Sometimes inspect.stack()[k][1] is an absolute path but sometimes it is a relative path like conf.py; the inspect.stack() function doesn't seem to document this but unutbu claims in a comment that it is relative to the Python startup directory. )
Sphinx does some unintentionally evil things like this comment:
# This file is execfile()d with the current directory set to its
# containing dir.
so os.path.abspath(filename) doesn't work, and
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('extensions'))
so sys.path[0] is corrupted by the time my code gets to it.
How do I find the startup directory in Python, if sys.path has been modified?
Or is there another way to get the path to the caller module?
If I run Jean-François Fabre's answer
for file,line,w1,w2 in traceback.extract_stack():
sys.stdout.write(' File "{}", line {}, in {}\n'.format(file,line,w1))
I get this:
File "c:\app\python\anaconda\1.6.0\Scripts\sphinx-build-script.py", line 5, in <module>
File "c:\app\python\anaconda\1.6.0\lib\site-packages\Sphinx-1.4.1-py2.7.egg\sphinx\__init__.py", line 51, in main
File "c:\app\python\anaconda\1.6.0\lib\site-packages\Sphinx-1.4.1-py2.7.egg\sphinx\__init__.py", line 92, in build_main
File "c:\app\python\anaconda\1.6.0\lib\site-packages\Sphinx-1.4.1-py2.7.egg\sphinx\cmdline.py", line 243, in main
File "c:\app\python\anaconda\1.6.0\lib\site-packages\Sphinx-1.4.1-py2.7.egg\sphinx\application.py", line 155, in __init__
File "conf.py", line 512, in setup
[more lines elided, the conf.py is the one that matters]
so the problem is that I need to find the path to conf.py but the current directory has been changed by Sphinx so I can't just do os.path.abspath(caller_filename)
you can get what you want using the traceback module. I've written this sample code in PyScripter:
import traceback,sys
def demo():
for file,line,w1,w2 in traceback.extract_stack():
sys.stdout.write(' File "{}", line {}, in {}\n'.format(file,line,w1))
def foo():
demo()
foo()
which gives on my Windows PC running PyScripter:
File "C:\Users\dartypc\AppData\Roaming\PyScripter\remserver.py", line 63, in <module>
File "C:\Users\dartypc\AppData\Roaming\PyScripter\remserver.py", line 60, in main
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\utils\server.py", line 227, in start
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\utils\server.py", line 139, in accept
File "C:\Users\dartypc\AppData\Roaming\PyScripter\remserver.py", line 14, in _accept_method
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\utils\server.py", line 191, in _serve_client
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 391, in serve_all
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 382, in serve
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 350, in _dispatch
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 298, in _dispatch_request
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 528, in _handle_call
File "<string>", line 420, in run_nodebug
File "C:\DATA\jff\data\python\stackoverflow\simple_traceback.py", line 10, in <module>
File "C:\DATA\jff\data\python\stackoverflow\simple_traceback.py", line 8, in foo
File "C:\DATA\jff\data\python\stackoverflow\simple_traceback.py", line 4, in demo
Bah, I'm just going to get around the issue by allowing callers to pass in their __file__ value :-(
my function:
def do_something(app, filename, relroot=None):
if relroot is None:
relroot = '.'
else:
relroot = os.path.dirname(relroot)
path = os.path.join(relroot, filename)
...
in conf.py:
def setup(app):
mymodule.do_something(app, 'path/to/file', relroot=__file__)
I'm working on a Python-programm that use the modul "pptx" (edit and manipulate powerpoint). The programm works without problem in the interpretor, but once I start it as a .exe file (built with cx_freeze or py2exe) I have this error-message when I click on the main-button :
Exception in Tkinter callback
Traceback (most recent call last):
File "Tkinter.pyc", line 1470, in __call__
File "pptx_02.py", line 187, in ButtonErstellen
File "pptx\api.pyc", line 29, in __init__
File "pptx\presentation.pyc", line 87, in __init__
File "pptx\presentation.pyc", line 170, in __open
File "pptx\packaging.pyc", line 88, in open
File "pptx\packaging.pyc", line 671, in __new__
PackageNotFoundError: Package not found at 'C:\Users\Moi\Programmation\Python\build\exe.win32-2.7\library.zip\pptx\templates\default.pptx'
The problem is that the file "default.pptx" actually exists at this adress.
While looking in the functions of "pptx", I may had an idea about the reason of this problem (but no idea of the solution).
For the last error : File "pptx\packaging.pyc", line 671, in new
is the following code:
class FileSystem(object):
"""
Factory for filesystem interface instances.
A FileSystem object provides access to on-disk package items via their URI
(e.g. ``/_rels/.rels`` or ``/ppt/presentation.xml``). This allows parts to
be accessed directly by part name, which for a part is identical to its
item URI. The complexities of translating URIs into file paths or zip item
names, and file and zip file access specifics are all hidden by the
filesystem class. |FileSystem| acts as the Factory, returning the
appropriate concrete filesystem class depending on what it finds at *path*.
"""
def __new__(cls, file):
# if *file* is a string, treat it as a path
if isinstance(file, basestring):
path = file
if is_zipfile(path):
fs = ZipFileSystem(path)
elif os.path.isdir(path):
fs = DirectoryFileSystem(path)
else:
raise PackageNotFoundError("Package not found at '%s'" % path)
else:
fs = ZipFileSystem(file)
return fs
The problem may be that the final file (default.pptx) isnot a zip but is in a zip. But I have no idea how I could fix it or if it realy is the problem.
If somebody as an idea ...
Thank you !
I'm developing django based site on a windows machine, but I git push my changes to a headless Ubuntu server for staging purposes. When I git pull the changes into the working directory on the staging server I then need to run manage.py collectstatic. However, when I do this the change monitor ('Monitoring Code Changes' section of Reloading Source Code) reports that it has detected changed files:
monitor (pid=26216): Starting change monitor.
You have requested to collect static files at the destination
location as specified in your settings.
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: monitor (pid=26216): Change detected to 'myproject/manage.py'.
monitor (pid=26216): Triggering process restart.
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/envs/myproject/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 453, in execute_from_command_line
utility.execute()
File "/envs/myproject/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/envs/myproject/local/lib/python2.7/site-packages/django/core/management/base.py", line 222, in run_from_argv
self.execute(*args, **options.__dict__)
File "/envs/myproject/local/lib/python2.7/site-packages/django/core/management/base.py", line 255, in execute
output = self.handle(*args, **options)
File "/envs/myproject/local/lib/python2.7/site-packages/django/core/management/base.py", line 385, in handle
return self.handle_noargs(**options)
File "/envs/myproject/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 160, in handle_noargs
% (destination_display, clear_display))
KeyboardInterrupt
I ran 'touch myproject/wsgi.py' to 'update the change cache' and then ran the collectstatic command again. This resulted in the same error.
I then entered the python console and ran the monitor starting command manually:
import myproject.monitor
myproject.monitor.start(1)
This made no difference. I also tried adding a conditional to the monitor so that it would ignore manage.py when checking for file changes but this just gave me the same error with myproject/__init__.py being the file it reported as having changed. DOes anyone know what is causing this?
I've worked out what was causing this. The legacy file modification times are held in a dictionary as unix timestamps with decimal places (e.g. 1366283624.92) while the actual modification times are returned by os.stat(path).st_mtime as integers. All I had to do was edit the monitor script at line 54, changing:
if mtime != _times[path]:
to
if int(mtime) != int(_times[path]):
I just hope this doesn't lead to any further problems down the road