We Keep Coding

WinError 32 keeps occuring when running script, even when a temporary directory is created. What am I doing wrong?

I am currently trying to write a python script that creates a backup of a ArcGIS Online Layer, but I keep getting the WinError 32 with this code and basically any script I run. I was having this issue before so I thought that making a temporary directory would help but the issue is still happening, I am not completely confident in knowing how directories work though so I may just be putting in unrelated code. The code I am currently using is as follows (also note I am very very new to coding):
# create a temporary directory since I keep getting a file in use error but this also doesnt fix it lol
import tempfile
with tempfile.TemporaryDirectory() as tmpdir:
print('created temporary directory', tmpdir)
# Import necessary modules
import arcgis
from arcgis.gis import GIS
# Create a GIS object
gis = GIS("insert_link", "username", "password")
# Get the item that represents the layer that you want to copy
layer_item = gis.content.get("layer_ID")
# Create a copy of the layer
copy_item = layer_item.copy()
# Find any old copies of the layer and delete them
for item in gis.content.search(query="title:'Copy of {}'".format(layer_item.title)):
if item.type == "Feature Layer":
item.delete()
# Save the new copy of the layer
copy_item.save()
# one day I will see this
print("File Downloaded")
This is the error I get:
C:\Users\SeanW\anaconda3\envs\MeganScriptTests\python.exe C:\Users\SeanW\PycharmProjects\AGOLBackup3\main.py
created temporary directory C:\Users\SeanW\AppData\Local\Temp\tmpj0yvbg64
Traceback (most recent call last):
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\importlib\_common.py", line 92, in _tempfile
os.write(fd, reader())
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\importlib\abc.py", line 371, in read_bytes
with self.open('rb') as strm:
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\importlib\_adapters.py", line 54, in open
raise ValueError()
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\SeanW\PycharmProjects\AGOLBackup3\main.py", line 7, in <module>
import arcgis
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\arcgis\__init__.py", line 3, in <module>
from arcgis.auth.tools import LazyLoader
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\arcgis\auth\__init__.py", line 1, in <module>
from .api import EsriSession
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\arcgis\auth\api.py", line 11, in <module>
certifi_win32.wincerts.verify_combined_pem()
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\certifi_win32\wincerts.py", line 65, in verify_combined_pem
with open(certifi_pem()) as certifi_pem_handle:
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\certifi_win32\wincerts.py", line 52, in certifi_pem
import certifi
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\wrapt\importer.py", line 177, in _exec_module
notify_module_loaded(module)
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\wrapt\decorators.py", line 470, in _synchronized
return wrapped(*args, **kwargs)
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\wrapt\importer.py", line 136, in notify_module_loaded
hook(module)
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\certifi_win32\wrapt_certifi.py", line 20, in apply_patches
certifi_win32.wincerts.CERTIFI_PEM = certifi.where()
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\site-packages\certifi\core.py", line 72, in where
_CACERT_PATH = str(_CACERT_CTX.__enter__())
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\contextlib.py", line 135, in __enter__
return next(self.gen)
File "C:\Users\SeanW\anaconda3\envs\MeganScriptTests\lib\importlib\_common.py", line 98, in _tempfile
_os_remove(raw_path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\SeanW\\AppData\\Local\\Temp\\tmpx9og0v3t'
Process finished with exit code 1
I have tried python 3.11 as well but it gives me another error where it cannot open an orphan path and I could find nothing about that online. I have also tried this on another computer in case my python was configured wrong and the same issue occurred. I have read it may be the python-certifi-win32 package but the same error occurs when I try to uninstall it. Indenting the code after making a temp directory also does not fix the winerror 32 issue.

You are using tempfile.TemporaryDirectory as a context manager (using with):
with tempfile.TemporaryDirectory() as tmpdir:
print('created temporary directory', tmpdir)
When used this way, the directory gets deleted automatically at the end of the indented block:
On completion of the context or destruction of the temporary directory object, the newly created temporary directory and all its contents are removed from the filesystem.
Whatever you want to do in the temporary directory needs to happen inside the indented block.

Pipeline code spanning multiple files in Apache Beam / Dataflow

After a lengthy search, I haven't found an example of a Dataflow / Beam pipeline that spans several files. Beam docs do suggest a file structure (under the section "Multiple File Dependencies"), but the Juliaset example they give has in effect a single code/source file (and the main file that calls it). Based on the Juliaset example, I need a similar file structure:
juliaset/__init__.py
juliaset/juliaset.py # actual code
juliaset/some_conf.py
__init__.py
juliaset_main.py
setup.py
Now I want to import .some_conf from juliaset/juliaset.py, which works when run locally but gives me an error when run on Dataflow
INFO:root:2017-12-15T17:34:09.333Z: JOB_MESSAGE_ERROR: (8cdf3e226105b90a): Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 706, in run
self._load_main_session(self.local_staging_directory)
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 446, in _load_main_session
pickler.load_session(session_file)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 247, in load_session
return dill.load_session(file_path)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 363, in load_session
module = unpickler.load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
value = func(*args)
File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 767, in _import_module
return getattr(__import__(module, None, None, [obj]), obj)
ImportError: No module named package_name.juliaset.some_conf
A full working example would be very much appreciated!

Can you verify your setup.py containing a structure like:
import setuptools
setuptools.setup(
name='My Project',
version='1.0',
install_requires=[],
packages=setuptools.find_packages(),
)
Import your modules like from juliaset.juliaset import SomeClass
And when you call the Python script, use python -m juliaset_main (without the .py)
Not sure if you already tried this, but just to be sure.

Is there a reliable way to get the path of the caller module from a Python function that is executed within a Sphinx conf.py?

I'm running some custom Python code in Sphinx and need to get the path to the caller module. (Essentially this is the caller's __file__ object; I need to interpret a filename relative to this location.)
I can get the filename from inspect.stack() as per How to use inspect to get the caller's info from callee in Python?, but apparently I need to interpret this filename in the context of the Python startup directory. (Sometimes inspect.stack()[k][1] is an absolute path but sometimes it is a relative path like conf.py; the inspect.stack() function doesn't seem to document this but unutbu claims in a comment that it is relative to the Python startup directory. )
Sphinx does some unintentionally evil things like this comment:
# This file is execfile()d with the current directory set to its
# containing dir.
so os.path.abspath(filename) doesn't work, and
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('extensions'))
so sys.path[0] is corrupted by the time my code gets to it.
How do I find the startup directory in Python, if sys.path has been modified?
Or is there another way to get the path to the caller module?
If I run Jean-François Fabre's answer
for file,line,w1,w2 in traceback.extract_stack():
sys.stdout.write(' File "{}", line {}, in {}\n'.format(file,line,w1))
I get this:
File "c:\app\python\anaconda\1.6.0\Scripts\sphinx-build-script.py", line 5, in <module>
File "c:\app\python\anaconda\1.6.0\lib\site-packages\Sphinx-1.4.1-py2.7.egg\sphinx\__init__.py", line 51, in main
File "c:\app\python\anaconda\1.6.0\lib\site-packages\Sphinx-1.4.1-py2.7.egg\sphinx\__init__.py", line 92, in build_main
File "c:\app\python\anaconda\1.6.0\lib\site-packages\Sphinx-1.4.1-py2.7.egg\sphinx\cmdline.py", line 243, in main
File "c:\app\python\anaconda\1.6.0\lib\site-packages\Sphinx-1.4.1-py2.7.egg\sphinx\application.py", line 155, in __init__
File "conf.py", line 512, in setup
[more lines elided, the conf.py is the one that matters]
so the problem is that I need to find the path to conf.py but the current directory has been changed by Sphinx so I can't just do os.path.abspath(caller_filename)

you can get what you want using the traceback module. I've written this sample code in PyScripter:
import traceback,sys
def demo():
for file,line,w1,w2 in traceback.extract_stack():
sys.stdout.write(' File "{}", line {}, in {}\n'.format(file,line,w1))
def foo():
demo()
foo()
which gives on my Windows PC running PyScripter:
File "C:\Users\dartypc\AppData\Roaming\PyScripter\remserver.py", line 63, in <module>
File "C:\Users\dartypc\AppData\Roaming\PyScripter\remserver.py", line 60, in main
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\utils\server.py", line 227, in start
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\utils\server.py", line 139, in accept
File "C:\Users\dartypc\AppData\Roaming\PyScripter\remserver.py", line 14, in _accept_method
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\utils\server.py", line 191, in _serve_client
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 391, in serve_all
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 382, in serve
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 350, in _dispatch
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 298, in _dispatch_request
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 528, in _handle_call
File "<string>", line 420, in run_nodebug
File "C:\DATA\jff\data\python\stackoverflow\simple_traceback.py", line 10, in <module>
File "C:\DATA\jff\data\python\stackoverflow\simple_traceback.py", line 8, in foo
File "C:\DATA\jff\data\python\stackoverflow\simple_traceback.py", line 4, in demo

Bah, I'm just going to get around the issue by allowing callers to pass in their __file__ value :-(
my function:
def do_something(app, filename, relroot=None):
if relroot is None:
relroot = '.'
else:
relroot = os.path.dirname(relroot)
path = os.path.join(relroot, filename)
...
in conf.py:
def setup(app):
mymodule.do_something(app, 'path/to/file', relroot=__file__)

When trying to build a Python executable with Pyinstaller, fails to find an existing scipy module

I have a build script for one of my established Python applications that uses Pyinstaller. This script has been working fine for over a year. Then today, I added to one of the source files for this application the line
import scipy.stats
because I want to use scipy.stats.linregress. This now causes the build script to crash with a long error traceback (apparently going back through a sequence of modules that import each other) ending with
File "C:\Users\462974\Documents\Local Sandbox\fof\TRUNK\programs\CDFParsing\build\pyi.win32\CDFGUI\outPYZ1.pyz/scipy.sparse.csgraph", line 148, in <module>
File "C:\Python27\pyinstaller-1.5\iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "C:\Python27\pyinstaller-1.5\iu.py", line 495, in doimport
mod = importfunc(nm)
File "C:\Python27\pyinstaller-1.5\iu.py", line 297, in getmod
mod = owner.getmod(nm)
File "C:\Python27\pyinstaller-1.5\archive.py", line 468, in getmod
return iu.DirOwner.getmod(self, self.prefix+'.'+nm)
File "C:\Python27\pyinstaller-1.5\iu.py", line 109, in getmod
mod = imp.load_module(nm, fp, attempt, (ext, mode, typ))
File "_shortest_path.pyx", line 18, in init scipy.sparse.csgraph._shortest_path (scipy\sparse\csgraph\_shortest_path.c:14224)
File "C:\Python27\pyinstaller-1.5\iu.py", line 455, in importHook
raise ImportError, "No module named %s" % fqname
ImportError: No module named scipy.sparse.csgraph._validation
This is puzzling because the module located at C:\Python27\Lib\site-packages\scipy\sparse\csgraph_validation.py very much exists. Why did adding scipy to my build break it (importing numpy works just fine), could it be failing to find it?

Not entirely sure why, but including the following definition in my code after the import statement fixed it:
def fix_dependencies():
from scipy.sparse.csgraph import _validation

django module already import and python egg warnings

Can anyone help me get rid of these warnings in Django please?
/usr/lib/python2.6/site-packages/simplejson-2.1.5-py2.6-linux-x86_64.egg/simplejson/_speedups.py:3: UserWarning: Module taggit was already imported from /var/www/html/matilah/taggit/__init__.py, but /usr/lib/python2.6/site-packages/django_taggit-0.9.3-py2.6.egg is being added to sys.path
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/usr/lib/python2.6/site-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/usr/lib/python2.6/site-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.6/site-packages/django/core/management/__init__.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/usr/lib/python2.6/site-packages/django/core/management/__init__.py", line 67, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/usr/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/usr/lib/python2.6/site-packages/haystack/__init__.py", line 154, in <module>
handle_registrations()
File "/usr/lib/python2.6/site-packages/haystack/__init__.py", line 151, in handle_registrations
search_sites_conf = importlib.import_module(settings.HAYSTACK_SITECONF)
File "/usr/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/var/www/html/matilah/search_sites.py", line 2, in <module>
haystack.autodiscover()
File "/usr/lib/python2.6/site-packages/haystack/__init__.py", line 106, in autodiscover
app_path = importlib.import_module(app).__path__
File "/usr/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/var/www/html/matilah/disqus/__init__.py", line 5, in <module>
from django.utils import simplejson as json
File "/usr/lib/python2.6/site-packages/django/utils/simplejson/__init__.py", line 111, in <module>
import simplejson
File "/usr/lib/python2.6/site-packages/simplejson-2.1.5-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 111, in <module>
File "/usr/lib/python2.6/site-packages/simplejson-2.1.5-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 7, in <module>
File "/usr/lib/python2.6/site-packages/simplejson-2.1.5-py2.6-linux-x86_64.egg/simplejson/scanner.py", line 10, in <module>
File "/usr/lib/python2.6/site-packages/simplejson-2.1.5-py2.6-linux-x86_64.egg/simplejson/scanner.py", line 6, in _import_c_make_scanner
File "/usr/lib/python2.6/site-packages/simplejson-2.1.5-py2.6-linux-x86_64.egg/simplejson/_speedups.py", line 7, in <module>
File "/usr/lib/python2.6/site-packages/simplejson-2.1.5-py2.6-linux-x86_64.egg/simplejson/_speedups.py", line 4, in __bootstrap__
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 841, in resource_filename
self, resource_name
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1310, in get_resource_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1332, in _extract_resource
self.egg_name, self._parts(zip_path)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 921, in get_cache_path
self.extraction_error()
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 887, in extraction_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Errno 13] Permission denied: '/home/interworx/.python-eggs'
The Python egg cache directory is currently set to:
/home/interworx/.python-eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
I'm not sure if the "Module was already imported" error has to do with the "cannot extract PYTHON EGG" error.
In my WSGI I already have the following line os.environ['PYTHON_EGG_CACHE'] = rel('python-eggs') and I know for sure that works. Otherwise WSGI won't load my app. So I'm not sure what to do here.
Thanks!

When you get a message like "Perhaps your account does not have write access to this directory?" it's best to do a quick test. Try changing it to "/tmp" (which is writable by everyone) and see if things start working. If that fixes it, you can then identify a more appropriate directory and make sure that you (or the server) have write access to it.
Update: You are correct in choosing to tackle the earliest error message first. Here is a quick utility, Python Module Dump, that I wrote a few years ago for tracking down a mis-included module error. You might try inserting it at interesting points along your initialization code path and see if you can spot when the earlier import occurred.

There's a chance that your initial problem "Module was already imported" does in fact have to do with the permissions error you have going. Your best option would be to fix the permissions/ownership settings on the /home/interworx/.python-eggs directory (as stated by Peter Rowell above).
I came across this great little refresher on permissions today at the Google Code University. Wouldn't hurt to skim through this when you have some time: http://code.google.com/edu/tools101/linux/ownership_permissions.html
Just read the comments above here is my thoughts:
I think the problem is that the .wsgi file is not setting the environment variable in time. Can you please post your .wsgi file? Also, how are you grabbing the packages? Is it a script? Manually via pip?