I'm running the Google App Engine devserver 1.3.3 on Windows 7.
Usually, this method works fine, but this time it gave an error:
def _deleteType(type):
results = type.all().fetch(1000)
while results:
db.delete(results)
results = type.all().fetch(1000)
The error:
File "src\modelutils.py", line 38, in _deleteType
db.delete(results)
File "C:\Program Files\Google\google_appengine\google\appengine\ext\db\__init__.py", line 1302, in delete
datastore.Delete(keys, rpc=rpc)
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.py", line 386, in Delete
'datastore_v3', 'Delete', req, datastore_pb.DeleteResponse(), rpc)
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.py", line 186, in _MakeSyncCall
rpc.check_success()
File "C:\Program Files\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 474, in check_success
self.__rpc.CheckSuccess()
File "C:\Program Files\Google\google_appengine\google\appengine\api\apiproxy_rpc.py", line 149, in _WaitImpl
self.request, self.response)
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_file_stub.py", line 667, in MakeSyncCall
response)
File "C:\Program Files\Google\google_appengine\google\appengine\api\apiproxy_stub.py", line 80, in MakeSyncCall
method(request, response)
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_file_stub.py", line 775, in _Dynamic_Delete
self.__WriteDatastore()
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_file_stub.py", line 610, in __WriteDatastore
self.__WritePickled(encoded, self.__datastore_file)
File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_file_stub.py", line 656, in __WritePickled
os.rename(tmpfile.name, filename)
WindowsError: [Error 183] Cannot create a file when that file already exists
What am I doing wrong? How could this have failed this time, but usually it doesn't?
UPDATE I restarted the devserver, and when it came back online, the datastore was empty.
Unfortunately, 1.3.3 is too far back for me to look at its sources and try to diagnose your problem precisely - the SDK has no 1.3.3 release tag and I can't guess which revision of the datastore_filestub.py was in 1.3.3. Can you upgrade to the current version, 1.3.5, and try again? Running old versions (especially 2+ versions back) is not recommended since they'll be possibly a little out of sync with what's actually available on Google's actual servers, anyway (and/or have bugs that are fixed in later versions). Anyway...
On Windows, os.rename doesn't work if the destination exists -- but the revisions I see are careful to catch the OSError that results (WindowsError derives from it), remove the existing file, and try renaming again. So I don't know what could explain your bug -- if the sources of the SDK you're running have that careful arrangement, and I think they do.
Plus, I'd recommend to --use_sqlite (see Nick Johnson's blog announcing it here) in lieu of the file-stub for your SDK datastore - it just seems to make more sense!-)
(disclaimer: i'm not answering your question but helping you optimize the code you're running)
your code seems to be massively deleting objects. in the SDK/dev server, you can accomplish wiping out the datastore using this command as a quicker and more convenient alternative:
$ dev_appserver.py -c helloworld
now, that is, if you want to wipe your entire SDK datastore. if not, then of course, don't use it. :-)
more importantly, you can make your code run faster and use less CPU on production if you change your query to be something like:
results = type.all(keys_only=True).fetch(SIZE)
this works the same as your's except it only fetches the keys as you don't need full entities retrieved from the datastore in order to delete them. also, your code is currently setting SIZE=1000, but you can make it larger than that, esp. if you have an idea of how many entities you have in your system... the 1000 result limit was lifted in 1.3.1 http://bit.ly/ahoLQp
one minor nit... try not to use type as a variable name... that's one of the most import objects and built-in/factory functions in Python. your code may act odd if do this -- in your case, it's only fractionally better since you're inside a function/method, but that's not going to be true as a global variable.
hope this helps!
Related
Over the last few days, I have been working on automating the generation of some pivot tables for a number of reports.
Boiled down to the minimum, the following code was working without issue:
import win32com.client
objExcelApp = win32com.client.gencache.EnsureDispatch('Excel.Application')
objExcelApp.Visible = 1
This would pop-up an instance of excel and I could continue working in Python. But suddenly, today my scripts are failing with the following:
>>>import win32com.client
>>> objExcelApp = win32com.client.gencache.EnsureDispatch('Excel.Application')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 534, in EnsureDispatch
mod = EnsureModule(tla[0], tla[1], tla[3], tla[4], bForDemand=bForDemand)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 391, in EnsureModule
module = GetModuleForTypelib(typelibCLSID, lcid, major, minor)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 266, in GetModuleForTypelib
AddModuleToCache(typelibCLSID, lcid, major, minor)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 552, in AddModuleToCache
dict = mod.CLSIDToClassMap
AttributeError: module 'win32com.gen_py.00020813-0000-0000-C000-000000000046x0x1x9' has no attribute 'CLSIDToClassMap'
The code has not changed from yesterday to today. I have no idea what is happening!!!.
Another interesting kicker. if I do the same code in the same session again I get a different error:
>>> objExcelApp = win32com.client.gencache.EnsureDispatch('Excel.Application')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 534, in EnsureDispatch
mod = EnsureModule(tla[0], tla[1], tla[3], tla[4], bForDemand=bForDemand)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 447, in EnsureModule
if module.MinorVersion != tlbAttributes[4] or genpy.makepy_version != module.makepy_version:
AttributeError: module 'win32com.gen_py.00020813-0000-0000-C000-000000000046x0x1x9' has no attribute 'MinorVersion'
>>>
So I jump over to a windows machine with a fresh windows install, install python37 and pip install pypiwin32. Run the very same lines and excel opens just like it did yesterday on my original machine.
I tried un-installing and re-installing with no luck. Any idea what is going on here?
NOTE:
Dynamic dispatch still works:
import win32com.client
objExcelApp = win32com.client.Dispatch("Excel.Application")
objExcelApp.Visible = 1
But I specifically need static dispatch as Pivot Tables won't work with a dynamically dispatched object (much later in my code):
objExcelPivotCache = objExcelWorkbook.PivotCaches().Create(SourceType=win32c.xlDatabase, SourceData=objExcelPivotSourceRange)
I had the same issue and I resolved it by following the instructions here: https://mail.python.org/pipermail/python-win32/2007-August/006147.html
Deleting the gen_py output directory and re-running makepy SUCCEEDS
and subsequently the test application runs OK again.
So the symptom is resolved, but any clues as to how this could have
happened. This is a VERY long running application (think 24x7 for
years) and I'm concerned that whatever caused this might occur again.
To find the output directory, run this in your python console / python session:
import win32com
print(win32com.__gen_path__)
Based on the exception message in your post, the directory you need to remove will be titled '00020813-0000-0000-C000-000000000046x0x1x9'. So delete this directory and re-run the code. And if you're nervous about deleting it (like I was) just cut the directory and paste it somewhere else.
💡Note that this directory is usually in your "TEMP" directory (copy-paste %TEMP%/gen_py in Windows File Explorer and you will arrive there directly).
I have no idea why this happens nor do I know how to prevent it from happening again, but the directions in the link I provided seemed to work for me.
A more straightforward solution was posted in a related question Issue in using win32com to access Excel file.
Basically, you just need to delete the folder C:\Users\<your username>\AppData\Local\Temp\gen_py and rerun your code.
💡TIP: You can also put in your Windows file explorer %TEMP%\gen_py to access it directly, and then delete its content.
Execute this command line in a powershell or cmd (NOT in Administrator mode => wouldn't work for me)
python -m win32com.client.makepy "Excel.Application"
It fixes all errors and you don't have to change your python code.
And keep using
win32com.client.gencache.EnsureDispatch("Excel.Application")
With gencache.EnsureDispatch you have access to the constants of the application loaded dynamically by makepy which must have the registered application (in our case Excel.Application).
If you have the same problem with Outlook, use "Outlook.Application" in above.
If still not working, reinstall pywin32 of your python distribution
<path to python root or venv>\pip.exe uninstall pywin32
<path to python root or venv>\pip.exe install pywin32
What has worked for me is:
excel = win32.gencache.EnsureDispatch('Excel.Application')
#change to =>
excel = win32.Dispatch('Excel.Application')
For me, it seems, the issue was that I have multiple processes that interact with Windows apps through win32com.
Since win32com creates the "gen_py" directory in win32api.GetTempPath() this can cause conflicts and the cache getting corrupted.
My solution is to set a custom location for "gen_py" for each process. A simple example:
from pathlib import Path
import win32com
gen_py_path = '/some/custom/location/gen_py'
Path(gen_py_path).mkdir(parents=True, exist_ok=True)
win32com.__gen_path__ = gen_py_path
# Any other imports/code that uses win32com
This way you don't have to delete the default "gen_py" folder and wonder what issues might arise. But if you still find you need to delete, you can just delete the custom folder and know you're deleting the cache just for that process.
To add to this discussion, for those receiving this error as part of an unsupervised automated process, you can fully automating the recovery process and allow any processes to continue unsupervised.
As answers by Ian and Qin mention, we need to delete the gen_py output directory and restart the process. As far as automation goes, there are two problems with this: the gen_py output directory is a temporary directory that can change, and currently running processes that rely on gen_py continue to fail even after it is regenerated.
To address these problems, we can dynamically look up the location, then we can completely nuke the whole process and restart it. I've tried deleting and re-importing win32com, but it seems a reference to the corrupted cache is still maintained, so the whole process needs to be restarted.
temp_data_dir = os.environ.get('LOCALAPPDATA'))
gen_py_dir = ''
for curr_path, dirs, files, in walk(temp_data_dir)
if 'gen_py' in dirs:
gen_py_dir = Path(curr_path).joinpath('gen_py')
shutil.rmtree(gen_py_dir)
execv(restart_args)
The restarted process should call win32com.client.gencache.EnsureDispatch('Excel.Application') again and a fresh copy of gen_py will now be utilized, then we can retry the failed Excel automation code.
The issue happens sporadically in a completely unreliable way that I can't replicate. My best guess is that the gen_py cache is sometimes corrupted somehow and just needs to be refreshed.
I want to learn the skill to log in websites with Python, so I copy other's code then try to understand them.
I've write several web crawls which don't need to post data to the websites by myself.
But this time, when I try to learn crawls that need post data, Python doesn't work.
It's the wrong message from Windows Command Line:
C:\Python26\lib\cookielib.py:1579: UserWarning: cookielib bug!
Traceback (most recent call last):
File "C:\Python26\lib\cookielib.py", line 1577, in make_cookies
parse_ns_headers(ns_hdrs), request)
File "C:\Python26\lib\cookielib.py", line 1534, in _cookies_from_attrs_set
cookie = self._cookie_from_cookie_tuple(tup, request)
File "C:\Python26\lib\cookielib.py", line 1527, in _cookie_from_cookie_tuple
rest)
File "C:\Python26\lib\cookielib.py", line 764, in __init__
self._rest = copy.copy(rest)
AttributeError: 'module' object has no attribute 'copy'
_warn_unhandled_exception()
Actually, I have tried to use Python2.7.6 and Python2.6 and refresh 2.7.6 for times.
The code that I copied is from http://www.oschina.net/code/snippet_1253232_26503.
This code is right because lots of people run this code .
I provide an account:
Email='13018015260'(Also can be telephone number in this site),password='qwerty'
The code is not the problem, but the Python itself. Once I use cookielib in code, Python on my computer doesn't work!
Sorry for my poor English, I want to know the answer, because I even can't find similar problems in Google!
Make sure that you do not have your own copy.py or copy.pyc in python path.
You can check that using following command in python interactive shell.
python -c "import copy; print(copy.__file__)"
It should print path something like:
c:\python27\lib\copy.pyc
If the above command print something that does not reference the file in standard library path, renamed the file (copy.py), and make sure you delete copy.pyc file.
I get a strange error in python. When I try to extract a password protected file using the zip module, I get an exception when trying to set "oy" as password. Everything else seems to work. A bug in ZipFile module?
import zipfile
zip = zipfile.ZipFile("file.zip", "r")
zip.setpassword("oy".encode('utf-8'))
zip.extractall() #Above password "oy" generates the error here
zip.close()
This is the exception I get:
Traceback (most recent call last):
File "unzip.py", line 4, in <module>
zip.extractall()
File "C:\Program Files\Python32\lib\zipfile.py", line 1002, in extrac
l
self.extract(zipinfo, path, pwd)
File "C:\Program Files\Python32\lib\zipfile.py", line 990, in extract
return self._extract_member(member, path, pwd)
File "C:\Program Files\Python32\lib\zipfile.py", line 1035, in _extra
member
shutil.copyfileobj(source, target)
File "C:\Program Files\Python32\lib\shutil.py", line 65, in copyfileo
buf = fsrc.read(length)
File "C:\Program Files\Python32\lib\zipfile.py", line 581, in read
data = self.read1(n - len(buf))
File "C:\Program Files\Python32\lib\zipfile.py", line 633, in read1
max(n - len_readbuffer, self.MIN_READ_SIZE)
zlib.error: Error -3 while decompressing: invalid block type
If I use UTF-16 as encoding I get this error:
zlib.error: Error -3 while decompressing: invalid distance too far back
EDIT
I have now tested on a virtual Linux machine with following stuff:
Python version: 2.6.5
I created a password protected zip file with zip -e file.zip
hello.txt
Now it seems the problem is something else. Now I can extract the zip file even if the password is wrong!
try:
zip.setpassword("ks") # "ks" is wrong password but it still extracts the zip
zip.extractall()
except RuntimeException:
print "wrong!"
Sometimes I can extract the zip file with an incorrect password. The file (inside the zip file) is then extracted but when I try to open it the information seems to be corrupted/decrypted.
If there's a problem with the password, usually you get the following exception:
RuntimeError: ('Bad password for file', <zipfile.ZipInfo object at 0xb76dec2c>)
Since your exception complains about block type, most probably your .zip archive is corrupted, have you tried to unpack it with standalone unzip utility?
Or maybe you have used something funny, like 7zip to create it, which makes incompatible .zip archives.
You don't provide enough information (OS version? Python version? ZIP archive creator and contents? are there many files in those archives or single file in single archive? do all those files give same errors, or you can unpack some of them?), so here's quick Q&A section, which should help you to find and remedy the problem.
Q1. Is this a bug in Python?
A1. Unlikely.
Q2. What might cause this behaviour?
A2. Broken zip files, incompatible zip compressors -- since you don't tell anything, it's hard to point the the exact cause.
Q3. How to find the cause?
A3. Try to isolate the problem, find the file which gives you an error, try to use zip.testzip() and/or decompress that particular file with different unzip utility, share the results. Only you have access to the problematic files, so nobody can help you unless you try to do something yourself.
Q4. How to fix this?
A4. You cannot. Use different zip extractor, ZipFile won't work.
Try using the testzip() method to check the file's integrity before extracting files.
It could be possibly a bug in zipfile, or a bug in your zip implementation. I noted that your line numbers do not match mine so I guess this is python 3.2 earlier than the current 3.2.3 release I have.
Now, as to your code, it does work for me on Python 3.2.3 on Linux. I suggest you update to the latest 3.2.x as there seem to be a number of bug fixes related to zipfile and zlib, including fixes for crashes.
I am trying to download some data from the datastore using the following
command:
appcfg.py download_data --config_file=bulkloader.yaml --application=myappname
--kind=mykindname --filename=myappname_mykindname.csv
--url=http://myappname.appspot.com/_ah/remote_api
When I didn't have much data in this particular kind/table I could
download the data in one shot - occasionally running into the
following error:
.................................[ERROR ] [Thread-11]
ExportProgressThread:
Traceback (most recent call last):
File "C:\Program Files\Google\google_appengine\google\appengine\tools
\bulkload
er.py", line 1448, in run
self.PerformWork()
File "C:\Program Files\Google\google_appengine\google\appengine\tools
\bulkload
er.py", line 2216, in PerformWork
item.key_end)
File "C:\Program Files\Google\google_appengine\google\appengine\tools
\bulkload
er.py", line 2011, in StoreKeys
(STATE_READ, unicode(kind), unicode(key_start), unicode(key_end)))
OperationalError: unable to open database file
This is what I see in the server log:
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/
ext/remote_api/handler.py", line 277, in post
response_data = self.ExecuteRequest(request)
File "/base/python_runtime/python_lib/versions/1/google/appengine/
ext/remote_api/handler.py", line 308, in ExecuteRequest
response_data)
File "/base/python_runtime/python_lib/versions/1/google/appengine/
api/apiproxy_stub_map.py", line 86, in MakeSyncCall
return stubmap.MakeSyncCall(service, call, request, response)
File "/base/python_runtime/python_lib/versions/1/google/appengine/
api/apiproxy_stub_map.py", line 286, in MakeSyncCall
rpc.CheckSuccess()
File "/base/python_runtime/python_lib/versions/1/google/appengine/
api/apiproxy_rpc.py", line 126, in CheckSuccess
raise self.exception
ApplicationError: ApplicationError: 4 no matching index found.
When that error appeared I would simply re-run the download and things
would work out well.
Of late, I am noticing that as the size of my kind increases, the
download tool fails much more often. For instance, with a kind with
~3500 entities I had to run to the command 5 times - only the last of
which succeeded. Is there a way around this error? Previously, my only
worry was I wouldn't be able to automate downloads in a script because
of the occasional failures - now I am scared I won't be able to get my
data out at all.
This issue was discussed previously here
but the post is old and I am not sure what the suggested flag does -
hence posting my similar query again.
Some additional details.
As mentioned here I tried the suggestion to proceed with interrupted downloads (in the section Downloading Data from App Engine ). When I resume after the interruption, I get no errors, but the number of rows that are downloaded are lesser than the entity count the datastore admin shows me.This is the message I get:
[INFO ] Have 3220 entities, 3220 previously transferred
[INFO ] 3220 entities (1003 bytes) transferred in 2.9 seconds
The datastore admin tells me this particular kind has ~4300 entities. Why aren't the remaining entities getting downloaded?
Thanks!
I am going to make a completely uneducated guess at this just based on the fact that I saw the word "unicode" in the first error; I had an issue that was related to my data being user generated from the web. A user put in a few unicode characters and a whole load of stuff started breaking - probably my fault - as I had implemented pretty looking repr functions and a load of other stuff. If you can, take a quick scan of your data via the console utility in your live app, maybe (if it's only 4k records), try converting all of the data to ascii strings to find any that don't conform.
And after that, I started "sanitising" user inputs (sorry, but my "public handle" field needs to be ascii only players!)
I'm working on a program to do some image wrangling in Python for work. I'm using FreeImagePy because PIL doesn't support multi-page TIFFs. Whenever I try to save a file with it from my program I get this error message (or something similar depending on which way I try to save):
Error returned. TIFF FreeImage_Save: failed to open file C:/OCRtmp/ocr page0
Traceback (most recent call last):
File "C:\Python25\Projects\OCRPageUnzipper\PageUnzipper.py", line 102, in <mod
ule> OCRBox.convertToPages("C:/OCRtmp/ocr page",FIPY.FIF_TIFF)
File "C:\Python25\lib\site-packages\FreeImagePy\FreeImagePy\FreeImagePy.py", l
ine 2080, in convertToPages self.Save(FIF, dib, fileNameOut, flags)
File "C:\Python25\lib\site-packages\FreeImagePy\FreeImagePy\FreeImagePy.py", l
ine 187, in Save return self.__lib.Save(typ, bitmap, fileName, flags)
WindowsError: exception: priviledged instruction
When I try and do the same things from IDLE, it works fine.
Looks like a permission issues, make sure you don't have the file open in another application, and that you have write permissions to the file location your trying to write to.
That's what I thought too, but I figured it out a couple hours ago. Apparently if the directory/file I'm trying to write to doesn't exist, FreeImagePy isn't smart enough to create it (most of the time. Creating a new multipage image seems to work fine) but i guess running it within IDLE, IDLE figures it out and takes care of it or something. I managed to work around it by using os.mkdir to explicitly make sure things that I need exist.