We Keep Coding

Unable to get complete git history of file with python

I am trying to get all the content from each revision - history of a file in my local repo i am using gitpython lib and here is the code:
import git,json
from pprint import pprint
repo = git.Repo()
path = "my_file_path"
revlist = (
(commit, (commit.tree / path).data_stream.read())
for commit in repo.iter_commits(paths=path)
)
for commit, filecontents in revlist:
filecontentsjs = json.loads(filecontents)
pprint(commit)
pprint(filecontentsjs["execution_status"])
pprint(filecontentsjs["execution_end_time"])
Problem: i am comparing our bitbucket history and the history i get from this script and the script comes up short, meaning the bitbucket history has more revisions of the file but when i clone the repo locally i get like half of the revisions with the script
am i missing something here? limitation or anything like that?

so it turns out it was my fault i didn't notice that the filename changed slightly and bitbucket did what it supposed to do since it thought "if the code is the same the file is the same" which is not true
so adding the --follow flag in the git log i was seeing the full "bad" history.
The real "good" history is without the --follow since i care only about the file with the same name

python-win32com excel com model started generating errors

Over the last few days, I have been working on automating the generation of some pivot tables for a number of reports.
Boiled down to the minimum, the following code was working without issue:
import win32com.client
objExcelApp = win32com.client.gencache.EnsureDispatch('Excel.Application')
objExcelApp.Visible = 1
This would pop-up an instance of excel and I could continue working in Python. But suddenly, today my scripts are failing with the following:
>>>import win32com.client
>>> objExcelApp = win32com.client.gencache.EnsureDispatch('Excel.Application')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 534, in EnsureDispatch
mod = EnsureModule(tla[0], tla[1], tla[3], tla[4], bForDemand=bForDemand)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 391, in EnsureModule
module = GetModuleForTypelib(typelibCLSID, lcid, major, minor)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 266, in GetModuleForTypelib
AddModuleToCache(typelibCLSID, lcid, major, minor)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 552, in AddModuleToCache
dict = mod.CLSIDToClassMap
AttributeError: module 'win32com.gen_py.00020813-0000-0000-C000-000000000046x0x1x9' has no attribute 'CLSIDToClassMap'
The code has not changed from yesterday to today. I have no idea what is happening!!!.
Another interesting kicker. if I do the same code in the same session again I get a different error:
>>> objExcelApp = win32com.client.gencache.EnsureDispatch('Excel.Application')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 534, in EnsureDispatch
mod = EnsureModule(tla[0], tla[1], tla[3], tla[4], bForDemand=bForDemand)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\win32com\client\gencache.py", line 447, in EnsureModule
if module.MinorVersion != tlbAttributes[4] or genpy.makepy_version != module.makepy_version:
AttributeError: module 'win32com.gen_py.00020813-0000-0000-C000-000000000046x0x1x9' has no attribute 'MinorVersion'
>>>
So I jump over to a windows machine with a fresh windows install, install python37 and pip install pypiwin32. Run the very same lines and excel opens just like it did yesterday on my original machine.
I tried un-installing and re-installing with no luck. Any idea what is going on here?
NOTE:
Dynamic dispatch still works:
import win32com.client
objExcelApp = win32com.client.Dispatch("Excel.Application")
objExcelApp.Visible = 1
But I specifically need static dispatch as Pivot Tables won't work with a dynamically dispatched object (much later in my code):
objExcelPivotCache = objExcelWorkbook.PivotCaches().Create(SourceType=win32c.xlDatabase, SourceData=objExcelPivotSourceRange)

I had the same issue and I resolved it by following the instructions here: https://mail.python.org/pipermail/python-win32/2007-August/006147.html
Deleting the gen_py output directory and re-running makepy SUCCEEDS
and subsequently the test application runs OK again.
So the symptom is resolved, but any clues as to how this could have
happened. This is a VERY long running application (think 24x7 for
years) and I'm concerned that whatever caused this might occur again.
To find the output directory, run this in your python console / python session:
import win32com
print(win32com.__gen_path__)
Based on the exception message in your post, the directory you need to remove will be titled '00020813-0000-0000-C000-000000000046x0x1x9'. So delete this directory and re-run the code. And if you're nervous about deleting it (like I was) just cut the directory and paste it somewhere else.
💡Note that this directory is usually in your "TEMP" directory (copy-paste %TEMP%/gen_py in Windows File Explorer and you will arrive there directly).
I have no idea why this happens nor do I know how to prevent it from happening again, but the directions in the link I provided seemed to work for me.

A more straightforward solution was posted in a related question Issue in using win32com to access Excel file.
Basically, you just need to delete the folder C:\Users\<your username>\AppData\Local\Temp\gen_py and rerun your code.
💡TIP: You can also put in your Windows file explorer %TEMP%\gen_py to access it directly, and then delete its content.

Execute this command line in a powershell or cmd (NOT in Administrator mode => wouldn't work for me)
python -m win32com.client.makepy "Excel.Application"
It fixes all errors and you don't have to change your python code.
And keep using
win32com.client.gencache.EnsureDispatch("Excel.Application")
With gencache.EnsureDispatch you have access to the constants of the application loaded dynamically by makepy which must have the registered application (in our case Excel.Application).
If you have the same problem with Outlook, use "Outlook.Application" in above.
If still not working, reinstall pywin32 of your python distribution
<path to python root or venv>\pip.exe uninstall pywin32
<path to python root or venv>\pip.exe install pywin32

What has worked for me is:
excel = win32.gencache.EnsureDispatch('Excel.Application')
#change to =>
excel = win32.Dispatch('Excel.Application')

For me, it seems, the issue was that I have multiple processes that interact with Windows apps through win32com.
Since win32com creates the "gen_py" directory in win32api.GetTempPath() this can cause conflicts and the cache getting corrupted.
My solution is to set a custom location for "gen_py" for each process. A simple example:
from pathlib import Path
import win32com
gen_py_path = '/some/custom/location/gen_py'
Path(gen_py_path).mkdir(parents=True, exist_ok=True)
win32com.__gen_path__ = gen_py_path
# Any other imports/code that uses win32com
This way you don't have to delete the default "gen_py" folder and wonder what issues might arise. But if you still find you need to delete, you can just delete the custom folder and know you're deleting the cache just for that process.

To add to this discussion, for those receiving this error as part of an unsupervised automated process, you can fully automating the recovery process and allow any processes to continue unsupervised.
As answers by Ian and Qin mention, we need to delete the gen_py output directory and restart the process. As far as automation goes, there are two problems with this: the gen_py output directory is a temporary directory that can change, and currently running processes that rely on gen_py continue to fail even after it is regenerated.
To address these problems, we can dynamically look up the location, then we can completely nuke the whole process and restart it. I've tried deleting and re-importing win32com, but it seems a reference to the corrupted cache is still maintained, so the whole process needs to be restarted.
temp_data_dir = os.environ.get('LOCALAPPDATA'))
gen_py_dir = ''
for curr_path, dirs, files, in walk(temp_data_dir)
if 'gen_py' in dirs:
gen_py_dir = Path(curr_path).joinpath('gen_py')
shutil.rmtree(gen_py_dir)
execv(restart_args)
The restarted process should call win32com.client.gencache.EnsureDispatch('Excel.Application') again and a fresh copy of gen_py will now be utilized, then we can retry the failed Excel automation code.
The issue happens sporadically in a completely unreliable way that I can't replicate. My best guess is that the gen_py cache is sometimes corrupted somehow and just needs to be refreshed.

How to get files written by the micro:bit to the PC?

I am new to BBC micro:bit, so I have the following Problem:
I want to read movements and write it to a file on the m:b and after all I want to download it from there to the pc to work on it.
I wrote the file like that:
from microbit import *
with open('FileName.txt', 'w') as my_file:
my_file.write('text to write down')
I couldn't see the file, when I used the m:b as USB- device.
But when I programmed the m:b to list all files it wrote short before the file was on it.
I know, m:b has no disk operating System, so I tried to pull it with python code, I started the following python code on a Windows pc:
(see: Docs to microfs)
import microfs
print microfs.ls()
But I got the error IOError: Could not find micro:bit.
The m:b is not found, I suppose.
What am I doing wrong? What else could I try?

Sometimes if the micro:bit cannot be found by scripts like uFlash or MicroFs it helps if you unplug the USB cable, wait a few seconds and plug it again (an additional note for Linux users, although I am aware that this is not the case for you, on Linux is also helpful to wait until the micro:bit drive has been mounted).
You are in the right track using MicroFs to access the MicroPython files, as they are in the microcontroller flash, and not accessible through the USB mass storage interface. Remember that writing a new program into the micro:bit does erase all of the flash contents, including any files your previous program might have created.
For easy of use I would recommend using the Mu editor (https://codewith.mu), as it offers you a GUI to move files in and out of the micro:bit. It's worth noting that Mu uses uFlash and MicroFs internally, so it will give you the same results as using those individual command line tools.

I, too, have spent several hours searching for the answer to getting microfs and mu to read a file from the microbit, after receiving the 'can't find it' message.
I have just found a solution: update the microbit firmware.
I have flashed an empty Python file from mu to the microbit and then used the command line ufs put <path to file>\main.py to copy over code that creates a text file and displays a heart on the microbit.
Now the Files option in mu correctly displays main.py and the created text file on my microbit.
I hope this helps.

As mentioned, you can use the command line ufs to put, get, list and delete files on the microbit.
pip install microfs --user
Then use ufs to list, delete, put and get files on to and off the microbit.
github site: https://github.com/ntoll/microfs.
The commands are:
ufs ls #list files on the card
ufs rm <filename> # remove a file
ufs put <filename> # write a file to the micro:bit
ufs get <filename> # get a file from the micro:bit
First of all put a blank .py file on to the micro:bit. In Linux you can create this using:
touch empty.py
Connect with microbit by double clicking on it using your file browser (e.g. Nautilus in Linux).
Use mu to flash empty.py onto the microbit.
Then write your Micropython code and call it main.py. Use ufs to write main.py to the micro:bit.
ufs main.py
This file will run at reset and restart. You can have your main.py file import and use other Micropython files on the micro:bit. Put these onto the micro:bit using ufs.
ufs <file to import to main.py>.py
e.g.
ufs put utilities.py
The files can be overwritten using ufs put.
You can't use ufs and a repl at the same time.
Now you are in a position to write and read a text file. Please find two example functions I have used for this.
def read_file(filename):
with open(filename, 'r') as my_file:
read_value = my_file.read()
return read_value
def write_file(filename, value):
with open(filename, 'w') as my_file:
my_file.write(str(value))
my_file.close()
The files are written to the microbit and the data remains intact after repowering the device. The data file can be seen using:
ufs ls
Then copied to your local machine using:
ufs get <filename>

Django TemporaryUploadedFile does not exist but nevertheless it is read successfully

I have the following situation here. My OS shows that django TemporaryUploadedFile which I got via the POST request does not exist anymore but somehow this uploaded file can be read.
Here is the code
text_file = request.FILES['text_file']
print(text_file.temporary_file_path())
os.system('ls -l ' + text_file.temporary_file_path())
fs = FileSystemStorage()
file_new =fs.save(text_file.name, text_file)
print(text_file.temporary_file_path())
os.system('ls -l ' + text_file.temporary_file_path())
fs.delete(file_new)
for chunk in text_file.chunks():
text += chunk.decode(encoding)
print('Got text OK.')
This gives the following output:
/tmp/tmp0tngal9t.upload foo.txt
-rw------- 1 mine machine 3072889 oct 18 19:29 /tmp/tmp0tngal9t.upload
/tmp/tmp0tngal9t.upload foo.txt
ls: cannot access '/tmp/tmp0tngal9t.upload': No such file or directory
Got text OK.
So TemporaryUploadedFile is disappeared after it was saved to file_new which later is also deleted. Anyway text_file is successfully read by chunks and I get all the text from uploaded foo.txt file. How it is possible? From where text_file.chunks() gets the data if text_file does not exist anymore?
I use:
python 3.5.2
django 1.10.2
ubuntu 16.04.1

I found out that this problem still remains for bare python, so it is not particularly related to django as in this example I just read text_file which were open in request.FILES['text_file'].
I re-asked the similar question here focusing on python only. It turned out that the problem is not so related with python either, but with Linux/Unix system file management. I quote here the answer of Jean-François Fabre:
Nothing to do with Python. In C, Fortran, or Visual Cobol you'd have
the same behaviour as long as the code gets its handle from open
system call.
On Linux/Unix systems, once a process has a handle on a file, it can
read it, even if the file is deleted. For more details check that
question (I wasn't sure if it was OK to do that, it seems to be)
On Windows you just wouldn't be able to delete the file as long as
it's locked by a process.

py2exe - generate single executable file

I thought I heard that py2exe was able to do this, but I never figured it out. Has anyone successfully done this? Can I see your setup.py file, and what command line options you used?
Basically I'm thinking of it giving me a single executable file that does something like unzips itself to maybe /temp and runs.

The way to do this using py2exe is to use the bundle_files option in your setup.py file. For a single file you will want to set bundle_files to 1, compressed to True, and set the zipfile option to None. That way it creates one compressed file for easy distribution.
Here is a more complete description of the bundle_file option quoted directly from the py2exe site*
Using "bundle_files" and "zipfile"
An easier (and better) way to create
single-file executables is to set
bundle_files to 1 or 2, and to set
zipfile to None. This approach does
not require extracting files to a
temporary location, which provides
much faster program startup.
Valid values for bundle_files are:
3 (default) don't bundle
2 bundle everything but the Python interpreter
1 bundle everything, including the Python interpreter
If zipfile is set to None, the files will be bundle
within the executable instead of library.zip.
Here is a sample setup.py:
from distutils.core import setup
import py2exe, sys, os
sys.argv.append('py2exe')
setup(
options = {'py2exe': {'bundle_files': 1, 'compressed': True}},
windows = [{'script': "single.py"}],
zipfile = None,
)

PyInstaller will create a single .exe file with no dependencies; use the --onefile option. It does this by packing all the needed shared libs into the executable, and unpacking them before it runs, just as you describe (EDIT: py2exe also has this feature, see minty's answer)
I use the version of PyInstaller from svn, since the latest release (1.3) is somewhat outdated. It's been working really well for an app which depends on PyQt, PyQwt, numpy, scipy and a few more.

As the other poster mention, py2exe, will generate an executable + some libraries to load. You can also have some data to add to your program.
Next step is to use an installer, to package all this into one easy-to-use installable/unistallable program.
I have used InnoSetup with delight for several years and for commercial programs, so I heartily recommend it.

I've been able to create a single exe file with all resources embeded into the exe.
I'm building on windows. so that will explain some of the os.system calls i'm using.
First I tried converting all my images into bitmats and then all my data files into text strings.
but this caused the final exe to be very very large.
After googleing for a week i figured out how to alter py2exe script to meet my needs.
here is the patch link on sourceforge i submitted, please post comments so we can get it included in
the next distribution.
http://sourceforge.net/tracker/index.php?func=detail&aid=3334760&group_id=15583&atid=315583
this explanes all the changes made, i've simply added a new option to the setup line.
here is my setup.py.
i'll try to comment it as best I can.
Please know that my setup.py is complex do to the fact that i'm access the images by filename.
so I must store a list to keep track of them.
this is from a want-to-b screen saver I was trying to make.
I use exec to generate my setup at run time, its easyer to cut and paste like that.
exec "setup(console=[{'script': 'launcher.py', 'icon_resources': [(0, 'ICON.ico')],\
'file_resources': [%s], 'other_resources': [(u'INDEX', 1, resource_string[:-1])]}],\
options={'py2exe': py2exe_options},\
zipfile = None )" % (bitmap_string[:-1])
breakdown
script = py script i want to turn to an exe
icon_resources = the icon for the exe
file_resources = files I want to embed into the exe
other_resources = a string to embed into the exe, in this case a file list.
options = py2exe options for creating everything into one exe file
bitmap_strings = a list of files to include
Please note that file_resources is not a valid option untill you edit your py2exe.py file as described in the link above.
first time i've tried to post code on this site, if I get it wrong don't flame me.
from distutils.core import setup
import py2exe ##UnusedImport
import os
#delete the old build drive
os.system("rmdir /s /q dist")
#setup my option for single file output
py2exe_options = dict( ascii=True, # Exclude encodings
excludes=['_ssl', # Exclude _ssl
'pyreadline', 'difflib', 'doctest', 'locale',
'optparse', 'pickle', 'calendar', 'pbd', 'unittest', 'inspect'], # Exclude standard library
dll_excludes=['msvcr71.dll', 'w9xpopen.exe',
'API-MS-Win-Core-LocalRegistry-L1-1-0.dll',
'API-MS-Win-Core-ProcessThreads-L1-1-0.dll',
'API-MS-Win-Security-Base-L1-1-0.dll',
'KERNELBASE.dll',
'POWRPROF.dll',
],
#compressed=None, # Compress library.zip
bundle_files = 1,
optimize = 2
)
#storage for the images
bitmap_string = ''
resource_string = ''
index = 0
print "compile image list"
for image_name in os.listdir('images/'):
if image_name.endswith('.jpg'):
bitmap_string += "( " + str(index+1) + "," + "'" + 'images/' + image_name + "'),"
resource_string += image_name + " "
index += 1
print "Starting build\n"
exec "setup(console=[{'script': 'launcher.py', 'icon_resources': [(0, 'ICON.ico')],\
'file_resources': [%s], 'other_resources': [(u'INDEX', 1, resource_string[:-1])]}],\
options={'py2exe': py2exe_options},\
zipfile = None )" % (bitmap_string[:-1])
print "Removing Trash"
os.system("rmdir /s /q build")
os.system("del /q *.pyc")
print "Build Complete"
ok, thats it for the setup.py
now the magic needed access the images.
I developed this app without py2exe in mind then added it later.
so you'll see access for both situations. if the image folder can't be found
it tries to pull the images from the exe resources. the code will explain it.
this is part of my sprite class and it uses a directx. but you can use any api you want or just access the raw data.
doesn't matter.
def init(self):
frame = self.env.frame
use_resource_builtin = True
if os.path.isdir(SPRITES_FOLDER):
use_resource_builtin = False
else:
image_list = LoadResource(0, u'INDEX', 1).split(' ')
for (model, file) in SPRITES.items():
texture = POINTER(IDirect3DTexture9)()
if use_resource_builtin:
data = LoadResource(0, win32con.RT_RCDATA, image_list.index(file)+1) #windll.kernel32.FindResourceW(hmod,typersc,idrsc)
d3dxdll.D3DXCreateTextureFromFileInMemory(frame.device, #Pointer to an IDirect3DDevice9 interface
data, #Pointer to the file in memory
len(data), #Size of the file in memory
byref(texture)) #ppTexture
else:
d3dxdll.D3DXCreateTextureFromFileA(frame.device, ##UndefinedVariable
SPRITES_FOLDER + file,
byref(texture))
self.model_sprites[model] = texture
#else:
# raise Exception("'sprites' folder is not present!")
Any questions fell free to ask.

You should create an installer, as mentioned before. Even though it is also possible to let py2exe bundle everything into a single executable, by setting bundle_files option to 1 and the zipfile keyword argument to None, I don't recommend this for PyGTK applications.
That's because of GTK+ tries to load its data files (locals, themes, etc.) from the directory it was loaded from. So you have to make sure that the directory of your executable contains also the libraries used by GTK+ and the directories lib, share and etc from your installation of GTK+. Otherwise you will get problems running your application on a machine where GTK+ is not installed system-wide.
For more details read my guide to py2exe for PyGTK applications. It also explains how to bundle everything, but GTK+.

I'm told bbfreeze will create a single file .EXE, and is newer than py2exe.

I recently used py2exe to create an executable for post-review for sending reviews to ReviewBoard.
This was the setup.py I used
from distutils.core import setup
import py2exe
setup(console=['post-review'])
It created a directory containing the exe file and the libraries needed. I don't think it is possible to use py2exe to get just a single .exe file. If you need that you will need to first use py2exe and then use some form of installer to make the final executable.
One thing to take care of is that any egg files you use in your application need to be unzipped, otherwise py2exe can't include them. This is covered in the py2exe docs.

try
c_x freeze
it can create a good standalone