I am using Python 3.6.3.
A problem remains in my script, which is fully operational.
Main modules are pandas and xslxwriter (with easygui for GUI).
From a single master file (Excel 2010), this script can generate dozens of Excel files (with xlsxwriter), each of them can contain hundreds of columns of data (depending of parameters of the master file).
Indentation, logic and results are OK.
But the last Excel file is not committed to disk, and I have to restart Python to get it.
For example, if one run produces 100 files, only 99 will be written on disk. The last one is calculated, but not visible.
If Python is not restarted, this file is written to disk at the beginning of a next run of the script.
I identified maybe a flush problem, and tried some solutions, but this problem still remains.
Are there some tricks to force the buffer? I am not allowed to modify the environment variables on my professional computer.
Thank you for your help and your time :)
Thank you Mark Tolonen
You were right : the file was not closed properly, and it was because I made a mistake.
My code was a bit difficult to summarize, and I could not post a résumé of my script.
First, a keyword continue (for the main loop) was bad-indented and I replaced it at the right place.
But just before this keyword, I was closing the xlsxwriter file with: workbook.close (only one in the script exists for the main loop).
But this was not mentioned as an error at run-time.
Each xlsxwriter file was committed to disk except the last one, as mentioned in my question above.
I then reviewed the documentation at https://xlsxwriter.readthedocs.io/workbook.html, and I noticed that parenthesis were missing when worbook is closed.
After correction by adding the missing parenthesis: workbook.close(), all is fine now :)
I would like to share this information, because some may have meet the same problem.
Than you also to progmatico for your information on flush properties.
Greetings from Paris, France :)
Related
I have two programs written in python and converted to one-file exe using auto-py-to-exe.
the first program writes to a file, which is read by the second program. The problem is when the second program wants to read the file the same time as it is being written, the code stops with a permission error.
The solutions that seemed to work are:
Using time management which is not useful in my case, since the reading and writing times are not constant.
I could check if the file is accessible, which might be a solution, however, I suppose it would raise an error if while reading the file, the writer tries to change the file.
I could use the size of the file to check if writing to the file has been finished, and then execute the reader, however, this does not seem to be both logical and pythonic!
I found some solutions using os.pipe(), but to be honest, I couldn't understand what the process does. If this is a solution, I would be glad to have it explained in simple English.
That's it. Any suggestions?
P.S: OS is windows and I am using Python 3.9
Solved:
Thanks to the replies and suggestions, I didn't know that the try except commands accept ErrorType. Thus, I solved the problem by using 'except' and 'PermissionError'. the code runs in a loop and it is checked again in a few seconds.
However, the drawback is this: the reading time should be less than the time the writer comes back to rewrite the file! In my case, as suggested by friends, I combined the two programs so they are run sequentially.
I have a project in mind, but there is a section that I don't know how to do. I'm using Python version 3.6 and windows 10. For example we have a file name of "example.txt" I want to prevent the name and its content of this file from being changed.
I did research on this topic, but I could not reach any research. Can we prevent the file's name (including its extension) from changing or its contents?To realize this, I think it is necessary to start as an administrator.
Thanks.
It is possible to stop another program from editing a file by locking it in python.
There is a module that does this called filelock. Take a look at the source code to see how it is done.
It is also worth noting that more advanced ransomware will try to stop processes so they can encrypt files, so this might not work in all cases.
The code inside of a python file randomly deleted, is there anyway to restore? It is a file of 3.70 KB but when opened and put into a text document there is nothing.
Open with python to see what it contains
with open('deleted.py', 'rb') as f:
print(repr(f.read()))
Since you are a new user I am assuming you are new to code development etc. Therefore, you should look at some versioning control tools like:
SVN
Github
Gitlab
There are some more, but these are the most common ones. They are used to store your code and to revert you code if you mess up. They are also used to merge codes when different programmers are changing it. For the moment this will not help but will help in the future.
For now you may look at some restore tools but I highly doubt it that you are able to recreate the file. Another possiblity is: when you haven IDE to look at your command history. Maybe you executed the script and you can find the executed script as commands in the command history.
I have a python script which starts by reading a few large files and then does something else. Since I want to run this script multiple times and change some of the code until I am happy with the result, it would be nice if the script did not have to read the files every time anew, because they will not change. So I mainly want to use this for debugging.
It happens to often, that I run scripts with bugs in them, but I only see the error message after minutes, because the reading took so long.
Are there any tricks to do something like this?
(If it is feasible, I create smaller test files)
I'm not good at Python, but it seems to be able to dynamically reload code from a changed module: How to re import an updated package while in Python Interpreter?
Some other suggestions not directly related to Python.
Firstly, try to create a smaller test file. Is the whole file required to demonstrate the bug you are observing? Most probably it is only a small part of your input file that is relevant.
Secondly, are these particular files required, or the problem will show up on any big amount of data? If it shows only on particular files, then once again most probably it is related to some feature of these files and will show also on a smaller file with the same feature. If the main reason is just big amount of data, you might be able to avoid reading it by generating some random data directly in a script.
Thirdly, what is a bottleneck of your reading the file? Is it just hard drive performance issue, or do you do some heavy processing of the read data in your script before actually coming to the part that generates problems? In the latter case, you might be able to do that processing once and write the results to a new file, and then modify your script to load this processed data instead of doing the processing each time anew.
If the hard drive performance is the issue, consider a faster filesystem. On Linux, for example, you might be able to use /dev/shm.
How can I see script errors for my python MIDI Remote Scripts in Ableton Live? I've seen references online to a hidden Python console, but no documentation on how to open it, if it would even help me!
Currently I type code, reload the script, and see what happens. If nothing happens, then something is wrong. Very tedious...
Edit: I should also point out that there isn't anything useful in the Log.txt file either, yet that file is being updated.
To debug the control surface, you can define your own log method like so:
def log(self, message):
sys.stderr.write("LOG: " + message.encode("utf-8"))
Usage example:
year = 1999
self.log("I'm gonna party like it's " + str(year))
This will append
21179419 ms. RemoteScriptError: LOG: Tonight I'm gonna party like it's 1999
to your Log.txt.
Also, it may be worth knowing that (at least as of Live 9.1) edited control surface files are recompiled every time you load a new song, no need to restart the application.
EDIT: changed the stderr write method so that it doesn't write two lines and an extra return for every log
I use the log.txt for a ton of debugging. It is a little hard to read, but here's my tips.
log writes top down (new info is on next line) and records all the events, more or less.
If you get an error you will get a large hex looking block - about 20+ lines looking like this:
280 ms. Exception: 0x00495580:0x00000000
Right above that block is the error or what may have thrown the script in the Ableton run time.
As you travel UP from that exception you will see something like a trace.
also - you can use :
self.log_message("STUFF")
to write into the log (there are other methods).
I have seen some log file parsers (real time) in my work. Not tried them yet.
Hope that hleps a little. O was ablt to write a PHP app for the python (yeah, sinful I know). http://modrn.dj/app
For the sake of completeness: on Mac OS X 10.9, the Log.txt file is in
/Users/-username-/Library/Preferences/Ableton/Live 9.1/
The Live 9.1 folder may of course be different for different versions of Live.
Note the Library folder is hidden. It took me a while to figure that out. An easy way to access it with Finder: click the Go menu item, then hold Alt and the Library folder will pop into the menu
what about http://remotescripts.blogspot.com/2010_03_01_archive.html
This has certainly changed over the years with different versions of Ableton Live.
I'm running Live 8 and first tried to get the LiveAPI stuff to spit out debug information: http://post.monome.org/comments.php?DiscussionID=4607&page=1
I started with an older version that didn't work with OSX, reported here: http://code.google.com/p/liveapi/issues/detail?id=4 I don't think the new version includes the telnet console.
SO, the best answer is to find a log file here, which contains debug information on a problematic MIDI Remote Scripts:
Library/Preferences/Ableton/Live 8.2.1/Log.txt
You can also use http://julienbayle.net/ableton-live-9-midi-remote-scripts for Ableton Live 9