So before I start I know this is not the proper way to go about doing this but this is the only method that I have for accessing the data I need on the fly.
I have a system which is writing telemetry data to a .csv file while it is running. I need to see some of this data while it is being written but it is not being broadcast in a manner which allows me to do this.
Question: How do I read from a CSV file which is being written to safely.
Typically I would open the file and look at the values but I am hoping to be able to write a python script which is able to examine the csv for me and report the most recent values written without compromising the systems ability to write to the file.
I have absolutely NO access to the system or the manner in which it is writing to the CSV I am only able to see that the CSV file is being updated as the system runs.
Again I know this is NOT the right way to do this but any help you could provide would be extremely helpful.
This is mostly being run in a Windows environment
You can do something like:
tailf csv_file | python your_script.py
and read from sys.stdin
Related
A python package that I'm using has data stored under a single file with a .pkz extension. How would I unzip (?) this file to view the format of data within?
Looks like what you are referencing is just a one-off file format used in sample data in scikit-learn. The .pkz is just a compressed version of a Python pickle file which usually has the extension .pkl.
Specifically you can see this in one of their sample files here along with the fact they are using the zlib_codec. To open it, you can go in reverse or try uncompressing from the command line.
Before attempting to open an PKZ file, you'll need to determine what kind of file you are dealing with and whether it is even possible to open or view the file format.
Files which are given the .PKZ extension are known as Winoncd Images Mask files, however other file types may also use this extension. If you are aware of any additional file formats that use the PKZ extension, please let us know.
How to open a PKZ file:
The best way to open an PKZ file is to simply double-click it and let the default assoisated application open the file. If you are unable to open the file this way, it may be because you do not have the correct application associated with the extension to view or edit the PKZ file.
If you can do it, great, you have a program installed that can do it, lets say that program is called pkzexecutor.exe, with python, you just have to do:
import subprocess
import os
path_to_notepad = 'C:\\Windows\\System32\\pkzexecutor.exe'
path_to_file = 'C:\\Users\\Desktop\\yourfile.pkz'
subprocess.call([path_to_notepad, path_to_file])
From the source code for fetch_olivetti_faces, the file appears to be downloaded from http://cs.nyu.edu/~roweis/data/ and originally has a .mat file extension, meaning it is actually a MATLAB file. If you have access to MATLAB or another program which can read those files, try opening it from there with the original file extension and see what that gives you.
(If you want to try opening this file in Python itself, then perhaps give this question a look: Read .mat files in Python )
I am new to python and have until now written only a few programs to help with my job (I'm a sysadmin). I am writing this script now which will write the output of a MySQL query to a file. While in-between looping, I want to check for an extra condition and if the condition does not match, I want to close the file that I am writing to without saving what it has already written to the file. Like 'exit without saving'. I wrote this simple code to see if not closing the file with a close() will exit without saving, but it is creating the file with the content after I run and exit this code. So, is there a legal way in Python to exit a file without saving?
#/usr/bin/python
fo=open('tempfile.txt','a')
fo.write('This content\n')
P.S:- Python version is 2.4.3 (sorry, cannot upgrade)
There is no such concept in programming.
For the vast majority of the programming languages out there, the write command will attempt to put data directly in the file. This may or may not occur instantly for various reasons so many languages also introduce the concept of flush which will guarantee that your data is written to the file
What you want to do instead is to write all your data to a huge buffer (a string) then conditionally write or skip writing to the file.
Use the tempfile module to create your temporary, then if you need to save it you can do so explicitly using shutil.copyfileobj.
See Quick way to save a python TempFile?
Note that this is only if you absolutely need a temporary file (large amounts of data, etc.); if your contents are small then just using a stringbuffer and only writing it if you need to is a better approach.
Check for the condition before opening the file:
#/usr/bin/python
if condition():
fo=open('tempfile.txt','a')
fo.write('This content\n')
The safest way to do this is not to write to, or even open, the file until you know you want to save it. You could, for example, save what you might eventually write to a string, or list of same, so you can write them once you've decided to.
I have a file that I want to read. The file may at any time be overwritten by another process. I do not want to block that writing. I am prepared to manage corruption to the data that I read, but do not want my reading to be in any way change the behaviour of the writing process.
The process that is writing the file is a delphi program running locally on the server. It opens the file using fmCreate. fmCreate tries to open the file exclusively and fails if there are any other handles on the file.
I am reading the file from a python script that accesses the file remotely across our network.
I am interested in whether there is a solution, independent of whether it is supported by python or delphi. I want to know if there is any way of achieving this under windows without modifying the writing program.
Edit: To reiterate, this is not a duplicate. The other question was trying to get read access to a file that is being written to. I want to the writer to have access to a file that I have open for reading. These are different questions (although I fear the answer will be similar, that it can't be done.)
I think the real answer here, all of these years later, is to use opportunistic locks. With this, you can open the file for read access, while telling the OS that you want to be notified if another program wants to access the file. Basically, you can use the file as long as you like, and then back off if someone else needs it. This avoids the sharing/access violation that the other program would normally get, if you had just opened the file "normally".
There is an MSDN article on Opportunistic Locks. Raymond Chen also has a blog article about this, complete with sample code: Using opportunistic locks to get out of the way if somebody wants the file
The key is calling the DeviceIoControl function, with the FSCTL_REQUEST_OPLOCK flag, and passing it the handle to an event that you previously created by calling CreateEvent.
It should be straightforward to use this from Delphi, since it supports calling Windows API functions. I am not so sure about Python. But, given the arrangement in the question, it should not be necessary to modify the Python code. Just make your Delphi code use the opportunistic lock when it opens the file, and let it get out of the way when the Python script needs the file.
Also much easier and lighter weight than a filter driver or the Volume Shadow Copy service.
You can setup a filter driver which can act in two ways: (1) modify the flags when the file is opened, and (2) it can capture the data when it's written to the file and save a copy of the data elsewhere.
This approach is much more lightweight and efficient than volume shadow copy service, mentioned in comments, however it requires having a filter driver. There exist several drivers on the market (i.e. those are products which include a driver and let you write business logic in user mode) yet they are costly and can be an overkill in your case. Still, if you need the thing for private use only, contact me privately for a license for our CallbackFilter.
Update: if you want to let the writer open the file which has been already opened, then a filter which will modify flags when the file is being opened is your only option.
I have a binary file which was created by a VBA file (I don't work with VBA or binary at all) but I need to get Python to read this binary file (which includes a list of inputs for a calculation) and then write these values into an xml file.
If I know the order of the inputs into the file that is created, is it possible to read the binary code line by line, to get input by input and then write into the xml?
I have to use Python rather than VBA since I am not authorised to change the original VBA files.
I apologise for the lack of information, I only know a bit of Python and have never worked with VBA or binary. I really appreciate any help anyone can give me! Thank you =)
If you have the VBA code that made the file you should be able to copy the data structure in python and then deserialize it.
Python is probably not the best tool, I would recommend VBA as it is going to be best suited to reading that file. Then, create your xml file as output.
I am writing a Python logger script which writes to a CSV file in the following manner:
Open the file
Append data
Close the file (I think this is necessary to save the changes, to be safe after every logging routine.)
PROBLEM:
The file is very much accessible through Windows Explorer (I'm using XP). If the file is opened in Excel, access to it is locked by Excel. When the script tries to append data, obviously it fails, then it aborts altogether.
OBJECTIVE:
Is there a way to lock a file using Python so that any access to it remains exclusive to the script? Or perhaps my methodology is poor in the first place?
Rather than closing and reopening the file after each access, just flush its buffer:
theloggingfile.flush()
This way, you keep it open for writing in Python, which should lock the file from other programs opening it for writing. I think Excel will be able to open it as read-only while it's open in Python, but I can't check that without rebooting into Windows.
EDIT: I don't think you need the step below. .flush() should send it to the operating system, and if you try to look at it in another program, the OS should give it the cached version. Use os.fsync to force the OS to really write it to the hard drive, e.g. if you're concerned about sudden power failures.
os.fsync(theloggingfile.fileno())
As far as I know, Windows does not support file locking. In other words, applications that don't know about your file being locked can't be prevented from reading a file.
But the remaining question is: how can Excel accomplish this?
You might want to try to write to a temporary file first (one that Excel does not know about) and replace the original file by it lateron.