Good day!
I am wondering if there is any way in which one can allow different programming languages to access a csv at the same time.
I am using c# to get live stock market data, and then Python does calculations on this data where it then returns the data to the csv file to be read by c# again, it works if I use multiple steps i.e. collect historical data predict the historical data and read the historical data, but when I try to do this in one step(live) I get the following error.
[Errno 13] Permission denied: 'CurrencyPair-Minute.csv'
I think this is the result of the file being used by the c# program.
Which I opened with the following parameters
File.Open(fiName, FileMode.Open, FileAccess.ReadWrite, FileShare.Write);
I only close the data streams when the program stops and to the streams are continually open for reading and writing in the c# program.
If I close the stream while the file is not being read or written the error I get is
Crashed in (MethodName) with ArgumentException: Stream was not readable.
Also, this will not work since the Python program continually needs to check the file for updates.
Any help would be appreciated
Thank you
You should be able to run the python script from your C# file after fetching the data and writing to the csv and closing the csv file in C#. See here: How do I run a Python script from C#?.
The flow would be something like this:
C#
Fetch data
Write to csv
Close file
Call python script
Python
Do calculation
Write to file
Close file
Exit
Related
I have this Python server, that connecting to sftp server, and pulling CSV files (there is a For loop that running in a nodeJS server, each time a different connection is coming)
In that Python server - i'm reading the CSV file with panda - like this
file = sftp.open(latestfile)
check = pd.read_csv(file).to_csv()
at the end, i return the check with the CSV file data inisde - and than i parse in the nodeJS server.
this process went really good and i managed to achieve a lot of data this way - but my Python server really crashed when he tried to read a big CSV file (22MB)
i searched online and tried to solve it with chunks and with .modin library and dask.dataframe but everytime i tried to use one of thsee methods i couldn't read the file content properly (.to_csv part)
I'm really lost right now because i can't get it to work (there can be larger files than that)
Here is a way to process your large csv files. It allows you to process groups of chunks at a time. You could modify it based on your requirements (such as through sftp etc).
Minimal example
import pandas as pd
chunksize = 10 ** 4
for chunk in pd.read_csv(latestfile, chunksize=chunksize):
process(chunk.to_csv())
I'm trying to test some file io and I was wondering if there's a way to emulate the following situation:
I have a block-storage device that is constantly being read/written from, but I want to notify the users of the proper error when they are trying to read/write from a file stored in the block-storage device but the block-storage service/device becomes unavailable or detached mid write. In which case, the read or write command would "timeout," or "hang."
I'm trying to write a test case that reads a file and I want to emulate that situation as closely as possible, meaning I don't want to use signal or just some timeout, I want to be able to make some kind of file that will hang a python file.read() statement or a file.write() statement.
Is this possible? I'm testing on a linux machine and mounting a blockstorage to a folder, pretty simple.
It seems to me that fsdisk is the right tool your looking for. It can bind your storage and inject errors.
I have a RAM tester that tests ram modules and generates a test report for each module after the test is finished(whether it passes or fails). The test report is your typical .txt file, and contains all the data I need to know about the RAM module, particularly the speed and the pass/fail condition. I am having a hard time figuring out a way to have python read the contents of the test report without blocking the RAM testing software from writing a test report.
My goal is to have python to run in the background and read the file and if the file contains the rams speed AND the word 'pass' I want python to write over the serial port where I will have a arduino waiting for a key character of my choosing to come over the serial line (that character will depend on the speed of the RAM detected). after the test report is read, and python has written a character over the serial to the arduino, I will need python to clear/truncate the .txt test report so it will be clear and ready for the next time the file is read from. that cycle will then go on indefinitely.
To get a bigger picture of the whole project I will explain the ultimate goal. the ram tester is a fully automated tester that loads it's self, tests, and ejects each module onto a conveyor belt. if the module fails the conveyor goes to the left and if it passes it goes to the right. I want to use an arduino to create a extra conveyor that will sort the tested passed ram by speed. so far everything seems doable, I'm just having a hard time with python reading the test report and clearing it without blocking the RAM tester from writing the test report. I've had someone suggest using pipe but I'm still not certain on how to do that. I will also include that the software that writes the test report is third-party software that I have no idea what language it is written it's just what came with the RAM tester. thanks ahead for taking the time to read through this and any suggestions are greatly appreciated.
If I move or delete the file it will just generate another in its place with the same name.
Do this:
Rename the file: mv foo foo-for-report
Generate the report from it.
Python can rename a file using os.rename.
import os
os.rename("foo", "foo-for-report")
I'm writing small bits of code inside org-mode files. This bits of code are slow (copy files from remote machines) and I wish to see how the copy progress (sometimes the connection to the remote machine fails and I wish to know). For that, I want to print the serial number of the currently accessed file.
Org-mode's code-block have two problems with this:
It places the either the printed messages or the returned variable in the results part of the block.
It does so only once the code ends.
Is there a way to get the printed output to a separated, live variable?
Reopening sys.stdout so that it is flushed should help.
See How to flush output of Python print?, and this blog post:
http://algorithmicallyrandom.blogspot.com.es/2009/10/python-tips-and-tricks-flushing-stdout.html
I'm trying to build an application that displays in a GUI the contents of a log file, written by a separate program that I call through subprocess. The application runs in Windows, and is a binary that I have no control over. Also, this application (Actel Designer if anyone cares) will write its output to a log file regardless of how I redirect the output of subprocess, so using a pipe for the output doesn't seem to be an option. The bottom line is that I seem to be forced into reading from a log file at the same time another thread may be writing to it. My question is if there is a way that I can keep the GUI's display of the log file's contents up to date in a robust way?
I've tried the following:
Naively opening the file for reading periodically while the child
process is running causes Python to crash (I'm guessing because the
child thread is writing to the file while I'm attempting to read its
contents)
Next I tried to open a file handle to the log filename before invoking the child process with GENERIC_READ, and SHARED_READ | SHARED_WRITE | SHARED_DELETE and reading back from that file. With this approach, the file appears empty
Thanks for any help you can provide - I'm not a professional programmer and I've been pulling my hair out over this for a week.
You should register for notifications on file change, the way tail -f does (you can find out what system calls it uses by executing strace tail -f logfile).
pyinotify provides a Python interface for these file change notifications.