I am working on a project to display sensor data on Python GUI. Sensor data is coming at a rate of 2kHz on serial port using Arduino. I was using pyserial (using readline()) to read the sensor data on my laptop. After hours of debugging I found that python was able to read around 400 samples/sec i.e. reading frequency is around 400Hz.
Is there any way to read the serial data at higher rate with the help of python?
Thanks in Advance.
Assuming the sensor is designed to transmit 2kHz data and is doing so properly, my guess is that the time your Python code is taking to read a data sample, process the data, update a plot, etc is the limiting factor. Are you reading and processing samples one at a time? Is there a smart way to read all of the available in a big chunk reducing the number of individual read/process steps?
Are you plotting the data in "real time"? If so, plot updates are slow.
Related
so I'm currently dealing with a fatigue test machine that perform cyclic loading on a specimen. The machine's software is made using LabView and the measurement data output is a WAV file since cyclic loading is basically just a time series of force amplitude. I've dealt with machines outputting .DAT and such but this is the first time I've dealt with WAV file and honestly I have no idea how to process this WAV file into readable series of numbers representing time and force.
So, does anyone know any program that can process it? Or if possible maybe do it with Python, since I'm quite familiar with it?
Thanks in advance!
I’m using a raspberry pi 4 to collect sensor data with a python script.
Like:
val=mcp.read_adc(0)
Which can read ten thousand data per second.
And now I want to save these data to influx for real-time analysis.
I have tried to save them to a log file while reading, and then use telegraf to collect as this blog did:
But it’s not working for my stream data as it is too slow.
Also I have tried to use python's influxdb module to write directly, like:
client.write(['interface,path=address,elementss=link value=3.14'],{'db':'db'},204,'line')
It's worse.
So how can I write these data into influxdb in time. Are there any solutions?
Thank you much appreciated!
Btw, I'm a beginner and can only use simple python, so sad.
InfluxDB OSS will process writes faster if you batch them. The python client has a batch parameter batch_size that you can use to do this. If you are reading ~10k points/s I would try a batch size of about 10k too. The batches should be compressed to speed transfer.
The write method also allows sending the tags path=address,elementss=link as a dictionary. Doing this should decrease parsing effort.
Are you also running InfluxDB on the raspberry pi or do you send the data off the Pi over a network connection?
I noticed that you said in the comments that nanosecond precision is very important but you did not include a timestamp in your line protocol point example. You should provide a timestamp yourself if the timing is this critical. Without an explicit timestamp in the data, InfluxDB will insert a timestamp at "when the data arrives" which is unpredictable.
As noted in the comments, you may want to consider preprocessing this data some before sending it to InfluxDB. We can't make a suggestion without knowing how you are processing the piezo data to detect footsteps. Usually ADC values are averaged in small batches (10 - 100 reads, depending) to reduce noise. Assuming your footstep detector runs continuously, you'll have over 750 million points per day from a single sensor. This is a lot of data to store and postprocess.
Please edit your question to include move information, if you are willing.
I'm trying to get timestamp data to match accelerometer and gyroscope readings.
I'm using a Raspberry Pi 3 B+ with python to pull accelerometer (Adxl345) and gyroscope (ITG3200) readings. I'm currently reading them through I2C as fast as I can, and I write a timestamp from the system time (time.time()) immediately before reading.
I thought this would be sufficiently accurate, but when I look at the resulting data the time is often not monotonic and/or just looks wrong. In fact, the result often seem to better match the motion I was tracking if I throw out all but the first timestamp and then synthetically create times based on the frequency of the device I'm collecting from!
That said, this approach clearly gives me wrong data, but I'm at a loss as to where to pull correct data. The accelerometer and gyro don't seem to include times in anything they do, and if I pull data as fast as I can I'm still bound to miss some from them at their highest rates meaning the times I use will always be somewhat wrong.
The accelerometer also has a FIFO cache it can store some values in, but again, if I pull from that case how do I know what timestamp goes with each value?? The documention mentions the cache storing values but nothing about the timestamp.
All of which leads me to believe I'm missing something. Is there a secret here I don't know? Or a standard practice I'm unaware of? Any thoughts or suggestions would be most welcome.
I am using rsync to transfer files using rsync in Python. I have the basic UI where user can selects the file and initiate the transfer. I want to show the Expected Time Duration to transfer all the files they selected. I know the total size of all the files in bytes. What's the smart way to show them the expected file transfer duration? It doesn't have to be exact precise.
To calculate an estimated time to completion for anything, you simply need to keep track of the amount of time taken to transfer the data currently completed and base your estimate for the rest of the data on the past speed. Once you get that basic method, there are all sorts of ways you can adjust your estimate to take account of acceleration, congestion and other effects - for example, taking the amount of data transferred in the last 100 seconds, breaking this down into 20s increments and calculating a weighted mean speed.
I'm not familiar with using rsync in Python. Are you just calling it using os.exec*() or are you using something like pysync (http://freecode.com/projects/pysync)? If you are spawning rsync processes, you'll struggle to get granular data (esp. if transferring large files). I suppose you could spawn rsync --progress and get/parse the progress lines in some sneaky way but that seems horridly awkward.
I am collecting data from two pieces of equipment using serial ports (scale and conductivity probe). I need to continuously collect data from the scale which I average between collection points of the conductivity probe (roughly a minuet).
Thus I need to run two processes at the same time. One that collects data from the scale, and other which waits for data from the conductivity probe, once it gets the data it would send a command to the other process in order to get the collected scale data, which is then time stamped and saved into .csv file.
I looked into subprocess but it I cant figure out how to reset a running script. Any suggestions on what to look into.
Instead of using threads you could also implementing your data sources as generators and just loop over them to consume the incoming data and do something with it. Perhaps using two different generators and zipping them together, actually would be a nice experiment I'm not entirely sure it can be done...