I recently started using the Arduino Due for a high-speed ADC project, and I need to be able to write ~3MBit/s to a PC with the overhead of the Arduino code to communicate with that ADC. I have been doing some testing of the native arduino speed using SerialUSB (arduino_USB_slow.ino), and have got communication to work with a simple python script using pySerial.
With increasing block size I am able to increase the data rate (from 1.22MBit/s writing one byte at a time to SerialUSB to 7.58Mbit/s using 8 bytes at a time). This is great, but I'm cycle-constrained and need to do better than this. However, if I try to send more than 8 bytes at a time with the SerialUSB.write() command, I get no data transferred over the serial port at all. My pySerial input buffer stays at 0 bytes after opening the port, which I check using port.in_waiting. There are no problems compiling the code on the arduino side, no problems uploading it, and no problems with either the arduino or pyserial when trying to write 8 bytes. The problem only exists when I try to write more than 8 bytes (I've tried 9, 10, 16, 64, none of them have worked).
There's no indication in the Serial library documentation of a limitation on the input byte array size, and as I understand it if the buffer isn't large enough, the SerialUSB library on the Arduino side will just block further execution until the hardware buffer is refilled enough times to get through all the data. Other people have managed to use much larger block size without difficulty.
This does appear to be a pySerial issue, because when I open the USB device with minicom and run:
sudo cat /dev/tty.usbmodem11414
I get a bunch of binary barf that isn't present when the device isn't transmitting. This tells me the problem isn't with the Arduino (it's sending over its data just fine), but with pySerial or how I am using it. Is there a word limitation on pyserial when receiving a stream of data? This seems really bizarre. The pyserial code is dead simple:
import serial
import numpy as np
maxBytes = 1000000
byteCounter = 0
dataBuffer = np.zeros(maxBytes + 1020)
arduino = serial.Serial()
arduino.port = '/dev/cu.usbmodem14141'
arduino.open()
while byteCounter < maxBytes:
bytesAvailable = arduino.in_waiting
dataBuffer[byteCounter:byteCounter+bytesAvailable] = np.frombuffer(arduino.read(bytesAvailable), dtype='uint8')
byteCounter += bytesAvailable
print(byteCounter)
And the Arduino code is below:
char adcData[16] = {'a', 'a', 'a', 'a','a', 'a', 'a', 'a','a', 'a', 'a', 'a','a', 'a', 'a', 'a'};
void setup() {
SerialUSB.begin(9600);
}
void loop() {
SerialUSB.write(adcData);
}
Related
Originally, I am using putty to display the output of my embedded device. However, I want to manipulate the data as I receive it, thus I used python to connect to the port(w timeout 0). All the C code does is print the buffer (lets say file length of size = 22112 bytes) as following:
for (x = 0; x < size; x++) {
printf("%x", buffer[x]);
}
printf("\r\n");
The python code just reads the data as a line since I end the C code with a new line after the loop:
ser_bytes = ser.readline()
print(len(ser_bytes))
But python prints the length to be 34742 when it should be 22112. As you may have noticed, I print the values in hex, other formats give other values.
So what is the correct way to receive the data to get the correct size?
And is there any other way to do this?
UPDATE:
Currently I am using:
write(1, fileBuffer, size);
To write the raw data, but Python STILL does not receive all the data. Out of 22112, only around 8000 is received. Printing to Putty shows the correct number of data though.
For some visuals, this is the end of the file (around 8000 bytes) which is stored by python when the data is received over serial.
This is the original file itself (directly from the SD card). I highlighted where the data stops exactly.
UPDATE:
The problem seems to be a buffer issue. Since I am running on windows, I used the following to increase the buffer size:
ser.set_buffer_size(rx_size = 25000, tx_size = 25000)
ser.inWaiting(), shows a total of 22208 in the buffer when it should be 22112. I read it and store it anyways using ser.read(ser.inWaiting()). To see where the extra bytes are coming from, I print two lines from the original file and from the file after it is received over serial:
There seems to be an 'extra' carriage return in the file over serial. Where did it come from?
I am trying to create a system a part of which requires a microphone to be connected to an arduino. I haven't worked with microphones a lot.
I have connected a microphone (Adafruit Electret Microphone Amplifier - MAX9814 with Auto Gain Control ) to an arduino nano. I want to record audio data from this.
void setup() {
Serial.begin(9600);
pinMode(A2, INPUT);
}
void loop() {
if(Serial.available())
{
Serial.println(analogRead(A2));
}
}
I send the data to the computer and record it using a python script and converted it into a WAV file to make sure that the microphone is working properly. I tried multiple things, using the ADC value, scaling the ADC value between -1 and 1, converting into voltage and then scaling it, but nothing seems to work. When I play it back all I can hear is static with a few clicks where the voice should be.
Below is the python code i wrote for the configuration where I am sending the ADC value using println. Here I collect the data using pyserial library and convert it into a float. Then I normalize it between -1 and 1. Then I save it in a wav file.
import serial
import matplotlib.pyplot as plt
import sounddevice as sd
import numpy as np
from scipy.io.wavfile import write
import pyaudio
import wave
def audnorm(aud):
normaud= -1+2*((aud-np.amin(aud))/(np.amax(aud)-np.amin(aud)))
return normaud
ser = serial.Serial('/dev/ttyACM0',115200)
ser.flushInput()
sound=[]
sound2=[]
while True:
try:
ser_bytes = ser.readline()
ser_bytes2= float(ser_bytes)
sound.append(ser_bytes2)
sound2.append(ser_bytes)
print(ser_bytes+"\t"+str(ser_bytes2))
print(type(ser_bytes))
except:
print("Keyboard Interrupt")
break
print(str(len(sound)))
soundnp= np.asarray(sound)
soundnp= soundnp - np.mean(soundnp)
soundnorm= audnorm(soundnp)
soundnormstr= [str(x) for x in soundnorm]
plt.plot(soundnp)
plt.show()
plt.plot(soundnorm)
plt.show()
wf = wave.open("output.wav", 'wb')
wf.setnchannels(1)
wf.setsampwidth(2)
wf.setframerate(10000)
wf.writeframes(b''.join(soundnormstr))
wf.close()
I have attached 2 images of the data I recorded using this code.
What am I doing wrong?
Raw Data
Normalized Data
To be recorded without distortion, the signal you're trying to record - I assume it's an audio signal - requires three things: 1) sampling at a uniform rate, 2) sampling at more than 8,000 samples per second to be able to barely understand a voice, and 3) transmitting or storing the data as fast as you acquire it.
re: 1 & 2) There is an instructable that goes into all the messy details of recording high-fidelity audio on the Arduino. It contains far more information than I could write here. See https://www.instructables.com/id/Arduino-Audio-Input/
If your application requires the Arduino to simply detect that there is a sound - such as a clapping pair of hands - you can get by with a lower and non-uniform sample rate. Search for "Arduino Clapper" to get some ideas.
I agree with Bradford. You will need to make a uniform sampling to acquire the audio signal and 8000 Hz is a minimum.
I think that you need to set a higher serial baud rate to achieve this sampling frequency. I have slightly modified your code to measure with an oscilloscope the "actual maximum frequency" of the serial "transmission" (plus the analogWrite).
void setup() {
// Serial.begin(9600);
Serial.begin(115200);
pinMode(A2, INPUT);
}
void loop() {
//if(Serial.available())
{
int val = analogRead(A2);
Serial.write(0);
}
}
On the oscilloscope, it is roughly 9 kHz frequency, sending simply zeros on the serial wire. see the attached figure. It might be doable (for speech, not for music).
This problem is quite complicated. Seemingly I have simple two-way communication beetwen 2 devices where reading side is Raspberry pi 3. I'm trying to transfer file. I'm sending it part by part (5kb part). It looks like:
1) Sending side send first 5kb part (exactly 5136 bytes where first 16 bytes are ID/SIZE/ADD_INFO/CRC)
2) RPi3 read 5136 bytes, and calculate CRC for chunk.
3) RPi3 compare CRC received from sending side and calculated by RPi
4a) If CRC doesn't match I switch lanes from RX to TX line using Dual Bus Buffer Gate With 3-State Outputs and set High State at TX line (I keep it for 30ms).
4b) If CRC match I just wait for next chunk of file
5) Sending side switch lanes too and read my TX state if state is HIGH/1 (when CRC doesn't match) it sends same chunk (retransmission) if state is LOW/0 sends another chunk. (changing state take 10 ms)
On osciloscope it looks like this (4a):
and this (4b):
Beetwen 5136 chunk there is free time to calculate CRC then we have on RPi3 side changing state if we have to because CRC doesn't match (red lines at 4a) and reading TX side from Sending side.
Ok now some simplified code from RPi:
def recv():
while True:
#Set line to low because I want to read data
GPIO.output(16, GPIO.LOW)
ID = ser.read(4)
SIZE = ser.read(4)
ADD_INFO = ser.read(4)
CRC = ser.read(4)
#get crc from sending side
crc_chunk = int.from_bytes(CRC, byteorder='little')
data = ser.read(5120)
#calculating CRC from chunk
rpiCRC = crcSTM(data)
while True:
<--- here I wait about 20ms to calculate CRC
if rpiCRC != crc_chunk:
#Crc doesn't match I want retransmission
GPIO.output(16, GPIO.HIGH)
<--- keep it HIGH for about 30ms
break
else:
#keep it low because crc match and we are waiting for next chunk
GPIO.output(16, GPIO.HIGH)
break
All this looks legit but to the point. I always get only first chunk after that RPi just stop reading bytes. It only happen in case '4b' I get first chunk properly and I'm waiting for next one and then RPi just stop reading bytes or give me just some scratches from time to time. I test what If i get first chunk properly and set retransmission thing but everything looks great and I was getting all the time retransmission of first chunk and get it all the time. I came to this that changing line on sending side when I have LOW/1 state affects on RPi and it just stop reading bytes properly. Don't know why it's messing it and don't know how to fight it. I tryied flushing buffer, closing port before sending side chaning line and open it after it changes line again but all this just do nothing what can I do more ?
P.S This waiting things i do in my own timer but there is no need to put here code like code from sending side because problem is on RPi side.
P.S sorry for chaotic language but I was trying do explane it as simply as i can
Ok I fight it. I have enabled option "Would you like a login shell to be accessible over serial" in sudo raspi-congif. Don't know why this was messing everything but disabling this fix it. This is quite strange because I'm playing with raspberry and serial some time and there was no problem if RPi3 was sending something via uart or when it was just reading without changing lines etc :)
I'm currently writing a small utility in python to monitor the communications on a serial line. This is being used to debug some hardware that is connected via rs232 so being able to see exactly what's going over the line is extremely important. How do I check for parity errors using pyserial?
Specifically I'm wondering if there is a platform independent way of finding the value of the parity bit using pyserial. I'd strongly prefer to not need termios to do this as this is used on some windows machines.
I monitored the parity bit by bit banging with the GPIO4 on my Pi.
Inspiration here
My solution is outputting the parity bit in a second byte and writing all into a file:
import time
import pigpio # http://abyz.me.uk/rpi/pigpio/python.html
RXD=4 # number of GPIO pin
pi = pigpio.pi()
if not pi.connected:
exit(0)
pigpio.exceptions = False # Ignore error if already set as bit bang read.
handle = pi.file_open("/home/pi/Documents/bit_bang_output.txt",pigpio.FILE_WRITE) #assuming that the file /opt/pigpio/access (yes without extension) contains a line /home/pi/Domcuments/* w
pi.bb_serial_read_open(RXD, 9600,9) # Set baud rate and number of data bits here. Reading 9 data bits will read the parity bit.
pigpio.exceptions = True
stop = time.time() + 5.0 # recording 5.0 seconds
while time.time() < stop:
(count, data) = pi.bb_serial_read(RXD)
if count:
#print(data.hex(),end="")
pi.file_write(handle, data.hex())
pi.bb_serial_read_close(RXD)
pi.stop()
Your help is badly needed...
I'm trying to read data and print it to the python console from a load cell. My setup is as follow:
The load cell is a MD type from Eilersen connected to a load cell signal converter of type MCE2040 Seriel Communication Module also from Eilersen. The MCE2040 is connected to my PC through a USB to seriel connector like this link_http://www.usbgear.com/USB-COM-I-SI.html (I'm only allowed two links) one.
The load cell is connected to COM 1.
I have tried to run this snippet:
import serial
ser = serial.Serial(0) # open first serial port
print ser.portstr # check which port was really used
#ser.write("hello") # write a string
ser.close()
...and that prints 'COM1' to the console so I guess my connection should be okay.
My problem is that I don't know how to proceed. In the end I'd like to plot a graph of the incoming data and output a data file with time stamps, but for starters I'd like to print some load cell data to the console.
Any help will be highly appreciated. If further information is needed, please let me know.
Thx in advance.
Edit:
I have some documentation re MCE2040:
3.1 EVC Mode (without time stamp)
Specification: RS232/RS4422
Baudrate: 115200 bps
38400 bps (select with SW1.5)
Data bits: 7
Parity: Even
Stop bits: 1
Protocol: EVC protocol described below (Transmit Only)
3.1.1 EVC Protocol Format
After each sample period a new weight telegram is transmitted. The transmitted telegram has the following format:
<LF>WWWWWWWW<CR>
Each telegram contains a line feed character, a weight result and a carriage return character. The telegram contains:
<LF> Line Feed character (ASCII 0Ah).
WWWWWWWW Weight value for the loadcell. The value is an 8 byte ASCII hex number with MSB first.
<CR> Carriage Return character (ASCII 0Dh).
I was able to get some output from the following code:
import serial
ser = serial.Serial(0, baudrate=115000 ,timeout=100)
print ser.portstr
x = ser.read(50)
print x
ser.close()
print 'close'
Output:
COM1
ÆÆÆÆA0·5
ÆÆÆÆA0·6
ÆÆÆÆA0·5
ÆÆÆÆA0·±
ÆÆÆÆA0·±
close
First of all make sure it's really your com port, since COM1 is used by a lot of computers i'm not sure it's your com port.
You can use a simple wire to loop back info by connecting TX to RX at the USB to Serial converter, it will result in an echo (you will read what you write) it's a very simple way to verify that you are talking with the right com port.
Regarding how to continue:
Useful basic commands:
ser.write("command") with this command you send to the device some command.
ser.read(n) is for read n bytes from the device
ser.readline() will read line until it reached \n (new line)
Steps:
Send a command to your device.
Read all the data by some end byte (Frame Synchronization).
Parse data to structure (list or something like that..)
Plot it to graph.
Useful Links:
pyserial docs
tips for reading serial
plotly for graphs in python