Originally, I am using putty to display the output of my embedded device. However, I want to manipulate the data as I receive it, thus I used python to connect to the port(w timeout 0). All the C code does is print the buffer (lets say file length of size = 22112 bytes) as following:
for (x = 0; x < size; x++) {
printf("%x", buffer[x]);
}
printf("\r\n");
The python code just reads the data as a line since I end the C code with a new line after the loop:
ser_bytes = ser.readline()
print(len(ser_bytes))
But python prints the length to be 34742 when it should be 22112. As you may have noticed, I print the values in hex, other formats give other values.
So what is the correct way to receive the data to get the correct size?
And is there any other way to do this?
UPDATE:
Currently I am using:
write(1, fileBuffer, size);
To write the raw data, but Python STILL does not receive all the data. Out of 22112, only around 8000 is received. Printing to Putty shows the correct number of data though.
For some visuals, this is the end of the file (around 8000 bytes) which is stored by python when the data is received over serial.
This is the original file itself (directly from the SD card). I highlighted where the data stops exactly.
UPDATE:
The problem seems to be a buffer issue. Since I am running on windows, I used the following to increase the buffer size:
ser.set_buffer_size(rx_size = 25000, tx_size = 25000)
ser.inWaiting(), shows a total of 22208 in the buffer when it should be 22112. I read it and store it anyways using ser.read(ser.inWaiting()). To see where the extra bytes are coming from, I print two lines from the original file and from the file after it is received over serial:
There seems to be an 'extra' carriage return in the file over serial. Where did it come from?
Related
I recently started using the Arduino Due for a high-speed ADC project, and I need to be able to write ~3MBit/s to a PC with the overhead of the Arduino code to communicate with that ADC. I have been doing some testing of the native arduino speed using SerialUSB (arduino_USB_slow.ino), and have got communication to work with a simple python script using pySerial.
With increasing block size I am able to increase the data rate (from 1.22MBit/s writing one byte at a time to SerialUSB to 7.58Mbit/s using 8 bytes at a time). This is great, but I'm cycle-constrained and need to do better than this. However, if I try to send more than 8 bytes at a time with the SerialUSB.write() command, I get no data transferred over the serial port at all. My pySerial input buffer stays at 0 bytes after opening the port, which I check using port.in_waiting. There are no problems compiling the code on the arduino side, no problems uploading it, and no problems with either the arduino or pyserial when trying to write 8 bytes. The problem only exists when I try to write more than 8 bytes (I've tried 9, 10, 16, 64, none of them have worked).
There's no indication in the Serial library documentation of a limitation on the input byte array size, and as I understand it if the buffer isn't large enough, the SerialUSB library on the Arduino side will just block further execution until the hardware buffer is refilled enough times to get through all the data. Other people have managed to use much larger block size without difficulty.
This does appear to be a pySerial issue, because when I open the USB device with minicom and run:
sudo cat /dev/tty.usbmodem11414
I get a bunch of binary barf that isn't present when the device isn't transmitting. This tells me the problem isn't with the Arduino (it's sending over its data just fine), but with pySerial or how I am using it. Is there a word limitation on pyserial when receiving a stream of data? This seems really bizarre. The pyserial code is dead simple:
import serial
import numpy as np
maxBytes = 1000000
byteCounter = 0
dataBuffer = np.zeros(maxBytes + 1020)
arduino = serial.Serial()
arduino.port = '/dev/cu.usbmodem14141'
arduino.open()
while byteCounter < maxBytes:
bytesAvailable = arduino.in_waiting
dataBuffer[byteCounter:byteCounter+bytesAvailable] = np.frombuffer(arduino.read(bytesAvailable), dtype='uint8')
byteCounter += bytesAvailable
print(byteCounter)
And the Arduino code is below:
char adcData[16] = {'a', 'a', 'a', 'a','a', 'a', 'a', 'a','a', 'a', 'a', 'a','a', 'a', 'a', 'a'};
void setup() {
SerialUSB.begin(9600);
}
void loop() {
SerialUSB.write(adcData);
}
This problem is quite complicated. Seemingly I have simple two-way communication beetwen 2 devices where reading side is Raspberry pi 3. I'm trying to transfer file. I'm sending it part by part (5kb part). It looks like:
1) Sending side send first 5kb part (exactly 5136 bytes where first 16 bytes are ID/SIZE/ADD_INFO/CRC)
2) RPi3 read 5136 bytes, and calculate CRC for chunk.
3) RPi3 compare CRC received from sending side and calculated by RPi
4a) If CRC doesn't match I switch lanes from RX to TX line using Dual Bus Buffer Gate With 3-State Outputs and set High State at TX line (I keep it for 30ms).
4b) If CRC match I just wait for next chunk of file
5) Sending side switch lanes too and read my TX state if state is HIGH/1 (when CRC doesn't match) it sends same chunk (retransmission) if state is LOW/0 sends another chunk. (changing state take 10 ms)
On osciloscope it looks like this (4a):
and this (4b):
Beetwen 5136 chunk there is free time to calculate CRC then we have on RPi3 side changing state if we have to because CRC doesn't match (red lines at 4a) and reading TX side from Sending side.
Ok now some simplified code from RPi:
def recv():
while True:
#Set line to low because I want to read data
GPIO.output(16, GPIO.LOW)
ID = ser.read(4)
SIZE = ser.read(4)
ADD_INFO = ser.read(4)
CRC = ser.read(4)
#get crc from sending side
crc_chunk = int.from_bytes(CRC, byteorder='little')
data = ser.read(5120)
#calculating CRC from chunk
rpiCRC = crcSTM(data)
while True:
<--- here I wait about 20ms to calculate CRC
if rpiCRC != crc_chunk:
#Crc doesn't match I want retransmission
GPIO.output(16, GPIO.HIGH)
<--- keep it HIGH for about 30ms
break
else:
#keep it low because crc match and we are waiting for next chunk
GPIO.output(16, GPIO.HIGH)
break
All this looks legit but to the point. I always get only first chunk after that RPi just stop reading bytes. It only happen in case '4b' I get first chunk properly and I'm waiting for next one and then RPi just stop reading bytes or give me just some scratches from time to time. I test what If i get first chunk properly and set retransmission thing but everything looks great and I was getting all the time retransmission of first chunk and get it all the time. I came to this that changing line on sending side when I have LOW/1 state affects on RPi and it just stop reading bytes properly. Don't know why it's messing it and don't know how to fight it. I tryied flushing buffer, closing port before sending side chaning line and open it after it changes line again but all this just do nothing what can I do more ?
P.S This waiting things i do in my own timer but there is no need to put here code like code from sending side because problem is on RPi side.
P.S sorry for chaotic language but I was trying do explane it as simply as i can
Ok I fight it. I have enabled option "Would you like a login shell to be accessible over serial" in sudo raspi-congif. Don't know why this was messing everything but disabling this fix it. This is quite strange because I'm playing with raspberry and serial some time and there was no problem if RPi3 was sending something via uart or when it was just reading without changing lines etc :)
I'm trying to read data from this barometric pressure sensor on a raspberry pi using python & i2c/smbus.
The sensor's data sheet (page 10) says it will output a digital value in the range 0-16383 (2**14). So far it seems like I have to read whole bytes, so I'm not sure how to get a 14 bit value. (I had a link to the data sheet, but SO says I need more reputation before I can add more links to posts.)
This sample uses Adafruit's I2C python library, which is basically a wrapper around SMBus.
import Adafruit_I2C
import time
# sensor returns a 14-bit reading
max_output = 2**14
# per data sheet, max_output == 1.6 bar
max_bar = 1.6
# i2c address specified in data sheet
sensor = Adafruit_I2C.Adafruit_I2C(0x78)
while True:
reading = sensor.readU16(0, little_endian=False)
# reading is sometimes, but not always, greater than 2**14
# this adjustment feels pretty hacky/wrong
while reading > max_output:
reading = reading >> 1
bar = reading / float(max_output) * max_bar
print bar
time.sleep(1)
I compare these readings to the output from my handheld GPS, which includes a barometer. I sometimes get readings which are somewhat close (1030 millibar when the GPS reads 1001 millibar), but the sensor then dips drastically (down to 930 millibar) for a few readings. I have a suspicion that this is due to how I'm reading the data, but no real evidence to back that up.
At this point, I'm not sure what to try next.
Some things I've guessed at, but would appreciate some more-informed help with:
How can I read just the 14 bits that the sensor is outputting?
What endian-ness are the returned values? Assuming big-endian produced values which seemed more sane, but I may be conflating multiple problems here.
How can I tell which register to read from? This isn't mentioned in the data-sheet anywhere. I guessed that register 0 is probably the only one.
You should be masking the output of the sensor, not shifting it. e.g. reading = reading & (max_output-1) should probably do it.
The top two bits are the status bits, so if they are set sometimes they could mean things like: normal mode or stale data indicator.
Your help is badly needed...
I'm trying to read data and print it to the python console from a load cell. My setup is as follow:
The load cell is a MD type from Eilersen connected to a load cell signal converter of type MCE2040 Seriel Communication Module also from Eilersen. The MCE2040 is connected to my PC through a USB to seriel connector like this link_http://www.usbgear.com/USB-COM-I-SI.html (I'm only allowed two links) one.
The load cell is connected to COM 1.
I have tried to run this snippet:
import serial
ser = serial.Serial(0) # open first serial port
print ser.portstr # check which port was really used
#ser.write("hello") # write a string
ser.close()
...and that prints 'COM1' to the console so I guess my connection should be okay.
My problem is that I don't know how to proceed. In the end I'd like to plot a graph of the incoming data and output a data file with time stamps, but for starters I'd like to print some load cell data to the console.
Any help will be highly appreciated. If further information is needed, please let me know.
Thx in advance.
Edit:
I have some documentation re MCE2040:
3.1 EVC Mode (without time stamp)
Specification: RS232/RS4422
Baudrate: 115200 bps
38400 bps (select with SW1.5)
Data bits: 7
Parity: Even
Stop bits: 1
Protocol: EVC protocol described below (Transmit Only)
3.1.1 EVC Protocol Format
After each sample period a new weight telegram is transmitted. The transmitted telegram has the following format:
<LF>WWWWWWWW<CR>
Each telegram contains a line feed character, a weight result and a carriage return character. The telegram contains:
<LF> Line Feed character (ASCII 0Ah).
WWWWWWWW Weight value for the loadcell. The value is an 8 byte ASCII hex number with MSB first.
<CR> Carriage Return character (ASCII 0Dh).
I was able to get some output from the following code:
import serial
ser = serial.Serial(0, baudrate=115000 ,timeout=100)
print ser.portstr
x = ser.read(50)
print x
ser.close()
print 'close'
Output:
COM1
ÆÆÆÆA0·5
ÆÆÆÆA0·6
ÆÆÆÆA0·5
ÆÆÆÆA0·±
ÆÆÆÆA0·±
close
First of all make sure it's really your com port, since COM1 is used by a lot of computers i'm not sure it's your com port.
You can use a simple wire to loop back info by connecting TX to RX at the USB to Serial converter, it will result in an echo (you will read what you write) it's a very simple way to verify that you are talking with the right com port.
Regarding how to continue:
Useful basic commands:
ser.write("command") with this command you send to the device some command.
ser.read(n) is for read n bytes from the device
ser.readline() will read line until it reached \n (new line)
Steps:
Send a command to your device.
Read all the data by some end byte (Frame Synchronization).
Parse data to structure (list or something like that..)
Plot it to graph.
Useful Links:
pyserial docs
tips for reading serial
plotly for graphs in python
I'm trying to use pywinusb to send output report to a pic18f4550. The device can receive data, and I've tested it with a C# application, which worked fine. Also, I can read data from the device with pywinusb just fine, but I have a problem trying to send data.
Here's the code I'm running:
from pywinusb import hid
filter = hid.HidDeviceFilter(vendor_id = 0x0777, product_id = 0x0077)
devices = filter.get_devices()
if devices:
device = devices[0]
print "success"
device.open()
out_report = device.find_output_reports()[0]
buffer= [0x00]*65
buffer[0]=0x0
buffer[1]=0x01
buffer[2]=0x00
buffer[3]=0x01
out_report.set_raw_data(buffer)
out_report.send()
dev.close()
It produces this error:
success
Traceback (most recent call last):
File "C:\Users\7User\Desktop\USB PIC18\out.py", line 24, in <module>
out_report.send()
File "build\bdist.win32\egg\pywinusb\hid\core.py", line 1451, in send
self.__prepare_raw_data()
File "build\bdist.win32\egg\pywinusb\hid\core.py", line 1406, in __prepare_raw_data
byref(self.__raw_data), self.__raw_report_size) )
File "build\bdist.win32\egg\pywinusb\hid\winapi.py", line 382, in __init__
raise helpers.HIDError("hidP error: %s" % self.error_message_dict[error_code])
HIDError: hidP error: data index not found
Here is my code and it works with an MSP430F chip running TI's datapipe USB stack. This is basically hid input and output endpoints that act as a custom data pipe allowing me to send 64 bytes in any format I want with the exception of the first byte being an ID number (defined by TI) decimal 63 and the second byte being the number of pertinent or useful bytes in the packet (64 byte max packet) with the first two bytes described above. It took me a while to figure this out mostly because of the lack of documentation. The few examples that come with pywinusb are hard to learn from at best. Anyways here is my code. It is working with my micro so this should help you.
filter = hid.HidDeviceFilter(vendor_id = 0x2048, product_id = 0x0302)
hid_device = filter.get_devices()
device = hid_device[0]
device.open()
print(hid_device)
target_usage = hid.get_full_usage_id(0x00, 0x3f)
device.set_raw_data_handler(sample_handler)
print(target_usage)
report = device.find_output_reports()
print(report)
print(report[0])
buffer = [0xFF]*64
buffer[0] = 63
print(buffer)
report[0].set_raw_data(buffer)
report[0].send()
One area that may be screwing you up is here:
out_report = device.find_output_reports()[0]
Try using "out_report = device.find_output_reports()" without the "[0]" at the end.
Then use
out_report[0].set_raw_data(buffer)
and finally
out_report[0].send()
Hope this helps you out.
HID is really powerful but nobody is using proper HID enumeration, HID provides a very flexible (not easy though) schema for describing the format on its reports.
For a simple device I'd recommend using a simple byte array usage to get started, this will give host applications to give context for your data items.
Anyway, raw reports here we go again...
Use starting_data = output_report.get_raw_data()[:] for any given output report, then change any 'raw' element directly.
Of course, ideally you'd have properly defined usage and you'd be able to change report items independently, without guessing bit widths and positions :-)