We Keep Coding

Python PiCamera - How do I format the coordinates for the Exif?

I just got my first Raspberry Pi product last week. I got a Raspberry Pi Zero W, and a PiCamera. I created a program (to be ran each day by crontab) that will take a photo, store the photo locally, transfer a copy of the photo to a local server, make a CSV log of the photo, and make an easily readable text daily report. I've gotten almost everything working correctly, but I have one particular issue I can't seem to figure out.
I'm using the Python PiCamera library to capture the photos everyday, and I wanted to set some static GPS data to each photo's EXIF metadata. (The GPS data needs to be static because the Pi is suction cupped to a window and wont be moving for at least a year) The issue I need help with is that I do not know how EXACTLY I need to format my GPS data for PiCamera's exif_tag property. Here are some of the formats I've tried and have not given me appropriate results:
Attempt #1 - Float
camera.exif_tags["GPS.GPSLatitude"] = 41.1027
camera.exif_tags["GPS.GPSLongitude"] = -85.1362
I get this error:
Traceback (most recent call last):
File "main.py", line 13, in <module>
camera.capture(local_path)
File "/usr/lib/python3/dist-packages/picamera/camera.py", line 1418, in capture
encoder.start(output)
File "/usr/lib/python3/dist-packages/picamera/encoders.py", line 1125, in start
self._add_exif_tag(tag, value)
File "/usr/lib/python3/dist-packages/picamera/encoders.py", line 1097, in _add_exif_tag
ct.sizeof(mmal.MMAL_PARAMETER_EXIF_T) + len(tag) + len(value) + 1)
TypeError: object of type 'float' has no len()
Attempt #2 - String
camera.exif_tags["GPS.GPSLatitude"] = "41.1027"
camera.exif_tags["GPS.GPSLongitude"] = "-85.1362"
No error is produced when running this code, however the output data from the exif after the image is captured is still inaccurate:
Latitude |0.040
Longitude |3162715.1775
Follow Up Attempts - Other Strings
# Style 1
camera.exif_tags["GPS.GPSLatitude"] = "(41) (6) (10.711)"
camera.exif_tags["GPS.GPSLongitude"] = "(85) (8) (8.968)"
# This produces
Latitude |1.00
Longitude |1.00
# Style 2
camera.exif_tags["GPS.GPSLatitude"] = "41 6 10.711"
camera.exif_tags["GPS.GPSLongitude"] = "85 8 8.968"
# This produces
Latitude |6.8
Longitude |10.6
# Style 3
camera.exif_tags["GPS.GPSLatitude"] = "41, 6, 10.711"
camera.exif_tags["GPS.GPSLongitude"] = "85, 8, 8.968"
#This produces
Latitude |6.8, 0.014, 1.000
Longitude |10.6, 0.008, 2.141837875
So, in short, can anyone help me by providing me with the means to format my strings? I am at a complete loss. The documentation for PiCamera only lists the different tags but does not explain how to format each of them.

Alright, so after several more test cases, a couple hours of research, and finally posting an issue on the PiCamera GitHub repo I finally figured it out. There is no direct documentation on this particular part of the PiCamera library as stated in my question. But someone on the repo was able to point me to the RaspiStill repo on adding GPS exif data, which is written in C. I've never completely learned C and my C++ skills are a bit rusty, but it gave me an idea. The end result came out to this:
camera.exif_tags["GPS.GPSLatitude"] = "41/1,6/1,9577/1000"
camera.exif_tags["GPS.GPSLongitude"] = "85/1,8/1,10223/1000"
When looking over the RaspiStill repo it got me thinking of how you would use the commands in the terminal to set exif data, so I looked up how to set the coordinates via RaspiStill and found this documentation which gave an example.
RaspiStill Documentation
Some examples of how to represent the data in the form of degrees, minutes, and seconds:
52 = 52 / 1 (52 units)
40.44 = 4044 / 100 (4044 hundredths)
52.97790 = 52977900 / 1000000 (52977900 millionths)
0 = 0/1
This is the source of where I found how to do the above "calculations":
Geotag - Exif Data
Hopefully this can help someone else too.

Receiving the correct length of data over serial

Originally, I am using putty to display the output of my embedded device. However, I want to manipulate the data as I receive it, thus I used python to connect to the port(w timeout 0). All the C code does is print the buffer (lets say file length of size = 22112 bytes) as following:
for (x = 0; x < size; x++) {
printf("%x", buffer[x]);
}
printf("\r\n");
The python code just reads the data as a line since I end the C code with a new line after the loop:
ser_bytes = ser.readline()
print(len(ser_bytes))
But python prints the length to be 34742 when it should be 22112. As you may have noticed, I print the values in hex, other formats give other values.
So what is the correct way to receive the data to get the correct size?
And is there any other way to do this?
UPDATE:
Currently I am using:
write(1, fileBuffer, size);
To write the raw data, but Python STILL does not receive all the data. Out of 22112, only around 8000 is received. Printing to Putty shows the correct number of data though.
For some visuals, this is the end of the file (around 8000 bytes) which is stored by python when the data is received over serial.
This is the original file itself (directly from the SD card). I highlighted where the data stops exactly.
UPDATE:
The problem seems to be a buffer issue. Since I am running on windows, I used the following to increase the buffer size:
ser.set_buffer_size(rx_size = 25000, tx_size = 25000)
ser.inWaiting(), shows a total of 22208 in the buffer when it should be 22112. I read it and store it anyways using ser.read(ser.inWaiting()). To see where the extra bytes are coming from, I print two lines from the original file and from the file after it is received over serial:
There seems to be an 'extra' carriage return in the file over serial. Where did it come from?

Stop reading bytes from serial port (Python/RPi/UART)

This problem is quite complicated. Seemingly I have simple two-way communication beetwen 2 devices where reading side is Raspberry pi 3. I'm trying to transfer file. I'm sending it part by part (5kb part). It looks like:
1) Sending side send first 5kb part (exactly 5136 bytes where first 16 bytes are ID/SIZE/ADD_INFO/CRC)
2) RPi3 read 5136 bytes, and calculate CRC for chunk.
3) RPi3 compare CRC received from sending side and calculated by RPi
4a) If CRC doesn't match I switch lanes from RX to TX line using Dual Bus Buffer Gate With 3-State Outputs and set High State at TX line (I keep it for 30ms).
4b) If CRC match I just wait for next chunk of file
5) Sending side switch lanes too and read my TX state if state is HIGH/1 (when CRC doesn't match) it sends same chunk (retransmission) if state is LOW/0 sends another chunk. (changing state take 10 ms)
On osciloscope it looks like this (4a):
and this (4b):
Beetwen 5136 chunk there is free time to calculate CRC then we have on RPi3 side changing state if we have to because CRC doesn't match (red lines at 4a) and reading TX side from Sending side.
Ok now some simplified code from RPi:
def recv():
while True:
#Set line to low because I want to read data
GPIO.output(16, GPIO.LOW)
ID = ser.read(4)
SIZE = ser.read(4)
ADD_INFO = ser.read(4)
CRC = ser.read(4)
#get crc from sending side
crc_chunk = int.from_bytes(CRC, byteorder='little')
data = ser.read(5120)
#calculating CRC from chunk
rpiCRC = crcSTM(data)
while True:
<--- here I wait about 20ms to calculate CRC
if rpiCRC != crc_chunk:
#Crc doesn't match I want retransmission
GPIO.output(16, GPIO.HIGH)
<--- keep it HIGH for about 30ms
break
else:
#keep it low because crc match and we are waiting for next chunk
GPIO.output(16, GPIO.HIGH)
break
All this looks legit but to the point. I always get only first chunk after that RPi just stop reading bytes. It only happen in case '4b' I get first chunk properly and I'm waiting for next one and then RPi just stop reading bytes or give me just some scratches from time to time. I test what If i get first chunk properly and set retransmission thing but everything looks great and I was getting all the time retransmission of first chunk and get it all the time. I came to this that changing line on sending side when I have LOW/1 state affects on RPi and it just stop reading bytes properly. Don't know why it's messing it and don't know how to fight it. I tryied flushing buffer, closing port before sending side chaning line and open it after it changes line again but all this just do nothing what can I do more ?
P.S This waiting things i do in my own timer but there is no need to put here code like code from sending side because problem is on RPi side.
P.S sorry for chaotic language but I was trying do explane it as simply as i can

Ok I fight it. I have enabled option "Would you like a login shell to be accessible over serial" in sudo raspi-congif. Don't know why this was messing everything but disabling this fix it. This is quite strange because I'm playing with raspberry and serial some time and there was no problem if RPi3 was sending something via uart or when it was just reading without changing lines etc :)

Checking for parity errors with pyserial

I'm currently writing a small utility in python to monitor the communications on a serial line. This is being used to debug some hardware that is connected via rs232 so being able to see exactly what's going over the line is extremely important. How do I check for parity errors using pyserial?
Specifically I'm wondering if there is a platform independent way of finding the value of the parity bit using pyserial. I'd strongly prefer to not need termios to do this as this is used on some windows machines.

I monitored the parity bit by bit banging with the GPIO4 on my Pi.
Inspiration here
My solution is outputting the parity bit in a second byte and writing all into a file:
import time
import pigpio # http://abyz.me.uk/rpi/pigpio/python.html
RXD=4 # number of GPIO pin
pi = pigpio.pi()
if not pi.connected:
exit(0)
pigpio.exceptions = False # Ignore error if already set as bit bang read.
handle = pi.file_open("/home/pi/Documents/bit_bang_output.txt",pigpio.FILE_WRITE) #assuming that the file /opt/pigpio/access (yes without extension) contains a line /home/pi/Domcuments/* w
pi.bb_serial_read_open(RXD, 9600,9) # Set baud rate and number of data bits here. Reading 9 data bits will read the parity bit.
pigpio.exceptions = True
stop = time.time() + 5.0 # recording 5.0 seconds
while time.time() < stop:
(count, data) = pi.bb_serial_read(RXD)
if count:
#print(data.hex(),end="")
pi.file_write(handle, data.hex())
pi.bb_serial_read_close(RXD)
pi.stop()

How to send hid data to device using python / pywinusb?

I'm trying to use pywinusb to send output report to a pic18f4550. The device can receive data, and I've tested it with a C# application, which worked fine. Also, I can read data from the device with pywinusb just fine, but I have a problem trying to send data.
Here's the code I'm running:
from pywinusb import hid
filter = hid.HidDeviceFilter(vendor_id = 0x0777, product_id = 0x0077)
devices = filter.get_devices()
if devices:
device = devices[0]
print "success"
device.open()
out_report = device.find_output_reports()[0]
buffer= [0x00]*65
buffer[0]=0x0
buffer[1]=0x01
buffer[2]=0x00
buffer[3]=0x01
out_report.set_raw_data(buffer)
out_report.send()
dev.close()
It produces this error:
success
Traceback (most recent call last):
File "C:\Users\7User\Desktop\USB PIC18\out.py", line 24, in <module>
out_report.send()
File "build\bdist.win32\egg\pywinusb\hid\core.py", line 1451, in send
self.__prepare_raw_data()
File "build\bdist.win32\egg\pywinusb\hid\core.py", line 1406, in __prepare_raw_data
byref(self.__raw_data), self.__raw_report_size) )
File "build\bdist.win32\egg\pywinusb\hid\winapi.py", line 382, in __init__
raise helpers.HIDError("hidP error: %s" % self.error_message_dict[error_code])
HIDError: hidP error: data index not found

Here is my code and it works with an MSP430F chip running TI's datapipe USB stack. This is basically hid input and output endpoints that act as a custom data pipe allowing me to send 64 bytes in any format I want with the exception of the first byte being an ID number (defined by TI) decimal 63 and the second byte being the number of pertinent or useful bytes in the packet (64 byte max packet) with the first two bytes described above. It took me a while to figure this out mostly because of the lack of documentation. The few examples that come with pywinusb are hard to learn from at best. Anyways here is my code. It is working with my micro so this should help you.
filter = hid.HidDeviceFilter(vendor_id = 0x2048, product_id = 0x0302)
hid_device = filter.get_devices()
device = hid_device[0]
device.open()
print(hid_device)
target_usage = hid.get_full_usage_id(0x00, 0x3f)
device.set_raw_data_handler(sample_handler)
print(target_usage)
report = device.find_output_reports()
print(report)
print(report[0])
buffer = [0xFF]*64
buffer[0] = 63
print(buffer)
report[0].set_raw_data(buffer)
report[0].send()
One area that may be screwing you up is here:
out_report = device.find_output_reports()[0]
Try using "out_report = device.find_output_reports()" without the "[0]" at the end.
Then use
out_report[0].set_raw_data(buffer)
and finally
out_report[0].send()
Hope this helps you out.

HID is really powerful but nobody is using proper HID enumeration, HID provides a very flexible (not easy though) schema for describing the format on its reports.
For a simple device I'd recommend using a simple byte array usage to get started, this will give host applications to give context for your data items.
Anyway, raw reports here we go again...
Use starting_data = output_report.get_raw_data()[:] for any given output report, then change any 'raw' element directly.
Of course, ideally you'd have properly defined usage and you'd be able to change report items independently, without guessing bit widths and positions :-)