We Keep Coding

Parsing and converting output from a GPS Dongle

I bought a cheap ublox7 GPS dongle and stuck it on my raspberry pi 3. When I looked at the output and tried to stick it into a map program I got weird results. Here is some sample output from the device after parsing with a library called "pynmea2".
$GPGLL,3745.81303,N,12214.62049,W,175033.00,A,D*7C
I did some research about how to convert this output to something useful and I found a formula that involved splitting the number up and dividing it by .6.
Doing GPS Conversion – Degrees to Latitude Longitude and vice versa
so I wrote a python program to try to capture and convert all of this, and the output is off by like a mile. I am trying to figure out what I am doing wrong, how could I be this close yet still off by about one mile?
from time import sleep
import pynmea2
import serial
import re
degree_sign = u"\N{DEGREE SIGN}"
ser = serial.Serial('/dev/ttyACM0', 9600, timeout=1.0)
sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser))
while True:
line = sio.readline()
msg = pynmea2.parse(line)
msg = str(msg)
if re.search("GPGLL", msg):
raw_nums = re.findall(r'\b\d*\.\d*', msg)
lat_whole = (raw_nums[0])
lat_part1 = lat_whole[0:2]
lat_part2 = lat_whole[2:4]
lat_part2 = int(lat_part2)
lat_part2 = lat_part2 / .6
lat_part2 = int(lat_part2)
lat_part2 = str(lat_part2)
lat_part3 = lat_whole[5:9]
lat_part3 = float(lat_part3)
lat_part3 = lat_part3 / .6
lat_part3 = round(lat_part3, 0)
lat_part3 = int(lat_part3)
lat_part3 = str(lat_part3)
lon_whole = raw_nums[1]
lon_part1 = lon_whole[0:3]
lon_part1 = int(lon_part1)
lon_part1 = -lon_part1
lon_part1 = str(lon_part1)
lon_part2 = lon_whole[3:5]
lon_part2 = int(lon_part2)
lon_part2 = lon_part2 / .6
lon_part2 = str(lon_part2)
lon_part2 = lon_part2[0:2]
lon_part3 = lon_whole[6:10]
lon_part3 = float(lon_part3)
lon_part3 = lon_part3 / .6
lon_part3 = round(lon_part3, 0)
lon_part3 = int(lon_part3)
lon_part3 = str(lon_part3)
print(lat_part1 + "." + lat_part2 + lat_part3 +"," , lon_part1 + "." + lon_part2 + lon_part3)
print(lat_part1+degree_sign+lat_part2+"'"+lat_part3+"\"" + "N", lon_part1 + degree_sign+ lon_part2 + "'" + lon_part3+"\"" + "W")
sleep(1)
Here is the list that regex generated using the pynmea2 output:
['3745.81246', '12214.61512', '224329.00'] assigned to raw_nums.
Output from the script:
37.7513540, -122.2310268
37°75'13540"N -122°23'10268"W
Entering the first bit of output into google maps brings up a place near me but about a mile away, the second number doesn't work on Google maps for some reason - but it works on apple maps.
My questions:
I know there must be at least 100 better ways to write this code, do you have suggestions for getting there quicker?
Does the formula make sense? Am I applying it correctly?
Do you see a reason why this should return a result that is close but no cigar?
Do you know why the second line of output would not work as input into google maps?
What accuracy should I expect from a ublox 7 GPS dongle I got from Amazon for $12?
Thanks in advance, I really appreciate it.
Update: I looked up my address on gps coordinates conversion
and the latitude they show for my address is 3745.50084 while my gps is reporting 3745.81246. So it just seems like I am starting with bad data...

If your parsed string from the GPS device is always of the form you specified, you can simply split the string on the commas like split_msg = msg.split(","). Then your lat will be split_msg[1] and your long split_msg[3]. With indexes 2 and 4 being the heading direction.
The lat is provided as DDmm.mm and long is provided as DDDmm.mm, which you seem to have captured above. So 3745.81246 would be 37 degrees and 45.81246 minutes. You can take the decimal portion of the minutes (i.e. 0.81246) and multiply times 60 to get seconds. So you would get 37 degrees, 45 minutes, and 48.75 seconds. As a sanity check, minutes and seconds should always be less than 60 as either of them being 60 would increment the next value (e.g. 60 minutes in a degree, 60 seconds in a minute).
To convert the minutes to a decimal degree number, simply divide the minutes number by 60 (45.81246/60=0.763541) then add that to your degrees. So 3745.81246 would become 37.763541.
So within the if statement:
split_msg = msg.split(",")
lat, lat_dir, long, long_dir = split_msg[1:5]
lat_d, lat_m = float(lat[:2]), float(lat[2:])
long_d, long_m = float(long[:3]), float(long[3:])
lat_dec = lat_d + lat_m/60
long_dec = long_d + long_m/60
lat_min = math.floor(lat_m)
lat_sec = 60*(lat_m - lat_min)
long_min = math.floor(long_m)
long_sec = 60*(long_m - long_min)
print(f"{lat_dec} {lat_dir}, {long_dec} {long_dir}")
print(f"{lat_d}{degree_sign} {lat_min}' {lat_sec}\" {lat_dir}, {long_d}{degree_sign} {long_min}' {long_sec}\" {long_dir}")
I have not tested the above code, but this is the general way I would approach this problem.

Since you're using the pynmea2 library you can make use of the object properties to ease the subsequent steps.
import pynmea2
line = "$GPGLL,3745.81303,N,12214.62049,W,175033.00,A,D*7C"
nmeaobj = pynmea2.parse(line)
coord = f'{nmeaobj.latitude} {nmeaobj.longitude}'
print(coord)
# 37.763551 -122.243675
The decimal degree difference between the script output and the library is around 0.012 which is similar to the precision length (1.1132 km) cited in the Degree precision versus length table. This would explain why you are seen a discrepancy of about a mile.
abs(-122.243675 - -122.2310268)
0.012648200000000998
abs(37.7635505 - 37.7513540)
0.01219650000000172
You could use the formula cited in the previous link to convert from decimal degree to DMS components, and this would yield a valid location. But notice that the directions of the coordinate (NS/WE) were left out of the final string formation.
def dd_to_dms(coord):
d = int(coord)
abs_d = abs(coord-d)
m = int(60 * abs_d)
s = 3600 * abs_d - 60 * m
return d,m,s
lat = '''%02d°%02d'%07.4f"''' % dd_to_dms(nmeaobj.latitude)
lon = '''%02d°%02d'%07.4f"''' % dd_to_dms(nmeaobj.longitude)
print(f'{lat} {lon}')
# 37°45'48.7818" -122°14'37.2294"
The second line of output would not work as input into google maps because, as already mentioned, minutes and seconds should always be less than 60. Moreover, adding the directions of the coordinate to a negative degree could make you (depending on the algorithm used to parse the string) "walk" in the opposite direction of the desired location as a consequence of having the degree sign (or the coordinate direction NS/WE) ignored, or, in Google maps case, simply not understanding the coordinate.
37°45'48.7818" -122°14'37.2294" # works
37°45'48.7818"N -122°14'37.2294"W # don't work
37°45'48.7818"N 122°14'37.2294"W # works

How do I get the least significant bits of many bytes?

I'm trying to get lsb from the line of an image,I managed to get here:
from PIL import Image
import sys
challengeImg = Image.open('image.png')
pixels = challengeImg.load()
for x in range(2944):
red = (pixels[x,310][0])
bred = format(red,"b")
#print(green)
#print(bred)
green = (pixels[x,310][1])
bgreen = format(green,"b")
#print(bgreen)
#print(green)
Well, until then I'm fine but now my problem, I managed to create the following code:
num = 10100001
n = 0
lsb = num >> n &1
print(lsb)
It works, but only with one byte, I suppose that with for I can achieve something but I am very beginner and I have not managed to make it work, how I can do to extract the lsb from each byte in the line of pixels of the red channel (or green, I guess it's the same procedure)?
It occurs to me that I could use a dictionary to group the bits in bytes (1: 10011001, 2: 01100110 ...) and then use the loop to apply the lsb code in each byte, anyway I do not know how I can do this and i dont think it's the best way (maybe it's not even valid).
I have a .png image of 2944x1912 that contains information hidden in the least significant bits, the first code that I put is the script that I am developing, and so far what it does is get the information of the pixels of the red channel in the line 310 and transform them into binary.
The second code is the code to get the lsb of a byte which I need to implement in the first code, so the second code should somehow group all the bits in 8 and select the last one for I save in a variable, resulting in (2944/8 = 368 = 368 bytes.)

The solution that came to me might not be the most optimal. I'll look for a better solution if it does not suffice, but in the meanwhile:
num = 10100001
num_string = str(num)
lsb_string = num_string[len(num_string)-1]
lsb = int(lsb_string)
print(lsb)
# output: 1

It works, thats the code;
from PIL import Image
import sys
challengeImg = Image.open('challenge.png')
pixels = challengeImg.load()
for x in range(2944):
red = (pixels[x,310][0])
bred = format(red,"b")
#print(green)
#print(bred)
green = (pixels[x,310][1])
bgreen = format(green,"b")
#print(bgreen)
#print(green)
rnum = format(red,"b")
rnum_string = str(rnum)
rlsb_string = rnum_string[len(rnum_string)-1]
rlsb = int(rlsb_string)
print(rlsb, end="")
Thanks!

Billing aliens via POS printer and image print

I am trying to create a prototype to print bitmap data for a text file to my LAN enabled epson pos printer TM-T88V.
While I have no problems to send text and text formatting instructions, I dont understand, what I have to do, to make my printer print the data of the Arecibo message.
first few lines:
00000010101010000000000
00101000001010000000100
10001000100010010110010
10101010101010100100100
00000000000000000000000
00000000000011000000000
00000000001101000000000
00000000001101000000000
00000000010101000000000
00000000011111000000000
00000000000000000000000
11000011100011000011000
10000000000000110010000
11010001100011000011010
11111011111011111011111
00000000000000000000000
00010000000000000000010
00000000000000000000000
00001000000000000000001
The message has 73 rows and 23 columns resulting in 1679 picture elements. Each of this elements is defined by either a 1 for black or a 0 as white and should be printed as a square of 8x8 (or 16x16) dots. the result would result in
(source: satsig.net)
From the printer's specifications:
While — as I said — the connecting and sending to the printer is no problem, I just dont get, what this instruction want to tell me. What would in the case of the Arecibo message be
What numbers do I have to send to the printer? Do I need to send every dot? What does nL, nH specify the number of dots of the image data in the horizontal direction as (nL + nH × 256). mean?
Here is my simple Python program I use for prototyping:
# -*- coding: utf-8 -*-
import struct
import socket
def sendInstructions(mySocket,l):
for x in l:
mySocket.send(struct.pack('h', *[x]),1)
def emphasizeOn(mySocket):
sendInstructions(mySocket,[27,33,48])
def emphasizeOff(mySocket):
sendInstructions(mySocket,[27,33,0])
def lineFeed(mySocket,number):
for i in range(number):
sendInstructions(mySocket,[0x0a,])
def paperCut(mySocket):
sendInstructions(mySocket,[29,86,0])
def sendText(mySocket,string):
mySocket.send(string.encode('UTF-8'))
def main():
mySocket = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
mySocket.connect(('192.168.1.15',9100))
lines = ["Hello,","World!"]
emphasizeOff(mySocket)
lineFeed(mySocket,2)
for l in lines:
if lines.index(l) == 0:
emphasizeOn(mySocket)
else:
emphasizeOff(mySocket)
sendText(mySocket,l)
lineFeed(mySocket,2)
lineFeed(mySocket,4)
paperCut(mySocket)
mySocket.close()
if __name__=="__main__":
main()

This command generates one horizontal strip of the image at a time. The strip is either 8 or 24 dots tall, depending on the value of m.
nL and nH are the low and high bytes of an integer that specifies the width in dots of the horizontal strip of image. That width is computed as nL + nH * 256, so if you wanted the image to be 550 dots wide, then nH=2 and nL=38.
The argument d is the bitmap data; if the image strip is 8 dots tall, then each byte represents one column in the strip. If the strip is 24 dots tall, then three bytes represent one column.
So let's say you have arecibo in a WxH numpy array of ints, 1 or 0. You would:
data = np.zeros((W, H), dtype=np.ubyte)
## (fill in data here)
## Use m=33 since this is apparently the only mode with
## square pixels and also the highest resolution
## (unless it prints too slowly for your liking)
m = 33
nH = W // 256 ## note this is integer division, but SO's
## syntax hilighting thinks it looks like a comment.
nL = W % 256
## Divide the array into sections with shape Wx24:
for n in range(data.shape[1] // 24):
## Note that if the image height is not a multiple of 24,
## you'll have to pad it with zeros somehow.
strip = data[:, n*24:(n+1)*24]
## Convert each strip into a string of bytes:
strip = strip.reshape(W, 3, 8)
bytes = (strip * (2**np.arange(8)[np.newaxis, np.newaxis, :])).sum(axis=2) # magic
byteString = bytes.astype(np.ubyte).tostring()
## Send the command to POS

How to do timecode calculation?

I got a question regarding calculating time code delta.
I read metadata from a movie file containing a timecode formated HH:MM:SS:FF
(FF = frame, 00->23 for example. So its like 00 to framerate-1)
So i get some data like 15:41:08:02 and from another refrence file i get 15:41:07:00
Now I have to calculate the timeoffset (like timedelta but just with frames).
How would i go around doing this?

framerate = 24
def timecode_to_frames(timecode):
return sum(f * int(t) for f,t in zip((3600*framerate, 60*framerate, framerate, 1), timecode.split(':')))
print timecode_to_frames('15:41:08:02') - timecode_to_frames('15:41:07:00')
# returns 26
def frames_to_timecode(frames):
return '{0:02d}:{1:02d}:{2:02d}:{3:02d}'.format(frames / (3600*framerate),
frames / (60*framerate) % 60,
frames / framerate % 60,
frames % framerate)
print frames_to_timecode(26)
# returns "00:00:01:02"

I'd just use gobal frame numbers for all computations, converting back to timecodes only for display
def tc_to_frame(hh, mm, ss, ff):
return ff + (ss + mm*60 + hh*3600) * frame_rate
def frame_to_tc(fn):
ff = fn % frame_rate
s = fn // frame_rate
return (s // 3600, s // 60 % 60, s % 60, ff)
for negative frame numbers I'd prepend a minus to the representation of the absolute value

If the timecode is SMPTE timecode, you may need to take into account drop frames. Drop-frame timecodes drop frame numbers 0 and 1 of the first second of every minute, except when the number of minutes is divisible by 10.
This page provides some history background with formulas to convert between timecodes and frame numbers.

Using the timecode module this is quite easy:
import timecode as tc
tc1 = tc.Timecode(24, "15:41:08:02")
tc2 = tc.Timecode(24, "15:41:07:00")
delta = tc1 - tc2
print(delta.frames)
gives you 26

How can I obtain the image size using a standard Python class (without using an external library)?

I am using Python 2.5. And using the standard classes from Python, I want to determine the image size of a file.
I've heard PIL (Python Image Library), but it requires installation to work.
How might I obtain an image's size without using any external library, just using Python 2.5's own modules?
Note I want to support common image formats, particularly JPG and PNG.

Here's a Python 3 script that returns a tuple containing an image height and width for .png, .gif and .jpeg without using any external libraries (i.e., what Kurt McKee referenced). It should be relatively easy to transfer it to Python 2.
import struct
import imghdr
def get_image_size(fname):
'''Determine the image type of fhandle and return its size.
from draco'''
with open(fname, 'rb') as fhandle:
head = fhandle.read(24)
if len(head) != 24:
return
if imghdr.what(fname) == 'png':
check = struct.unpack('>i', head[4:8])[0]
if check != 0x0d0a1a0a:
return
width, height = struct.unpack('>ii', head[16:24])
elif imghdr.what(fname) == 'gif':
width, height = struct.unpack('<HH', head[6:10])
elif imghdr.what(fname) == 'jpeg':
try:
fhandle.seek(0) # Read 0xff next
size = 2
ftype = 0
while not 0xc0 <= ftype <= 0xcf:
fhandle.seek(size, 1)
byte = fhandle.read(1)
while ord(byte) == 0xff:
byte = fhandle.read(1)
ftype = ord(byte)
size = struct.unpack('>H', fhandle.read(2))[0] - 2
# We are at a SOFn block
fhandle.seek(1, 1) # Skip `precision' byte.
height, width = struct.unpack('>HH', fhandle.read(4))
except Exception: #IGNORE:W0703
return
else:
return
return width, height

Kurt's answer needed to be slightly modified to work for me.
First, on Ubuntu: sudo apt-get install python-imaging
Then:
from PIL import Image
im = Image.open(filepath)
im.size # (width,height) tuple
Check out the handbook for more information.

Here's a way to get dimensions of a PNG file without needing a third-party module. From Python - verify a PNG file and get image dimensions:
import struct
def get_image_info(data):
if is_png(data):
w, h = struct.unpack('>LL', data[16:24])
width = int(w)
height = int(h)
else:
raise Exception('not a png image')
return width, height
def is_png(data):
return (data[:8] == '\211PNG\r\n\032\n'and (data[12:16] == 'IHDR'))
if __name__ == '__main__':
with open('foo.png', 'rb') as f:
data = f.read()
print is_png(data)
print get_image_info(data)
When you run this, it will return:
True
(x, y)
And another example that includes handling of JPEGs as well:
http://markasread.net/post/17551554979/get-image-size-info-using-pure-python-code

While it's possible to call open(filename, 'rb') and check through the binary image headers for the dimensions, it seems much more useful to install PIL and spend your time writing great new software! You gain greater file format support and the reliability that comes from widespread usage. From the PIL documentation, it appears that the code you would need to complete your task would be:
from PIL import Image
im = Image.open('filename.png')
print 'width: %d - height: %d' % im.size # returns (width, height) tuple
As for writing code yourself, I'm not aware of a module in the Python standard library that will do what you want. You'll have to open() the image in binary mode and start decoding it yourself. You can read about the formats at:
PNG file format documentation
Notes on the JPEG file format headers

Regarding Fred the Fantastic's answer:
Not every JPEG marker between C0-CF are SOF markers; I excluded DHT (C4), DNL (C8) and DAC (CC). Note that I haven't looked into whether it is even possible to parse any frames other than C0 and C2 in this manner. However, the other ones seem to be fairly rare (I personally haven't encountered any other than C0 and C2).
Either way, this solves the problem mentioned in comments by Malandy with Bangles.jpg (DHT erroneously parsed as SOF).
The other problem mentioned with 1431588037-WgsI3vK.jpg is due to imghdr only being able detect the APP0 (EXIF) and APP1 (JFIF) headers.
This can be fixed by adding a more lax test to imghdr (e.g. simply FFD8 or maybe FFD8FF?) or something much more complex (possibly even data validation). With a more complex approach I've only found issues with: APP14 (FFEE) (Adobe); the first marker being DQT (FFDB); and APP2 and issues with embedded ICC_PROFILEs.
Revised code below, also altered the call to imghdr.what() slightly:
import struct
import imghdr
def test_jpeg(h, f):
# SOI APP2 + ICC_PROFILE
if h[0:4] == '\xff\xd8\xff\xe2' and h[6:17] == b'ICC_PROFILE':
print "A"
return 'jpeg'
# SOI APP14 + Adobe
if h[0:4] == '\xff\xd8\xff\xee' and h[6:11] == b'Adobe':
return 'jpeg'
# SOI DQT
if h[0:4] == '\xff\xd8\xff\xdb':
return 'jpeg'
imghdr.tests.append(test_jpeg)
def get_image_size(fname):
'''Determine the image type of fhandle and return its size.
from draco'''
with open(fname, 'rb') as fhandle:
head = fhandle.read(24)
if len(head) != 24:
return
what = imghdr.what(None, head)
if what == 'png':
check = struct.unpack('>i', head[4:8])[0]
if check != 0x0d0a1a0a:
return
width, height = struct.unpack('>ii', head[16:24])
elif what == 'gif':
width, height = struct.unpack('<HH', head[6:10])
elif what == 'jpeg':
try:
fhandle.seek(0) # Read 0xff next
size = 2
ftype = 0
while not 0xc0 <= ftype <= 0xcf or ftype in (0xc4, 0xc8, 0xcc):
fhandle.seek(size, 1)
byte = fhandle.read(1)
while ord(byte) == 0xff:
byte = fhandle.read(1)
ftype = ord(byte)
size = struct.unpack('>H', fhandle.read(2))[0] - 2
# We are at a SOFn block
fhandle.seek(1, 1) # Skip `precision' byte.
height, width = struct.unpack('>HH', fhandle.read(4))
except Exception: #IGNORE:W0703
return
else:
return
return width, height
Note: Created a full answer instead of a comment, since I'm not yet allowed to.

If you happen to have ImageMagick installed, then you can use 'identify'. For example, you can call it like this:
path = "//folder/image.jpg"
dim = subprocess.Popen(["identify","-format","\"%w,%h\"",path], stdout=subprocess.PIPE).communicate()[0]
(width, height) = [ int(x) for x in re.sub('[\t\r\n"]', '', dim).split(',') ]

I found a nice solution in another Stack Overflow post (using only standard libraries + dealing with JPEG as well): JohnTESlade answer
And another solution (the quick way) for those who can afford running the 'file' command within the Python interpreter, run:
import os
info = os.popen("file foo.jpg").read()
print info
Output:
foo.jpg: JPEG image data...density 28x28, segment length 16, baseline, precision 8, 352x198, frames 3
All you have got to do now is to format the output to capture the dimensions. 352x198 in my case.

That code does accomplish two things:
Getting the image dimension
Find the real EOF of a JPEG file
Well, when googling, I was more interested in the latter one.
The task was to cut out a JPEG file from a data stream. Since I didn't find any way to use Pythons 'image' to a way to get the EOF of so a JPEG file, I made up this.
Interesting things /changes/notes in this sample:
extending the normal Python file class with the method uInt16,
making the source code better readable and maintainable.
Messing around with struct.unpack() quickly makes the code look ugly
Replaced read over'uninteresting' areas/chunk with seek
In case you just like to get the dimensions, you may remove the line:
hasChunk = ord(byte) not in range(0xD0, 0xDA) + [0x00]
-> since that only gets important when reading over the image data chunk
and comment in
#break
to stop reading as soon as the dimension were found.
...but smile what I'm telling. You're the voder ;)
import struct
import io, os
class myFile(file):
def byte(self):
return file.read(self, 1);
def uInt16(self):
tmp = file.read(self, 2)
return struct.unpack(">H", tmp)[0];
jpeg = myFile('grafx_ui.s00_\\08521678_Unknown.jpg', 'rb')
try:
height = -1
width = -1
EOI = -1
type_check = jpeg.read(2)
if type_check != b'\xff\xd8':
print("Not a JPG")
else:
byte = jpeg.byte()
while byte != b"":
while byte != b'\xff': byte = jpeg.byte()
while byte == b'\xff': byte = jpeg.byte()
# FF D8 SOI Start of Image
# FF D0..7 RST DRI Define Restart Interval inside CompressedData
# FF 00 Masked FF inside CompressedData
# FF D9 EOI End of Image
# http://en.wikipedia.org/wiki/JPEG#Syntax_and_structure
hasChunk = ord(byte) not in range(0xD0, 0xDA) + [0x00]
if hasChunk:
ChunkSize = jpeg.uInt16() - 2
ChunkOffset = jpeg.tell()
Next_ChunkOffset = ChunkOffset + ChunkSize
# Find bytes \xFF \xC0..C3. That marks the start of the frame
if (byte >= b'\xC0' and byte <= b'\xC3'):
# Found SOF1..3 data chunk - Read it and quit
jpeg.seek(1, os.SEEK_CUR)
h = jpeg.uInt16()
w = jpeg.uInt16()
#break
elif (byte == b'\xD9'):
# Found end of image
EOI = jpeg.tell()
break
else:
# Seek to the next data chunk
print "Pos: %.4x %x" % (jpeg.tell(), ChunkSize)
if hasChunk:
jpeg.seek(Next_ChunkOffset)
byte = jpeg.byte()
width = int(w)
height = int(h)
print("Width: %s, Height: %s JpgFileDataSize: %x" % (width, height, EOI))
finally:
jpeg.close()

It depends on the output of file which I am not sure is standardized on all systems. Some JPEGs don't report the image size
import subprocess, re
image_size = list(map(int, re.findall('(\d+)x(\d+)', subprocess.getoutput("file" + filename))[-1]))

I stumbled upon this one, but you can get it by using the following as long as you import NumPy.
import numpy as np
[y, x] = np.shape(img[:, :, 0])
It works because you ignore all but one color, and then the image is just 2D, so shape() tells you how big it is. I am still kind of new to Python, but it seems like a simple way to do it.