Parsing and converting output from a GPS Dongle - python

I bought a cheap ublox7 GPS dongle and stuck it on my raspberry pi 3. When I looked at the output and tried to stick it into a map program I got weird results. Here is some sample output from the device after parsing with a library called "pynmea2".
$GPGLL,3745.81303,N,12214.62049,W,175033.00,A,D*7C
I did some research about how to convert this output to something useful and I found a formula that involved splitting the number up and dividing it by .6.
Doing GPS Conversion – Degrees to Latitude Longitude and vice versa
so I wrote a python program to try to capture and convert all of this, and the output is off by like a mile. I am trying to figure out what I am doing wrong, how could I be this close yet still off by about one mile?
from time import sleep
import pynmea2
import serial
import re
degree_sign = u"\N{DEGREE SIGN}"
ser = serial.Serial('/dev/ttyACM0', 9600, timeout=1.0)
sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser))
while True:
line = sio.readline()
msg = pynmea2.parse(line)
msg = str(msg)
if re.search("GPGLL", msg):
raw_nums = re.findall(r'\b\d*\.\d*', msg)
lat_whole = (raw_nums[0])
lat_part1 = lat_whole[0:2]
lat_part2 = lat_whole[2:4]
lat_part2 = int(lat_part2)
lat_part2 = lat_part2 / .6
lat_part2 = int(lat_part2)
lat_part2 = str(lat_part2)
lat_part3 = lat_whole[5:9]
lat_part3 = float(lat_part3)
lat_part3 = lat_part3 / .6
lat_part3 = round(lat_part3, 0)
lat_part3 = int(lat_part3)
lat_part3 = str(lat_part3)
lon_whole = raw_nums[1]
lon_part1 = lon_whole[0:3]
lon_part1 = int(lon_part1)
lon_part1 = -lon_part1
lon_part1 = str(lon_part1)
lon_part2 = lon_whole[3:5]
lon_part2 = int(lon_part2)
lon_part2 = lon_part2 / .6
lon_part2 = str(lon_part2)
lon_part2 = lon_part2[0:2]
lon_part3 = lon_whole[6:10]
lon_part3 = float(lon_part3)
lon_part3 = lon_part3 / .6
lon_part3 = round(lon_part3, 0)
lon_part3 = int(lon_part3)
lon_part3 = str(lon_part3)
print(lat_part1 + "." + lat_part2 + lat_part3 +"," , lon_part1 + "." + lon_part2 + lon_part3)
print(lat_part1+degree_sign+lat_part2+"'"+lat_part3+"\"" + "N", lon_part1 + degree_sign+ lon_part2 + "'" + lon_part3+"\"" + "W")
sleep(1)
Here is the list that regex generated using the pynmea2 output:
['3745.81246', '12214.61512', '224329.00'] assigned to raw_nums.
Output from the script:
37.7513540, -122.2310268
37°75'13540"N -122°23'10268"W
Entering the first bit of output into google maps brings up a place near me but about a mile away, the second number doesn't work on Google maps for some reason - but it works on apple maps.
My questions:
I know there must be at least 100 better ways to write this code, do you have suggestions for getting there quicker?
Does the formula make sense? Am I applying it correctly?
Do you see a reason why this should return a result that is close but no cigar?
Do you know why the second line of output would not work as input into google maps?
What accuracy should I expect from a ublox 7 GPS dongle I got from Amazon for $12?
Thanks in advance, I really appreciate it.
Update: I looked up my address on gps coordinates conversion
and the latitude they show for my address is 3745.50084 while my gps is reporting 3745.81246. So it just seems like I am starting with bad data...

If your parsed string from the GPS device is always of the form you specified, you can simply split the string on the commas like split_msg = msg.split(","). Then your lat will be split_msg[1] and your long split_msg[3]. With indexes 2 and 4 being the heading direction.
The lat is provided as DDmm.mm and long is provided as DDDmm.mm, which you seem to have captured above. So 3745.81246 would be 37 degrees and 45.81246 minutes. You can take the decimal portion of the minutes (i.e. 0.81246) and multiply times 60 to get seconds. So you would get 37 degrees, 45 minutes, and 48.75 seconds. As a sanity check, minutes and seconds should always be less than 60 as either of them being 60 would increment the next value (e.g. 60 minutes in a degree, 60 seconds in a minute).
To convert the minutes to a decimal degree number, simply divide the minutes number by 60 (45.81246/60=0.763541) then add that to your degrees. So 3745.81246 would become 37.763541.
So within the if statement:
split_msg = msg.split(",")
lat, lat_dir, long, long_dir = split_msg[1:5]
lat_d, lat_m = float(lat[:2]), float(lat[2:])
long_d, long_m = float(long[:3]), float(long[3:])
lat_dec = lat_d + lat_m/60
long_dec = long_d + long_m/60
lat_min = math.floor(lat_m)
lat_sec = 60*(lat_m - lat_min)
long_min = math.floor(long_m)
long_sec = 60*(long_m - long_min)
print(f"{lat_dec} {lat_dir}, {long_dec} {long_dir}")
print(f"{lat_d}{degree_sign} {lat_min}' {lat_sec}\" {lat_dir}, {long_d}{degree_sign} {long_min}' {long_sec}\" {long_dir}")
I have not tested the above code, but this is the general way I would approach this problem.

Since you're using the pynmea2 library you can make use of the object properties to ease the subsequent steps.
import pynmea2
line = "$GPGLL,3745.81303,N,12214.62049,W,175033.00,A,D*7C"
nmeaobj = pynmea2.parse(line)
coord = f'{nmeaobj.latitude} {nmeaobj.longitude}'
print(coord)
# 37.763551 -122.243675
The decimal degree difference between the script output and the library is around 0.012 which is similar to the precision length (1.1132 km) cited in the Degree precision versus length table. This would explain why you are seen a discrepancy of about a mile.
abs(-122.243675 - -122.2310268)
0.012648200000000998
abs(37.7635505 - 37.7513540)
0.01219650000000172
You could use the formula cited in the previous link to convert from decimal degree to DMS components, and this would yield a valid location. But notice that the directions of the coordinate (NS/WE) were left out of the final string formation.
def dd_to_dms(coord):
d = int(coord)
abs_d = abs(coord-d)
m = int(60 * abs_d)
s = 3600 * abs_d - 60 * m
return d,m,s
lat = '''%02d°%02d'%07.4f"''' % dd_to_dms(nmeaobj.latitude)
lon = '''%02d°%02d'%07.4f"''' % dd_to_dms(nmeaobj.longitude)
print(f'{lat} {lon}')
# 37°45'48.7818" -122°14'37.2294"
The second line of output would not work as input into google maps because, as already mentioned, minutes and seconds should always be less than 60. Moreover, adding the directions of the coordinate to a negative degree could make you (depending on the algorithm used to parse the string) "walk" in the opposite direction of the desired location as a consequence of having the degree sign (or the coordinate direction NS/WE) ignored, or, in Google maps case, simply not understanding the coordinate.
37°45'48.7818" -122°14'37.2294" # works
37°45'48.7818"N -122°14'37.2294"W # don't work
37°45'48.7818"N 122°14'37.2294"W # works

Related

I want to get the output in the form of HH:MM:SS, so for example get 5 seconds to be expressed as 05 seconds? Python

Ive written this code and ive obtained the desired output however it is not in the correct format. Essentially the codewars challenge was to take a number of seconds up to around 350000 and then split it into hours then minutes then seconds. For example my code would take x seconds then express it as y:z:p (where y, z and p represent single digit integers) however i would like my code to express it as 0y:0z:0p unless y,z or p are already two digit integers.
Here is my code:
secs = 350000
mins = secs/60
hrs = mins/60
hrs_hol = int(hrs)
print(hrs_hol, hrs)
hrs_rem = hrs-hrs_hol
print(hrs_rem)
mins_from_hrs_rem = hrs_rem*60
mins_hol = int(mins_from_hrs_rem)
mins_rem = mins_from_hrs_rem - mins_hol
secs_from_min_rem = mins_rem*60
secs_final = int(secs_from_min_rem)
H = str(hrs_hol)
M = str(mins_hol)
S = str(secs_final)
print(H+':'+M+':'+S)
Thankyou!
Try this:
H = str(hrs_hol).zfill(2)
M = str(mins_hol).zfill(2)
S = str(secs_final).zfill(2)
Remove the assignments to H, M & S then:
print(f'{hrs_hol:02d}:{mins_hol:02d}:{secs_final:02d}')
Here we use an f-string and a format specifier that indicates a minimum of two digits (left-padded with zero if necessary)

How to sum strings of hours and minutes without the use of any date or time libraries

I should not use any date or time libraries! Actually I was able to sum them as normal integers but I couldn't find how to sum them as hours and minutes so as an example "03:50" + "04:20" should be "08:10" but actually I got "7:70" because I still couldn't find valid approach. Any hint or approach is much appreciated. Thank you.
import re
def add_time(time, duration):
elems1 = re.split(r"\s", time)
elems2 = re.split(r":", elems1[0])
elems2.append(elems1[1])
elems3 = re.split(r":", duration)
resH = eval(elems2[0]) + eval(elems3[0])
resM = eval(elems2[1]) + eval(elems3[1])
return f'{str(resH)}:{str(resM)} {elems2[2]}'
print(add_time("3:50 AM", "4:20"))
Actual output
7:70 AM
Desired output
08:10 AM
You can implement it by yourself by using % to keep track of hours and minutes format, but keep in mind that the remainder of the minutes, if came across 60, should passed to the hours.
take a look in this implementation:
import re
import numpy as np
def add_time(time, duration):
elems1 = re.split(r"\s", time)
elems2 = re.split(r":", elems1[0])
elems2.append(elems1[1])
elems3 = re.split(r":", duration)
remainder_from_minutes = int(np.floor((int(elems2[1]) + int(elems3[1]))/60))
resH = (int(elems2[0]) + int(elems3[0]))%24 + remainder_from_minutes
resM = (int(elems2[1]) + int(elems3[1]))%60
am_or_pm = elems2[2] if int(resH/12)%2==0 else chr(145-ord(elems2[2][0])) + "M"
return str(resH).rjust(2,"0") + ":" + str(resM).rjust(2,"0") + " " + str(am_or_pm)
print(add_time("3:50 AM", "8:50"))
print(add_time("3:20 PM", "8:50"))
output:
12:40 PM
12:10 AM
fixed AM/PM notation as well, with some ascii table tricks, in such way that also 12:40 AM and 03:50 PM will be acceptable
i think you should add just condition for handling when minutes is greater than 59 and when hour is greater than 12.
import re
def add_time(time, duration):
elems1 = re.split(r"\s", time)
elems2 = re.split(r":", elems1[0])
elems2.append(elems1[1])
elems3 = re.split(r":", duration)
cf = 0
resM = eval(elems2[1]) + eval(elems3[1])
if (resM > 60):
cf = 1
resM = resM - 60
resH = eval(elems2[0]) + eval(elems3[0]) + cf
if(resH > 12):
resH = resH - 12
return f'{str(resH)}:{str(resM)} {elems2[2]}'
I think a good way to do this is to convert everything to minutes, sum them and convert them to hours + minutes back. This is good because this way you will be able to scale your system in future, for example, make it possible to add more than just two time values (which is not possible in one of the other answers). Because you are using Python which has built-in long arithmetic I don't see any disadvantages of this solution

How to get accurate timing using microphone in python

I'm trying to make beat detection using PC microphone and then with timestamp of beat calculate distance between multiple successive beats. I have chosen python because there is plenty of material available and it's quick to develop. By searching the internet I have come up with this simple code (no advanced peak detection or anything yet, this comes later if need be):
import pyaudio
import struct
import math
import time
SHORT_NORMALIZE = (1.0/32768.0)
def get_rms(block):
# RMS amplitude is defined as the square root of the
# mean over time of the square of the amplitude.
# so we need to convert this string of bytes into
# a string of 16-bit samples...
# we will get one short out for each
# two chars in the string.
count = len(block)/2
format = "%dh" % (count)
shorts = struct.unpack(format, block)
# iterate over the block.
sum_squares = 0.0
for sample in shorts:
# sample is a signed short in +/- 32768.
# normalize it to 1.0
n = sample * SHORT_NORMALIZE
sum_squares += n*n
return math.sqrt(sum_squares / count)
CHUNK = 32
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 44100
p = pyaudio.PyAudio()
stream = p.open(format=FORMAT,
channels=CHANNELS,
rate=RATE,
input=True,
frames_per_buffer=CHUNK)
elapsed_time = 0
prev_detect_time = 0
while True:
data = stream.read(CHUNK)
amplitude = get_rms(data)
if amplitude > 0.05: # value set by observing graphed data captured from mic
elapsed_time = time.perf_counter() - prev_detect_time
if elapsed_time > 0.1: # guard against multiple spikes at beat point
print(elapsed_time)
prev_detect_time = time.perf_counter()
def close_stream():
stream.stop_stream()
stream.close()
p.terminate()
The code works pretty good in silence, and I have been pretty satisfied the first two moments I ran it, but then I tried how accurate it was and I was a little bit less satisfied. To test this I used two methods: phone with metronome set to 60bpm (emits tic toc sounds into microphone) and an Arduino hooked to a beeper, which is triggered at 1Hz rate by accurate Chronodot RTC. The beeper beeps into microphone, triggering a detection. With both methods results look similar (numbers represent distance between two beat detections in seconds):
0.9956681643835616
1.0056331689497717
0.9956100091324198
1.0058207853881278
0.9953449497716891
1.0052103013698623
1.0049350136986295
0.9859074337899543
1.004996383561644
0.9954095342465745
1.0061518904109583
0.9953025753424658
1.0051235068493156
1.0057199634703196
0.984839305936072
1.00610396347032
0.9951862648401821
1.0053146301369864
0.9960100821917806
1.0053391780821919
0.9947373881278523
1.0058608219178105
1.0056580091324214
0.9852110319634697
1.0054473059360731
0.9950465753424638
1.0058237077625556
0.995704694063928
1.0054566575342463
0.9851026118721435
1.0059882374429243
1.0052523835616398
0.9956161461187207
1.0050863926940607
0.9955758173515932
1.0058052968036577
0.9953960913242028
1.0048014611872205
1.006336876712325
0.9847434520547935
1.0059712876712297
Now I'm pretty confident that at least Arduino is accurate to 1 msec (which is targeted accuracy). The results tend to be off by +- 5msec, but now and then even 15ms, which is unacceptable. Is there a way to achieve greater accuracy or is this limitation of python / soundcard / something else? Thank you!
EDIT:
After incorporating tom10 and barny's suggestions into the code, the code looks like this:
import pyaudio
import struct
import math
import psutil
import os
def set_high_priority():
p = psutil.Process(os.getpid())
p.nice(psutil.HIGH_PRIORITY_CLASS)
SHORT_NORMALIZE = (1.0/32768.0)
def get_rms(block):
# RMS amplitude is defined as the square root of the
# mean over time of the square of the amplitude.
# so we need to convert this string of bytes into
# a string of 16-bit samples...
# we will get one short out for each
# two chars in the string.
count = len(block)/2
format = "%dh" % (count)
shorts = struct.unpack(format, block)
# iterate over the block.
sum_squares = 0.0
for sample in shorts:
# sample is a signed short in +/- 32768.
# normalize it to 1.0
n = sample * SHORT_NORMALIZE
sum_squares += n*n
return math.sqrt(sum_squares / count)
CHUNK = 4096
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 44100
RUNTIME_SECONDS = 10
set_high_priority()
p = pyaudio.PyAudio()
stream = p.open(format=FORMAT,
channels=CHANNELS,
rate=RATE,
input=True,
frames_per_buffer=CHUNK)
elapsed_time = 0
prev_detect_time = 0
TIME_PER_CHUNK = 1000 / RATE * CHUNK
SAMPLE_GROUP_SIZE = 32 # 1 sample = 2 bytes, group is closest to 1 msec elapsing
TIME_PER_GROUP = 1000 / RATE * SAMPLE_GROUP_SIZE
for i in range(0, int(RATE / CHUNK * RUNTIME_SECONDS)):
data = stream.read(CHUNK)
time_in_chunk = 0
group_index = 0
for j in range(0, len(data), (SAMPLE_GROUP_SIZE * 2)):
group = data[j:(j + (SAMPLE_GROUP_SIZE * 2))]
amplitude = get_rms(group)
amplitudes.append(amplitude)
if amplitude > 0.02:
current_time = (elapsed_time + time_in_chunk)
time_since_last_beat = current_time - prev_detect_time
if time_since_last_beat > 500:
print(time_since_last_beat)
prev_detect_time = current_time
time_in_chunk = (group_index+1) * TIME_PER_GROUP
group_index += 1
elapsed_time = (i+1) * TIME_PER_CHUNK
stream.stop_stream()
stream.close()
p.terminate()
With this code I achieved the following results (units are this time milliseconds instead of seconds):
999.909297052154
999.9092970521542
999.9092970521542
999.9092970521542
999.9092970521542
1000.6349206349205
999.9092970521551
999.9092970521524
999.9092970521542
999.909297052156
999.9092970521542
999.9092970521542
999.9092970521524
999.9092970521542
Which, if I didn't make any mistake, looks a lot better than before and has achieved sub-millisecond accuracy. I thank tom10 and barny for their help.
The reason you're not getting the right timing for the beats is that you're missing chunks of the audio data. That is, the chunks are being read by the soundcard, but you're not collecting the data before it's overwritten with the next chunk.
First, though, for this problem you need to distinguish between the ideas of timing accuracy and real-time response.
The timing accuracy of a sound card should be very good, much better than a ms, and you should be able to capture all of this accuracy in the data you read from the soundcard. The real-time responsiveness of your computer's OS should be very bad, much worse than a ms. That is, you should easily be able to identify audio events (such as beats) to within a ms, but not identify them at the time they happen (instead, 30-200ms later depending on your system). This arrangement usually works for computers because general human perception of the timing of events is much greater than a ms (except for rare specialized percepetual systems, like comparing auditory events between the two ears, etc).
The specific problem with your code is that CHUNKS is much too small for the OS to query the sound card at each sample. You have it at 32, so at 44100Hz, the OS needs to get to the sound card every 0.7ms, which is too short of a time for a computer that's tasked with doing many other things. If you OS doesn't get the chunk before the next one comes in, the original chunk is overwritten and lost.
To get this working so it's consistent with the constraints above, make CHUNKS much larger than 32, and more like 1024 (as in the PyAudio examples). Depending on your computer and what it's doing, even that my not be long enough.
If this type of approach won't work for you, you will probably need a dedicated real-time system like an Arduino. (Generally, though, this isn't necessary, so think twice before you decide that you need to use the Arduino. Usually, when I've seen people need true real-time it's when trying to do something very quantitave interactive with the human, like flash a light, have the person tap a button, flash another light, have the person tap another button, etc, to measure response times.)

Calculating remaining time of file transfer in Paramiko

Here is my code:
def send_file(self,source,destination):
self.client.open_sftp().put(source,destination,self.upload_status)
def upload_status(self,sent,size):
sent_mb=round(sent/1000000,1)
remaining_mb=round((size-sent)/1000000,1)
size=round(size/1000000,1)
sys.stdout.write("Total size:{0} MB|Sent:{1} MB|Remaining:{2} MB".
format(size,sent_mb,remaining_mb))
sys.stdout.write('\r')
I get the following output:
Total size:30.6|Sent:30.5 MB|Remaining:0.1 MB
My expected output is:
Total size:30.6|Sent:30.5 MB|Remaining:0.1 MB|Time remaining:00:00:01
Is there any module in Paramiko that can give me time stamp? If not, how can I achieve this?
Remember time, when the transfer started;
On each update, calculate how long the transfer is already taking place;
Based on that, calculate a transfer speed;
Based on that, calculate, how long will it take to transfer the rest.
import datetime
# ...
start = datetime.datetime.now()
def upload_status(self, sent, size):
sent_mb = round(float(sent) / 1000000, 1)
remaining_mb = round(float(size - sent) / 1000000, 1)
size_mb = round(size / 1000000, 1)
time = datetime.datetime.now()
elapsed = time - start
if sent > 0:
remaining_seconds = elapsed.total_seconds() * (float(size - sent) / sent)
else:
remaining_seconds = 0
remaining_hours, remaining_remainder = divmod(remaining_seconds, 3600)
remaining_minutes, remaining_seconds = divmod(remaining_remainder, 60)
print(
("Total size:{0} MB|Sent:{1} MB|Remaining:{2} MB|" +
"Time remaining:{3:02}:{4:02}:{5:02}").
format(
size_mb, sent_mb, remaining_mb,
int(remaining_hours), int(remaining_minutes), int(remaining_seconds)))
Also note that your MB calculation works in Python 3 only. In Python 2, you would be striping all digits. I've fixed that by a cast to float.

Python: Can I read two bytes in one transaction with the smbus module?

I'm trying to read the temperature and humidity using a Texas Instruments HDC1008 from Adafruit, product 2635. I'm on a rasberry pi 2, using the smbus module. According to TI's PDF, when getting a reading, the number will be sent in two bytes that you put together. I found this code that does what I'm trying to do with micropython, where they have a recv function that seems to simply sends them back a list with two bytes. The SMBus module doesn't seem to have any equivalent for what I'm trying to do. Here's some of my code.
class HDC1008:
I2C_BUS = 1
#Registers
REG_TEMP = 0
REG_HUMID = 1
REG_CONFIG = 2
#Configuration bits
CFG_RST = 1<<15
CFG_MODE_SINGLE = 0 << 12
CFG_MODE_BOTH = 1 << 12
ADDRESS = 0x40
def __init__(self, bus_num=I2C_BUS):
self.bus=smbus.SMBus(bus_num)
def readTemperature(self):
#configure the HDC1008 for one reading
config = 0
config |= self.CFG_MODE_SINGLE
self.bus.write_byte_data(self.ADDRESS, self.REG_CONFIG, config)
#tell the thing to take a reading
self.bus.write_byte(self.ADDRESS, self.REG_TEMP)
time.sleep(0.015)
#get the reading back from the thing
raw = self.bus.read_byte(self.ADDRESS)
raw = (raw<<8) + self.bus.read_byte(self.ADDRESS)
#use TI's formula to turn it into people numbers
temperature = (raw/65536.0)*165.0 - 40
#convert temp to f
temperature = temperature * (9.0/5.0) + 32
return temperature
When I'm getting the value for raw from bus.read_byte, I'm able to get the first half of the temperature bits, but the second reading is just zeros, presumably because the first transaction is over. How do I get two bytes in one transaction?
tnx a lot for sharing this code. I'm happy to get it working in Python.
I do not exactly understand the problem, I can read the Temperature and Humidity with our code (the only Python code I could find and works)
I did change it a little bit (make it a Class):
import smbus
class HDC:
#Registers
REG_TEMP = 0
REG_HUMID = 1
REG_CONFIG = 2
I2C_BUS = 2 #2 for PCDuino, 1 for PI
#Configuration bits
CFG_RST = 1<<15
CFG_MODE_SINGLE = 0 << 12
CFG_MODE_BOTH = 1 << 12
ADDRESS = 0x40
def __init__(self, bus_num=I2C_BUS):
self.bus=smbus.SMBus(bus_num)
def readTemperature(self):
#configure the HDC1008 for one reading
config = 0
config |= self.CFG_MODE_SINGLE
self.bus.write_byte_data(self.ADDRESS, self.REG_CONFIG, config)
#tell the thing to take a reading
self.bus.write_byte(self.ADDRESS, self.REG_TEMP)
time.sleep(0.015)
#get the reading back from the thing
raw = self.bus.read_byte(self.ADDRESS)
raw = (raw<<8) + self.bus.read_byte(self.ADDRESS)
#use TI's formula to turn it into people numbers
temperature = (raw/65536.0)* 165 - 40
#convert temp to farenheid
#temperature = temperature * (9.0/5.0) + 32
return temperature
def readHum(self):
#configure the HDC1008 for one reading
config = 0
config |= self.CFG_MODE_SINGLE
self.bus.write_byte_data(self.ADDRESS, self.REG_CONFIG, config)
#tell the thing to take a reading
self.bus.write_byte(self.ADDRESS, self.REG_HUMID)
time.sleep(0.015)
#get the reading back from the thing
raw = self.bus.read_byte(self.ADDRESS)
raw = (raw<<8) + self.bus.read_byte(self.ADDRESS)
hum=(raw/(2.0**16))*100
return hum
In the program:
from hdc1008 import HDC
HDC1008=HDC()
print HDC1008.readTemperature()
print HDC1008.readHum()

Categories