We Keep Coding

How to add incoming characters from UART to a string?

How do I add up incoming individual characters over UART to form a string?
For exmaple:
The characters from UART are printed in following format:
\x02
1
2
3
4
5
6
7
\X03
\x02
a
b
c
d
e
f
g
\x03
and I would like output to be something like:
1234567
abcdefg
I have tried this so far:
#!/usr/bin/env python
import time
import serial
ser = serial.Serial('/dev/ttyUSB0',38400)
txt = ""
ser.flushInput()
ser.flushOutput()
while 1:
bytesToRead = ser.inWaiting()
data_raw = ser.read(1)
while 1:
if data_raw !='\x02' or data_raw !='\x03':
txt += data_raw
elif data_raw == '\x03':
break
print txt
Any ideas on how to do that? I am getting no output using this.

First of all, you don't need to call inWaiting: read will block until data is available unless you explicitly set a read timeout. Secondly, if you do insist on using it, keep in mind that the function inWaiting has been replaced by the property in_waiting.
The string /x03 is a 4-character string containing all printable characters. The string \x03, on the other hand, contains only a single non printable character with ASCII code 3. Backslash is the escape character in Python strings. \x followed by two digits is an ASCII character code. Please use backslashes where they belong. This is the immediate reason you are not seeing output: the four character string can never appear in a one-character read.
That being out of the way, the most important thing to remember is to clear your buffer when you encounter a terminator character. Let's say you want to use the inefficient method of adding to your string in place. When you reach \x03, you should print txt and just reset it back to '' instead of breaking out of the loop. A better way might be to use bytearray, which is a mutable sequence. Keep in mind also that read returns bytes, not strings in Python 3.x. This means that you should be decoding the result if you want text: txt = txt.decode('ascii').
I would suggest a further improvement, and create an infinite generator function to split your steam into strings. You could use that generator to print the strings or do anything else you wanted with them:
def getstrings(port):
buf = bytearray()
while True:
b = port.read(1)
if b == b'\x02':
del buf[:]
elif b == b'\x03':
yield buf.decode('ascii')
else:
buf.append(b)
for item in getstring(Serial(...)):
print(item)

Here's one way that I think would help you a bit
l = []
while 1:
data_raw = ser.read(1)
if data_raw !='/x02' or data_raw !='/x03':
l.append(data_raw)
elif data_raw == '/x03':
txt = "-".join(l)
print txt
I begin by creating an empty list and each time you receive a new raw_data. you append it to the list. Once you reach your ending character, you create your string and print it.
I took the liberty of removing one of the loop here to give you a simpler approach. The code will not break by itself at the end (just add a break after the print if you want to). It will print out the result whenever you reach the ending character and then wait for a next data stream to start.
If you want to see an intermediate result, you can print out each data_raw and could add right after a print of the currently joined list.
Be sure to set a timeout value to None when you open your port. That way, you will simply wait to receive one bit, process it and then return to read one bit at a time.
ser = serial.Serial(port='/dev/ttyUSB0', baudrate=38400, timeout=None)
http://pyserial.readthedocs.io/en/latest/pyserial_api.html for mor information on that.

Interpreting a string received from a socket

I am trying to interpret a string that I have received from a socket. The first set of data is seen below:
2 -> 1
1 -> 2
2 -> 0
0 -> 2
0 -> 2
1 -> 2
2 -> 0
I am using the following code to get the numerical values:
for i in range(0,len(data)-1):
if data[i] == "-":
n1 = data[i-2]
n2 = data[i+3]
moves.append([int(n1),int(n2)])
But when a number greater than 9 appears in the data, the program only takes the second digit of that number (eg. with 10 the program would get 0). How would I get both of the digits from the code while maintaining the ability to get single digit numbers?

Well you just grab one character on each side ..
for the second value you can make it like this: data[i+3,len(data)-1]
for the first one: : data[0,i-2]

Use the split() function
numlist = data[i].split('->')
moves.append([int(numlist[0]),int(numlist[1])])

I assume each line is available as a (byte) string in a variable named line. If it's a whole bunch of lines then you can split it into individual lines with
lines = data.splitlines()
and work on each line inside a for statement:
for line in lines:
# do something with the line
If you are confident the lines will always be correctly formatted the easiest way to get the values you want uses the string split method. A full code starting from the data would then read like this.
lines = data.splitlines()
for line in lines:
first, _, second = line.split()
moves.append([int(first), int(second)])

"TypeError: byte indices must be integers or slices, not str" Converting bytes to ints

I am using a different program (ffmpeg) to grab the length of a youtube video that was downloaded in order to randomize a specific point in the video. However I am getting this error when I am trying to execute this code:
def grabTimeOfDownloadedYoutubeVideo(youtubeVideo):
process = subprocess.Popen(['/usr/local/bin/ffmpeg', '-i', youtubeVideo], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout, stderr = process.communicate()
matches = str(re.search(b"Duration:\s{1}(?P<hours>\d+?):(?P<minutes>\d+?):(?P<seconds>\d+\.\d+?),", stdout, re.DOTALL).groupdict()).encode()
print(matches)
hours = int(matches['hours'])
minutes = int(matches['minutes'])
seconds = int(matches['seconds'])
total = 0
total += 60 * 60 * hours
total += 60 * minutes
total += seconds
print(total)
The matches variable prints out to this:
b"{'minutes': b'04', 'hours': b'00', 'seconds': b'24.94'}"
So all of the output comes out with a 'b' at the beginning of it. How do I remove the 'b' and just get the number?
Full error message here:
Traceback (most recent call last):
File "bot.py", line 87, in <module>
grabTimeOfDownloadedYoutubeVideo("videos/1.mp4")
File "bot.py", line 77, in grabTimeOfDownloadedYoutubeVideo
hours = int(matches['hours'])
TypeError: byte indices must be integers or slices, not str

It seems you have a byte object there. In order to use it you can do the following**:
Decode it:
matches = matches.decode("utf-8")
Then, by using ast.literal_eval, translate the str to what it truly is, a dict:
matches = ast.literal_eval(matches)
Then you can access the contents of matches as you normally would:
int(matches['hours']) # returns 0
**Of course this simply fixes an error that really shouldn't be here in the first place as #Tim points out.

matches = str(re.search(b"Duration:\s{1}(?P<hours>\d+?):(?P<minutes>\d+?):(?P<seconds>\d+\.\d+?),", stdout, re.DOTALL).groupdict()).encode()
is weird. By converting the result of the regex match to a string, you're causing the error (because now matches['hours'] will fail).
By encoding that string to a bytes object (why?), you're complicating things even further.
matches = re.search(r"Duration:\s(?P<hours>\d+?):(?P<minutes>\d+?):(?P<seconds>\d+\.\d+?),", stdout).groupdict()
should do (although I'm uncertain about using stdout as input...)

decode error ValueError: invalid literal for int() with base 16: '\xe8' in python

I have a unicode string thus I wanna check if the character is continuation bit or starting bit so as tot count the number of unicode character through simple programme as
#!/usr/bin/env python
# -*- coding: utf-8 -*-
def arg(str):
i = 0
j = 0
print i
for test in str:
print test
value = int(test,16)
if (value & 0xc0) != 0x80:
j=j+1
print "hello"
print j
#return j
def main():
print "inside main"
new = "象形字"
charlen = len(new)
print charlen
tes = new.decode('utf-8')
declen = len(tes)
print declen
data = tes.encode('utf-8')
# print self_len
enclen = len(data)
print enclen
print data
arg(data)
if __name__ == "__main__":
main()
running the code gives the error as
象形字[Decode error - output not utf-8]
Traceback (most recent call last):
File "/Users/laxmi518/Documents/laxmi/code/C/python-c/python_unicode.py", line 69, in <module>
main()
File "/Users/laxmi518/Documents/laxmi/code/C/python-c/python_unicode.py", line 52, in main
arg(data)
File "/Users/laxmi518/Documents/laxmi/code/C/python-c/python_unicode.py", line 16, in arg
value = int(test,16)
ValueError: invalid literal for int() with base 16: '\xe8'
[Finished in 0.1s with exit code 1]

UTF-8 bytes are not hex strings. They are just bytes, and Python will display bytes outside the ASCII printable range by using the literal escape syntax. This is just a debugging display notation.
Use the ord() function to get the numerical value of a byte:
value = ord(test)
With that change, running your script in a terminal on Mac OS X (configured for UTF-8) outputs:
inside main
9
3
9
象形字
0
?
hello
?
?
?
hello
?
?
?
hello
?
?
3
The question marks are generated by the terminal; printing a single byte from a UTF-8 bytestream means you are printing incomplete UTF-8 code units, so the terminal doesn't know what to do with those and produces a placeholder character instead.
Instead of printing test directly, print the output of the repr() function:
print repr(test)
to get a \xhh hex notation for those bytes instead:
inside main
9
3
9
象形字
0
'\xe8'
hello
'\xb1'
'\xa1'
'\xe5'
hello
'\xbd'
'\xa2'
'\xe5'
hello
'\xad'
'\x97'
3

How does len function actually work for files?

The python docs say: Return the length (the number of items) of an object. The argument may be a sequence (string, tuple or list) or a mapping (dictionary).
Code:
from sys import argv
script, from_file = argv
input = open(from_file)
indata = input.read()
print "The input file is %d bytes long" % len(indata)
Contents of the file:
One two three
Upon running this simple program I get as output: The input file is 14 bytes long
Qutestion:
I don't understand, if my file has written in it only 11 characters(One two three) how can len return me 14 bytes and not just simply 11?(what's with the bytes by the way?) In the python interpreter if I type s = "One two three" and then len(s) I get 13, so I am very confused.

"One two three" is indeed 13 chars (11 letters + 2 spaces).
>>> open("file.txt", 'w').write("one two three")
>>> len(open("file.txt").read())
13
Most likely you have an extra char for the endline, which explains the 14.

One two three
one = 3 characters
two = 3 characters
three = 5 characters
and than you have two spaces. So a total of 13 characters.
when reading from file there is an extra space in your file so 14 characters.
In your python interpreter do this:
st = " "
len(st)
output = 1

I used your code and created file similar to your by content. Result of running: indeed you have extra non-printable character in your "One two three" file. It's the only explanation. Space and line break - most obvious things to look up for.