We Keep Coding

Loop through all serial addresses to locate hardware

I'm new to python and I'm trying to loop through all 255 addresses to locate a specific bit of hardware over RS485. The hardware is attached, and it's address is supposed to be 0x25, but this is not the case, so I need to find what its address is.
So far I've got:
def Init_Sync_4 ():
GPIO.output(18,GPIO.HIGH)
ser.write([CID,0x17,0x20,0x01,0x14,0x1c,0x04,0x00,0x00,0x00,0x00,0x00,0x32,0x00,0x01,0x00,0x00,0x00,0x32,0x00,0x02,0x00,0x00,0$
time.sleep(0.007)
GPIO.output(18,GPIO.LOW)
and
ser=serial.Serial(port='/dev/serial0',baudrate=38400,stopbits=2,timeout=1)
GPIO.setmode(GPIO.BCM)
GPIO.setup(18,GPIO.OUT)
GPIO.output(18,GPIO.LOW)
for i in range(0xff):
CID = '0x{:02x}'.format(i).encode('ascii')
print "at address %s" % CID
Init_Sync_4()
time.sleep(0.05)
Init_2()
time.sleep(0.05)
Here CID is the address being built, which Init_Sync_4() uses in it's byte array in ser.write, but I keep getting the error:
at address 0x00
Traceback (most recent call last):
File "rs485_test_2.py", line 97, in <module>
Init_4()
File "rs485_test_2.py", line 40, in Init_4
ser.write([CID,0x17,0x20,0x01,0x14,0x1c,0x04,0x00,0x00,0x00,0x00,0x00,0x32,0x00,0x01,0x00,0x00,0x00,0x32,0x00,0x02,0x00,0x00,0x00,0x00])
File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 518, in write
d = to_bytes(data)
File "/usr/lib/python2.7/dist-packages/serial/serialutil.py", line 66, in to_bytes
return bytes(bytearray(seq))
ValueError: string must be of size 1
I'm assuming that it's still being passed as a string, when it needs to be a single byte, but I'm lost at how the conversion would work. I've checked out some SO pages, that specify using .encode with or without params, but I'm either using it incorrectly, or it's not quite what I'm after. Any help is greatly appreciated! Thanks.

ser.write([CID,0x17,0x20,0x01, ...
CID is going into a list with a bunch of integers. But itself is a string.
You want to leave CID as an integer.
# CID = '0x{:02x}'.format(i).encode('ascii')
CID = i

IMAPClient - How to get subject and sender?

I'm trying to make a simple program to check and show unread messages, but I have problem while trying to get subject and sender adress.
For sender I've tried this method:
import email
m = server.fetch([a], ['RFC822'])
#a is variable with email id
msg = email.message_from_string(m[a], ['RFC822'])
print msg['from']
from email.utils import parseaddr
print parseaddr(msg['from'])
But it didn't work. I was getting this error:
Traceback (most recent call last):
File "C:/Users/ExampleUser/AppData/Local/Programs/Python/Python35-32/myprogram.py", line 20, in <module>
msg = email.message_from_string(m[a], ['RFC822'])
File "C:\Users\ExampleUser\AppData\Local\Programs\Python\Python35-32\lib\email\__init__.py", line 38, in message_from_string
return Parser(*args, **kws).parsestr(s)
File "C:\Users\ExampleUser\AppData\Local\Programs\Python\Python35-32\lib\email\parser.py", line 68, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
TypeError: initial_value must be str or None, not dict
I also used this:
print(server.fetch([a], ['BODY[HEADER.FIELDS (FROM)]']))
but the result was like:
defaultdict(<class 'dict'>, {410: {b'BODY[HEADER.FIELDS ("FROM")]': b'From: "=?utf-8?q?senderexample?=" <sender#example.com>\r\n\r\n', b'SEQ': 357}, 357: {b'SEQ': 357, b'FLAGS': (b'\\Seen',)}})
Is there a way to repair the first method, or make the result of second look like:
Sender Example <sender#example.com>
?
And I also don't know how to get email subject. But I guess it's the same as sender, but with other arguments. So the only thing I need are these arguments.

You should start by reviewing various IMAP libraries which are available for Python and use one which fits your needs. There are multiple ways of fetching the data you need in IMAP (the protocol), and by extension also in Python (and its libraries).
For example, the most straightforward way of getting the data you need in IMAP the protocol is through fetching the ENVELOPE object. You will still have to perform decoding of RFC2047 encoding of the non-ASCII data (that's that =?utf-8?q?... bit that you're seeing), but at least it would save you from parsing RFC5322 header structure with multiple decades of compatibility syntax rules.

Strange UnicodeEncodeError/AttributeError in my script

Currently I am writing a script in Python 2.7 that works fine except for after running it for a few seconds it runs into an error:
Enter Shopify website URL (without HTTP): store.highsnobiety.com
Scraping! Check log file # z:\shopify_output.txt to see output.
!!! Also make sure to clear file every hour or so !!!
Copper Bracelet - 3mm - Polished ['3723603267']
Traceback (most recent call last):
File "shopify_sitemap_scraper.py", line 38, in <module>
print(prod, variants).encode('utf-8')
AttributeError: 'NoneType' object has no attribute 'encode'
The script is to get data from a Shopify website and then print it to console. Code here:
# -*- coding: utf-8 -*-
from __future__ import print_function
from lxml.html import fromstring
import requests
import time
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
# Log file location, change "z://shopify_output.txt" to your location.
logFileLocation = "z:\shopify_output.txt"
log = open(logFileLocation, "w")
# URL of Shopify website from user input (for testing, just use store.highsnobiety.com during input)
url = 'http://' + raw_input("Enter Shopify website URL (without HTTP): ") + '/sitemap_products_1.xml'
print ('Scraping! Check log file # ' + logFileLocation + ' to see output.')
print ("!!! Also make sure to clear file every hour or so !!!")
while True :
page = requests.get(url)
tree = fromstring(page.content)
# skip first url tag with no image:title
url_tags = tree.xpath("//url[position() > 1]")
data = [(e.xpath("./image/title//text()")[0],e.xpath("./loc/text()")[0]) for e in url_tags]
for prod, url in data:
# add xml extension to url
page = requests.get(url + ".xml")
tree = fromstring(page.content)
variants = tree.xpath("//variants[#type='array']//id[#type='integer']//text()")
print(prod, variants).encode('utf-8')
The most crazy part about it is that when I take out the .encode('utf-8') it gives me a UnicodeEncodeError seen here:
Enter Shopify website URL (without HTTP): store.highsnobiety.com
Scraping! Check log file # z:\shopify_output.txt to see output.
!!! Also make sure to clear file every hour or so !!!
Copper Bracelet - 3mm - Polished ['3723603267']
Copper Bracelet - 5mm - Brushed ['3726247811']
Copper Bracelet - 7mm - Polished ['3726253635']
Highsnobiety x EARLY - Leather Pouch ['14541472963', '14541473027', '14541473091']
Traceback (most recent call last):
File "shopify_sitemap_scraper.py", line 38, in <module>
print(prod, variants)
File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xae' in position 13: character maps to <undefined>'
Any ideas? Have no idea what else to try after hours of googling.

snakecharmerb almost got it, but missed the cause of your first error. Your code
print(prod, variants).encode('utf-8')
means you print the values of the prod and variants variables, then try to run the encode() function on the output of print. Unfortunately, print() (as a function in Python 2 and always in Python 3) returns None. To fix it, use the following instead:
print(prod.encode("utf-8"), variants)

Your console has a default encoding of cp437, and cp437 is unable to represent the character u'\xae'.
>>> print (u'\xae')
®
>>> print (u'\xae'.encode('utf-8'))
b'\xc2\xae'
>>> print (u'\xae'.encode('cp437'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/encodings/cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character '\xae' in position 0: character maps to <undefined>
You can see that it's trying to convert to cp437 in the traceback:
File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
(I reproduced the problem in Python3.5, but it's the same issue in both versions of Python)

Python3 and hmac . How to handle string not being binary

I had a script in Python2 that was working great.
def _generate_signature(data):
return hmac.new('key', data, hashlib.sha256).hexdigest()
Where data was the output of json.dumps.
Now, if I try to run the same kind of code in Python 3, I get the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.4/hmac.py", line 144, in new
return HMAC(key, msg, digestmod)
File "/usr/lib/python3.4/hmac.py", line 42, in __init__
raise TypeError("key: expected bytes or bytearray, but got %r" %type(key).__name__)
TypeError: key: expected bytes or bytearray, but got 'str'
If I try something like transforming the key to bytes like so:
bytes('key')
I get
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: string argument without an encoding
I'm still struggling to understand the encodings in Python 3.

You can use bytes literal: b'key'
def _generate_signature(data):
return hmac.new(b'key', data, hashlib.sha256).hexdigest()
In addition to that, make sure data is also bytes. For example, if it is read from file, you need to use binary mode (rb) when opening the file.

Not to resurrect an old question but I did want to add something I feel is missing from this answer, to which I had trouble finding an appropriate explanation/example of anywhere else:
Aquiles Carattino was pretty close with his attempt at converting the string to bytes, but was missing the second argument, the encoding of the string to be converted to bytes.
If someone would like to convert a string to bytes through some other means than static assignment (such as reading from a config file or a DB), the following should work:
(Python 3+ only, not compatible with Python 2)
import hmac, hashlib
def _generate_signature(data):
key = 'key' # Defined as a simple string.
key_bytes= bytes(key , 'latin-1') # Commonly 'latin-1' or 'ascii'
data_bytes = bytes(data, 'latin-1') # Assumes `data` is also an ascii string.
return hmac.new(key_bytes, data_bytes , hashlib.sha256).hexdigest()
print(
_generate_signature('this is my string of data')
)

try
codecs.encode()
which can be used both in python2.7.12 and 3.5.2
import hashlib
import codecs
import hmac
a = "aaaaaaa"
b = "bbbbbbb"
hmac.new(codecs.encode(a), msg=codecs.encode(b), digestmod=hashlib.sha256).hexdigest()

for python3 this is how i solved it.
import codecs
import hmac
def _generate_signature(data):
return hmac.new(codecs.encode(key), codecs.encode(data), codecs.encode(hashlib.sha256)).hexdigest()

Receiving a Windows message in Python - UnicodeEncodeError ...?

I have a little Python 3.3 script that successfully sends (SendMessage) a WM_COPYDATA message (inspired from here , works with XYplorer):
import win32api
import win32gui
import struct
import array
def sendScript(window, message):
CopyDataStruct = "IIP"
dwData = 0x00400001 #value required by XYplorer
buffer = array.array("u", message)
cds = struct.pack(CopyDataStruct, dwData, buffer.buffer_info()[1] * 2 + 1, buffer.buffer_info()[0])
win32api.SendMessage(window, 0x004A, 0, cds) #0x004A is the WM_COPYDATA id
message = "helloworld"
sendScript(window, message) #I write manually the hwnd during debug
Now I need to write a receiver script, still in Python. The script in this answer seems to work (after correcting all the print statements in a print() form). Seems because it prints out all properties of the received message (hwnd, wparam, lparam, etc) except the content of the message.
I get an error instead, UnicodeEncodeError. More specifically,
Python WNDPROC handler failed
Traceback (most recent call last):
File "C:\Python\xxx.py", line 45, in OnCopyData
print(ctypes.wstring_at(pCDS.contents.lpData))
File "C:\Python\python-3.3.2\lib\encodings\cp850.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 10-13: character maps to <undefined>
I don't know how to fix it, also because I'm not using "fancy" characters in the message so I really can't see why I get this error. I also tried setting a different length of message in print(ctypes.wstring_at(pCDS.contents.lpData)) as well as using simply string_at, but without success (in the latter case I obtain a binary string).

ctypes.wstring (in the line print (ctypes.wstring_at(pCDS.contents.lpData))) may not be the string type that the sender sent. Try changing it to:
print (ctypes.string_at(pCDS.contents.lpData))