We Keep Coding

Python sockets & Opencv - ConnectionResetError: [WinError 10054]

Edit: To clarify: it does compile, it just crashes almost immediately after the stream loads. It does connect properly.
So, I've been trying for a very long time to complete this project of mine. What I'm trying to do is send a video feed over sockets using cv2. It works over LAN, not over WAN. I get the following error:
"ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host"
Code for client(sending video over):
import cv2
import numpy as np
import socket
import pickle
host = "<insert public ip of recipient>"
port = 7643
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # declares s object with two parameters
s.connect((host, port)) # connects to the host & port
cap = cv2.VideoCapture(1)
while cap.isOpened(): # while camera is being used
ret, frame = cap.read() # reads each frame from webcam
if ret:
encoded = pickle.dumps(cv2.imencode(".jpg", frame)[1]) # encoding each frame, instead of sending live video it is sending pictures one by one
s.sendall(encoded)
if cv2.waitKey(1) & 0xFF == ord("q"): # wait until key was pressed once and
break
cap.release()
cv2.destroyAllWindows()
Code for recipient(receiving video):
import cv2
import socket
import pickle
host = "192.168.1.186"
port = 7643
boo = True
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # declares s object with two parameters
s.bind((host, port)) # tells my socket object to connect to this host & port "binds it to it"
s.listen(10) # tells the socket how much data it will be receiving.
conn, addr = s.accept()
while boo:
try:
pictures = conn.recv(256000) # creates a pictures variable that receives the pictures with a max amount of 128000 data it can receive
decoded = pickle.loads(pictures) # decodes the pictures
frame = cv2.imdecode(decoded, cv2.IMREAD_COLOR) # translates decoded into frames that we can see!
cv2.imshow("unique", frame)
if cv2.waitKey(1) & 0xFF == ord("q"): # wait until q key was pressed once and
break
except:
print("Something is broken...")
boo = False
cv2.destroyAllWindows()
s.close()

You apparently got lucky when running this over your LAN. Your code is not correctly sending a stream of images from sender to recipient, because stream sockets like TCP are a little more complicated to use by their nature. The main issue is that your sender is not communicating where each image ends and the next begins, and your recipient similarly is not organizing the data it reads into individual full images.
That is to say, socket.sendall() does not communicate the end of its data to the recipient; you need to include that information in the actual data that you send.
Error handling
But before fixing that, you should fix your error handling on the recipient so that you get more useful error messages. When you write
except:
print("Something is broken...")
You're throwing away something that would have helped you more, like "EOFError: Ran out of input" or "_pickle.UnpicklingError". Don't throw that information away. Instead, print it:
except:
traceback.print_exc()
or re-raise it:
except Exception as err:
# do whatever you want to do first
raise err
or, since you want to let it crash your program, and just want to do cleanup first, do your cleanup in a finally clause, no need for except:
try:
# your code
finally:
# the cleanup
Stream sockets and the sender
Back to your socket code, you're using stream sockets. They send a stream of bytes, and while you can count on them arriving in the correct order, you can't count on when they'll arrive. If you send b"something" and then b"something else", you could receive b"somethingsomething else" all at once, b"somet" and then later b"hing", etc. Your receiver needs to know where the dividing line is between each message, so step one is making there be dividing lines between the messages. There are a few ways to do this:
Making all messages be the same size. Since you're encoding them as JPEGs which can have different sizes based on how it's compressed, that would be a little complicated and maybe not what you want anyway.
Sending an actual marker in bytes, like a newline b"\n" or b"\n\r". This is more complicated to make work for your situation.
Sending the size of each message before you send it. This should be the easiest for your case.
Of course if you're now sending the size of the message, that's just like another message, and your recipient needs to know where this size message ends. Once again you could end the size message with a newline:
s.sendall("{}\n".format(len(encoded)).encode("ascii"))
Or you could pack it into a fixed-length number of bytes, for example 4:
s.sendall(struct.pack("!i", len(encoded)))
The receiver
Your receiver code now needs to read full messages, despite the fact that socket.recv() can return partial messages, or parts of multiple messages together. You can keep a buffer of the incoming data. Add to the end, and then remove full messages from the front:
buf = ''
while boo:
new_data = s.recv(4096)
if not new_data:
# exit, because the socket has been closed
buf += new_data
# if there's a full message at the beginning of buf:
# remove that message, but leave the rest in buf
# process that message
# else:
# nothing, just go back to receiving more
Of course, to find your full message, first you need to get the full size message. If you encoded all your size messages as 4 bytes with struct.pack, just receive data until buf is 4 or more bytes long, then split it into the size and any remaining data:
message_size = struct.unpack("!i", buf[:4])[0]
buf = buf[4:]
Then do the same thing with the image message. Receive data until you have at least message_size bytes of data, split your buffer into the first image message, which you can decode and display, and keep the remainder in the buffer.
Security
The documentation for pickle says:
Warning: The pickle module is not secure. Only unpickle data you trust.
It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.
In your case, someone else could in theory connect to your IP on your chosen port and send whatever they wanted to your recipient. If this is just a toy project that wouldn't be left running all the time, the odds are low.

Handshake frames manipulation

I intend to create a fake AP in order to send the frames received from the client to the real AP and,
of course, the frames received from the real AP to the client... all passing through the fake AP.
I will use hostapd to create the fake AP and scapy to sniff and manipulate the packets (changing only source and destination).
Basically, using python, this is what I'm going to do:
Client->FakeAP->RealAP
1) sniff the packet
2) change source and destination (src=FakeAP, dst=RealAP)
3) send the packet to the Real AP
RealAP->FakeAP->Client
1) sniff the packet
2) change source and destination (src=FakeAP, dst=Client)
3) send the packet to the Client
Here's the code:
from scapy.all import *
def manipulateAndSend(pkt):
pkt[Ether].src= new_src_mac
pkt[Ether].dst= new_dst_mac
pkt[IP].src= new_src_ip
pkt[IP].dst= new_dst_ip
send(pkt)
sniff(iface="wlan0", prn=manipulateAndSend, filter="tcp", store=0)
My two questions are:
1) In hostapd I'm going to set a "whatever" WPA key, so how can I avoid to reject the authentication?
2) How can I edit the IP src and dst of the packet if the IP association starts after the authentication is complete?

scapy sniff and decode diameter

I am trying to do scapy/python sniffer for Diameter messages and parse Diameter part to get AVP's from Raw.load.
After some fails I get back to basic python/scapy script like this:
from scapy.all import *
def pkt_diam(pkt):
raw = pkt.getlayer(Raw).load
print raw
# pkt.show()
sniff(iface="eth0", filter="port 3868", store=0, prn=pkt_diam)
By printing raw.load I have received just some AVP's but very unreadable. If I use
pkt.show()
I receive whole packet, Ethernet, IP, TCP and Raw part but Raw.load is almost unusable.
###[ Raw ]###
load = '\x01\x00\x00\xec#\x00\x01/\x01\x00\x00\x00\x07K\x12\xca\x07K\x12\xca\x00\x00\x01\x07#\x00\x00 00000001;000001;61de2650\x00\x00\x01\x04#\x00\x00 \x00\x00\x01\n#\x00\x00\x0c\x00\x00(\xaf\x00\x00\x01\x02#\x00\x00\x0c\x01\x00\x00\x00\x00\x00\x01\x15#\x00\x00\x0c\x00\x00\x00\x01\x00\x00\x01\x08#\x00\x00\x1dtest.a-server.org\x00\x00\x00\x00\x00\x01(#\x00\x00\x14a-server.org\x00\x00\x01)#\x00\x00 \x00\x00\x01\n#\x00\x00\x0c\x00\x00(\xaf\x00\x00\x01*#\x00\x00\x0c\x00\x00\x13\x89\x00\x00\x02t\x80\x00\x008\x00\x00(\xaf\x00\x00\x01\n#\x00\x00\x0c\x00\x00(\xaf\x00\x00\x02u\x80\x00\x00\x10\x00\x00(\xaf\x00\x00\x00\x01\x00\x00\x02v\x80\x00\x00\x10\x00\x00(\xaf\x00\x00\x00\x05'
I need some help to parse and decode Diameter Raw.load message.
Thx in advance

The best way to do it is to define the Diameter header yourself, following the link that I just gave you which is the section of the main Scapy documentation that details the step-by-step guide on how to build your own protocol type (header).
Once you have the Diameter() header defined correctly, dissecting the Diameter packets will become a breeze.
The wikipedia page on the Diameter protocol seems to be a very good reference regarding the Diameter packet header.

As part of the current Scapy pull requests https://bitbucket.org/secdev/scapy/pull-requests/ , number #109 provides support for the Diameter layer (parsing and generation).
Download the latest Scapy sources and the diameter.py file which should be placed in the 'contribution' directory (this file will not fully work with the current 2.3.1 Scapy version)

scapy is very useful.
from scapy.all import *
packets = rdpcap('/path/to/rx.pcap')
def generatePacket():
'''
Generate a packet.
'''
IP()/TCP()/DiamG()
def dissectPacket():
'''
dissect a packet.
'''
packet[0][DiamG]
The above shows the idea. and you can use print(repr(packet[0][DiamG])) to see result. Of course in order to check the packet is a Diameter packet, you might want to check at first like:
x = packet[0]
while x.payload:
x = x.payload
if x.name == 'Diameter' # it has diameter message.
# dissect it like above.
And how to ensemble and send a Diameter packet, one can check:
building diameter message

How to send raw Frame using python

I have a raw ethernet Frame that i want to send
How should i do that? i tried to send hex values of a frame but i still cant control packet header that contains src/dst address and ports
import socket
# the public network interface
HOST = socket.gethostbyname(socket.gethostname())
addr = ('46.165.204.237', 10000)
# create a raw socket and bind it to the public interface
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
s.bind((HOST, 0))
netpacket = '\xDE\xB0\x7B\xE5\xA7\xCD\x4C\x17\xEB\x07\x0D\xBC\x08\x00\x45\x00\x00\x92\x68\x94\x40\x00\x78\x06\xDC\x94\x2E\xA5\xCC\xED\xC0\xA8\x01\x02\x27\x10\x07\xC8\x04\xD7\xEA\xEA\xC3\x2A\x4E\xA2\x50\x18\x01\x02\x39\xB0\x00\x00\x3C\x3F\x78\x6D\x6C\x20\x76\x65\x72\x73\x69\x6F\x6E\x3D\x22\x31\x2E\x30\x22\x3F\x3E\x3C\x50\x61\x63\x6B\x65\x74\x3E\x3C\x6F\x70\x65\x72\x61\x74\x69\x6F\x6E\x3E\x33\x3C\x2F\x6F\x70\x65\x72\x61\x74\x69\x6F\x6E\x3E\x3C\x64\x61\x74\x61\x3E\x33\x24\x30\x24\x30\x24\x30\x24\x30\x24\x30\x3C\x2F\x64\x61\x74\x61\x3E\x3C\x65\x78\x74\x64\x61\x74\x61\x3E\x3C\x2F\x65\x78\x74\x64\x61\x74\x61\x3E\x3C\x2F\x50\x61\x63\x6B\x65\x74\x3E'
#netpaket = netpacket.encode('UTF-8')
s.sendto(netpacket.encode('UTF-8'), addr)
Is there in Python a function like sendRaw() or sendRawFrame()?
I know scapy can handle this, but i need to do that many many times, and every time with various payload data. How scapy can be automated? I mean python script that launch scapy creates packet with some payload and send it.
scapy.py
packet1 = IP(dst='46.165.204.237')/TCP(sport=1992, dport=10000)/'<?xml version="1.0"?><Packet><operation>99</operation><data><![CDATA[8 fast]]></data><extdata><![CDATA[]]></extdata></Packet>.'
send(packet1)
The goal for it is to send packet from a port that already in use. If there a better solution for that problem?
Offtopic: maybe someone knows how to send packets through the Open socket id in windows (not only in python)?

You can provide Scapy with raw input by using the Raw layer.
netpacket = Raw('\xDE\xB0...')
To send packets at the ethernet layer - see the documentation for sendp.
sendp(netpacket, iface="eth1")

Read data from load cell

Your help is badly needed...
I'm trying to read data and print it to the python console from a load cell. My setup is as follow:
The load cell is a MD type from Eilersen connected to a load cell signal converter of type MCE2040 Seriel Communication Module also from Eilersen. The MCE2040 is connected to my PC through a USB to seriel connector like this link_http://www.usbgear.com/USB-COM-I-SI.html (I'm only allowed two links) one.
The load cell is connected to COM 1.
I have tried to run this snippet:
import serial
ser = serial.Serial(0) # open first serial port
print ser.portstr # check which port was really used
#ser.write("hello") # write a string
ser.close()
...and that prints 'COM1' to the console so I guess my connection should be okay.
My problem is that I don't know how to proceed. In the end I'd like to plot a graph of the incoming data and output a data file with time stamps, but for starters I'd like to print some load cell data to the console.
Any help will be highly appreciated. If further information is needed, please let me know.
Thx in advance.
Edit:
I have some documentation re MCE2040:
3.1 EVC Mode (without time stamp)
Specification: RS232/RS4422
Baudrate: 115200 bps
38400 bps (select with SW1.5)
Data bits: 7
Parity: Even
Stop bits: 1
Protocol: EVC protocol described below (Transmit Only)
3.1.1 EVC Protocol Format
After each sample period a new weight telegram is transmitted. The transmitted telegram has the following format:
<LF>WWWWWWWW<CR>
Each telegram contains a line feed character, a weight result and a carriage return character. The telegram contains:
<LF> Line Feed character (ASCII 0Ah).
WWWWWWWW Weight value for the loadcell. The value is an 8 byte ASCII hex number with MSB first.
<CR> Carriage Return character (ASCII 0Dh).
I was able to get some output from the following code:
import serial
ser = serial.Serial(0, baudrate=115000 ,timeout=100)
print ser.portstr
x = ser.read(50)
print x
ser.close()
print 'close'
Output:
COM1
ÆÆÆÆA0·5
ÆÆÆÆA0·6
ÆÆÆÆA0·5
ÆÆÆÆA0·±
ÆÆÆÆA0·±
close

First of all make sure it's really your com port, since COM1 is used by a lot of computers i'm not sure it's your com port.
You can use a simple wire to loop back info by connecting TX to RX at the USB to Serial converter, it will result in an echo (you will read what you write) it's a very simple way to verify that you are talking with the right com port.
Regarding how to continue:
Useful basic commands:
ser.write("command") with this command you send to the device some command.
ser.read(n) is for read n bytes from the device
ser.readline() will read line until it reached \n (new line)
Steps:
Send a command to your device.
Read all the data by some end byte (Frame Synchronization).
Parse data to structure (list or something like that..)
Plot it to graph.
Useful Links:
pyserial docs
tips for reading serial
plotly for graphs in python