I'm having trouble receiving a UDP packet sent from an FPGA in a python program. I've checked similar questions and did the following:
Checked that Wireshark can the see UDP packet
Disabled windows firewall in PC
Used sock.bind() since it's UDP packets
Manually set the destination MAC address on Ethernet frame since FPGA does not support ARP
Set dest IP to broadcast 10.10.255.255 for testing, no packets received
Set the UDP checksum of the packet from the sender to 0x0000
Here's the python receiver code:
import socket
import sys
UDP_IP = "10.10.10.87"
UDP_PORT = 4660
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((UDP_IP, UDP_PORT))
print("Socket: "+str(sock.getsockname()))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print(data)
print(addr)
sys.stdout.flush()
When tested against another python script that sends to 10.10.10.87:4660
(from another PC in the 10.10.10 network) the receiver script works fine. I've even tried to recreate the UDP pcket byte-by-byte in the FPGA from packets that I know are received OK (Differences are source IP, port & MAC, checksums (disabled), identification).
Here's the output for both packets from Wireshark:
Wireshark UDP packet (Python UDP packet that gets received OK on the left, Xilinx FPGA packet that is not received by python on the right)
I'm not sure what else to try. Any help would be appreciated.
Apparently the IPv4 header checksum from an FPGA calculation was wrong. It can get confusing to follow since the TTL (Time to Live) changes on a router hop, and the new TTL will also change the IPv4 header, forcing a new checksum per hop until it gets to Wireshark on the receiver end. By default Wireshark has IPv4 checksum validation disabled (as can be seen in the question screenshot), the answer is easier to spot with validation on.
I set the IPv4 checksum during packet construction to zero (x0000). It gets recalculated correctly at the router, and with the correct checksum Python can receive the packet.
I also tested a direct connection (without router) from the FPGA to the host PC. The IPv4 header also gets recalculated correctly (I'm not sure where, probably the PC's NIC?)
Hope this is useful to someone with similar problems.
Related
I have an application that sends data over a TCP connection to a production server. I need to sniff the contents of that TCP connection and resend it to a debug server.
I've gotten quite close with this:
from scapy.all import *
packets = 0
def dup_pkt(pkt):
global packets
read = raw(pkt[TCP].payload)
print(str(packets))
s.sendall(read)
print("connecting")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("x.x.x.x", 12345))
print("connected")
print("sniffing")
pkts = sniff(prn=dup_pkt, filter="tcp dst port 12345 and not host x.x.x.x") # don't sniff the packets we're resending
The problem is that the packets appear to be missing data in the first two headers. I have set the debug server to save each received packet to a file, and set the application to connect directly to the debug server to compare the known good packet data with what the sniffer is sending. In the first packet, the first 1546/2079 bytes are good, but after that, each byte is zero instead of full of the correct data. In the second packet, the first 11 bytes are missing, but the rest is good.
Interestingly, after those initial two big setup packets, the remainder of the TCP packets seem to be sniffed properly - perhaps because they are usually far simpler and less than 40 bytes each.
Is there a better way to read packet data? Am I missing something? Unfortunately I don't have access to the source of the application, so I can't tell if it's doing anything special with those two big packets I'm having trouble with.
The issue with the first packet could indicate a problem in your operating system's TCP stack w.r.t. fragment reassembly.
In any case, try using another tool like tcpdump or wireshark to capture the packets. If they have the same problem, the problem lies with your operating system. If not, it could be a bug or configuration issue with scapy.
It could also mean that your IP packets are fragmented. Scapy does not automatically defragment packets, but fragments them.
you need to use the defrag function, or defragment (have a look at help(defrag) ), on the received packet list.
Maybe the packet you are checking is a fragment
I am new to Python and trying to send a byte array as a raw packet using a socket. My IP is 192.168.0.116 and the device I am sending this to is 192.168.0.64. This client device is a microcontroller based unit which is executing my code to simply sniff the Ethernet packets and check for a particular pattern. I am using UDP packets and tried the client side firmware by using 'Ostinato' in my PC to send a raw UDP packet. I am using Wireshark to monitor the network packet flow. The client seems to work fine with packets sent by Ostinato.
However when I try to send the same packet with Python (using raw packets as follows), it doesn't seem to spit out the bytes as I cant see anything on Wireshark nor the client get any data. But the return value from sendto() is correct. So it seems that Python feeds the byte array to a buffer to be sent (by OS?) but stops there.
import socket
CLNT_UDP_IP = '192.168.0.64'
CLNT_UDP_PORT = 5005
svr_sock = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_RAW)
send_data = [0x00, 0x11...]
send_data_arr = bytearray (send_data)
svr_sock.bind (('192.168.0.116',0))
bytes_send = svr_sock.sendto (send_data_arr, (CLNT_UDP_IP, CLNT_UDP_PORT))
svr_sock.close()
I have taken out the try-except blocks for clarity.
Another thing I noted is that when the socket is closing, it takes a bit of time. If I comment out the sendto statement, it exits immediately. So it seems like the socket close is trying to flush the send buffers, which failed to send the packet.
Any ideas?
Ilya is right, you should be opening a UDP socket with
socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
You're binding to port 0, which is invalid. If this is not an inbound port, you don't need to bind it. This may explain why your sendto call is blocking.
The send_data array should contain only your data, not "the full ethernet packet".
The send_data array must be under the MTU size for your network, otherwise it may be dropped silently. It varies but under 1300 should work, over 1500 almost certainly won't work.
I've been pulling my hair out over this one. I'm trying to write a SOCKS5 server in Python to tunnel UDP traffic. I bind to a port, and receive data fine. I then parse the SOCKS5 UDP header (not the typical UDP header), and forward the datagram to the requested endpoint.
All is good. I then listen for a response from the endpoint (resending if timeout), and get a response. Great!
Here is where I'm losing my mind-
I get the datagram back from the endpoint. I re-encapsulate the returned datagram according to the SOCKS5 RFC, which is the same UDP header as before, except I have now changed the destination address and port to the original caller. I use:
sock_client = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock_client.sendto(packed_datagram, (self.client_ip, self.client_port))
to send the datagram back to client. The client never receives the reply! Ever!
Looking at WireShark, it says this: Header checksum: 0x0000 [incorrect, should be 0xcd1f (may be caused by "IP checksum offload"?)]
Shouldn't the python socket implementation, with socket.DGRAM set, automatically pack my data correctly in a UDP header and calculate the appropriate checksum? Why is it being set to 0x0000? I checked the payload in hex, the checksum is indeed set wrong. What the heck is going on?
The checksum calculation is done by the drivers in the operating system. In may cases, the calculation is done by the network card itself. IIRC, Wireshark grabs local packets just before they are handed off to the network stack. It's common to checksum errors for all locally generated packets.
I was not following two pieces of the SOCKS5 spec accurately.
A UDP association terminates when the TCP connection that the UDP
ASSOCIATE request arrived on terminates.
I was terminating the TCP socket immediately after completing UDP relay handshake. TCP must stay open until the back-and-forth is finished.
When a UDP relay server receives a reply datagram from a remote
host, it MUST encapsulate that datagram using the above UDP request
header, and any authentication-method-dependent encapsulation.
I was using the client's IP and port as the values in the datagram. It needs to be the remote server's IP and port, essentially the UDP header encapsulation is a clone of what the client passed in.
With these two issues solved, the SOCKS5 server works as anticipated.
If I understand your part of code correctly you create a new socket to send the data back to the client. Thus this will be a new socket with a random source IP, e.g. from the view of the client you have the following packet flow:
client_ip:client_port -> socks5_ip:socks_port
client_ip:client_port <- socks5_ip:random_port
The client has probably a connected socket to the socks5 server and thus expects replies coming from socks5_ip:socks_port, not the random_port.
So you should not create a new socket to the client but instead reply using the existing socket where you received the data from the client.
turn off tx-checksumming using the linux command:
ethtool -K eth0 tx off
OR use this function to calculate the checksum
def checksum(data):
s = 0
n = len(data) % 2
for i in range(0, len(data)-n, 2):
s+= ord(data[i]) + (ord(data[i+1]) << 8)
if n:
s+= ord(data[i+1])
while (s >> 16):
s = (s & 0xFFFF) + (s >> 16)
s = ~s & 0xffff
return s
Where the data is the pseudo header
I am using Python to send UDP packets to an FPGA board and am fairly new to python. I am using wireshark to monitor the packets being sent and I know the the FPGA is receiving all the packets correctly. The problem is with what packets are being sent. The computer does not recognize what the FPGA is when it is connected so it leaves it as an unidentified network and assigns it an IP address (169.254.233.200). I am using this code to try and send a UDP packet that simply says "Test" to the FPGA (Sorry for the terrible formatting).
import sys
import socket
def main():
UDP_IP="169.254.233.201"
UDP_PORT=9
MESSAGE='Test'
print ("UDP target IP:", UDP_IP)
print ("UDP target port:", UDP_PORT)
print ("message:", MESSAGE)
sock = socket.socket( socket.AF_INET, socket.SOCK_DGRAM ) # UDP
sock.sendto( bytes(MESSAGE, 'UTF-8'), (bytes(UDP_IP, 'UTF-8'), UDP_PORT) )
However, instead of sending a UDP packet when this script is run, the computer broadcasts a message saying "Who has 169.254.233.201? Tell 169.254.233.200". I'm guessing this a problem caused by the unidentified network, or I am just addressing the packet incorrectly. I am trying to keep the HDL on the FPGA as simple as possible, so I would prefer if it didn't have to send any packets to the computer to verify the connection. This is why I wanted to use UDP over TCP.
Any Help would be appreciated.
Graeme
SOLVED: I noticed that the computer was sending messages to port 169.254.255.255 when trying to set up the connection. I changed UDP_IP="169.254.233.201" to UDP_IP="169.254.255.255" and the UDP packets started going through.
LESSON: I am bad at IP addresses.
For the reference/future readers, the problem was the your FPGA lacks the ARP protocol support.
The Ethernet frames cannot be sent directly to IP address, only to the MAC address. Thus, the sender computer must somehow learn IP<->MAC address mapping. Normally, the "ARP" protocol does that -- the message "Who has 169.254.233.201? Tell 169.254.233.200". is ARP request.
The proper workaround is to add ARP entry manually, for example on Linux, one would write arp -s 169.254.233.201 00:00:11:22:33:44 to tell computer the MAC address of the FPGA board. Even better way is to teach FPGA board to respond with proper ARP reply.
The address 169.254.255.255 is a "broadcast address", which goes to all computers at once (hence no need for ARP). This is OK to use for testing, but if you were to put it to large network and send a lot of traffic, then this traffic will be received by every computer on the network (i.e. it will drain battery on your ipad faster if you put it on home network)
So, the very simple code that I'm trying to use is here: http://wiki.python.org/moin/UdpCommunication
(also here):
Sending:
import socket
UDP_IP = "127.0.0.1"
UDP_PORT = 5005
MESSAGE = "Hello, World!"
print "UDP target IP:", UDP_IP
print "UDP target port:", UDP_PORT
print "message:", MESSAGE
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))
Receiving:
import socket
UDP_IP = "127.0.0.1"
UDP_PORT = 5005
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "received message:", data
The code works fine when I run both applications on my computer. I put the sending code on my laptop with:
UDP_IP="IP address for my computer"
Everything else is the same. Yet, nothing happens. What am I doing incorrectly? I used wireshark and determined that the packet is being sent and received; however, the python program isn't receiving the packet. I'm very confused.
Any help is much appreciated.
Thanks in advance.
Try binding to all local interfaces on the receiving side:
sock.bind(("", UDP_PORT)) # could also use "0.0.0.0"
Note that the behavior of operating systems is not entirely logical (nor consistent) in terms of binding when receiving UDP packets, especially for multicast traffic. This is the behavior you get:
Linux: Binding to a specific IP will filter the incoming UDP packets and only the ones targeted at this specific IP will get through the filter. This means for example that multicast UDP packets received by an interface with IP 192.168.1.100 will not be received when binding to IP 192.168.1.100. On Linux the normal bind does not bind to an interface. Use setsockopt(SO_BINDTODEVICE) for this. Binding to 0.0.0.0 (or "" on Python) will always receive all UDP packets received by the machine on all interfaces, regardless of the destination IP, so this is usually the most useful option on Linux.
Windows: Binding to a specific IP will bind to the interface belonging to this IP, pretty much like setsockopt(SO_BINDTODEVICE) does on Linux. Incoming UDP packets are not filtered by this IP, so multicast traffic can be received even when binding to a concrete IP. (This is probably the first time the Windows behavior seems more consistent to me than the Linux behavior.)
Python does not abstract these OS specific differences away for sockets (as it does in other areas). As long as you have no explicit reason not to do so I suggest to always bind to 0.0.0.0.
eventually figured out my issue and it was pretty complex and highly localized,
I had a very similar problem happen to me. I realize that you have already solved this problem, however I thought it would be good to share how I solved the issue for me.
The issue I found was with my firewall settings. I found that the packets were being blocked by Windows Firewall.
I too had used Wireshark which showed that packets were being sent and received. It is important to note that Wireshark grabs packets at a much lower level than a Python application.
By running my code locally with a listener on one port and a client on another port on the same PC, the firewall wasn't blocking the packets. When I moved to interfacing with an external machine, the firewall rules came into play blocking the incoming packets.
Changing the firewall policy fixed this issue. There are numerous ways and inherent security risks to changing the firewall to make this work so I will leave that part up to the IT professionals. :-)
Make sure that the server port is open while trying to execute recvfrom call. If the destination port from which the socket reading was down then we get this error.
I got the same error and fixed by reading this link - http://www.linuxsa.org.au/mailing-list/2001-04/668.html
So if I want to send a message AND receive a response then how would the code look? Like this?
import socket
UDP_IP = "127.0.0.1"
UDP_PORT = 5005
MESSAGE = "Hello, World!"
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "received message:", data