We Keep Coding

How to create a dual-stack (v4+v6) python UDP server?

By searching for 'python dual-stack', I found https://bugs.python.org/issue25667 where someone mentions that this was already resolved in https://bugs.python.org/issue17561. In the latter issue, one of the last messages helpfully mentions what the implemented solution is:
# IPv4 only
>>> socket.create_server(addr)
# IPv6 only
>>> socket.create_server(addr, family=socket.AF_INET6)
# IPv4 + IPv6
>>> socket.create_server(addr, family=socket.AF_INET6, dualstack_ipv6=True)
However, I was looking for a generic solution. Or rather, I was looking for UDP but figured that, because this is on the IP layer, any solution would be generic. It turns out that create_server() is TCP-only.
Binding to both v4 and v6 manually means I cannot just call recvfrom() and let it block until a packet comes in, because then a v6 packet might be queued while it's blocking on the v4 recvfrom call or vice versa. So I guess I need to use threading?
What is the best way to make a dual-stack UDP server in python 3.x? For me, a Linux solution would be sufficient, but of course platform-independent answers are even better.
Basically, how to write this in dual-stack:
import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.bind(('0.0.0.0', 53))
while True:
message, address = server_socket.recvfrom(1400)

At least under Linux, creating an IPv6 socket listens for both IPv6 and IPv4 connections. That is, given this code:
import socket
s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
s.bind(('::', 9090))
while True:
message, address = s.recvfrom(1024)
print(address, "says: ", message.decode())
I can connect using IPv4:
echo hello world | nc -u 127.0.0.1 9090
Which results in:
('::ffff:127.0.0.1', 36694, 0, 0) says: hello world
Or I can connect using IPv6:
echo hello world | nc -u ::1 9090
Which results in:
('::1', 51880, 0, 0) says: hello world

How can I get live established UDP connections with python using psutil

I am trying to get the destination IP (remote host) addresses connected to my machine that are using UDP protocol but i get Null results using psutil
the script I wrote
import psutil
def GetServerIP():
PROCNAME = "theprocessname.exe"
for proc in psutil.process_iter():
if proc.name() == PROCNAME:
pinfo = proc.as_dict(attrs=['pid', 'name', 'create_time'])
pidnumber = pinfo["pid"]
print("Process is runnging on number: %r" % (pidnumber))
for connection in psutil.net_connections(kind='udp4'):
if pidnumber in connection:
print(connection.raddr)
GetServerIP()
The script works for TCP connections but gives nothing on UDP connections that are established on my local machine.
I read through psutil documentation however still cant figure out why it gives no results back on UDP
I can verify there are established UDP packets being sent and received using wireshark
if psutil does not work well with UDP is there an alternative solution

It seems like psutil was not the right solution and path to take, turned out UDP connections are not established like TCP, so instead I switched to python scapy to capture UDP packets and it helped me resolve the destination IP address

Python TCP Server blocks forever in send to client if client is stopped (SIGSTOP)

I have an extremely simple tcp server in python the code for which is below:
#!/usr/bin/env python
import socket
sock = socket.socket()
sock.bind(('',3912))
sock.listen(100)
num_cons = 10
cons = []
for i in range(num_cons):
con, addr = sock.accept()
cons.append(con)
while True:
for con in cons:
msg = "a"* 1000
num_sent = con.send(msg.encode())
print("sent: {} bytes of msg:{}".format(str(num_sent), msg))
The corresponding client code is
#!/usr/bin/env python
import socket
sock = socket.socket()
sock.connect(('',3912)) # in reality here I use the IP of the host where
# I run the server since I launch the clients on a different host
while True:
data = sock.recv(1000)
print("received data: {} ".format(str(data)))
Now, if I start the server with
./server.py
and 10 clients in parallel from a different host:
for i in `seq 1 10`; do ./client.py 2>/dev/null 1>/dev/null & done
And I send kill -SIGSTOP %1 to the first client, I expect the server to successfully keep trying to send data because it cannot know that the client has been stopped. Instead, the server blocks when it tries to send the data to client 1. I can understand the behaviour if the clients were on the same host as the server: we tried to write data, but the kernel buffers are full, so we block in the server, but the client never reads, so the buffer is never freed. However, if the clients are on a different machine, the kernel buffers of the server host should only be full temporarily and then the kernel should send the data over the network card and free them. So why is my server blocking on the send call? I have not verified if the same behaviour is seen when using a different language (C for example)

It is weird because 1000 characters is a small size for TCP. I have no available Linux machine but on a FreeBSD box, I could successfully send 130000 bytes on a TCP connection where the peer was stopped before the sender blocks. And more that 1000000 on Windows.
But as TCP is a connected protocol, a send call will block if it cannot queue its data because the internal TCP stack queue is full.

The gist of your problem seems to be that you're creating a SOCK_STREAM socket (i.e. TCP), and then abruptly terminating the client. As discussed in the Python Socket Programming HOWTO, a hang is expected in this situation.
TCP is a reliable protocol, meaning that every transmitted packet has to be acked. If the receiving side is dead, the sender will block waiting for that acknowledgement. Try setting a timeout and see if your send raises a socket.timeout after the expected time.

Receiving multicast data on specific interface

tcmpdump can view all the multicast traffic to specific group and port on eth2, but my Python program cannot. The Python program, running on Ubuntu 12.04:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# Multicast port is 52122
sock.bind(('', 52122))
# Interface eth2 IP is 1.2.3.4, multicast group is 6.7.8.9
mreq = socket.inet_aton('6.7.8.9')+socket.inet_aton('1.2.3.4')
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
while True:
print '\nwaiting to receive message'
data, address = sock.recvfrom(1024)
print data
When I use another program to send a multicast packet to eth2, it works and prints the packet. But it fails to see all the current multicast traffic. If I run tcpdump on eth2 on the same port and group as the above program:
sudo tcpdump -i eth2 host 6.7.8.9 and port 52122
it sees both the packets I send from another program AND all the current multicast traffic. It's output looks likes this...
# Packet sent from my other program
09:52:51.952714 IP 1.2.3.4.57940 > 6.7.8.9.52122: UDP, length 19
# Packet send from the outside world
09:52:52.143339 IP 9.9.9.9.39295 > 6.7.8.9.52122: UDP, length 62
Why can't my program see the packets from the outside world? How can I modify it (or something else) to fix this?
Edit:
I should have mentioned, the interface this going over is not eth2 but eth2.200 a VLAN. (The local IP and the tcpdump commands are all run with eth2.200, I just changed that in this question to make it simpler.) Based on this answer that could be the problem?
Edit #2:
netstat -ng when the program is running shows eth2.200 subscribed to 224.0.0.1 and 6.7.8.9`.
tshark -i eth2.200 igmp shows three repeated 1.2.3.4 -> 6.7.8.9 IGMP 46 V2 Membership Report / Join group 6.7.8.9 when the program first starts. When the program process is killed, it shows 1.2.3.4 -> 224.0.0.2 IGMP 46 V2 Leave group 6.7.8.9. There is also an infrequent 1.2.3.1 -> 224.0.0.1 IGMP 60 V2 Membership Query, general, where 1.2.3.1 is 1.2.3.4's gateway.
Not sure if it will help, but the routing table looks like:
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 1.2.5.6 0.0.0.0 UG 0 0 0 eth1
1.2.3.0 0.0.0.0 255.255.255.240 U 0 0 0 eth2.200
Thank you!

Finally! Found this question on ServerFault that addresses the same thing. Basically the kernel was not forwarding on / was filtering out the packets because it thought the sourced address was spoofed.
Changed the settings in /etc/sysctl.conf to match:
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.ip_forward = 1
Rebooted and everything works.

Stripping payload from a tcpdump?

Is there an automated way (either in tcpdump or via a helper app Out There) to generate a pcap file that contains only Ethernet, IP and Layer 4 (TCP in my case) headers, so that there is no payload/application data in the resulting pcap? I've found that since header sizes often vary, it's impossible to pick a capture size that won't catch any payload data.

You can strip out the TCP payload very easily with Python's scapy module
BEFORE
[mpenning#hotcoffee tshark_wd]$ tcpdump -n -r sample.pcap
reading from file sample.pcap, link-type EN10MB (Ethernet)
00:25:42.443559 IP 192.168.12.237.1052 > 192.168.12.236.22: Flags [P.],
seq 2445372969:2445373021, ack 1889447842, win 63432, length 52
00:25:42.443607 IP 192.168.12.236.22 > 192.168.12.237.1052: Flags [.],
ack 52, win 65535, length 0
00:25:42.443980 IP 192.168.12.236.22 > 192.168.12.237.1052: Flags [P.],
seq 1:389, ack 52, win 65535, length 388
PAYLOAD STRIPPING
Running this as root in linux...
#!/usr/bin/env python
from scapy.all import *
INFILE = 'sample.pcap'
OUTFILE = 'stripped.pcap'
paks = rdpcap(INFILE)
for pak in paks:
pak[TCP].remove_payload()
wrpcap(OUTFILE, paks)
AFTER
[mpenning#hotcoffee tshark_wd]$ tcpdump -n -r stripped.pcap
reading from file sample.pcap, link-type EN10MB (Ethernet)
00:25:42.443559 IP truncated-ip - 52 bytes missing! 192.168.12.237.1052
> 192.168.12.236.22: Flags [P.], seq 2445372969:2445373021,
ack 1889447842, win 63432, length 52
00:25:42.443607 IP 192.168.12.236.22 > 192.168.12.237.1052: Flags [.],
ack 52, win 65535, length 0
00:25:42.443980 IP truncated-ip - 388 bytes missing! 192.168.12.236.22
> 192.168.12.237.1052: Flags [P.], seq 1:389,
ack 52, win 65535, length 388
In the tcpdump above, notice the "XX bytes missing!" messages. That is because we have removed the TCP payload.

If simple truncate would work for you, you could use:
tcpdump -i eth0 -s 96 -w test1.pcap
Later on you can analyze it with wireshark.

My solution was as follows. I'd love to hear how others do it without external libraries or truncation. I'd love to hear how others performed this, because I was unable to find the remove_payload() function in the Scapy documentation, making this answer unusable.
#read pcap file
pkts = rdpcap("packet-capture.pcap")
#write packet with payload "XXXXXXXXXX"
for pkt in pkts:
pkt.load = "XXXXXXXXXX"
#write new pcap
wrpcap("new.pcap", pkts)
The problem with this is that, when read with tcpdump, it leaves a bytes missing! for the src IP. I can verify the infromation is still there using scapy via
pkts[_packet_num].load
Is there a way to regenerate the whole capture so it looks as if it was unaltered?