We Keep Coding

Building a packet sniffer with Python + sockets

So I'm trying to build a packet sniffer in Python to deepen my understanding of networking. Thing is, it has turned out to be a tad bit more confusing than I initially anticipated. The problem is that all resources with thorough explanations cover the scenario of creating sockets for client/server data sending/receiving purposes.
At this point, I've successfully created some classes that handle packet header decoding for IPv4 and ICMP. Now, since my socket code only seemed to capture ICMP packets, I've been trying to configure it so that I can catch all traffic reaching my wifi interface, but I still almost exclusively see ICMP packets (with localhost as both source and destination).
So, I have some questions which I'd like to get answered. But first, my code:
import socket
import sys
from protocols.ipv4 import IPv4
PACKET_SIZE = 65535
sock = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
sock.bind(("0.0.0.0", 0))
try:
while True:
# read in a packet
raw_buffer = sock.recvfrom(PACKET_SIZE)[0]
# create an IP packet object
ip_header = IPv4(raw_buffer)
# print the packet
print(ip_header)
except KeyboardInterrupt:
print("\nExiting...")
sock.close()
sys.exit(0)
This is how I've understood it:
First I'm creating a socket with socket.socket, where I specify address family, socket type and protocol. In my case, I'm selecting the AF_INET family which I don't really understand very well, but it seems to yield packets from the network layer. The socket type is set to SOCK_RAW meaning that I want the raw sockets as opposed to using SOCK_STREAM for TCP connections and SOCK_DGRAM for UDP. The last argument IPPROTO_IP just indicates that I want IP packets only.
Then, I'm binding the socket to 0.0.0.0 which supposedly means "any address" as described here.
What I don't understand:
Initially, I saw some examples of creating a sniffer socket which used the AF_PACKET address family. I soon found out that this address family is not available on macos (which I'm using). Why is that? What is an address family how does it relate to sockets? Is there an alternative way to catch packets from lower levels? In Wireshark I can see ethernet datagrams, so it seems possible.
As I've stated, I want to sniff all the traffic reaching my wifi interface. How does the socket know which interface I want it to operate on? Also I've learned that network interfaces can be put into different modes like monitor or promiscuous, how does that relate to sockets and my goal of catching packets?
Why am I almost only catching ICMP packets? What is the purpose of these packets with localhost both as destination and source?
I know there are lots of gaps in my current understanding of this. I'm not sure if I'll be able to get this to work, but I'm curious and I'd be grateful for any kind of answer or even just some good resources to check out.
Edit: My main question is where can I find out more about sockets in the context of packet sniffing?

Python - Twisted - Simple UDP forwarder. Preserve source IP?

I have this basic UDP forward script in Python 3.
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
class Forward(DatagramProtocol):
def __init__(self, targetTuples):
print ('in init, targetTuples are ', targetTuples)
self._targetTuples = targetTuples
def datagramReceived(self, data, hostAndPort):
print ('self._targetTuples is ', self._targetTuples)
for (targetHost, targetPort) in self._targetTuples:
self.transport.write(data, (targetHost, targetPort))
reactor.listenUDP(5005, Forward([('10.35.203.24', 5000), ('10.35.200.251', 5005)]))
reactor.run()
So I'm listening on port 5005 UDP, and forwarding those packets to the two IP addresses and different ports.
My question is this -
How do I preserve the original IP address that twisted gets while listening on port 5005?
Source IP (10.1.1.1) --> Twisted (10.30.1.1) --> Multiple Destinations
How can I get Multiple Destinations to see the packet source preserved from the Source IP of (10.1.1.1) ?

When sending UDP datagrams using the BSD socket API (around which, as a first approximation, Twisted is a wrapper), the source address is set to the address the socket is bound to. You can specify the IP of the bind address for a UDP socket in Twisted by passing a value for the interface argument to reactor.listenTCP. However, you are typically restricted in what addresses you are allowed to bind to. Typically the only values allowed are addresses which are assigned to a local network interface. If you are forwarding traffic for 10.1.1.1 but you are on host 10.30.1.1 then you probably cannot set the source address of the UDP packets you send to 10.1.1.1 because that address isn't assigned to a network interface on the host doing the forwarding. If you assigned it to one, routing on your network would probably break in weird ways because then two different hosts would have the same IP address.
This doesn't mean it's not possible to do what you want - but it does mean you probably cannot do it using Twisted's basic UDP support. There are a number of other approaches you could take. For example, you can rewrite source addresses using iptables on Linux. Or you can operate on the IP level and parse and generate full UDP datagrams yourself letting you specify any source address you want (you can do this with Twisted on Linux, too, using twisted.pair.tuntap). There are probably a number of other ways. The best solution for you may depend on which platforms you're targeting and the particular reasons you want to do this kind of forwarding.

Python client server how UDP is supposed to work?

I have a client-server "snake" game working really well with TCP connections, and I would like to try it the UDP way.
I wonder how it is supposed to be used ? I know how UDP works, how to make a simple ECHO example, but I wonder how to do the following :
For instance with TCP, every TICK (1/15 second) server sends to the client the new Snake head position.
With UDP, am I supposed to do something like this :
Client SIDE :
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
serverAddr = (('localhost', PORT))
while 1:
client.sendto('askForNewHead', serverAddr)
msg, addrServer = client.recvfrom(1024)
game.addPosition(msg)
Server SIDE :
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind((HOST, PORT))
while 1:
data, addr = server.recvfrom(1024)
server.sendto(headPosition, addr)
So here Client has to ask server to get the new head position, and then server sends the answer. I managed to make it work this way, but I can't figure out if it is a good way of doing.
Seems weird that client has to ask udp for an update while with my TCP connection, client has just to wait untill he receives a message.

There are differences between TCP and UDP but not the way you describe. Like with TCP the client can recvfrom to get messages from the server without asking each time for new data. The differences are:
With TCP the initial connect includes a packet exchange between client and server. Unless the client socket was already bound to an IP and port it will be bound to the clients IP and a free port will be allocated. Because of the handshake between client and server the server knows where to contact the client and thus can send data to the packet without getting data from the client before.
With UDP there is no initial handshake. Unless already bound, the socket will be bound to clients IP and a free port when sending the first packet to the server. Only when receiving this packet the server knows the IP and port of the client and can send data back.
Which means, that you don't need to 'askForNewHead' all the time. Instead the client has to send only a single packet to the server so that the server knows where to send all future packets.
But there are other important differences between TCP and UDP:
With UDP packets may be lost or could arrive in a different order. With TCP you have a guaranteed delivery.
With UDP there is no real connection, only an exchange of packets between two peers. With TCP you have the start and end of a connection. This is relevant for packet filters in firewalls or router, which often need to maintain the state of a connection. Because UDP has no end-of-connection the packet filters will just use a simple timeout, often as low as 30 seconds. Thus, if the client is inside a home network and waits passively for data from server, it might wait forever if the packet filter closed the state because of the timeout. To work around this data have to be transmitted in regular intervals so that the state does not time out.
One often finds the argument, that UDP is faster then TCP. This is plain wrong. But you might see latency problems if packets get lost because TCP will notice packet loss and send the packet again and also reduce wire speed to loose less packets. With UDP instead you have to deal with the packet loss and other congestion problems yourself. There are situations like real time audio, where it is ok to loose some packets but low latency is important. These are situations where UDP is good, but in most other situations TCP is better.

UDP is different to TCP, and I believe with python the client does have to ask for an update from the server.
Although it is fun to learn and use a different way of communicating over the internet, for python I would really recommend sticking with TCP.

You don't have to ask the server for a update. But since UDP is connection-less the server can send head-positions without being asked. But the client should send i'm-alive-packets to the server, but this could happen every 10 seconds or so.

Serial data over UDP Sockets in Python

I may be going about this the wrong way but that's why I'm asking the question.
I have a source of serial data that is connected to a SOC then streams the serial data up to a socket on my server over UDP. The baud rate of the raw data is 57600, I'm trying to use Python to receive and parse the data. I tested that I'm receiving the data successfully on the port via the script below (found here: https://wiki.python.org/moin/UdpCommunication)
import socket
UDP_IP = "MY IP"
UDP_PORT = My PORT
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "received message:", data
Since I'm not reading the data with the .serial lib in Python or setting the baud rate to read at it comes all garbled, as would be expected. My end goal is to be able to receive and parse the data for server side processing and also have another client connect to the raw data stream piped back out from the server (proxy) which is why I'm not processing the data directly from the serial port on the device.
So my question is, how can I have Python treat the socket as a serial port that I can set a baud rate on and #import serial and .read from? I can't seem to find any examples online which makes me think I'm overlooking something simple or am trying to do something stupid.
sadface

You can't treat a socket as a serial line. A socket can only send and receive data (data stream for TCP, packets for UDP). If you would need a facility to control the serial line on the SOC you would need to build an appropriate control protocol over the socket, i.e. either use another socket for control like FTP does or use in-band control and distinguish between controlling and data like HTTP does. And of course both sides of the connection have to understand this protocol.

Build on facts
A first thing to start with is to summarise facts -- begining from the very SystemOnChip (SOC) all the way up ...:
an originator serial-bitstream parameters ::= 57600 Bd, X-<dataBIT>-s, Y-<stopBIT>, Z-<parityBIT>,
a mediator receiving process de-framing <data> from a raw-bitstream
a mediator upStream sending to server-process integration needs specification ( FSA-model of a multi-party hand-shaking, fault-detection, failure-state-resilience, logging, ... )
T.B.C. ...
Design as per a valid requirement list
A few things work as a just bunch of SLOC one-liners. Design carefully against the validated Requirement List as a principle. It saves both your side and the cooperating Team(s).
Test on models
Worth a time to test on-the-fly, during the efforts to progress from simple parts to more complex, multiparty scenarios.
Integrate on smarter frameworks
Definitely a waste of time to reinvent wheel. Using a smart framework for server-side integration will unlock a lot of your time and energy on your ProblemDOMAIN-specific tasks, rather than to waste both the former and the latter for writing your just-another-socket-framework ( destined in majority of cases to failure ... )
Try ZeroMQ or a nanomsg Scale-able Formal Communication Patterns Framework for smart-services to send de-framed data from your serial-bitStream source and you are almost there.

Finding source IP-address when binding to 0.0.0.0

When binding a UDP socket to ("", 1234) or ("0.0.0.0", 1234), is it possible to find out what IP-address it will actually send from?
As you can see in the code below, getsockname only tells me what I bound to. But when I send a packet, I can see that the IP-address is, in my case, 10.0.0.2.
Do I have to infer this address myself by looking at my network interfaces? If so, that is fine, but is there a robust way of doing so?
from socket import *
s = socket(AF_INET, SOCK_DGRAM)
s.bind(("", 1234))
print(s.getsockname()) # prints ("0.0.0.0", 1234)
s.sendto("hello", ("10.0.0.3", 1234)) # sends from 10.0.0.2
I've tried doing
import socket
print(socket.gethostbyname(socket.gethostname()))
but that doesn't seem to be very reliable (in the case where I expected 10.0.0.2, I got 127.0.1.1).
I realize that by binding to 0.0.0.0, I bind to all local network interfaces. Does that mean that my source IP-address will be determined by the routing tables when I try to send something? If so, can I still get that address in a robust way from Python?

The IP address used when sending will be determined by the routing table as the packet is sent.
There might be platform specific ways of querying that routing table, but a fairly portable way is to
connect() the socket first.
You can use another socket just for querying this information too. e.g.
from socket import *
s = socket(AF_INET, SOCK_DGRAM)
s.bind(("", 1234))
print(s.getsockname()) # prints ("0.0.0.0", 1234)
sq = socket(AF_INET, SOCK_DGRAM)
sq.connect(("10.0.0.3", 1234))
print(sq.getsockname()[0])
sq.close()
s.sendto("hello", ("10.0.0.3", 1234)) # sends from 10.0.0.2

This is more a usually-you-do-not-need-it answer. It may not correspond to your usecase.
Have alook at socket.gethostbyname_ex(socket.gethostname()). It shows all possible IP-addresses and the host name. You can receive from all of those since you did not bind to any specific one of those. They will be your source ip addresses.
It is not necessairy for you to know the exact address you send from. The receiver may see another one if it goes behind a NAT, into the internet or through a VPN.
The receiver will then know where the packet came from and can send answers.
#Joachim_Pileborg is also right. It is not usually done.
If you need a specific interface, bind to it. If not, you probaply do not need it.