We Keep Coding

Building a packet sniffer with Python + sockets

So I'm trying to build a packet sniffer in Python to deepen my understanding of networking. Thing is, it has turned out to be a tad bit more confusing than I initially anticipated. The problem is that all resources with thorough explanations cover the scenario of creating sockets for client/server data sending/receiving purposes.
At this point, I've successfully created some classes that handle packet header decoding for IPv4 and ICMP. Now, since my socket code only seemed to capture ICMP packets, I've been trying to configure it so that I can catch all traffic reaching my wifi interface, but I still almost exclusively see ICMP packets (with localhost as both source and destination).
So, I have some questions which I'd like to get answered. But first, my code:
import socket
import sys
from protocols.ipv4 import IPv4
PACKET_SIZE = 65535
sock = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
sock.bind(("0.0.0.0", 0))
try:
while True:
# read in a packet
raw_buffer = sock.recvfrom(PACKET_SIZE)[0]
# create an IP packet object
ip_header = IPv4(raw_buffer)
# print the packet
print(ip_header)
except KeyboardInterrupt:
print("\nExiting...")
sock.close()
sys.exit(0)
This is how I've understood it:
First I'm creating a socket with socket.socket, where I specify address family, socket type and protocol. In my case, I'm selecting the AF_INET family which I don't really understand very well, but it seems to yield packets from the network layer. The socket type is set to SOCK_RAW meaning that I want the raw sockets as opposed to using SOCK_STREAM for TCP connections and SOCK_DGRAM for UDP. The last argument IPPROTO_IP just indicates that I want IP packets only.
Then, I'm binding the socket to 0.0.0.0 which supposedly means "any address" as described here.
What I don't understand:
Initially, I saw some examples of creating a sniffer socket which used the AF_PACKET address family. I soon found out that this address family is not available on macos (which I'm using). Why is that? What is an address family how does it relate to sockets? Is there an alternative way to catch packets from lower levels? In Wireshark I can see ethernet datagrams, so it seems possible.
As I've stated, I want to sniff all the traffic reaching my wifi interface. How does the socket know which interface I want it to operate on? Also I've learned that network interfaces can be put into different modes like monitor or promiscuous, how does that relate to sockets and my goal of catching packets?
Why am I almost only catching ICMP packets? What is the purpose of these packets with localhost both as destination and source?
I know there are lots of gaps in my current understanding of this. I'm not sure if I'll be able to get this to work, but I'm curious and I'd be grateful for any kind of answer or even just some good resources to check out.
Edit: My main question is where can I find out more about sockets in the context of packet sniffing?

What is the most efficient way to transfer data on Unix via Ethernet？

I'm working on a project that requires the PC to transmit a 4-bit array with another device running Linux. This transmission happens inside a nested for loop in Python, so I hope each transmission has a low delay. I found a lot resources online suggesting transferring files by netcat. However, I'm a bit worried that opening/closing files each time introduces a long delay.
For simple data transmission like this, what's the most efficient way to transmit data?

First of all, make sure that you don't establish a new connection every time you want to send some data. If you reconnect every time you run your loop, it will certainly add lots of delay to your communication (no matter what tool you use). Moreover, you might run out of ports on your system based on how fast your loop is and your system setting for TIME_WAIT (this is the time needed for the system to get rid of the unused network connection - if you keep opening too many of them too fast, you will run into problems eventually as you will reach maximum number of open connections).
You can create a socket outside of your loop and write/read it in your loop - thus having the connection establishment delay only once.
Now depending on how time critical your application is, you might want to choose correct network protocol for your communication - 4 bits is awfully little amount of information which is wrapped in half a dozens of layers that contain all kinds of info required by network protocols: network addresses of the source and the destination, control checksums, sequence numbers etc. and so each individual packet can be quite big despite containing only 4 bits of useful info. On the other hand, if you are not running an IOT device, but a computer on ethernet network, then you probably don't need to care about that.
That being said, if you want to be as effective as possible, you probably will want to send the information at the lowest possible level. Take the look at https://docs.python.org/3/howto/sockets.html
Quick summary
On the server side do:
# create an INET, STREAMing socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# bind the socket to a public host, and a well-known port
serversocket.bind((socket.gethostname(), port_number))
# become a server socket
serversocket.listen(1)
# accept connection from outside
(clientsocket, address) = serversocket.accept()
# now do something with the clientsocket
# in this case, we'll pretend this is a threaded server
while True:
# receive at most 1 byte of data
data = clientsocket.recv(1)
if (not data):
break
do_something_with( data )
On the client side do:
# create an INET, STREAMing socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect to the server on specific port
s.connect((server_ip_or_hostname, port_number))
data = get_your_for_bits_as_bytestring()
# send the data until there is no more of them or we lose the socket
while (s and data):
s.sendall(data)
data = get_your_for_bits_as_bytestring()

Python Read ethernet frames using socket on Windows?

I'm trying to read Ethernet (IEEE 802.2 / 3) frames using primarily socket.
The application shuld just sniff ethernet frames and depending on the content, act on it. However, there are almost no information on how to do this on Windows, the default (unix way) being socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(0x0800)). This is nonexistent in winsock apparently. So how do I sniff eth frames?
I suspect I need to bind to a MAC using socket.bind() instead of IP.
My current piece of code:
def _receive(interface): #Receive Eth packets.
#Interface = '192.168.0.10'
sock2 = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_RAW)
sock2.setsockopt(socket.SOL_IP, socket.IP_HDRINCL, 1))
sock2.bind((interface, 0))
while True:
data, sender = sock2.recvfrom(1500)
handle_data(sender, data)
Gets me nowhere. I see packets on Local connection in Wireshark, but it's not picked up in python..
On linux, I can do sock_raw = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_802_2)) , then bind and setsockopt(sock_raw, SOL_PACKET, PACKET_ADD_MEMBERSHIP, &mreq, sizeof(mreq))
I would like to not have to depend on too many external libraries becuase this is supposed to be distributed and thus pretty lightweight. pip install-able packages are OK though, they can be bundled with the installer..

Python's socket doesn't come with sniffing capabilites. Simple as that.
The idea of having a network stack in your operating system is that programs "register" for specific types of packets to be delivered to them – typically, this is something like listening on a IP port, ie. one to two levels above raw ethernet packets.
To get all raw ethernet packets, your operating system's network stack needs some kind of driver/interface to support you with that. That's why wireshark needs WinPcap.
My guess is you're going to be pretty happy with pypcap, which probably is PyPi/pip installable.

Send messages to ZeroMQ server using conventional TCP, possible?

I'm not sure if I'm doing this right, but I would like to be able to send messages to my server running ZMQ from normal TCP connections. The server is running Python ZMQ on port 5555 using a TCP transport. I would like to be able to send messages to it using different clients (Python, Java, PHP) that use conventional TCP. This is what I have so far:
SERVER
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
message = socket.recv()
print message
socket.send('{"name":"someone"}')
CLIENT
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 5555))
s.send('Hello, World!')
data = s.recv(1024)
print data
Printing data on the client does not give me the message I am expecting. I get this: �. I tried doing bytes(data).decode('utf8') thinking what I'm getting was an array of bytes, but I get the following error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
I am just wondering: Is this possible at all? Or am I doing something wrong? Also, is it recommended?
My reason for not using ZMQ on the clients is that I want to reduce the number of dependencies (ZeroMQ being one).
Thank you for your help.

ZeroMQ is a protocol that sits on top of TCP (well, technically, ZMTP is a protocol; 0MQ is the library that implements the ZMTP protocol, and ZeroMQ is the Python implementation of 0MQ…). All the signaling it does, and all the framing (dividing the stream of bytes into separate messages), is done by sending bytes over the socket. A client that doesn't know how to do ZeroMQ signaling and framing is just going to see a bunch of garbage.
What you're trying to do is exactly like trying to write a web client that just reads off a socket without knowing anything about HTTP. You're going to get a bunch of framing stuff that you don't know what to do with. The only difference is that in the case of HTTP, the framing is sometimes (but not always—you can have MIME envelopes, gzip transport-encoding, chunked transport, …) just a bunch of human-readable ASCII that comes before the data, while with ZMTP it's never human-readable…
If you want to send data over a plain TCP socket, you have to do that by creating a plain TCP socket on the server side and calling its send (or sendall, etc.) method. If you want to send data over a ZMTP channel, you have to do that by parsing ZMTP, or using a library that does it for you (like ZeroMQ), on the client side.
One more thing to keep in mind is that, unlike ZMTP, TCP is not a message-oriented protocol; all TCP sends is a stream of bytes. From the receiving side, there's no way to know when one send ends and the next one begins. So, for almost anything but a "send a request, get a response, hang up" protocol, you need to write some kind of framing of your own. This can be as simple as "messages are strings that have no newlines in them, and each message is separated by a newline" (in which case you can just use socket.makefile), but often the message format has to be more complicated, or you have to send "commands" rather than just data, etc.
Since tdelaney's answer has been deleted (which I think means it's invisible to anyone under 10K rep), and had a useful suggestion, I'll repeat it here, with credit as due: You can (using the ZeroMQ library) write a piece of middleware that talks ZMTP to the server but your own simple TCP-based protocol to the clients. ZeroMQ has been specifically designed to make this reasonably easy; as tdelaney put it, it's "kinda like Lego, you build a robust communication infrastructure by building different communicating parts. Not all of the parts need to be zeromq."

Python UDP socket misses packets

I'm implementing client-server communication using UDP that's used for FTP. First off, you don't need to tell me that UDP is unreliable, I know. My approach is: client asks for a file, server blasts the client with udp packets with sequence numbers, then says "what'd you miss?", resending those. On a local network, packet loss is < 1%. I'm pretty new to socket programming, so I'm not familiar with all the socket options (of which most examples found on google are for tcp).
My problem is why my client's receiving of this data.
PACKET_SIZE = 9216
mysocket.sendto('GO!', server_addr)
while True:
resp = mysocket.recv(PACKET_SIZE)
worker_thread.enqeue_packet(resp)
But by the time it gets back up to .recv(), it's missed a few udp packets (that I've confirmed are being sent using wireshark). I can fix this by making the server send slightly slower (actually, including logging statements is enough of a delay to make everything function).
How can i make sure that socket.recv doesn't miss anything in the time it takes to process a packet? I've tried pushing the data out to a separate thread that pushes it into a queue, but it's still not enough.
Any ideas? select, recv_into, setblocking?

While you already know, that UDP is not reliable, you maybe missed the other advantages of TCP. Relevant for you is that TCP has flow control and automatically scales down if the receiver is unable to cope with the senders speed (e.g. packet loss). So for normal connections TCP should be preferred for data transfer. For high latency connections (satellite link) it behaves too bad in the default configuration, so that some people design there custom transfer protocols (mostly with UDP), while others just tune the existing TCP stack.
I don't know why you use UDP, but if you want to continue to use it you should add some kind of back channel to the sender to inform it from current packet loss, so that it can scale down. Maybe you should have a look at RTCP, which accompanies RTP (used for VoIP etc).