I read the paper on TCP hole punching available here.
In order to do this one has to bind the sockets which are used for making TCP connections to a remote host and that which the local host uses to listen for connections to the same port. I have been able to do this in Java but cannot in Python even when the SO_REUSEADDR flag is set for the given sockets. Can someone explain to me why? Is it because Python is in inherently single-threaded?
From far as I tested/studied TCP Hole Punching is not a viable technique that will work in every situation.
First, what TCP Hole Punching does is not well supported by NATs and their behavior is unpredictable.
In resume it relies on sending a TCP SYN packet and recieving TCP SYN packet (when in a normal conversation you would respond with SYN+ACK) so that the NAT would open a connection between the two hosts. Some NATs may open this connection while others dont.
The best way I know to acomplish NAT Traversal is to use UDP. Since UDP is not connection oriented, you can start sending packet and reciving so the NAT will think one packet is a reply form the other.
See
UDP Hole Punching
Also, to make UDP as reliable as TCP you can use an implementation of TCP over UDP.
See UDT
I am sorry I didnt answer your question, but why in Java worked and why in Python did not, is hard to know, it has something to do with virtual machines implementation and system calls or even the NAT you are using.
Related
I have an application, foo which takes in data, does stuff to it, and then publishes the new treated data over AMQ for another downstream application to grab. Until this point, foo has always gotten its data by connecting to another AMQ server which another script is publishing packetized data to (a lot of handwaving here, but the specifics don't really matter).
Recently a change has been made, and foo needs to be able to grab its data from a UDP socket. Is AMQ able to connect to this socket and receive/listen to the data being transmitted over it? From my understanding, AMQ uses TCP to establish connection to the client, and some initial research points me to this UDP Transport documentation from Apache, but not much else.
Alternatively, I could develop a rough UDP socket listener in Python, and then publish those messages to AMQ for foo to grab, but it would be optimal to have it all included in foo itself.
Not necessarily looking for an exhaustive solution here; quick and dirty would be enough to get me started.
Thanks!
ActiveMQ itself is a broker and therefore doesn't connect to sockets and listen for messages. It is the job of a client to connect to the broker and send and/or receive messages.
The UDP transport documentation is just theoretical as far as I know. It is technically possible to use UDP as the base of a traditional messaging protcol, but I've never actually seen it done since UDP is unreliable. The documentation even says, "Note that by default UDP is not reliable; datagrams can be lost so you should add a reliability layer to ensure the JMS contract can be implemented on a non-reliable transport." Adding a "reliability layer" is impractical when TCP can simply be used instead. All of the protocols which ActiveMQ supports (i.e. AMQP, STOMP, MQTT, OpenWire) fundamentally require a reliable network transport.
I definitely think you'll need some kind of intermediary process to read the data from the UDP socket and push it to the broker.
So I'm trying to build a packet sniffer in Python to deepen my understanding of networking. Thing is, it has turned out to be a tad bit more confusing than I initially anticipated. The problem is that all resources with thorough explanations cover the scenario of creating sockets for client/server data sending/receiving purposes.
At this point, I've successfully created some classes that handle packet header decoding for IPv4 and ICMP. Now, since my socket code only seemed to capture ICMP packets, I've been trying to configure it so that I can catch all traffic reaching my wifi interface, but I still almost exclusively see ICMP packets (with localhost as both source and destination).
So, I have some questions which I'd like to get answered. But first, my code:
import socket
import sys
from protocols.ipv4 import IPv4
PACKET_SIZE = 65535
sock = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
sock.bind(("0.0.0.0", 0))
try:
while True:
# read in a packet
raw_buffer = sock.recvfrom(PACKET_SIZE)[0]
# create an IP packet object
ip_header = IPv4(raw_buffer)
# print the packet
print(ip_header)
except KeyboardInterrupt:
print("\nExiting...")
sock.close()
sys.exit(0)
This is how I've understood it:
First I'm creating a socket with socket.socket, where I specify address family, socket type and protocol. In my case, I'm selecting the AF_INET family which I don't really understand very well, but it seems to yield packets from the network layer. The socket type is set to SOCK_RAW meaning that I want the raw sockets as opposed to using SOCK_STREAM for TCP connections and SOCK_DGRAM for UDP. The last argument IPPROTO_IP just indicates that I want IP packets only.
Then, I'm binding the socket to 0.0.0.0 which supposedly means "any address" as described here.
What I don't understand:
Initially, I saw some examples of creating a sniffer socket which used the AF_PACKET address family. I soon found out that this address family is not available on macos (which I'm using). Why is that? What is an address family how does it relate to sockets? Is there an alternative way to catch packets from lower levels? In Wireshark I can see ethernet datagrams, so it seems possible.
As I've stated, I want to sniff all the traffic reaching my wifi interface. How does the socket know which interface I want it to operate on? Also I've learned that network interfaces can be put into different modes like monitor or promiscuous, how does that relate to sockets and my goal of catching packets?
Why am I almost only catching ICMP packets? What is the purpose of these packets with localhost both as destination and source?
I know there are lots of gaps in my current understanding of this. I'm not sure if I'll be able to get this to work, but I'm curious and I'd be grateful for any kind of answer or even just some good resources to check out.
Edit: My main question is where can I find out more about sockets in the context of packet sniffing?
I have a client written using python-twisted (http://pastebin.com/X7UYYLWJ) which sends a UDP packet to a UDP Server written in C using libuv. When the client sends a packet to the server, it is successfully received by the server and it sends a response back to the python client. But the client not receiving any response, what could be the reason ?
Unfortunately for you, there are many possibilities.
Your code uses connect to set up a "connected UDP" socket. Connected UDP sockets filter the packets they receive. If packets are received from any address other than the one to which the socket is connected, they are dropped. It may be that the server sends its responses from a different address than you've connected to (perhaps it uses another port or perhaps it is multi-homed and uses a different IP).
Another possibility is that a NAT device is blocking the return packets. UDP NAT hole punching has come a long way but it's still not perfect. It could be that the server's response arrives at the NAT device and gets discarded or misrouted.
Related to this is the possibility that an intentionally configured firewall is blocking the return packets.
Another possibility is that the packets are simply lost. UDP is not a reliable protocol. A congested router, faulty networking gear, or various other esoteric (often transient) concerns might be resulting in the packet getting dropped at some point, instead of forwarded to the next hop.
Your first step in debugging this should be to make your application as permissive as possible. Get rid of the use of connected UDP so that all packets that make it to your process get delivered to your application code.
If that doesn't help, use tcpdump or wireshark or a similar tool to determine if the packets make it to your computer at all. If they do but your application isn't seeing them, look for a local firewall configuration that might reject them.
If they're not making it to your computer, see if they make it to your router. Use whatever diagnostic tools are available (along the lines of tcpdump) on your router to see whether packets make it that far or not. Or if there are no such tools, remove the router from the equation. If you see packets making it to your router but no further, look for firewall or NAT configuration issues there.
If packets don't make it as far as your router, move to the next hop you have access to. This is where things might get difficult since you may not have access to the next hop or the next hop might be the server (with many intervening hops - which you have to just hope are all working).
Does the server actually generate a reply? What addressing information is on that reply? Does it match the client's expectations? Does it get dropped at the server's outgoing interface because of congestion or a firewall?
Hopefully you'll discover something interesting at one of these steps and be able to fix the problem.
I had a similar problem. The problem was windows firewall. In firewall allowed programs settings, allowing the communication for pythonw/python did solve the problem. My python program was:
from socket import *
import time
address = ( '192.168.1.104', 42) #Defind who you are talking to (must match arduino IP and port)
client_socket = socket(AF_INET, SOCK_DGRAM) #Set Up the Socket
client_socket.bind(('', 45)) # arduino sending to port 45
client_socket.settimeout(1) #only wait 1 second for a response
data = "xyz"
client_socket.sendto(data, address)
try:
rec_data, addr = client_socket.recvfrom(2048) #Read response from arduino
print rec_data #Print the response from Arduino
except:
pass
while(1):
pass
I'm developping a client-server game in Python, and each quantum, the server has to send the state of the game to the clients.
I developed it with both UDP and TCP connections. UDP ensures the speed sending of the game states, and TCP is used for the reliability part.
Is this a good way of doing ?
So each quantum server sends data like this :
while playing:
data = computeGameData()
sendNewPlayUDP(data)
sendNewPlayTCP(data)
time.sleep(sleeptime)
I tested it, and it seems to work well, but I wonder if the thread can block because of TCP struggling. There is maybe a better way of doing.
According to :
http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/
you should not use TCP at all. This articles recommends using UDP and adding extra logic for the packets you absolutely want to be received and acknowledged. This article also states that TCP packets may interfere with UDP packets, increasing UDP packet loss rate.
You may also have a look to :
https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking
Loosing packets can be tolerated in many cases. It looks like a bit overkill to send the same data on both TCP and UDP channels.
I'm implementing client-server communication using UDP that's used for FTP. First off, you don't need to tell me that UDP is unreliable, I know. My approach is: client asks for a file, server blasts the client with udp packets with sequence numbers, then says "what'd you miss?", resending those. On a local network, packet loss is < 1%. I'm pretty new to socket programming, so I'm not familiar with all the socket options (of which most examples found on google are for tcp).
My problem is why my client's receiving of this data.
PACKET_SIZE = 9216
mysocket.sendto('GO!', server_addr)
while True:
resp = mysocket.recv(PACKET_SIZE)
worker_thread.enqeue_packet(resp)
But by the time it gets back up to .recv(), it's missed a few udp packets (that I've confirmed are being sent using wireshark). I can fix this by making the server send slightly slower (actually, including logging statements is enough of a delay to make everything function).
How can i make sure that socket.recv doesn't miss anything in the time it takes to process a packet? I've tried pushing the data out to a separate thread that pushes it into a queue, but it's still not enough.
Any ideas? select, recv_into, setblocking?
While you already know, that UDP is not reliable, you maybe missed the other advantages of TCP. Relevant for you is that TCP has flow control and automatically scales down if the receiver is unable to cope with the senders speed (e.g. packet loss). So for normal connections TCP should be preferred for data transfer. For high latency connections (satellite link) it behaves too bad in the default configuration, so that some people design there custom transfer protocols (mostly with UDP), while others just tune the existing TCP stack.
I don't know why you use UDP, but if you want to continue to use it you should add some kind of back channel to the sender to inform it from current packet loss, so that it can scale down. Maybe you should have a look at RTCP, which accompanies RTP (used for VoIP etc).