I stumbled upon some code in the book PyMOTW3 (https://pymotw.com/3/socket/multicast.html) and I could not understand why the time to live(ttl) argument was packed as struct.pack('b',1).
I tried searching the manuals to see if the arguments are supposed to be packed but it states that it can be integers. I tried inputting a normal integer and it seemed to work fine. It gave the same output as the code below. Then is there a specific reason why it has been packed like this ?
I do know that packing it as 1 is unnecessary because the default value is 1, but what if I need to use some other number. Do I need to pack it?
I have included the code from the book below.
import socket
import struct
import sys
message = b'very important data'
multicast_group = ('224.3.29.71', 10000)
# Create the datagram socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Set a timeout so the socket does not block
# indefinitely when trying to receive data.
sock.settimeout(0.2)
# Set the time-to-live for messages to 1 so they do not
# go past the local network segment.
ttl = struct.pack('b', 1)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, ttl)
print(s.getsockopt(socket.IPPROTO_IP,socket.IP_MULTICAST_TTL))
try:
# Send data to the multicast group
print('sending {!r}'.format(message))
sent = sock.sendto(message, multicast_group)
# Look for responses from all recipients
while True:
print('waiting to receive')
try:
data, server = sock.recvfrom(16)
except socket.timeout:
print('timed out, no more responses')
break
else:
print('received {!r} from {}'.format(
data, server))
finally:
print('closing socket')
sock.close()
This is the output I get whether I pack it or use a normal integer.
1
Sending very important message
Timed Out!
Closing Socket
All packets consist of bytes.
For the AF_INET or an IPv4 you can see the distribution of bits among the properties of the packet in this picture. You can see in the IPv4 header, TTL is the 9th octet of 20. TTL is an 8-bit (1-byte) field and can be from 0 to 255.
That is, each property should take a certain number of bits.
For this purpose, Python 3.7 has a module struct. This module performs conversions between Python values and C structs represented as Python bytes objects.
Since TTL is a positive integer of 1 byte in size, we have to select the unsigned char type, which is 'B'.
And so now the code looks like this:
ttl = struct.pack('B', 25) # unsigned char as 1-byte object
sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, ttl)
Related
I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1
As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.
One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.
You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()
While trying to attempt to go for python based socket, I have 2 questions which I am not able to resolve. Kindly help me. Here is my sample socket code:
import socket
import threading
import chardet
bind_ip = '0.0.0.0'
bind_port = 9999
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(1) # max backlog of connections
print (('Listening on {}:{}').format(bind_ip, bind_port))
def handle_client_connection(client_socket):
request = client_socket.recv(4096 )
result = chardet.detect(request)
print(result)
print (request.decode(result['encoding']))
client_socket.send('ACK!'.encode(result['encoding']))
client_socket.close()
while True:
client_sock, address = server.accept()
print (('Accepted connection from {}:{}').format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,) # without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
The above one is server and the below is client:
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('127.0.0.1', 9999))
client.send(str('test data').encode("utf-16"))
response = client.recv(4096)
print(response.decode("utf-16"))
Now the questions:
1) What is the meaning of the number in this statement: client.recv(4096)? What is 4096, is it bit or byte of kilobyte for data receiving?
2) Can I receive infinite data stream through the socket? Meaning, as this statement client.recv(4096), whatever is the menaing of 4096, may be byte, then the socket will receive the 4096 bytes of data only. I do not have a control on the size of data received through the socket, hence, can I generalize it to accept any size of data through socket?
Please help me get the answers to above queries. I tried the python documentation for socket but didn't found much. I guess I miss something. Please help me get through it.
According to documentation
"If a message is too long to fit in the supplied buffer, excess bytes may be discarded depending on the type of socket the message is received from."
found here
"https://linux.die.net/man/2/recv"
which was from python 3.6 docs
"he maximum amount of data to be received at once is specified by bufsize. See the Unix manual page recv(2)"
found here
"https://docs.python.org/3.6/library/socket.html"
so it is a bytes object and it may actually truncate the message depending on the protocol. So message integrity is handled by the next layer up. So presumably you would loose part of a UDP packet, and you would get either a retry or an additional packet for TCP if the message was too large to fit in the buffer.
I have been trying to figure this out and can't seem to wrap my head around it.
import socket
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server = '192.168.1.7'
port = 8000
buffer_size = 4096
request = '1|gkdgdfg49|sdfsdfsd*|sdsfsf*|'+server
request = request.encode()
s.connect((server,port))
s.send(request)
x = s.recv(buffer_size)
The documentation for the server side is:
https://www.sharekhan.com/Upload/General/TradeTigerAPIForClient.pdf
The log of api server shows:
[I] "API Client Socket Connected with error Code=0"2018-09-22 16:07:23.555+05:30
[I] "Hurraay You have Connected."2018-09-22 16:07:23.555+05:30
[I] "Received Bytes Count = 49 Data Length = 1801944113 Thread State = Background, WaitSleepJoin , Queue Count = 0"2018-09-22 16:07:23.555+05:30
[I] "API Client Request Data Length 1801944113 bytes needs to read"
2018-09-22 16:08:21.984+05:30
[I] "Received Bytes Count = 49 Data Length = 1801944113 Thread State = Background, WaitSleepJoin , Queue Count = 0"
2018-09-22 16:08:21.984+05:30
[I] "API Client Request Data Length 1801944113 bytes needs to read"|Reserved = |
Is the request correct? Is the data being transferred?
If I rerun
s.send(request)
x = s.recv(buffer_size)
Nothing happens. The interpreter is stuck at running the above two lines.
I would appreciate any help. Very new to `sockets and the documentation is not very helpful.
As I pointed out in your other question, this is a binary protocol.
You're sending it ASCII text. Note that big number (1801944113), when interpreted as text, is equal to 'kg|1'. This is (the big-endian ordering of) the first four characters you sent it. In other words, it's taken the first four bytes you sent it, interpreted them as a binary value, and is now using it as the message length value -- which is obviously all wrong.
You need to use the python struct module's pack and unpack methods to encode and decode binary values to build up your buffer.
That would look something like this:
import struct
trans_code = 1 # LOGIN
login_id = b'MyName'
mpassword = b'MyPass'
tpassword = b'MyTradingPass'
my_ip = b'192.168.1.31' # No idea what the point of this is
reserved = b''
msg_length = 196
msg_buffer = struct.pack("!IH30s20s20s20s100s", msg_length, trans_code,
login_id, mpassword, tpassword, my_ip, reserved)
assert len(msg_buffer) == msg_length
print("Login request to send '{}'".format(msg_buffer))
You will need to learn to use the struct module. It's well-specified in the python documentation.
Your results (in particular that big number) establish that the server is expecting numeric fields to be encoded in big-endian byte order (AKA "network byte order").
I'm trying to create a simple server to client based chat program and the issue is that when I try to execute c.sendto(data,client) this error appears saying that Client is an int but it's a tuple containing the port number and the address. I'm I supposed to convert the tuple to bytes so I can send the message to the clients?
Server Script
import socket
clients = []
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1",7999))
s.listen()
print("Waiting for connection")
c, addr = s.accept()
while True:
data , addr = c.recvfrom(1024)
print(data)
if addr not in clients:
clients.append(addr)
print(clients[0])
if data:
for client in clients:
print(client)
c.sendto(data,client)
s.close()
Client Script
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
addr = ("127.0.0.1",7999)
s.connect(addr)
while True:
string = input(">>")
s.send(string.encode("utf-8"))
data =s.recv(1024)
s.close()
Server Output
The problem is that you're using sendto() with a connection-mode socket. I think you want c.send(data) instead.
Details:
The Python docs for sendto say "The socket should not be connected to a remote socket, since the destination socket is specified by address." Also the man page for sendto says "If sendto() is used on a connection-mode (SOCK_STREAM, SOCK_SEQPACKET) socket, the arguments dest_addr and addrlen are ignored (and the error EISCONN may be returned when they are not NULL and 0)." I somewhat suspect that this is happening and Python is misreporting the error in a confusing way.
The sockets interface and networking in general can be pretty confusing but basically sendto() is reserved for SOCK_DGRAM which is UDP/IP type internet traffic, which you can think of as sending letters or postcards to a recipient. Each one goes out with a recipient address on it and there's no guarantee on order of receipt. On the other hand, connection-mode sockets like SOCK_STREAM use TCP/IP which is a bit more like a phone call in that you make a connection for a certain duration and and each thing you send is delivered in order at each end.
Since your code seems to be designed for communication over a connection I think you just want c.send(data) and not sendto.
You must first convert the string that contains the IP address into a byte or a string of bytes and then start communicating.
According to the code below, your error will be resolved.
Make sure your code is working correctly overall.
string = '192.168.1.102'
new_string = bytearray(string,"ascii")
ip_receiver = new_string
s.sendto(text.encode(), (ip_receiver, 5252))
server.py is fine and I use array to send data:
for i in range(0,len(outputdata)):
connectionSocket.send(outputdata[i])
connectionSocket.close()
But my client don't work(print nothing):
#import socket module
from socket import *
serverName = '127.0.0.1'
serverPort = 9999
clientSocket = socket(AF_INET,SOCK_STREAM)
clientSocket.connect((serverName,serverPort))
request = raw_input('Input the filename:')
clientSocket.send(request)
while (clientSocket.recv(1024)):
print clientSocket.recv(1024)
clientSocket.close()
Why doesn't this work?
The command clientSocket.recv(1024) is a command that will fetch available data.
from Python Docs:
Receive data from the socket. The return value is a string
representing the data received. The maximum amount of data to be
received at once is specified by bufsize. See the Unix manual page
recv(2) for the meaning of the optional argument flags; it defaults to
zero.
To understand it better, your server is sending some data, the client receives that data and holds it in a buffer, the command will go to that buffer and pop the data (with max of 1024 at your code) so if you have 1024 bytes waiting in the buffer the first recv(1024) command will get all the data in the buffer and the second recv(1024) will be empty.
A good solution was suggested by #gnibbler.
Remember calling clientSocket.recv(1024) without saving it's return value it's like popping from a stack without saving the value.
The first clientSocket.recv(1024) is throwing away those bytes
while (clientSocket.recv(1024)):
print clientSocket.recv(1024)
You need to save them into a variable like this
while True:
data = clientSocket.recv(1024)
if not data:
break
print data