I am trying to run the following python server under windows:
"""
An echo server that uses select to handle multiple clients at a time.
Entering any line of input at the terminal will exit the server.
"""
import select
import socket
import sys
host = ''
port = 50000
backlog = 5
size = 1024
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(backlog)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
server.close()
I get the error message (10038, 'An operation was attempted on something that is not a socket'). This probably relates back to the remark in the python documentation that "File objects on Windows are not acceptable, but sockets are. On Windows, the underlying select() function is provided by the WinSock library, and does not handle file descriptors that don’t originate from WinSock.". On internet there are quite some posts on this topic, but they are either too technical for me or simply not clear. So my question is: is there any way the select() statement in python can be used under windows? Please add a little example or modify my code above. Thanks!
Look like it does not like sys.stdin
If you change input to this
input = [server]
the exception will go away.
This is from the doc
Note:
File objects on Windows are not acceptable, but sockets are. On Windows, the
underlying select() function is provided by the WinSock library, and does not
handle file descriptors that don’t originate from WinSock.
I don't know if your code has other problems, but the error you're getting is because of passing input to select.select(), the problem is that it contains sys.stdin which is not a socket. Under Windows, select only works with sockets.
As a side note, input is a python function, it's not a good idea to use it as a variable.
Of course and the answers given are right...
you just have to remove the sys.stdin from the input but still use it in the iteration:
for s in inputready+[sys.stdin]:
Related
I'm new to socket programming in python. Here is an example of opening a TCP socket in a Mininet host and sending a photo from one host to another. In fact I changed the code that I had used to send a simple message to another host (writing the received data to a text file) in order to meet my requirements. Although when I implement this revised code, there is no error and it seems to transfer correctly, I am not sure whether this is a correct way to do this transmission or not. Since I'm running both hosts on the same machine, I thought it may have an influence on the result. I wanted to ask you to check whether this is a correct way to transfer or I should add or remove something.
mininetSocketTest.py
#!/usr/bin/python
from mininet.topo import Topo, SingleSwitchTopo
from mininet.net import Mininet
from mininet.log import lg, info
from mininet.cli import CLI
def main():
lg.setLogLevel('info')
net = Mininet(SingleSwitchTopo(k=2))
net.start()
h1 = net.get('h1')
p1 = h1.popen('python myClient2.py')
h2 = net.get('h2')
h2.cmd('python myServer2.py')
CLI( net )
#p1.terminate()
net.stop()
if __name__ == '__main__':
main()
myServer2.py
import socket
import sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('10.0.0.1', 12345))
buf = 1024
f = open("2.jpg",'wb')
s.listen(1)
conn , addr = s.accept()
while 1:
data = conn.recv(buf)
print(data[:10])
#print "PACKAGE RECEIVED..."
f.write(data)
if not data: break
#conn.send(data)
conn.close()
s.close()
myClient2.py:
import socket
import sys
f=open ("1.jpg", "rb")
print sys.getsizeof(f)
buf = 1024
data = f.read(buf)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('10.0.0.1',12345))
while (data):
if(s.sendall(data)):
#print "sending ..."
data = f.read(buf)
print(f.tell(), data[:10])
else:
s.close()
s.close()
This loop in client2 is wrong:
while (data):
if(s.send(data)):
print "sending ..."
data = f.read(buf)
As the send
docs say:
Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data. For further information on this topic, consult the Socket Programming HOWTO.
You're not even attempting to do this. So, while it probably works on localhost, on a lightly-loaded machine, with smallish files, it's going to break as soon as you try to use it for real.
As the help says, you need to do something to deliver the rest of the buffer. Since there's probably no good reason you can't just block until it's all sent, the simplest thing to do is to call sendall:
Unlike send(), this method continues to send data from bytes until either all data has been sent or an error occurs. None is returned on success. On error, an exception is raised…
And this brings up the next problem: You're not doing any exception handling anywhere. Maybe that's OK, but usually it isn't. For example, if one of your sockets goes down, but the other one is still up, do you want to abort the whole program and hard-drop your connection, or do you maybe want to finish sending whatever you have first?
You should at least probably use a with clause of a finally, to make sure you close your sockets cleanly, so the other side will get a nice EOF instead of an exception.
Also, your server code just serves a single client and then quits. Is that actually what you wanted? Usually, even if you don't need concurrent clients, you at least want to loop around accepting and servicing them one by one.
Finally, a server almost always wants to do this:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Without this, if you try to run the server again within a few seconds after it finished (a platform-specific number of seconds, which may even depend whether it finished with an exception instead of a clean shutdown), the bind will fail, in the same way as if you tried to bind a socket that's actually in use by another program.
First of all, you should use TCP and not UDP. TCP will ensure that your client/server has received the whole photo properly. UDP is more used for content streaming.
Absolutely not your use case.
def mp_worker(row):
ip = row[0]
ip_address = ip
tcp_port = 2112
buffer_size = 1024
# Read the reset message sent from the sign when a new connection is established
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
print('Connecting to terminal: {0}'.format(ip_address))
s.connect((ip_address, tcp_port))
#Putting a breakpoint on this call in debug makes the script work
s.send(":08a8RV;")
#data = recv_timeout(s)
data = s.recv(buffer_size)
strip = data.split("$", 1)[-1].rstrip()
strip = strip[:-1]
print(strip)
termStat = [ip_address, strip]
terminals.append(termStat)
except Exception as exc:
print("Exception connecting to: " + ip_address)
print(exc)
The above code is the section of the script that is causing the problem. It's a pretty simple function that connects to a socket based on a passed in IP from a DB query and receives a response that indicates the hardware's firmware version.
Now, the issue is that when I run it in debug with a breakpoint on the socket I get the entire expected response from the hardware, but if I don't have a breakpoint in there or I full on Run the script it only responds with part of the expected message. I tried both putting a time.sleep() in after the send to see if it would get the entire response and I tried using the commented out recv_timeout() method in there which uses a non-blocking socket and timeout to try to get an entire response, both with the exact same results.
As another note, this works in a script with everything in one main code block, but I need this part separated into a function so I can use it with the multiprocessing library. I've tried running it on both my local Windows 7 machine and on a Unix server with the same results.
I'll expands and reiterate on what I've put into a comment moment ago. I am still not entirely sure what is behind the different behavior in either scenario (apart from timing guess apparently disproved by an attempt to include sleep.
However, it's somewhat immaterial as stream sockets do not guarantee you get all the requested data at once and in chunks as requested. This is up for an application to deal with. If the server closes the socket after full response was sent, you could replace:
data = s.recv(buffer_size)
with recv() until zero bytes were received, this would be equivalent of getting 0 (EOF) from from the syscall:
data = ''
while True:
received = s.recv(buffer_size)
if len(received) == 0:
break
data += received
If that is not the case, you would have to rely on fixed or known (sent in the beginning) size you want to consider together. Or deal with this on protocol level (look for characters, sequences used to signal message boundaries.
I just recently found out a solution here, and thought I'd post it in case anyone else has issue, I just decided to try and call socket.recv() before calling socket.send() and then calling socket.recv() again afterwards and it seems to have fixed the issue; I couldn't really tell you why it works though.
data = s.recv(buffer_size)
s.send(":08a8RV;")
data = s.recv(buffer_size)
im writting an app using python and sockets, here is piece of the server code:
while True:
c = random.choice(temp_deck)
temp_deck.remove(c)
if hakem == p1:
p1.send(pickle.dumps(('{} for {}'.format(c,'you'),False)))
p2.send(pickle.dumps(('{} for {}'.format(c,'other'),False)))
else:
p1.send(pickle.dumps(('{} for {}'.format(c,'other'),False)))
p2.send(pickle.dumps(('{} for {}'.format(c,'you'),False)))
if c in ['A♠','A♣','A♦','A♥']:
if hakem == p1:
p1.send(pickle.dumps(('You are Hakem!',False)))
p2.send(pickle.dumps(('Other Player is Hakem!',False)))
break
else:
p1.send(pickle.dumps(('Other Player is Hakem!',False)))
p2.send(pickle.dumps(('You are Hakem!',False)))
break
if hakem == p1:
hakem = p2
other = p1
else:
hakem = p1
other = p2
this needs two clients to connect, everything is fine except clients don't receive full data:
for example one gets:
3♠ for other
2♠ for you
10♣ for other
10♦ for you
A♣ for other
the other gets:
2♠ for you
10♣ for other
10♦ for you
A♣ for other
what should i do?
client code:
import socket
import pickle
s = socket.socket()
host = socket.gethostname()
port = 12345
s.connect((host, port))
while True:
o = pickle.loads(s.recv(1024))
print(o[0])
if o[1] == True:
s.send(pickle.dumps(input(">")))
s.close
The problem is that TCP sockets are byte streams, not message streams. When you send some data and the client does a recv, there's no guarantee that it will receive everything you sent. It may get half the message. It may get multiple messages at once.
I've explained this at some length in a blog post—but fortunately, you're actually only hitting half the problem, and it's ultimately the simpler half. You've chosen to use a stream of pickle messages as your protocol, and pickle is a self-delimiting (aka framed) protocol.
pickle.load can load pickle after pickle out of anything with a file-like interface. And if your client and server are built around blocking I/O (e.g., using a thread for each direction on the socket), you can simulate read by doing blocking recv calls and appending them onto a buffer until you have enough bytes to satisfy the read.
And, even better, you don't have to do that yourself, because that's exactly what the builtin socket.makefile does. I haven't done any more than a quick test with this, so I won't promise it's bulletproof, but…
On the client side, you probably have something like this:
sock.connect(...)
# more stuff
# in a loop somewhere
buf = sock.recv(16384)
msg = pickle.loads(buf)
# later
sock.close()
Change it to this:
sock.connect(...)
rfile = socket.makefile('rb')
# more stuff
# in a loop somewhere
msg = pickle.load(rfile)
# later
rfile.close()
sock.close()
And it just works.
Again, you should test this. And you should read either my blog post, or a more complete primer on sockets programming and TCP, to understand what's going on. And really, you're probably better off designing your app around a higher-level framework (asyncio is really cool, especially with the syntactic support in Python 3.5+, or I think Twisted already has a pickle protocol class pre-written for you…). But this may be enough to get you started.
I'm testing UDP punching using code from here. It works on Linux however reports error on Windows. Here's the code snippet where the error occurs:
while True:
rfds, _, _ = select([0, sockfd], [], []) # sockfd is a socket
if 0 in rfds:
data = sys.stdin.readline()
if not data:
break
sockfd.sendto(data, target)
elif sockfd in rfds:
data, addr = sockfd.recvfrom(1024)
sys.stdout.write(data)
And error msg:
Traceback (most recent call last):
File "udp_punch_client.py", line 64, in <module>
main()
File "udp_punch_client.py", line 50, in main
rfds, _, _ = select([0, sockfd], [], [])
select.error: (10038, '')
I know this error has some thing to do with the select implementation on Windows, and everyone quote this:
Note File objects on Windows are not acceptable, but sockets are. On
Windows, the underlying select() function is provided by the WinSock
library, and does not handle file descriptors that don’t originate
from WinSock.
So I got two questions:
What does 0 in [0, sockfd] mean? Is this some sort often-used technique?
If select only works with socket on Windows, How to make the code Windows compatible?
Thank you.
Unfortunately, select will not help you to process stdin and network events in one thread, as select can't work with streams on Windows. What you need is a way to read stdin without blocking. You may use:
An extra thread for stdin. That should work fine and be the easiest way to do the job. Python threads support is quite ok if what you need is just waiting for I/O events.
A greenlet-like mechanism like in gevent that patches threads support and most of I/O functions of the standard library to prevent them from blocking the greenlets. There also are libraries like twisted (see the comments) that offer non-blocking file I/O. This way is the most consistent one, but it should require to write the whole application using a style that matches your framework (twisted or gevent, the difference is not significant). However, I suspect twisted wrappers are not capable of async input from stdin on Windows (quite sure they can do that on *nix, as probably they use the same select).
Some other trick. However, most of the possible tricks are rather ugly.
As the answer suggests, I create another thread to handle input stream and it works.
Here's the modified code:
sock_send = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def send_msg(sock):
while True:
data = sys.stdin.readline()
sock.sendto(data, target)
def recv_msg(sock):
while True:
data, addr = sock.recvfrom(1024)
sys.stdout.write(data)
Thread(target=send_msg, args=(sock_send,)).start()
Thread(target=recv_msg, args=(sockfd,)).start()
I've been trying to create a TCP server with gevent without (any major) success so far. I think that the problem lies within Windows ( I've had some issues with sockets under Windows before ). I'm using Python2.7, gevent0.13 under Windows7. Here's my code:
from gevent import socket
from gevent.server import StreamServer
def handle_echo(sock, address):
try:
fp = sock.makefile()
while True:
# Just echos whatever it receives
try:
line = fp.readline()
except Exception:
break
if line:
try:
fp.write(line)
fp.flush()
except Exception:
break
else:
break
finally:
sock.shutdown(socket.SHUT_WR)
sock.close()
server = StreamServer(("", 2345), handle_echo)
server.server_forever()
This implementation is similar to the one you can find here:
http://blog.pythonisito.com/2012/08/building-tcp-servers-with-gevent.html
Now there are no errors and the server seems to work correctly, however it is not reading ( and thus sending ) anything. Is it possible that sock.makefile() does not work correctly under Windows7? Or maybe the problem lies somewhere else?
I've tried to replace sock.makefile() with simple
while True:
line = sock.recv(2048)
but this operation obviously blocks.
I've also tried to mix gevent's spawn with sock.setblocking(0). Now this was better and it worked, however it would not handle more then ~300 connections at a time.
I'm going to do some tests on Linux and see if it makes difference. In the meantime if you have any ideas, then feel free to share them with me. Cheers!
UPDATE Original code does the same thing under Ubuntu 12.04. So how should I implement gevent TCP server??
What did you send to the server? Make sure it's terminated by newline.. otherwise readline() won't work.
You could also use tcpdump or wireshark to see what's happening at TCP layer if you think you're doing correct things in your code.