As an extension to a previous post that unfortunately seems to have died a death:
select.select issue for sockets and pipes. Since this post I have been trying various things to no avail and I wanted to see if anyone has any idea where I am going wrong. I'm using the select() module to identify when data is present on either a pipe or a socket. The socket seems to be working fine but the pipe is proving problematic.
I have set up the pipe as follows:
pipe_name = 'testpipe'
if not os.path.exists(pipe_name):
os.mkfifo(pipe_name)
and the pipe read is:
pipein = open(pipe_name, 'r')
line = pipein.readline()[:-1]
pipein.close()
It works perfectly as a stand alone piece of code but when I try and link it to the select.select function it fails:
inputdata,outputdata,exceptions = select.select([tcpCliSock,xxxx],[],[])
I have tried entering 'pipe_name', 'testpipe' and 'pipein' in the inputdata argument but I always get a 'not defined' error. Looking at various other posts I thought it might be because the pipe does not have an object identifier so I tried:
pipein = os.open(pipe_name, 'r')
fo = pipein.fileno()
and put 'fo' in the select.select arguments but got a TypeError: an integer is required. I have also had a Error 9: Bad file descriptor when using this configuration of 'fo'. Any ideas what I have done wrong would be appreciated.
EDITED CODE:
I have managed to find a way to resolve it although not sure it is particularly neat - I would be interested in any comments-
Revised pipe setup:
pipe_name = 'testpipe'
pipein = os.open(pipe_name, os.O_RDONLY)
if not os.path.exists(pipe_name):
os.mkfifo(pipe_name)
Pipe Read:
def readPipe()
line = os.read(pipein, 1094)
if not line:
return
else:
print line
Main loop to monitor events:
inputdata, outputdata,exceptions = select.select([tcpCliSock,pipein],[],[])
if tcpCliSock in inputdata:
readTCP() #function and declarations not shown
if pipein in inputdata:
readPipe()
It all works well, my only problem now is getting the code to read from the socket before any event monitoring from select gets underway. As soon as connection is made to the TCP server a command is sent via the socket and I seem to have to wait until the pipe has been read for the first time before this command comes through.
According to the docs, select needs a file descriptor from os.open or similar. So, you should use select.select([pipein], [], []) as your command.
Alternatively, you can use epoll if you are on a linux system.
poller = epoll.fromfd(pipein)
events = poller.poll()
for fileno, event in events:
if event is select.EPOLLIN:
print "We can read from", fileno
Related
The problem is, my script won't work (it's printing empty lane), but it works in python interactive console.
import telnetlib
tn = telnetlib.Telnet("killermud.pl", 4000)
data = tn.read_very_eager()
data = data.decode()
print(data)
tn.close()
What is the reason of such behavior?
I just took a look at the documentation for the read_very_eager method, which says:
Read all data available already queued or on the socket,
without blocking.
It is likely that at the time you call this method that there is no data "already available or queued on the socket", so you're getting nothing back. You probably want to use something like the read_until method, which will read data until it finds a specific string. For example:
data = tn.read_until('Podaj swoje imie')
According to tlnetlib documentation, Telnet.read_very_eager() Raises EOFError if connection closed and no cooked data available. Return '' if no cooked data available otherwise. Do not block unless in the midst of an IAC sequence.
If you do data=="", returns true, Therefore, it means that no cooked data is available
I'm testing UDP punching using code from here. It works on Linux however reports error on Windows. Here's the code snippet where the error occurs:
while True:
rfds, _, _ = select([0, sockfd], [], []) # sockfd is a socket
if 0 in rfds:
data = sys.stdin.readline()
if not data:
break
sockfd.sendto(data, target)
elif sockfd in rfds:
data, addr = sockfd.recvfrom(1024)
sys.stdout.write(data)
And error msg:
Traceback (most recent call last):
File "udp_punch_client.py", line 64, in <module>
main()
File "udp_punch_client.py", line 50, in main
rfds, _, _ = select([0, sockfd], [], [])
select.error: (10038, '')
I know this error has some thing to do with the select implementation on Windows, and everyone quote this:
Note File objects on Windows are not acceptable, but sockets are. On
Windows, the underlying select() function is provided by the WinSock
library, and does not handle file descriptors that don’t originate
from WinSock.
So I got two questions:
What does 0 in [0, sockfd] mean? Is this some sort often-used technique?
If select only works with socket on Windows, How to make the code Windows compatible?
Thank you.
Unfortunately, select will not help you to process stdin and network events in one thread, as select can't work with streams on Windows. What you need is a way to read stdin without blocking. You may use:
An extra thread for stdin. That should work fine and be the easiest way to do the job. Python threads support is quite ok if what you need is just waiting for I/O events.
A greenlet-like mechanism like in gevent that patches threads support and most of I/O functions of the standard library to prevent them from blocking the greenlets. There also are libraries like twisted (see the comments) that offer non-blocking file I/O. This way is the most consistent one, but it should require to write the whole application using a style that matches your framework (twisted or gevent, the difference is not significant). However, I suspect twisted wrappers are not capable of async input from stdin on Windows (quite sure they can do that on *nix, as probably they use the same select).
Some other trick. However, most of the possible tricks are rather ugly.
As the answer suggests, I create another thread to handle input stream and it works.
Here's the modified code:
sock_send = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def send_msg(sock):
while True:
data = sys.stdin.readline()
sock.sendto(data, target)
def recv_msg(sock):
while True:
data, addr = sock.recvfrom(1024)
sys.stdout.write(data)
Thread(target=send_msg, args=(sock_send,)).start()
Thread(target=recv_msg, args=(sockfd,)).start()
I am trying to run the following python server under windows:
"""
An echo server that uses select to handle multiple clients at a time.
Entering any line of input at the terminal will exit the server.
"""
import select
import socket
import sys
host = ''
port = 50000
backlog = 5
size = 1024
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(backlog)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
server.close()
I get the error message (10038, 'An operation was attempted on something that is not a socket'). This probably relates back to the remark in the python documentation that "File objects on Windows are not acceptable, but sockets are. On Windows, the underlying select() function is provided by the WinSock library, and does not handle file descriptors that don’t originate from WinSock.". On internet there are quite some posts on this topic, but they are either too technical for me or simply not clear. So my question is: is there any way the select() statement in python can be used under windows? Please add a little example or modify my code above. Thanks!
Look like it does not like sys.stdin
If you change input to this
input = [server]
the exception will go away.
This is from the doc
Note:
File objects on Windows are not acceptable, but sockets are. On Windows, the
underlying select() function is provided by the WinSock library, and does not
handle file descriptors that don’t originate from WinSock.
I don't know if your code has other problems, but the error you're getting is because of passing input to select.select(), the problem is that it contains sys.stdin which is not a socket. Under Windows, select only works with sockets.
As a side note, input is a python function, it's not a good idea to use it as a variable.
Of course and the answers given are right...
you just have to remove the sys.stdin from the input but still use it in the iteration:
for s in inputready+[sys.stdin]:
I will write a SSH communicator class on Python. I have telnet communicator class and I should use functions like at telnet. Telnet communicator have read_until and read_very_eager functions.
read_until : Read until a given string is encountered or until timeout.
read_very_eager : Read everything that's possible without blocking in I/O (eager).
I couldn't find these functions for SSH communicator. Any idea?
You didn't state it in the question, but I am assuming you are using Paramiko as per the tag.
read_until: Read until a given string is encountered or until timeout.
This seems like a very specialized function for a particular high level task. I think you will need to implement this one. You can set a timeout using paramiko.Channel.settimeout and then read in a loop until you get either the string you want or a timeout exception.
read_very_eager: Read everything that's possible without blocking in I/O (eager).
Paramiko doesn't directly provide this, but it does provide primitives for non-blocking I/O and you can easily put this in a loop to slurp in everything that's available on the channel. Have you tried something like this?
channel.setblocking(True)
resultlist = []
while True:
try:
chunk = channel.recv(1024)
except socket.timeout:
break
resultlist.append(chunk)
return ''.join(resultlist)
Hi there even i was searching solution for the same problem.
I think it might help you ....
one observation, tell me if you find solution.
I wont get output if i remove 6th line.
I was actually printing 6th line to know the status, later i found recv_exit_status() should be called for execution of this code.
import paramiko,sys
trans = paramiko.Transport((host, 22))
trans.connect(username = user, password = passwd)
session = trans.open_channel("session")
session.exec_command('grep -rE print .')
session.recv_exit_status()
while session.recv_ready():
temp = session.recv(1024)
print temp
1.Read until > search for the data you are searching for and break the loop
2.Read_very_eager > use the above mentioned code.
I'm trying to use a unix named pipe to output statistics of a running service. I intend to provide a similar interface as /proc where one can see live stats by catting a file.
I'm using a code similar to this in my python code:
while True:
f = open('/tmp/readstatshere', 'w')
f.write('some interesting stats\n')
f.close()
/tmp/readstatshere is a named pipe created by mknod.
I then cat it to see the stats:
$ cat /tmp/readstatshere
some interesting stats
It works fine most of the time. However, if I cat the entry several times in quick successions, sometimes I get multiple lines of some interesting stats instead of one. Once or twice, it has even gone into an infinite loop printing that line forever until I killed it. The only fix that I've got so far is to put a delay of let's say 500ms after f.close() to prevent this issue.
I'd like to know why exactly this happens and if there is a better way of dealing with it.
Thanks in advance
A pipe is simply the wrong solution here. If you want to present a consistent snapshot of the internal state of your process, write that to a temporary file and then rename it to the "public" name. This will prevent all issues that can arise from other processes reading the state while you're updating it. Also, do NOT do that in a busy loop, but ideally in a thread that sleeps for at least one second between updates.
What about a UNIX socket instead of a pipe?
In this case, you can react on each connect by providing fresh data just in time.
The only downside is that you cannot cat the data; you'll have to create a new socket handle and connect() to the socket file.
MYSOCKETFILE = '/tmp/mysocket'
import socket
import os
try:
os.unlink(MYSOCKETFILE)
except OSError: pass
s = socket.socket(socket.AF_UNIX)
s.bind(MYSOCKETFILE)
s.listen(10)
while True:
s2, peeraddr = s.accept()
s2.send('These are my actual data')
s2.close()
Program querying this socket:
MYSOCKETFILE = '/tmp/mysocket'
import socket
import os
s = socket.socket(socket.AF_UNIX)
s.connect(MYSOCKETFILE)
while True:
d = s.recv(100)
if not d: break
print d
s.close()
I think you should use fuse.
it has python bindings, see http://pypi.python.org/pypi/fuse-python/
this allows you to compose answers to questions formulated as posix filesystem system calls
Don't write to an actual file. That's not what /proc does. Procfs presents a virtual (non-disk-backed) filesystem which produces the information you want on demand. You can do the same thing, but it'll be easier if it's not tied to the filesystem. Instead, just run a web service inside your Python program, and keep your statistics in memory. When a request comes in for the stats, formulate them into a nice string and return them. Most of the time you won't need to waste cycles updating a file which may not even be read before the next update.
You need to unlink the pipe after you issue the close. I think this is because there is a race condition where the pipe can be opened for reading again before cat finishes and it thus sees more data and reads it out, leading to multiples of "some interesting stats."
Basically you want something like:
while True:
os.mkfifo(the_pipe)
f = open(the_pipe, 'w')
f.write('some interesting stats')
f.close()
os.unlink(the_pipe)
Update 1: call to mkfifo
Update 2: as noted in the comments, there is a race condition in this code as well with multiple consumers.