We Keep Coding

Multithreading gethostbyaddr causes wifi to drop

So I'm writing a python script that takes a large amount of IPs in a set (~20k) and the goal is to resolve the hostname of each IP using gethostbyaddr. In an attempt to make the script run as fast as possible I implemented multithreading. Note that my computer running the script is on a wired ethernet connection. The issue is that while running the script, I've noticed that my wifi connection drops (on my phone, on every other household device) however the script will continue to run fine and once completed, wifi is restored. I've tweaked around with the amount of threads active at once and this is likely the issue. If I set the limit to say 500 threads, the script will complete in 2 minutes, but the wifi will be dropped. If I lower it down to <50 threads the wifi is fine, but the script will take ages to complete. As the limit goes higher, the wifi connection suffers until a point of just dropping. So I see that I am essentially DDoSing myself so my question is if there is a way to efficiently complete the script without bottlenecking my wifi. Or maybe there is something wrong with my implementation. My code is below:
def lookup_IP(ip):
result = tuple()
try:
host, aliases, _ = socket.gethostbyaddr(ip)
result = (ip, host)
except socket.herror:
result = (ip, 'no host found')
except Exception:
result = (ip, 'error finding host')
return result
with futures.ThreadPoolExecutor(500) as executor:
submitted_threads = {executor.submit(lookup_IP, ip): ip for ip in all_IPs}
for thread in futures.as_completed(submitted_threads):
try:
data = thread.result()
#add data to a dictionary
except Exception as err:
#print error
#rest of code

python socket programming for transferring a photo

I'm new to socket programming in python. Here is an example of opening a TCP socket in a Mininet host and sending a photo from one host to another. In fact I changed the code that I had used to send a simple message to another host (writing the received data to a text file) in order to meet my requirements. Although when I implement this revised code, there is no error and it seems to transfer correctly, I am not sure whether this is a correct way to do this transmission or not. Since I'm running both hosts on the same machine, I thought it may have an influence on the result. I wanted to ask you to check whether this is a correct way to transfer or I should add or remove something.
mininetSocketTest.py
#!/usr/bin/python
from mininet.topo import Topo, SingleSwitchTopo
from mininet.net import Mininet
from mininet.log import lg, info
from mininet.cli import CLI
def main():
lg.setLogLevel('info')
net = Mininet(SingleSwitchTopo(k=2))
net.start()
h1 = net.get('h1')
p1 = h1.popen('python myClient2.py')
h2 = net.get('h2')
h2.cmd('python myServer2.py')
CLI( net )
#p1.terminate()
net.stop()
if __name__ == '__main__':
main()
myServer2.py
import socket
import sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('10.0.0.1', 12345))
buf = 1024
f = open("2.jpg",'wb')
s.listen(1)
conn , addr = s.accept()
while 1:
data = conn.recv(buf)
print(data[:10])
#print "PACKAGE RECEIVED..."
f.write(data)
if not data: break
#conn.send(data)
conn.close()
s.close()
myClient2.py:
import socket
import sys
f=open ("1.jpg", "rb")
print sys.getsizeof(f)
buf = 1024
data = f.read(buf)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('10.0.0.1',12345))
while (data):
if(s.sendall(data)):
#print "sending ..."
data = f.read(buf)
print(f.tell(), data[:10])
else:
s.close()
s.close()

This loop in client2 is wrong:
while (data):
if(s.send(data)):
print "sending ..."
data = f.read(buf)
As the send
docs say:
Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data. For further information on this topic, consult the Socket Programming HOWTO.
You're not even attempting to do this. So, while it probably works on localhost, on a lightly-loaded machine, with smallish files, it's going to break as soon as you try to use it for real.
As the help says, you need to do something to deliver the rest of the buffer. Since there's probably no good reason you can't just block until it's all sent, the simplest thing to do is to call sendall:
Unlike send(), this method continues to send data from bytes until either all data has been sent or an error occurs. None is returned on success. On error, an exception is raised…
And this brings up the next problem: You're not doing any exception handling anywhere. Maybe that's OK, but usually it isn't. For example, if one of your sockets goes down, but the other one is still up, do you want to abort the whole program and hard-drop your connection, or do you maybe want to finish sending whatever you have first?
You should at least probably use a with clause of a finally, to make sure you close your sockets cleanly, so the other side will get a nice EOF instead of an exception.
Also, your server code just serves a single client and then quits. Is that actually what you wanted? Usually, even if you don't need concurrent clients, you at least want to loop around accepting and servicing them one by one.
Finally, a server almost always wants to do this:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Without this, if you try to run the server again within a few seconds after it finished (a platform-specific number of seconds, which may even depend whether it finished with an exception instead of a clean shutdown), the bind will fail, in the same way as if you tried to bind a socket that's actually in use by another program.

First of all, you should use TCP and not UDP. TCP will ensure that your client/server has received the whole photo properly. UDP is more used for content streaming.
Absolutely not your use case.

Can select() be used with files in Python under Windows?

I am trying to run the following python server under windows:
"""
An echo server that uses select to handle multiple clients at a time.
Entering any line of input at the terminal will exit the server.
"""
import select
import socket
import sys
host = ''
port = 50000
backlog = 5
size = 1024
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(backlog)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
server.close()
I get the error message (10038, 'An operation was attempted on something that is not a socket'). This probably relates back to the remark in the python documentation that "File objects on Windows are not acceptable, but sockets are. On Windows, the underlying select() function is provided by the WinSock library, and does not handle file descriptors that don’t originate from WinSock.". On internet there are quite some posts on this topic, but they are either too technical for me or simply not clear. So my question is: is there any way the select() statement in python can be used under windows? Please add a little example or modify my code above. Thanks!

Look like it does not like sys.stdin
If you change input to this
input = [server]
the exception will go away.
This is from the doc
Note:
File objects on Windows are not acceptable, but sockets are. On Windows, the
underlying select() function is provided by the WinSock library, and does not
handle file descriptors that don’t originate from WinSock.

I don't know if your code has other problems, but the error you're getting is because of passing input to select.select(), the problem is that it contains sys.stdin which is not a socket. Under Windows, select only works with sockets.
As a side note, input is a python function, it's not a good idea to use it as a variable.

Of course and the answers given are right...
you just have to remove the sys.stdin from the input but still use it in the iteration:
for s in inputready+[sys.stdin]:

Python for loop slows and evenutally hangs

I'm totally new to Python (as of half an hour ago) and trying to write a simple script to enumerate users on an SMTP server.
The users file is a simple list (one per line) of usernames.
The script runs fine but with each iteration of the loop it slows until, around loop 14, it seems to hang completely. No error - I have to ^c.
Can anyone shed some light on the problem please?
TIA,
Tom
#!/usr/bin/python
import socket
import sys
if len(sys.argv) != 2:
print "Usage: vrfy.py <username file>"
sys.exit(0)
#open user file
file=open(sys.argv[1], 'r')
users=[x.strip() for x in file.readlines()]
file.close
#Just for debugging
print users
# Create a Socket
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Connect to the Server
connect=s.connect(('192.168.13.222',25))
for x in users:
# VRFY a user
s.send('VRFY ' + x + '\r\n')
result=s.recv(1024)
print result
# Close the socket
s.close()

Most likely your SMTP server is tarpitting your client connection. This is a defense against runaway clients, or clients which submit large volumes of "junk" commands. From the manpage for Postfix smtpd:
smtpd_junk_command_limit (normal: 100, stress: 1)
The number of junk commands (NOOP, VRFY, ETRN or RSET) that a
remote SMTP client can send before the Postfix SMTP server
starts to increment the error counter with each junk command.
The smtpd daemon will insert a 1-second delay before replying after a certain amount of junk is seen. If you have root access to the smtp server in question, try an strace to see if nanosleep syscalls are being issued by the server.
Here is a trace from running your script against my local server. After 100 VRFY commands it starts sleeping between commands. Your server may have a lower limit of ~15 junk commands:
nanosleep({1, 0}, 0x7fffda9a67a0) = 0
poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
write(9, "252 2.0.0 pat\r\n", 15) = 15
poll([{fd=9, events=POLLIN}], 1, 300000) = 1 ([{fd=9, revents=POLLIN}])
read(9, "VRFY pat\r\n", 4096) = 10

s.recv blocks so if you have no more data on the socket then it will block forever.
You have to keep track of how much data you are receiving. You need to know this ahead of time so the client and the server can agree on the size.

Solving the exact same problem I also ran into the issue.
I'm almost sure #samplebias is right. I found I could work around the "tarpitting" by abusing the poor system even more, tearing down and rebuilding every connection:
#[ ...Snip... ]
import smtplib
#[ ...Snip... ]
for USER in open(opts.USERS,'r'):
smtpserver = smtplib.SMTP(HOST,PORT)
smtpserver.ehlo()
verifyuser = smtpserver.verify(USER)
print("%s %s: %s") % (HOST.rstrip(), USER.rstrip(), verifyuser)
smtpserver.quit()
I'm curious whether this particular type of hammering would work in a live environment, but too certain it would make some people very unhappy.
PS, python: batteries included.

In a glance, your code has no bugs. However, you shall notice that TCP isn't a "message" oriented protocol. So, you can't use socket.send in a loop assuming that one message will be actually sent through the medium at every call. Thus, if some calls starts to get buffered in the output buffer, and you just call socket.recv after it, your program will stuck in a deadlock.
What you should do is a threaded or asynchronous code. Maybe Twisted Framework may help you.

get open TCP port in Python [duplicate]

This question already has answers here:
On localhost, how do I pick a free port number?
(5 answers)
Closed 1 year ago.
I want to get any random open TCP port on localhost in Python. What is the easiest way?

My current solution:
def get_open_port():
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("",0))
s.listen(1)
port = s.getsockname()[1]
s.close()
return port
Not very nice and also not 100% correct but it works for now.

The free port can be found by binding a socket to a port selected by the operating system. After the operating system selects a port the socket can be disposed. However, this solution is not resistant to race conditions - in the short time between getting the free port number and using this port other process may use this port.
import socket
def find_free_port():
with socket.socket() as s:
s.bind(('', 0)) # Bind to a free port provided by the host.
return s.getsockname()[1] # Return the port number assigned.
https://godbolt.org/z/fs13K13dG

I actually use the following in one of my programs:
port = random.randint(10000,60000)
Of course, this is even more prone to collisions than the code you have. But I've never had a problem with it. The point is, at any given time, most of those high-numbered ports are not in use and if you just pick one at random, having a conflict with another process is pretty unlikely. If you do something like the solution you posted in your answer (opening a socket and grabbing its port number), it's almost certain that the port isn't going to conflict. So if this is something that you'll only be using for yourself (as opposed to something you're going to release to the public), think about whether it's worth coming up with a truly bulletproof solution. Odds are it'll never make a difference.
Motivated by Marcelo Cantos' comment on your question, I will add that the standard solution in cases like this is to have the process that will be using the port bind to it and then share that information with any other program that needs it. Typically it'll do something like writing a temporary file containing the port number to some standard location in the filesystem. Since the process you're working with doesn't do that, in some sense whatever solution you come up with will be a bit of a hack. But again, if it's just for your own use, that's probably fine.

The ephemeral ports basically lie in range 49152 - 65535.
if you want to check ports in bigger range then just change the values in randint.
import pustil
from random import randint
def getfreeport():
port = randint(49152,65535)
portsinuse=[]
while True:
conns = pstuil.net_connections()
for conn in conns:
portsinuse.append(con.laddr[1])
if port in portsinuse:
port = randint(49152,65535)
else:
break
return port

This is my version, however its not really random if you specify a port range. This will also suffer from race conditions, but this is the best way I know if you need to know the port ahead of time.
import socket
import errno
import contextlib
reserved_ports = set()
def get_open_port(lowest_port = 0, highest_port = None, bind_address = '', *socket_args, **socket_kwargs):
if highest_port is None:
highest_port = lowest_port + 100
while lowest_port < highest_port:
if lowest_port not in reserved_ports:
try:
with contextlib.closing(socket.socket(*socket_args, **socket_kwargs)) as my_socket:
my_socket.bind((bind_address, lowest_port))
this_port = my_socket.getsockname()[1]
reserved_ports.add(this_port)
return this_port
except socket.error as error:
if not error.errno == errno.EADDRINUSE:
raise
assert not lowest_port == 0
reserved_ports.add(lowest_port)
lowest_port += 1
raise Exception('Could not find open port')