We Keep Coding

Scapy variable sniff stop

i found a similar problem:
(Instance variables not being updated Python when using Multiprocessing),
but still do not know the solutionn for my task.
The task is to stop a scapy sniff function after the completness of a testskript. the running duration of single testscripts can vary greatly (from some seconds till hours). My sniff function runs in a separate threat. The testscript calls an init Funktion in the beginning which calls the sniff Function from an other modul.
#classmethod
def SaveFullTrafficPcap(self, TestCase, Termination):
try:
Full_Traffic = []
PktList = []
FullPcapName = Settings['GeneralSettings']['ResultsPath']+TestCase.TestCaseName +"Full_Traffic_PCAP.pcap"
#while Term.Termination < 1:
Full_Traffic = sniff(lfilter=None, iface=str(Settings['GeneralSettings']['EthInterface']), store=True, prn = lambda x: Full_Traffic.append(x), count=0, timeout=Term.Termination)
print(Full_Traffic)
wrpcap(FullPcapName, Full_Traffic)
except(Exception):
SYS.ABS_print("No full traffic PCAP file wirtten!\n")
At the end of the testscript an exit function is called. In the exit function I set Term.Termination parameter to 1 and wait for 5 sec, but it doesnt work. The sniff function is stoped by the system and i get no file"FullPCAPName"
If count or timeout get a value, the code works without problemms and i get my FullPCAPName file with he complet traffic on my Interface.
Have anybody hinds how i can stopt the sniff function regulary after finisching the testscript?

Use of the stop_filter command as specified here worked for me. I've duplicated HenningCash's code below for convenience:
import time, threading
from scapy.all import sniff
e = threading.Event()
def _sniff(e):
a = sniff(filter="tcp port 80", stop_filter=lambda p: e.is_set())
print("Stopped after %i packets" % len(a))
print("Start capturing thread")
t = threading.Thread(target=_sniff, args=(e,))
t.start()
time.sleep(3)
print("Try to shutdown capturing...")
e.set()
# This will run until you send a HTTP request somewhere
# There is no way to exit clean if no package is received
while True:
t.join(2)
if t.is_alive():
print("Thread is still running...")
else:
break
print("Shutdown complete!")
However, you still have to wait for a final packet to be sniffed, which might not be ideal in your scenario.

now i solved the problem with global variables. It is not nice, but it works well.
Nevertheless I am interested in a better solution for the variable sniff stop.

stop_var = ""
def stop():
global stop_var
stop_var.stop()
def start():
"""
your code
"""
global stop_var
stop_var = AsyncSniffer(**arg)
stop_var=start()

Multiple stdout w/ flush going on in Python threading

I have a small piece of code that I made to test out and hopefully debug the problem without having to modify the code in my main applet in Python. This has let me to build this code:
#!/usr/bin/env python
import sys, threading, time
def loop1():
count = 0
while True:
sys.stdout.write('\r thread 1: ' + str(count))
sys.stdout.flush()
count = count + 1
time.sleep(.3)
pass
pass
def loop2():
count = 0
print ""
while True:
sys.stdout.write('\r thread 2: ' + str(count))
sys.stdout.flush()
count = count + 2
time.sleep(.3)
pass
if __name__ == '__main__':
try:
th = threading.Thread(target=loop1)
th.start()
th1 = threading.Thread(target=loop2)
th1.start()
pass
except KeyboardInterrupt:
print ""
pass
pass
My goal with this code is to be able to have both of these threads displaying output in stdout format (with flushing) at the same time and have then side by side or something. problem is that I assume since it is flushing each one, it flushes the other string by default. I don't quite know how to get this to work if it is even possible.
If you just run one of the threads, it works fine. However I want to be able to run both threads with their own string running at the same time in the terminal output. Here is a picture displaying what I'm getting:
terminal screenshot
let me know if you need more info. thanks in advance.

Instead of allowing each thread to output to stdout, a better solution is to have one thread control stdout exclusively. Then provide a threadsafe channel for the other threads to dispatch data to be output.
One good method to achieve this is to share a Queue between all threads. Ensure that only the output thread is accessing data after it has been added to the queue.
The output thread can store the last message from each other thread and use that data to format stdout nicely. This can include clearing output to display something like this, and update it as each thread generates new data.
Threads
#1: 0
#2: 0
Example
Some decisions were made to simplify this example:
There are gotchas to be wary of when giving arguments to threads.
Daemon threads terminate themselves when the main thread exits. They are used to avoid adding complexity to this answer. Using them on long-running or large applications can pose problems. Other
questions discuss how to exit a multithreaded application without leaking memory or locking system resources. You will need to think about how your program needs to signal an exit. Consider using asyncio to save yourself these considerations.
No newlines are used because \r carriage returns cannot clear the whole console. They only allow the current line to be rewritten.
import queue, threading
import time, sys
q = queue.Queue()
keepRunning = True
def loop_output():
thread_outputs = dict()
while keepRunning:
try:
thread_id, data = q.get_nowait()
thread_outputs[thread_id] = data
except queue.Empty:
# because the queue is used to update, there's no need to wait or block.
pass
pretty_output = ""
for thread_id, data in thread_outputs.items():
pretty_output += '({}:{}) '.format(thread_id, str(data))
sys.stdout.write('\r' + pretty_output)
sys.stdout.flush()
time.sleep(1)
def loop_count(thread_id, increment):
count = 0
while keepRunning:
msg = (thread_id, count)
try:
q.put_nowait(msg)
except queue.Full:
pass
count = count + increment
time.sleep(.3)
pass
pass
if __name__ == '__main__':
try:
th_out = threading.Thread(target=loop_output)
th_out.start()
# make sure to use args, not pass arguments directly
th0 = threading.Thread(target=loop_count, args=("Thread0", 1))
th0.daemon = True
th0.start()
th1 = threading.Thread(target=loop_count, args=("Thread1", 3))
th1.daemon = True
th1.start()
# Keep the main thread alive to wait for KeyboardInterrupt
while True:
time.sleep(.1)
except KeyboardInterrupt:
print("Ended by keyboard stroke")
keepRunning = False
for th in [th0, th1]:
th.join()
Example Output:
(Thread0:110) (Thread1:330)

Remove threads usage from script

The next script I'm using is used to listen to IMAP connection using IMAP IDLE and depends heavily on threads. What's the easiest way for me to eliminate the treads call and just use the main thread?
As a new python developer I tried editing def __init__(self, conn): method but just got more and more errors
A code sample would help me a lot
#!/usr/local/bin/python2.7
print "Content-type: text/html\r\n\r\n";
import socket, ssl, json, struct, re
import imaplib2, time
from threading import *
# enter gmail login details here
USER="username#gmail.com"
PASSWORD="password"
# enter device token here
deviceToken = 'my device token x x x x x'
deviceToken = deviceToken.replace(' ','').decode('hex')
currentBadgeNum = -1
def getUnseen():
(resp, data) = M.status("INBOX", '(UNSEEN)')
print data
return int(re.findall("UNSEEN (\d)*\)", data[0])[0])
def sendPushNotification(badgeNum):
global currentBadgeNum, deviceToken
if badgeNum != currentBadgeNum:
currentBadgeNum = badgeNum
thePayLoad = {
'aps': {
'alert':'Hello world!',
'sound':'',
'badge': badgeNum,
},
'test_data': { 'foo': 'bar' },
}
theCertfile = 'certif.pem'
theHost = ('gateway.push.apple.com', 2195)
data = json.dumps(thePayLoad)
theFormat = '!BH32sH%ds' % len(data)
theNotification = struct.pack(theFormat, 0, 32,
deviceToken, len(data), data)
ssl_sock = ssl.wrap_socket(socket.socket(socket.AF_INET,
socket.SOCK_STREAM), certfile=theCertfile)
ssl_sock.connect(theHost)
ssl_sock.write(theNotification)
ssl_sock.close()
print "Sent Push alert."
# This is the threading object that does all the waiting on
# the event
class Idler(object):
def __init__(self, conn):
self.thread = Thread(target=self.idle)
self.M = conn
self.event = Event()
def start(self):
self.thread.start()
def stop(self):
# This is a neat trick to make thread end. Took me a
# while to figure that one out!
self.event.set()
def join(self):
self.thread.join()
def idle(self):
# Starting an unending loop here
while True:
# This is part of the trick to make the loop stop
# when the stop() command is given
if self.event.isSet():
return
self.needsync = False
# A callback method that gets called when a new
# email arrives. Very basic, but that's good.
def callback(args):
if not self.event.isSet():
self.needsync = True
self.event.set()
# Do the actual idle call. This returns immediately,
# since it's asynchronous.
self.M.idle(callback=callback)
# This waits until the event is set. The event is
# set by the callback, when the server 'answers'
# the idle call and the callback function gets
# called.
self.event.wait()
# Because the function sets the needsync variable,
# this helps escape the loop without doing
# anything if the stop() is called. Kinda neat
# solution.
if self.needsync:
self.event.clear()
self.dosync()
# The method that gets called when a new email arrives.
# Replace it with something better.
def dosync(self):
print "Got an event!"
numUnseen = getUnseen()
sendPushNotification(numUnseen)
# Had to do this stuff in a try-finally, since some testing
# went a little wrong.....
while True:
try:
# Set the following two lines to your creds and server
M = imaplib2.IMAP4_SSL("imap.gmail.com")
M.login(USER, PASSWORD)
M.debug = 4
# We need to get out of the AUTH state, so we just select
# the INBOX.
M.select("INBOX")
numUnseen = getUnseen()
sendPushNotification(numUnseen)
typ, data = M.fetch(1, '(RFC822)')
raw_email = data[0][1]
import email
email_message = email.message_from_string(raw_email)
print email_message['Subject']
#print M.status("INBOX", '(UNSEEN)')
# Start the Idler thread
idler = Idler(M)
idler.start()
# Sleep forever, one minute at a time
while True:
time.sleep(60)
except imaplib2.IMAP4.abort:
print("Disconnected. Trying again.")
finally:
# Clean up.
#idler.stop() #Commented out to see the real error
#idler.join() #Commented out to see the real error
#M.close() #Commented out to see the real error
# This is important!
M.logout()

As far as I can tell, this code is hopelessly confused because the author used the "imaplib2" project library which forces a threading model which this code then never uses.
Only one thread is ever created, which wouldn't need to be a thread but for the choice of imaplib2. However, as the imaplib2 documentation notes:
This module presents an almost identical API as that provided by the standard python library module imaplib, the main difference being that this version allows parallel execution of commands on the IMAP4 server, and implements the IMAP4rev1 IDLE extension. (imaplib2 can be substituted for imaplib in existing clients with no changes in the code, but see the caveat below.)
Which makes it appear that you should be able to throw out much of class Idler and just use the connection M. I recommend that you look at Doug Hellman's excellent Python Module Of The Week for module imaplib prior to looking at the official documentation. You'll need to reverse engineer the code to find out its intent, but it looks to me like:
Open a connection to GMail
check for unseen messages in Inbox
count unseen messages from (2)
send a dummy message to some service at gateway.push.apple.com
Wait for notice, goto (2)
Perhaps the most interesting thing about the code is that it doesn't appear to do anything, although what sendPushNotification (step 4) does is a mystery, and the one line that uses an imaplib2 specific service:
self.M.idle(callback=callback)
uses a named argument that I don't see in the module documentation. Do you know if this code ever actually ran?
Aside from unneeded complexity, there's another reason to drop imaplib2: it exists independently on sourceforge and PyPi which one maintainer claimed two years ago "An attempt will be made to keep it up-to-date with the original". Which one do you have? Which would you install?

Don't do it
Since you are trying to remove the Thread usage solely because you didn't find how to handle the exceptions from the server, I don't recommend removing the Thread usage, because of the async nature of the library itself - the Idler handles it more smoothly than a one thread could.
Solution
You need to wrap the self.M.idle(callback=callback) with try-except and then re-raise it in the main thread. Then you handle the exception by re-running the code in the main thread to restart the connection.
You can find more details of the solution and possible reasons in this answer: https://stackoverflow.com/a/50163971/1544154
Complete solution is here: https://www.github.com/Elijas/email-notifier

python watchdog for threads

Im writing simple app, which reads (about a million) lines from file, copy those lines into list, and if next line will be different then previous it runs a thread, to do some job with that list. Thread job is based on tcp sockets, sending and receiving commands via telnet lib.
Sometimes my application hangs and does nothing. All telnet operations I wrapped into try-except statements, also read and write into sockets has timeouts.
I thought about writing watchdog, which will do sys.exit() or something similiar on that hang condtition. But, for now I'm thinking how to create it, and still got no idea how to do it. So if You can trace me, it would be great.
For that file I'm creating 40 threads. Pseudo code looks:
lock = threading.Lock()
no_of_jobs = 0
class DoJob(threading.Thread):
def start(self, cond, work):
self.work = work
threading.Thread.start(self)
def run(self)
global lock
global no_of_jobs
lock.acquire()
no_of_jobs += 1
lock.release()
# do some job, if error or if finished, decrement no_of_jobs under lock
(...)
main:
#starting conditions:
with open(sys.argv[1]) as targetsfile:
head = [targetsfile.next() for x in xrange(1)]
s = head[0]
prev_cond = s[0]
work = []
for line in open(sys.argv[1], "r"):
cond = line([0])
if prev_cond != cond:
while(no_of_jobs>= MAX_THREADS):
time.sleep(1)
DoJob(cond, work)
prev_cond = cond
work = None
work = []
work.append(line)
#last job:
DoJob(cond, work)
while threading.activeCount() > 1:
time.sleep(1)
best regards
J

I have successfully used code like below in the past (from a python 3 program I wrote):
import threading
def die():
print('ran for too long. quitting.')
for thread in threading.enumerate():
if thread.isAlive():
try:
thread._stop()
except:
pass
sys.exit(1)
if __name__ == '__main__':
#bunch of app-specific code...
# setup max runtime
die = threading.Timer(2.0, die) #quit after 2 seconds
die.daemon = True
die.start()
#after work is done
die.cancel()

Help adding threading for GUI progress

I have an FTP function that traces the progress of running upload but my understanding of threading is limited and i have been unable to implement a working solution... I'd like to add a GUI progress bar to my current Application by using threading. Can someone show me a basic function using asynchronous threads that can be updated from another running thread?
def ftpUploader():
BLOCKSIZE = 57344 # size 56 kB
ftp = ftplib.FTP()
ftp.connect(host)
ftp.login(login, passwd)
ftp.voidcmd("TYPE I")
f = open(zipname, 'rb')
datasock, esize = ftp.ntransfercmd(
'STOR %s' % os.path.basename(zipname))
size = os.stat(zipname)[6]
bytes_so_far = 0
print 'started'
while 1:
buf = f.read(BLOCKSIZE)
if not buf:
break
datasock.sendall(buf)
bytes_so_far += len(buf)
print "\rSent %d of %d bytes %.1f%%\r" % (
bytes_so_far, size, 100 * bytes_so_far / size)
sys.stdout.flush()
datasock.close()
f.close()
ftp.voidresp()
ftp.quit()
print 'Complete...'

Here's a quick overview of threading, just in case :) I won't go into too much detail into the GUI stuff, other than to say that you should check out wxWidgets. Whenever you do something that takes a long time, like:
from time import sleep
for i in range(5):
sleep(10)
You'll notice that to the user, the entire block of code seems to take 50 seconds. In those 5 seconds, your application can't do anything like update the interface, and so it looks like it's frozen. To solve this problem, we use threading.
Usually there are two parts to this problem; the overall set of things you want to process, and the operation that takes a while, that we'd like to chop up. In this case, the overall set is the for loop and the operation we want chopped up is the sleep(10) function.
Here's a quick template for the threading code, based on our previous example. You should be able to work your code into this example.
from threading import Thread
from time import sleep
# Threading.
# The amount of seconds to wait before checking for an unpause condition.
# Sleeping is necessary because if we don't, we'll block the os and make the
# program look like it's frozen.
PAUSE_SLEEP = 5
# The number of iterations we want.
TOTAL_ITERATIONS = 5
class myThread(Thread):
'''
A thread used to do some stuff.
'''
def __init__(self, gui, otherStuff):
'''
Constructor. We pass in a reference to the GUI object we want
to update here, as well as any other variables we want this
thread to be aware of.
'''
# Construct the parent instance.
Thread.__init__(self)
# Store the gui, so that we can update it later.
self.gui = gui
# Store any other variables we want this thread to have access to.
self.myStuff = otherStuff
# Tracks the paused and stopped states of the thread.
self.isPaused = False
self.isStopped = False
def pause(self):
'''
Called to pause the thread.
'''
self.isPaused = True
def unpause(self):
'''
Called to unpause the thread.
'''
self.isPaused = False
def stop(self):
'''
Called to stop the thread.
'''
self.isStopped = True
def run(self):
'''
The main thread code.
'''
# The current iteration.
currentIteration = 0
# Keep going if the job is active.
while self.isStopped == False:
try:
# Check for a pause.
if self.isPaused:
# Sleep to let the os schedule other tasks.
sleep(PAUSE_SLEEP)
# Continue with the loop.
continue
# Check to see if we're still processing the set of
# things we want to do.
if currentIteration < TOTAL_ITERATIONS:
# Do the individual thing we want to do.
sleep(10)
# Update the count.
currentIteration += 1
# Update the gui.
self.gui.update(currentIteration,TOTAL_ITERATIONS)
else:
# Stop the loop.
self.isStopped = True
except Exception as exception:
# If anything bad happens, report the error. It won't
# get written to stderr.
print exception
# Stop the loop.
self.isStopped = True
# Tell the gui we're done.
self.gui.stop()
To call this thread, all you have to do is:
aThread = myThread(myGui,myOtherStuff)
aThread.start()