I've come across an unusual problem in regards to updating variables. I've built a simple class object to help me with some network sniffing. I wanted to make a parallel process which allows me to run some network tests and capture the traffic generated using python so I can extend the program to do amazing things. I'm using scapy's sniffing function to help with the interface sniffing.
Scapy's sniffer allows you to pass a function into itself function that allows you to create a 'stop sniffing' condition. In my case I've created function stop_filter and I wish to stop the Scapy sniff function by simply updating the self.stop_sniffing instance variable. I've presented the program output below, which shows self.stop_sniffing getting set to True in Function stop, but is then set back to False (or is not updated at all) when printed in stop_filter. I have no clue why this is happening and no solution comes to mind as it's such a weird problem.
If anyone with fresh eyes can see what insane thing I've done here it would be greatly appreciated!
from scapy.all import *
from multiprocessing import Process
class DatasetSniffer:
def __init__(self, iface, local_dir='.'):
self.iface = iface
self.master = None
self.local_dir = local_dir
self.stop_sniffing = False # Never updates! why!?
self.writer = PcapWriter(local_dir+"/master.pcap", append=True, sync=True)
def stop_filter(self, p):
# Note: 'p' gets passed in by Scapy function 'sniff'
print self.stop_sniffing
# Return 'True' to stop sniffer
return self.stop_sniffing
def sniff(self):
sniff(store=0, prn=self.writer.write, iface=self.iface, stop_filter=self.stop_filter)
def start(self):
self.master = Process(target=self.sniff)
self.master.start()
def stop(self):
self.stop_sniffing = True
# Shows that self.stop_sniffing is 'True'
print self.stop_sniffing
self.master.join()
if __name__ == "__main__":
interface = 'en3'
sniffer = DatasetSniffer(interface)
sniffer.start()
# some process
time.sleep(5)
sniffer.stop()
Shell output:
sudo python sniffing.py
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
True
False
False
False
False
The Problem
You are not using multiple threads in this example code you are using multiple processes.
Here you have two separate processes, that do not share memory:
the original process
a new process, started by multiprocessing.Process.start
this process will have been started by forking the original process, creating a copy of its memory at the time of the fork. They do not "share" memory.
Now, when you call DatasetSniffer.stop within your original process, this will not alter the value of stop_sniffing in the new ("master") process.
How to Communicate Then?
When using multiprocessing, you can communicate using a Pipe. Something like this:
readable_pipe, writable_pipe = multiprocessing.Pipe(duplex=False)
process = Process(target=do_something)
Now, our original process can send a message by writing to the pipe:
writable_pipe.send("stop")
while the new process can check for messages using:
if readable_pipe.poll():
msg = readable_pipe.recv()
Try working this into your code.
Thanks for all your suggestions. After a glass of inspiration I managed to knock up this script. Probably a nicer way to approach my problem without making too many changes. So this code allows the threads to use the stop function outside the class, thus allowing all the asynchronous tasks to use the stop_filter.
Found this information in the link below. Hopfully this post will be useful to someone else!
http://www.tutorialspoint.com/python/python_multithreading.htm
Cheers!
import threading
from scapy.all import *
from datetime import datetime
directory = str(datetime.now().strftime("%Y%m%d%H%M%S"))
os.makedirs(directory)
DatasetSnifferExit = 0
class DatasetSniffer(threading.Thread):
def __init__(self, iface, local_dir='.', filename=str(datetime.now())):
self.iface = iface
self.filename = filename
self.local_dir = local_dir
self.stop_sniffing = False
self.writer = PcapWriter(local_dir+"/"+filename+".pcap", append=True, sync=True)
threading.Thread.__init__(self)
def run(self):
sniff_interface(self.writer.write, self.iface)
def stop_filter(p):
if DatasetSnifferExit:
return True
else:
return False
def sniff_interface(write, iface):
sniff(store=0, prn=write, iface=iface, stop_filter=stop_filter)
if __name__ == "__main__":
DatasetSnifferExit = False
# Create new threads
pcap1 = DatasetSniffer('en3', directory, "master")
pcap2 = DatasetSniffer('en0', directory, "slave")
# Start new Threads
pcap1.start()
pcap2.start()
# Do stuff
time.sleep(10)
# Finished doing stuff
DatasetSnifferExit = True
Related
I am pretty new to Python and have a question about threading.
I have one function that is called pretty often. This function starts another function in a new Thread.
def calledOften(id):
t = threading.Thread(target=doit, args=(id))
t.start()
def doit(arg):
while true:
#Long running function that is using arg
When calledOften is called everytime a new Thread is created. My goal is to always terminate the last running thread --> At all times there should be only one running doit() Function.
What I tried:
How to stop a looping thread in Python?
def calledOften(id):
t = threading.Thread(target=doit, args=(id,))
t.start()
time.sleep(5)
t.do_run = False
This code (with a modified doit Function) worked for me to stop the thread after 5 seconds.
but i can not call t.do_run = False before I start the new thread... Thats pretty obvious because it is not defined...
Does somebody know how to stop the last running thread and start a new one?
Thank you ;)
I think you can decide when to terminate the execution of a thread from inside the thread by yourself. That should not be creating any problems for you. You can think of a Threading manager approach - something like below
import threading
class DoIt(threading.Thread):
def __init__(self, id, stop_flag):
super().__init__()
self.id = id
self.stop_flag = stop_flag
def run(self):
while not self.stop_flag():
pass # do something
class CalledOftenManager:
__stop_run = False
__instance = None
def _stop_flag(self):
return CalledOftenManager.__stop_run
def calledOften(self, id):
if CalledOftenManager.__instance is not None:
CalledOftenManager.__stop_run = True
while CalledOftenManager.__instance.isAlive():
pass # wait for the thread to terminate
CalledOftenManager.__stop_run = False
CalledOftenManager.__instance = DoIt(id, CalledOftenManager._stop_flag)
CalledOftenManager.__instance.start()
# Call Manager always
CalledOftenManager.calledOften(1)
CalledOftenManager.calledOften(2)
CalledOftenManager.calledOften(3)
Now, what I tried here is to make a controller for calling the thread DoIt. Its one approach to achieve what you need.
In Python I have my main program which is basically a console 'gui' class. Its main method is a running thread that keeps the mains screen with options printed, and keeps waitin for user to input a choice.
The gui class has another object initialized that has its own running Threads. One of the Threads in this other object basically requires the main GUI thread to switch to a different mode / stop, but because it always 'pauses' on any input(), it will never switch.
So is there way, since I have access to the gui thread from the other object, to send an empty stdin to the gui thread so it gets over input()?
Example: (not actual code)
from threading import Thread
class Gui:
def __init__(self):
self.mainthread = Thread(target=self.console_loop, daemon=True)
self.server = Server(self.mainthread)
self.mainthread.start()
self.mainthread.join()
def console_loop(self):
while True:
if some_terminating_condition:
break
while self.server.guiloop:
"""
.
. various code
.
"""
a = input()
if a == "Some input":
self.server.guiloop = False
self.server.do_stuff()
while not self.server.guiloop:
"""
More code
"""
class Server:
def __init__(self, guithread):
self.guiloop = True
self.gui_thread = guithread
def do_stuff(self):
self.guiloop = False
"""
Code to stop gui waiting for input <-- What I need
"""
# Arbitrary code
Sorry for any mistakes, wrote it on the fly, since my code is too big to copy over
I am trying to simulate an environment with vms and trying to run an object method in background thread. My code looks like the following.
hyper_v.py file :
import random
from threading import Thread
from virtual_machine import VirtualMachine
class HyperV(object):
def __init__(self, hyperv_name):
self.hyperv_name = hyperv_name
self.vms_created = {}
def create_vm(self, vm_name):
if vm_name not in self.vms_created:
vm1 = VirtualMachine({'vm_name': vm_name})
self.vms_created[vm_name] = vm1
vm1.boot()
else:
print('VM:', vm_name, 'already exists')
def get_vm_stats(self, vm_name):
print('vm stats of ', vm_name)
print(self.vms_created[vm_name].get_values())
if __name__ == '__main__':
hv = HyperV('temp')
vm_name = 'test-vm'
hv.create_vm(vm_name)
print('getting vm stats')
th2 = Thread(name='vm1_stats', target=hv.get_vm_stats(vm_name) )
th2.start()
virtual_machine.py file in the same directory:
import random, time, uuid, json
from threading import Thread
class VirtualMachine(object):
def __init__(self, interval = 2, *args, **kwargs):
self.vm_id = str(uuid.uuid4())
#self.vm_name = kwargs['vm_name']
self.cpu_percentage = 0
self.ram_percentage = 0
self.disk_percentage = 0
self.interval = interval
def boot(self):
print('Bootingup', self.vm_id)
th = Thread(name='vm1', target=self.update() )
th.daemon = True #Setting the thread as daemon thread to run in background
print(th.isDaemon()) #This prints true
th.start()
def update(self):
# This method needs to run in the background simulating an actual vm with changing values.
i = 0
while(i < 5 ): #Added counter for debugging, ideally this would be while(True)
i+=1
time.sleep(self.interval)
print('updating', self.vm_id)
self.cpu_percentage = round(random.uniform(0,100),2)
self.ram_percentage = round(random.uniform(0,100),2)
self.disk_percentage = round(random.uniform(0,100),2)
def get_values(self):
return_json = {'cpu_percentage': self.cpu_percentage,
'ram_percentage': self.ram_percentage,
'disk_percentage': self.disk_percentage}
return json.dumps(return_json)
The idea is to create a thread that keeps on updating the values and on request, we read the values of the vm object by calling the vm_obj.get_values() we would be creating multiple vm_objects to simulate multiple vms running in parallel and we need to get the information from a particular vm on request.
The problem, that I am facing, is that the update() function of the vm doesnot run in the background (even though the thread is set as daemon thread).
The method call hv.get_vm_stats(vm_name) waits until the completion of vm_object.update() (which is called by vm_object.boot()) and then prints the stats. I would like to get the stats of the vm on request by keeping the vm_object.update() running in the background forever.
Please share your thoughts if I am overlooking anything related to the basics. I tried looking into the issues related to the python threading library but I could not come to any conclusion. Any help is greatly appreciated. The next steps would be to have a REST api to call these functions to get the data of any vm but I am struck with this problem.
Thanks in advance,
As pointed out by #Klaus D in the comments, my mistake was using the braces when specifying the target function in the thread definition, which resulted in the function being called right away.
target=self.update() will call the method right away. Remove the () to
hand the method over to the thread without calling it.
I'm currently in the process of writing some server code in python and I have a method which begins an infinite loop to serve requests ie:
class s:
def serve(self):
while True:
# do stuff
When I call this code I do something like:
a = s()
a.serve()
My question is -- is there way to send a message to 'serve' to disrupt the loop from outside the method. I don't want to simply kill the program. Help much appreciated.
One option I've thought of is rather than:
while True:
// do something
Could do:
while self.serving: # and then update self.serving as appropriate
But there's probably a better solution.
Yes this is possible:
class Server:
def __init__():
self.alive = True
def serve(self):
while self.alive:
# do stuff
def die():
self.alive = False
Now just call die() outside the thread running serve(..).
srv = Server()
thread = Thread(target=srv.serve)
thread.start()
time.sleep(some_time) # or do something else.
srv.die() # or do this from some outside process.
I don't understand exactly what you're trying to do but could you have a simple flag rather than:
while True:
it could be
example = True
while example:
Then set example to false when you want to end the loop.
I started learning python recently, and I am facing a situation that I do not even know if it is expected, or if something is wrong.
I am learning parallel threading to have two independent processes on the same program (UI control on one thread, image processing on another)
So, to test this I created this simple code:
(Camera is a custom class that connects to a usb webcam)
import thread
from vii.camera import Camera
class Process(object):
def __init__(self, width=800, height=600):
self._cam = Camera(width, height)
self._is_running = False
self._current_image = None
def start(self):
thread.start_new(self._run(), (self))
def _run(self):
self._cam.start()
self._is_running = True
while self._is_running:
self._current_image = self._cam.update()
self._current_image.show()
def get_image(self):
return self._current_image
def stop(self):
self._is_running = False
self._cam.close()
thread.exit()
process = Process()
process.start()
print("You will never see this output")
while (True):
key = raw_input()
if key == 'q':
process.stop()
break
The thread is created with success, and I am able to see the image. Now, I need to be able to affect it (stop it, get data from it) from the main thread. But the problem is that the code never enters in the while loop.
Is this behaviour expected? If it is, is there a way for me to achieve the functionality I need?