We Keep Coding

Python output boolean variable to file (read, check, write)

I am new to python.
Tell me how to implement saving the value of a variable to a file so that you don't receive unnecessary notifications when the program is restarted.
The program pings the servers and sends a message when the status changes. When you restart the program, it does not save the history of its checks. It is very uncomfortable.
I hope I explained the problem correctly?
I need to constantly save the check result to a file and use this data when comparing new checks.
def ping_host(address):
status = ping_url(address.address)
if status != address.status:
send_message(( "! " if status is None else "+ " if status else
"- ") + address.comment)
address.status = status
This function checks the status, if it has changed, then a new message is sent.

If your file does not need to be portable the simplest solution is to use python pickling. The drawback is that you cannot inspect the file manually or modify it for debuging purpose vs text based saving (eg ini files, json or simple txt). The main advantage is the ease of use as you can serialyze this way any python basic type.
Here is a simple example on how to use it:
import pickle
def get_status():
with open('status','rb') as f:
status = pickle.load(f)
return status
def set_status(status:bool):
with open('status','wb') as f:
pickle.dump(status,f)
set_status(True)
s = get_status()
assert s
set_status(False)
s = get_status()
assert not s

You can make a file history.txt, and then on startup open it, and read the last state, and if its different overwrite that state in the file and save.

from what you wrote in comments I would change it to this:
import json
ping_data = dict()
with open('C:\ping_data.json') as file:
data = json.load(file)
def ping_host(address):
status = ping_url(address.address)
if data['address.status'] != status:
ping_data['address.status'] = status
send_message(("! " if status is None else "+ " if status else "- ") + address.comment)
ping_host(youraddress)
with open('C:\ping_data.json', 'w') as file:
json.dump(ping_data, file, indent=2)
the way I would do this is using json library
import json
next thing I would create a dictionary in your script
saved_data = dict()
then whenever I receive an update I would store the value in dictionary
saved_data['info'] = updated_info
and export? it to a json file
with open('saved_data.json', 'w') as file:
json.dump(saved_data, file, indent=2)
now whenever I open the program it would read that file like this
with open('saved_data.json') as file:
data = json.load(file)
and then I would access variable data as a dictionary
for k in data:
for info in data[k]:
if info != updated_info
saved_data['info'] = updated_info

Python Scapy wireless scan and match a mac address stored in text file

I have some code that will scan for wireless packets and displays the mac address from each of them. What i would like to do is have a text file of mac addresses and for the code to alert me with a message when one of the addresses in the file is picked up on the wireless scan. I can not think of a way to implement this, here is the code for the wiresless scan and below is an example of the text file.
import sys
from scapy.all import *
devices = set()
def PacketHandler(pkt):
if pkt.haslayer(Dot11):
dot11_layer = pkt.getlayer(Dot11)
if dot11_layer.addr2 and (dot11_layer.addr2 not in devices):
devices.add(dot11_layer.addr2)
print dot11_layer.addr2
sniff(iface = sys.argv[1], count = int(sys.argv[2]), prn = PacketHandler)
here is example of the text file.
00:11:22:33:44:55
AA:BB:CC:DD:EE:FF

Create a function that reads from a .txt and store each line (matching a MAC address) in a list.
def getListMac() -> list: # you can put the path for your .txt file as argument
with open('MAClist.txt', 'r+') as file:
res = [x.rstrip('\n') for x in file.readlines()]
return res
And then check in your packetHandler function if the mac if in this list.
Here you have two choice :
Call getListMac() at the start of your program, store it in a global variable. Go for this if your .txt file won't change after launching your program.
MACLIST = getListMac()
...
# in your PacketHandler function
if mac in MACLIST:
print("mac found!") #or whatever your want to do
Call the function each time a packet is sniffed. Go for this option if the list of MAC addresses frequently changes and you need it updated when your program is running. Be careful with it as this will slow your program, especially if your list is very long.
# in your PacketHandler function:
if mac in getListMac():
print("mac found!") # or whatever your want to do
Finally, i will finish this post by advising you to use a real DBMS, which will be much more efficient than reading a txt file. ;)
EDIT
To answer your comment :
Modify the getListMac function in order to store the information in a dictionnary.
Here is an exemple assuming you use " - " as separator between MAC - Time - Username
def getListMac() -> dict: # you can put the path for your .txt file as argument
with open('MAClist.txt', 'r+') as file:
res = {x.rstrip('\n').split(" - ")[0]: x.rstrip('\n').split(" - ")[2] for x in file.readlines()}
return res
Access the data in the dictionary like this:
if MAC in MACLIST:
print(f"MAC found -> {MAC}, Username -> {MACLIST[MAC]}")

Python replace line in text file

I am trying to manage a host file with a python script. I am new to python and I am having a hard time with figuring out how to replace a line if I find a match. For example, if the address gets changed in a host file for a website I want the script to find it and change it back. Thanks for your help.
import os
import time
#location to the host file to read and write to
hosts_path=r"C:\Windows\System32\drivers\etc\hosts"
#the address I want for the sites
redirect="0.0.0.0"
#the websites that I will set the address for
website_list=["portal.citidirect.com","www.bcinet.nc","secure.banque-tahiti.pf","www.bancatlan.hn","www.bancentro.com.ni","www.davivienda.com.sv","www.davivienda.cr","cmo.cibc.com","www.bi.com.gt","empresas.banistmo.com","online.belizebank.com","online.westernunion.com","archive.clickatell.com"]
#continuous loop
while True:
with open(hosts_path,'r+') as file:
content=file.read()
#for each of the websites in the list above make sure they are in the host file with the correct address
for website in website_list:
site=redirect+" "+ website
#here is where I have an issue, if the website is in the host file but with the wrong address I want to write over the line, instead the program is adding it to the end of the file
if website in content:
if site in content:
pass
else:
file.write(site)
else:
file.write("\n"+site)
time.sleep(300)
os.system('ipconfig /flushdns')

You need to read the file into a list, then changes the index of the list if it needs to be, then writes the list back to the file. What you are doing was just writing to the end of the file. You can’t change a file directly like that. You need to record the changes in a list then write the list. I ended up having to re-write a lot of the code. Here's the full script. I wasn't sure what the os.system('ipconfig /flushdns') was accomplishing, so I removed it. You can easily add it back where you want.
#!/usr/bin/env python3.6
import time
hosts_path = r"C:\\Windows\\System32\\drivers\\etc\\hosts"
redirect = "0.0.0.0"
website_list = [
"portal.citidirect.com",
"www.bcinet.nc",
"secure.banque-tahiti.pf",
"www.bancatlan.hn",
"www.bancentro.com.ni",
"www.davivienda.com.sv",
"www.davivienda.cr",
"cmo.cibc.com",
"www.bi.com.gt",
"empresas.banistmo.com",
"online.belizebank.com",
"online.westernunion.com",
"archive.clickatell.com"]
def substring_in_list(the_list, substring):
for s in the_list:
if substring in s:
return True
return False
def write_websites():
with open(hosts_path, 'r') as file:
content = file.readlines()
for website in website_list:
site = "{} {}\n".format(redirect, website)
if not substring_in_list(content, website):
content.append(site)
else:
for line in content:
if site in line:
pass
elif website in line:
line = site
with open(hosts_path, "w") as file:
file.writelines(content)
while True:
write_websites()
time.sleep(300)

So, you're going to assign the same IP address to every site that doesn't appear in your websites list?
The following would replace what's inside your outermost while loop:
# Read in all the lines from the host file,
# splitting each into hostname, IPaddr and aliases (if any),
# and trimming off leading and trailing whitespace from
# each of these components.
host_lines = [[component.strip() for component in line.split(None, 2)] for line in open(host_path).readlines()]
# Process each of the original lines.
for line in host_lines:
# Is the site in our list?
if line[1] in website_list:
# Make sure the address is correct ...
if line[0] != redirect:
line[0] == redirect
# We can remove this from the websites list.
website_list.remove(line[1])
# Whatever sites are left in websites don't appear
# in the hosts file. Add lines for these to host_lines
host_lines.extend([[redirect, site] for site in website_list])
# Write the host_lines back out to the hosts file:
open(hosts_path, 'w').write("\n".join([" ".join(line) for line in host_lines]))
The rightmost join glues the components of each line back together into a single string. The join to the left of it glues all of these strings together with newline characters between them, and writes this entire string to the file.
I have to say, this looks like a rather complicated and even dangerous way to make sure your hosts file stays up-to-date and accurate. Wouldn't it be better to just have a cron job scp a known-good hosts file from a trusted host every five minutes instead?

I ended up mixing some of the responses to create a new file to replace the current host file using functions as shown below. In addition to this code I am using pyinstaller to turn it into an exe then I setup that exe to run as a auto-start service.
#!/usr/bin/env python3.6
import os
import shutil
import time
temp_file = r"c:\temp\Web\hosts"
temp_directory="c:\temp\Web"
hosts_path = r"C:\Windows\System32\drivers\etc\hosts"
websites = ('''# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host
# localhost name resolution is handled within DNS itself.
# 127.0.0.1 localhost
# ::1 localhost
0.0.0.0 portal.citidirect.com
0.0.0.0 www.bcinet.nc
0.0.0.0 secure.banque-tahiti.pf
0.0.0.0 www.bancatlan.hn
0.0.0.0 www.bancentro.com.ni
0.0.0.0 www.davivienda.com.sv
0.0.0.0 www.davivienda.cr
0.0.0.0 cmo.cibc.com
0.0.0.0 www.bi.com.gt
0.0.0.0 empresas.banistmo.com
0.0.0.0 online.belizebank.com
0.0.0.0 online.westernunion.com
0.0.0.0 archive.clickatell.com''')
def write_websites():
with open(temp_file, 'w+') as file:
file.write(websites)
while True:
if not os.path.exists(temp_directory):
os.makedirs(temp_directory)
try:
os.remove(temp_file)
except OSError:
pass
write_websites()
try:
os.remove(hosts_path)
except OSError:
pass
try:
shutil.move(temp_file,hosts_path)
except OSError:
pass
os.system('ipconfig /flushdns')
time.sleep(300)

Using a USB flash drive with Raspberry Pi/Linux [duplicate]

I'm writing a python module for a device that interacts with a user supplied USB memory stick. The user can insert a USB memory stick in the device USB slot, and the device will dump data onto the memory stick without user intervention. If the device is running when the user inserts the USB stick, I have hooked into D-Bus and have an auto mount routine all worked out. The new issue is, what if the stick is inserted while the device is powered off? I get no D-Bus insertion event, or any the associated nuggets of information about the memory stick after the device is powered on.
I have worked out a way to derive the device node ( /dev/sd? ) from scanning the USB devices in /proc, by calling:
ls /proc/scsi/usb-storage
this gives the scsi device info if you cat each of the files in that folder.
I then take the Vendor, Product, and Serial Number fields from the usb-storage records, generate an identifier string that I then use in
ll /dev/disc/by-id/usb_[vendor]_[product]_[serial_number]-0:0
So I can parse through the result to get the relative path
../../sdc
Then, I can mount the USB stick.
This is a cumbersome procedure, pretty much all text based, and ready for bugs when someone introduces a weird character, or non-standard serial number string. It works with all 2 of the USB memory sticks I own. I have tried to map output from /var/log/messages but that ends up being text comparisons as well. Output from lsusb, fdisk, udevinfo, lsmod, and others only show half of the required data.
My question: how do I determine, in the absence of a D-Bus message, the /dev device assigned to a USB memory stick without user intervention, or knowing in advance the specifics of the inserted device?
Thanks, sorry about the novel.

This seems to work combining /proc/partitions and the /sys/class/block approach ephimient took.
#!/usr/bin/python
import os
partitionsFile = open("/proc/partitions")
lines = partitionsFile.readlines()[2:]#Skips the header lines
for line in lines:
words = [x.strip() for x in line.split()]
minorNumber = int(words[1])
deviceName = words[3]
if minorNumber % 16 == 0:
path = "/sys/class/block/" + deviceName
if os.path.islink(path):
if os.path.realpath(path).find("/usb") > 0:
print "/dev/%s" % deviceName
I'm not sure how portable or reliable this is, but it works for my USB stick. Of course find("/usb") could be made into a more rigorous regular expression. Doing mod 16 may also not be the best approach to find the disk itself and filter out the partitions, but it works for me so far.

I'm not entirely certain how portable this is. Also, this information would presumably also be available over D-Bus from udisks or HAL but neither of those is present on my system so I can't try. It seems to be reasonably accurate here regardless:
$ for i in /sys/class/block/*; do
> /sbin/udevadm info -a -p $i | grep -qx ' SUBSYSTEMS=="usb"' &&
> echo ${i##*/}
> done
sde
sdf
sdg
sdh
sdi
sdj
sdj1
$ cd /sys/class/block/
$ for i in *; do [[ $(cd $i; pwd -P) = */usb*/* ]] && echo $i; done
sde
sdf
sdg
sdh
sdi
sdj
sdj1

After looking at this thread about doing what ubuntu does with nautilus, i found a few recommendations and decided to go with accessing udisks through shell commands.
The Mass storage device class is what you want. Just give it the device file. ie: /dev/sdb
you can then do d.mount() and d.mount_point to get where it has been mounted.
After that is also a class for finding many identical USB devices to control mounting, un-mounting and ejecting a large list of devices that all have the same label.
(if you run is with no argument, it will apply this to all SD devices. Could be handy for a "just auto mount everything" script
import re
import subprocess
#used as a quick way to handle shell commands
def getFromShell_raw(command):
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return p.stdout.readlines()
def getFromShell(command):
result = getFromShell_raw(command)
for i in range(len(result)):
result[i] = result[i].strip() # strip out white space
return result
class Mass_storage_device(object):
def __init__(self, device_file):
self.device_file = device_file
self.mount_point = None
def as_string(self):
return "%s -> %s" % (self.device_file, self.mount_point)
""" check if we are already mounted"""
def is_mounted(self):
result = getFromShell('mount | grep %s' % self.device_file)
if result:
dev, on, self.mount_point, null = result[0].split(' ', 3)
return True
return False
""" If not mounted, attempt to mount """
def mount(self):
if not self.is_mounted():
result = getFromShell('udisks --mount %s' % self.device_file)[0] #print result
if re.match('^Mounted',result):
mounted, dev, at, self.mount_point = result.split(' ')
return self.mount_point
def unmount(self):
if self.is_mounted():
result = getFromShell('udisks --unmount %s' % self.device_file) #print result
self.mount_point=None
def eject(self):
if self.is_mounted():
self.unmount()
result = getFromShell('udisks --eject %s' % self.device_file) #print result
self.mount_point=None
class Mass_storage_management(object):
def __init__(self, label=None):
self.label = label
self.devices = []
self.devices_with_label(label=label)
def refresh(self):
self.devices_with_label(self.label)
""" Uses udisks to retrieve a raw list of all the /dev/sd* devices """
def get_sd_list(self):
devices = []
for d in getFromShell('udisks --enumerate-device-files'):
if re.match('^/dev/sd.$',d):
devices.append(Mass_storage_device(device_file=d))
return devices
""" takes a list of devices and uses udisks --show-info
to find their labels, then returns a filtered list"""
def devices_with_label(self, label=None):
self.devices = []
for d in self.get_sd_list():
if label is None:
self.devices.append(d)
else:
match_string = 'label:\s+%s' % (label)
for info in getFromShell('udisks --show-info %s' % d.device_file):
if re.match(match_string,info): self.devices.append(d)
return self
def as_string(self):
string = ""
for d in self.devices:
string+=d.as_string()+'\n'
return string
def mount_all(self):
for d in self.devices: d.mount()
def unmount_all(self):
for d in self.devices: d.unmount()
def eject_all(self):
for d in self.devices: d.eject()
self.devices = []
if __name__ == '__main__':
name = 'my devices'
m = Mass_storage_management(name)
print m.as_string()
print "mounting"
m.mount_all()
print m.as_string()
print "un mounting"
m.unmount_all()
print m.as_string()
print "ejecting"
m.eject_all()
print m.as_string()

why don't you simply use an udev rule? i had to deal with a similar situation, and my solution was to create a file in /etc/udev/rules.d containing following rule:
SUBSYSTEMS=="scsi", KERNEL=="sd[b-h]1", RUN+="/bin/mount -o umask=000 /dev/%k /media/usbdrive"
one assumption here is that nobody ever inserts more than one usb stick at time. it has however the advantage that i know in advance where the stick will be mounted (/media/usbdrive).
you can quite surely elaborate it a bit to make it smarter, but personally i never had to change it and it still works on several computers.
however, as i understand, you want to be alerted somehow when a stick is inserted, and perhaps this strategy gives you some trouble on that side, i don't know, didn't investigate...

I think the easiest way is to use lsblk:
lsblk -d -o NAME,TRAN | grep usb

Incremental parse of appended data to external XML file in Python

I've got a log file on external computer in my LAN network. Log is an XML file. File is not accessible from http, and is updating every second.
Currently i'm copying log file into my computer and run parser, but I want to parse file directly from external host.
How can I do it in Python? Is it possible, to parse whole file once, and later parse only new content added to the end in future versions?

You can use paramiko and xml.sax's default parser, xml.sax.expatreader, which implements xml.sax.xmlreader.IncrementalParser.
I ran the following script on local virtual machine to produce XML.
#!/bin/bash
echo "<root>" > data.xml
I=0
while sleep 2; do
echo "<entry><a>value $I</a><b foo='bar' /></entry>" >> data.xml;
I=$((I + 1));
done
Here's incremental consumer.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import time
import xml.sax
from contextlib import closing
import paramiko.client
class StreamHandler(xml.sax.handler.ContentHandler):
lastEntry = None
lastName = None
def startElement(self, name, attrs):
self.lastName = name
if name == 'entry':
self.lastEntry = {}
elif name != 'root':
self.lastEntry[name] = {'attrs': attrs, 'content': ''}
def endElement(self, name):
if name == 'entry':
print({
'a' : self.lastEntry['a']['content'],
'b' : self.lastEntry['b']['attrs'].getValue('foo')
})
self.lastEntry = None
def characters(self, content):
if self.lastEntry:
self.lastEntry[self.lastName]['content'] += content
if __name__ == '__main__':
# use default ``xml.sax.expatreader``
parser = xml.sax.make_parser()
parser.setContentHandler(StreamHandler())
client = paramiko.client.SSHClient()
# or use ``client.load_system_host_keys()`` if appropriate
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('192.168.122.40', username = 'root', password = 'pass')
with closing(client) as ssh:
with closing(ssh.open_sftp()) as sftp:
with closing(sftp.open('/root/data.xml')) as f:
while True:
buffer = f.read(4096)
if buffer:
parser.feed(buffer)
else:
time.sleep(2)

I am assuming that another process of which you don't have access to is maintaining the xml as an object being updated every so often, and then dumping the result.
If you don't have access to the source of the program dumping the XML, you will need a fancy diffing between the two XML versions to get an incremental update to send over the network.
And I think you would have to parse the new XML each time to be able to have that diff.
So maybe you could have a python process watching the file, parsing the new version, diffing it (for instance using solutions from this article), and then you can send that difference over the network using a tool like xmlrpc. If you want to save bandwidth it'll probably help. Although I think I would send directly the raw diff via network, patch and parse the file in the local machine.
However, if only some of your XML values are changing (no node deletion or insertion) there may be a faster solution. Or, if the only operation on your xml file is to append new trees, then you should be able to parse only these new trees and send them over (diff first, then parse in the server, send to client, merge in client).