We Keep Coding

difference between 2 files and get the changes

I have 2 files 1st one is original data and 2nd one is after some modification (eg - network/storage level). So I want to compare 1st file with new one and create a file with what new added / modified lines.
for eg:
file1 - original file
Route table:
Route table:
10.208.85.0/26 dev eth3 proto kernel scope link src 10.1.108.12
148.89.140.0/22 via 192.168.10.1 dev eth0
148.89.168.0/21 via 192.168.10.1 dev eth0
151.87.44.0/23 via 192.168.10.1 dev eth0
151.87.46.0/23 via 192.168.10.1 dev eth0
192.55.16.128/28 via 192.168.10.1 dev eth0
192.55.28.32/27 via 192.168.10.1 dev eth0
192.55.28.64/28 via 192.168.10.1 dev eth0
192.55.192.96/27 via 192.168.10.1 dev eth0
192.60.136.0/22 via 192.168.10.1 dev eth0
192.62.212.64/26 via 192.168.10.1 dev eth0
Disks:
Disk /dev/sda: 96 GiB, 103079215104 bytes, 201326592 sectors
/dev/sda1 * 2048 1060863 1058816 517M 83 Linux
/dev/sda2 1060864 201326591 200265728 95.5G 8e Linux LVM
file2 - Modified
Route table:
10.208.85.0/26 dev eth3 proto kernel scope link src 10.1.108.12
10.123.17.64/26 via 10.1.208.1 dev eth2
148.89.140.0/22 via 192.168.10.1 dev eth0
148.89.168.0/21 via 192.168.10.1 dev eth0
151.87.44.0/23 via 192.168.10.1 dev eth0
151.87.46.0/23 via 192.168.10.1 dev eth0
192.55.16.128/28 via 192.168.10.1 dev eth0
192.55.28.32/27 via 192.168.10.1 dev eth0
192.55.28.64/28 via 192.168.10.1 dev eth0
192.55.192.96/27 via 192.168.10.1 dev eth0
192.60.136.0/22 via 192.168.10.1 dev eth0
192.62.212.64/26 via 192.168.10.1 dev eth0
Disks:
Disk /dev/sda: 128 GiB, 137438953472 bytes, 201326592 sectors
/dev/sda1 * 2048 1060863 1058816 517M 83 Linux
/dev/sda2 1060864 201326591 200265728 95.5G 8e Linux LVM
I am looking an output something like (another file )
Added : 10.123.17.64/26 via 10.1.208.1 dev eth2
Modified : Disk /dev/sda: 128 GiB, 137438953472 bytes, 201326592 sectors
Please help

As starting point, you can use difflib:
from difflib import Differ
with (open('file1.txt') as file_1,
open('file2.txt') as file_2):
differ = Differ()
for line in differ.compare(file_1.readlines(), file_2.readlines()):
if line[0] in list('+-?'):
print(line.strip())
Output:
+ 10.123.17.64/26 via 10.1.208.1 dev eth2
- Disk /dev/sda: 96 GiB, 103079215104 bytes, 201326592 sectors
? ^^ - - -----
+ Disk /dev/sda: 128 GiB, 137438953472 bytes, 201326592 sectors
? ^^^ +++ ++++

In linux you can use diff to compare files line by line.
Let us consider the 2 example files provided in your question.
The Command:
diff /tmp/file1.txt /tmp/file2.txt
The Output:
2a3
> 10.123.17.64/26 via 10.1.208.1 dev eth2
15c16
< Disk /dev/sda: 96 GiB, 103079215104 bytes, 201326592 sectors
---
> Disk /dev/sda: 128 GiB, 137438953472 bytes, 201326592 sectors
The Command to generate the desired output:
diff /tmp/file1.txt /tmp/file2.txt | sed -e ':begin;$!N;s/---\n>/Modified: /;tbegin' -e 's/>/Added: /g' | egrep 'Added|Modified'
The Desired Output:
Added: 10.123.17.64/26 via 10.1.208.1 dev eth2
Modified: Disk /dev/sda: 128 GiB, 137438953472 bytes, 201326592 sectors

OSX El Capitan sqlalchemy Can't connect to MySQL server on '127.0.0.1' (111)

The python in this app is running from a docker container. The container is running in network mode: "host".
I'm having trouble connecting to a MySQL database through the python package sqlalchemy. I get the following error.
OperationalError: (OperationalError) (2003, "Can't connect to MySQL
server on '127.0.0.1' (111)") None None
The following terminal command works fine
mysql -h 127.0.0.1 -u my_user --password='password' db -e "SHOW TABLES;"
enter code here
Maybe sqlalchemy isn't using the correct configuration you say? I have printed out the configuration in a debug line right before the first connection to MySQL:
print config.get('repository', 'host') // 127.0.0.1
print config.get('repository', 'user') // my_user
print config.get('repository', 'passwd') // password
Maybe that configuration still isn't making it into sqlalchemy? Lets print out the engine string
engine = getUnaffiliatedEngine()
print engine //Engine(mysql://my_user:***#127.0.0.1:3306)
with engine.connect() as connection:
for s in statements:
if s.strip():
connection.execute(s)
Maybe there's more than 1 version of mysql running? Only one process is running:
ps -ef | grep mysql
74 15459 1 0 10:14AM ?? 0:00.80
/usr/local/mysql/bin/mysqld --user=_mysql
Maybe that user doesn't have access to the database?
mysql> select User, Host from mysql.user;
+------+--------------------------+
| User | Host |
+------+--------------------------+
|my_user 127.0.0.1 |
| root | 127.0.0.1 |
| root | ::1 |
| | localhost |
| root | localhost |
+------+--------------------------+
Similarly 'show databases' shows that the db exists. I have granted all privs to that user for every database. I have flushed the privs too.
Maybe some kind of firewall rule is stopping the connection? I use little snitch, and the entire firewall has been taken down to test this.
I don't even know what else to debug at this point. Here is a reduced version of the python script throwing the connection error:
DSN_FORMAT = "mysql://%(user)s:%(passwd)s#%(host)s:%(port)s"
def getDSN():
return DSN_FORMAT % dict(config.items("repository"))
def getUnaffiliatedEngine():
return create_engine(getDSN())
def reset(offline=False):
config.loadConfig()
dbName = config.get('repository', 'db')
print config.get('repository', 'host')
print config.get('repository', 'user')
print config.get('repository', 'passwd')
resetDatabaseSQL = (
"DROP DATABASE IF EXISTS %(database)s; "
"CREATE DATABASE %(database)s;" % {"database": dbName})
statements = resetDatabaseSQL.split(";")
engine = getUnaffiliatedEngine()
print engine
with engine.connect() as connection:
for s in statements:
if s.strip():
connection.execute(s)
Here is the output for tcpdump -i lo0 port 3306 :
11:44:41.224036 IP localhost.58797 > localhost.mysql: Flags [P.], seq 3915736486:3915736498, ack 2134634265, win 12519, options [nop,nop,TS val 980567261 ecr 980503692], length 12
11:44:41.224105 IP localhost.mysql > localhost.58797: Flags [.], ack 12, win 12737, options [nop,nop,TS val 980567261 ecr 980567261], length 0
11:44:41.224178 IP localhost.mysql > localhost.58797: Flags [P.], seq 1:19, ack 12, win 12737, options [nop,nop,TS val 980567261 ecr 980567261], length 18
11:44:41.224218 IP localhost.58797 > localhost.mysql: Flags [.], ack 19, win 12519, options [nop,nop,TS val 980567261 ecr 980567261], length 0
11:45:07.422776 IP localhost.58796 > localhost.mysql: Flags [P.], seq 2953728354:2953728366, ack 432872138, win 12483, options [nop,nop,TS val 980593366 ecr 980533534], length 12
11:45:07.422807 IP localhost.mysql > localhost.58796: Flags [.], ack 12, win 12729, options [nop,nop,TS val 980593366 ecr 980593366], length 0
11:45:07.422856 IP localhost.mysql > localhost.58796: Flags [P.], seq 1:19, ack 12, win 12729, options [nop,nop,TS val 980593366 ecr 980593366], length 18
11:45:07.422877 IP localhost.58796 > localhost.mysql: Flags [.], ack 19, win 12482, options [nop,nop,TS val 980593366 ecr 980593366], length 0
MySQL 5.6
OSX El Capitan 10.11.6

That last comment you wrote is very important and probably the reason of the problem.
I'm pretty sure that your Docker container has its own loopback interface with an address 127.0.0.1 that is different than your OSX loopback interface where you have running the MySQL.
I recommend you put your MySQL listening on an address that is visible from inside your container. You can debug that config easily making telnet ip_address 3306 from inside your container.

'RTNETLINK answers: File exists'

When I added the delay command, paramiko returns back "2" as exit status(while it returns "0" for bandwidth command setting). Is 2 alright? (Googling didn't help with this, I am guessing 2 must be an error of some sort).
The command I use is:-
delay_cmd = "sudo tc qdisc add dev eth0 parent 1: handle 1: netem delay %dms" %(delay)
We were basically trying to follow this:
# tc qdisc add dev eth0 root netem delay 100ms
from an online tutorial.
The block of commands which should set the bw,filter and delay is this. (I am guessing I might be messing up with the parent/class ids). Can you see what I might be doing wrong?
cmd2 = "sudo tc class add dev %s parent 1: classid 1:1 htb rate %dmbps ceil %dmbps" % (interface, bandwidth, 2*bandwidth )
filter_cmd = "sudo tc filter add dev %s protocol ip parent 1:0 prio 1 u32 match ip dst %s/%d flowid 1:1" % (interface, ip, subnetmasklength)
delay_cmd = "sudo tc qdisc add dev eth0 parent 1:0 handle 1: netem delay %dms" %(delay)
This is the error I receive:
'RTNETLINK answers: File exists'
This is our script for setting the delay and bandwidth:
def exec_bw_config2(ssh, interface, bandwidth, ip, subnetmasklength, delay):
clear_bw_config2(ssh, interface)
# create a qdisc (queuing discipline), 12 is default class
cmd1 = "sudo tc qdisc add dev %s root handle 1: htb default 12" % interface
print cmd1
block_exec(ssh, cmd1)
# define the performance for default class
cmd2 = "sudo tc class add dev %s parent 1: classid 1:1 htb rate %dmbps ceil %dmbps" % (interface, bandwidth, 2*bandwidth )
print cmd2
block_exec(ssh, cmd2)
filter_cmd = "sudo tc filter add dev %s protocol ip parent 1:0 prio 1 u32 match ip dst %s/%d flowid 1:1" % (interface, ip, subnetmasklength)
print filter_cmd
block_exec(ssh, filter_cmd)
delay_cmd = "sudo tc qdisc add dev eth0 parent 1: handle 1: netem delay %dms" %(delay)
print delay_cmd
block_exec(ssh, delay_cmd)
So is anything wrong with delay_cmd above?
This is how we call it in our code:
def main():
myhosts = ["10.0.1.192", "10.0.1.191", "10.0.1.190"]
username="ubuntu"
port=22
#key = get_private_key()
for host in myhosts:
ssh = get_ssh(username, host, port)
clear_bw_config2(ssh, "eth0")
exec_bw_config2(ssh, "eth0", int(sys.argv[1]) , "10.0.1.0", 24, int(sys.argv[2]))
# iterate over hosts here
# for everyhost,
# 1. create ssh connection
# 2. run the exec_bw_config with params
return
On a further note what's the difference between parent 1: handle and parent 1:2 handle per se?

It's not a fatal error: 'File exists' is just saying that such emulation is already defined and 'Invalid argument' that such emulation is not known to kernel or already deleted.
Quick small sample:
# ~ $ sudo tc qdisc add dev eth0 root netem delay 0ms
# ~ $ sudo tc qdisc add dev eth0 root netem delay 0ms
RTNETLINK answers: File exists
# ~ $ sudo tc qdisc del dev eth0 root netem delay 0ms
# ~ $ sudo tc qdisc del dev eth0 root netem delay 0ms
RTNETLINK answers: Invalid argument

use:
sudo tc qdisc replace dev eth0 root netem delay 100ms
or:
sudo tc qdisc del dev eth0 root
sudo tc qdisc add dev eth0 root netem delay 100ms

How can I put mac os x en1 interface into monitor mode to use with python3 scapy?

On my mac the wireless interface is the en1 interface. I can put the interface into monitor mode using mac's airport application but then it doesn't work with the scapy module when i use python 3. How can i make this work?
Thanks in advance
ifconfig output
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=1<PERFORMNUD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=10b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV>
nd6 options=1<PERFORMNUD>
media: autoselect (none)
status: inactive
fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078
lladdr 00:3e:e1:ff:fe:0f:0a:4a
nd6 options=1<PERFORMNUD>
media: autoselect <full-duplex>
status: inactive
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet6 fe80::7ed1:c3ff:fe6e:eeda%en1 prefixlen 64 scopeid 0x6
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=1<PERFORMNUD>
media: autoselect
status: active
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
media: autoselect <full-duplex>
status: inactive
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
media: autoselect
status: inactive
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1452
inet6 fe80::18b8:64ff:fec8:85%awdl0 prefixlen 64 scopeid 0x9
nd6 options=1<PERFORMNUD>
media: autoselect
status: active
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x2
member: en2 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 7 priority 0 path cost 0
nd6 options=1<PERFORMNUD>
media: <unknown type>
status: inactive
Python Script To Detected Packets (After putting en1 into mon mode using airport)
from scapy.all import *
def pktIdentifier(pkt):
if pkt.haslayer(Dot11Beacon):
print ("[+] Detected 802.11 Beacon Frame")
elif pkt.haslayer(Dot11ProbeReq):
print ("[+] Detected 802.11 Probe Frame")
elif pkt.haslayer(TCP):
print ("[+] Detected TCP Packet")
elif pky.haslayer(UDP):
print ("[+] Detected UDP Packet")
conf.iface = 'en1'
sniff(prn=pktIdentifier)
Output of conf.route
Network Netmask Gateway Iface Output IP
0.0.0.0 0.0.0.0 192.168.0.1 en1 192.168.0.7
127.0.0.0 255.0.0.0 0.0.0.0 lo0 127.0.0.1
127.0.0.1 255.255.255.255 0.0.0.0 lo0 127.0.0.1
169.254.0.0 255.255.0.0 0.0.0.0 en1 192.168.0.7
192.168.0.0 255.255.255.0 0.0.0.0 en1 192.168.0.7
192.168.0.1 255.255.255.255 0.0.0.0 en1 192.168.0.7
192.168.0.1 255.255.255.255 0.0.0.0 en1 192.168.0.7
192.168.0.7 255.255.255.255 0.0.0.0 en1 192.168.0.7
192.168.0.255 255.255.255.255 0.0.0.0 en1 192.168.0.7

Short Answer: You could MonkeyPatch the _PcapWrapper_pypcap class. An example Code is provided below.
Slightly Longer Answer: On Mac OS X scapy sniffs on interfaces through libpcap. Instead of calling pcap_open_live we call pcap_create, pcap_set_rfmon and pcap_activate (in this order). This will set the interface in monitor mode and start capturing. I tested the following MonkeyPatch under scapy-python3 (0.21) and macOS Sierra 10.12.6. Make sure you run this Code with admin rights.
from scapy.all import *
import scapy.arch.pcapdnet
from ctypes import POINTER, byref, create_string_buffer
from ctypes.util import find_library
class _PcapWrapper_pypcap_monkeypatched(scapy.arch.pcapdnet._PcapWrapper_pypcap):
def __init__(self, device, snaplen, promisc, to_ms):
self.errbuf = create_string_buffer(PCAP_ERRBUF_SIZE)
self.iface = create_string_buffer(device.encode('ascii'))
#self.pcap = pcap_open_live(self.iface, snaplen, promisc, to_ms, self.errbuf)
STRING = c_char_p
_lib_name = find_library("pcap")
if not _lib_name:
raise OSError("Cannot fine libpcap.so library")
_lib=CDLL(_lib_name)
pcap_create = _lib.pcap_create
pcap_create.restype = POINTER(pcap_t)
pcap_create.argtypes = [STRING, STRING]
pcap_set_rfmon = _lib.pcap_set_rfmon
pcap_set_rfmon.restype = c_int
pcap_set_rfmon.argtypes = [POINTER(pcap_t), c_int]
pcap_activate = _lib.pcap_activate
pcap_activate.restype = c_int
pcap_activate.argtypes = [POINTER(pcap_t)]
self.pcap = pcap_create(self.iface, self.errbuf)
pcap_set_rfmon(self.pcap, 1)
pcap_activate(self.pcap)
self.header = POINTER(pcap_pkthdr)()
self.pkt_data = POINTER(c_ubyte)()
self.bpf_program = bpf_program()
scapy.arch.pcapdnet._PcapWrapper_pypcap = _PcapWrapper_pypcap_monkeypatched
def pktIdentifier(pkt):
if pkt.haslayer(Dot11Beacon):
print("[+] Detected 802.11 Beacon Frame")
elif pkt.haslayer(Dot11ProbeReq):
print("[+] Detected 802.11 Probe Frame")
sniff(iface="en0", prn=pktIdentifier)

When using the sniff function setting monitor=True on Mac OS Catalina always works for me. Example: scapy.all.sniff(iface='en0, monitor=True) then obviously what ever other functions you want.

This is a possible answer: http://www.cqure.net/wp/2014/04/scapy-with-wifi-monitor-rfmon-mode-on-os-x/
If you will file a bug on http://github.com/phaethon/scapy I will assist with patching part.

Python to run Linux commands

I want to execute few linux commands using python
these are my commands.
modprobe ipv6
ip tunnel add he-ipv6 mode sit remote 216.218.221.6 local 117.211.75.3 ttl 255
ip link set he-ipv6 up
ip addr add 2001:470:18:f3::2/64 dev he-ipv6
ip route add ::/0 dev he-ipv6
ip -f inet6 addr
216.218.221.6
117.211.75.3
2001:470:18:f3::2/64
these ip's are the inputs from the user. Commands also need root privileges.
My Code upto now.
import os
print("Enter Server Ipv4 Address")
serverip4=input()
print("Enter Local Ipv4 Address")
localip4=input()
print("Enter Client Ipv6 Address")
clientip4=input()

Like this:
import sys
import os
os.system("ip tunnel add he-ipv6 mode sit remote %s local %s ttl 255" % (whicheveripvariableisfirst), (whicheveripvariableisnext)))
If you need it run at sudo level then put sudo in the command section or make sure to run the python script as sudo.

I guess, subprocess would be best choice in this scenario as you want to get all command results and use it.
You can refer this page for that: https://docs.python.org/2/library/subprocess.html
Here is the code:
import subprocess
#To use the sudo -> echo "password" | sudo <command>
ipv6_command_list = "echo 'password' | sudo 'ip tunnel add he-ipv6 mode sit remote 216.218.221.6 local 117.211.75.3 ttl 255'"
ip_link_list = "echo 'password' | sudo 'ip link set he-ipv6 up'"
ip_addr_list = "echo 'password' | sudo 'ip addr add 2001:470:18:f3::2/64 dev he-ipv6'"
ip_route_list = "echo 'password' |sudo 'ip route add ::/0 dev he-ipv6'"
ip_inet_list = "echo 'password' | sudo 'ip -f inet6 addr'"
for ip_command in [ip_link_list,ip_addr_list,ip_route_list,ip_inet_list]:
proc = subprocess.check_output(ip_command, shell=True)