For a project I need to find the size of the certificate packet that is sent by server to client when setting up the ssl connection.
This happens over TCP and thus the packet can be split up over multiple segments. I have to do this on the top 100 Alexa sites. I am able to get the list of 100 sites and then running this code:
import ssl
import urllib
import os
import subprocess
from subprocess import call
import re
f = open("urls.txt", "r")
array = []
for line in f:
line = line[:-1]
array.append(line)
for a in array:
str = ":443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ~/Desktop/thesis/certificates/" + a + ".cert"
subprocess.getoutput('echo -n | openssl s_client -connect '+ a + str)
However this second for loop hangs after six iterations, so in my certificates folder there are only six certs.. Could someone tell me why this hangs and/or how to fix it? Thanks
Related
I'm starting with python3 and stuck with following problem. I will be thankful for any help to unblock me.
Requirement:
run python script on host server to do following:
from host server, ssh to a remote server.
search for a file if its present on the remote server.
if present, send its name to the host server.
else, print following on host server "file not found on remote server".
Sample code:
import sys
import getopt
import os
from os import path
import re
import copy
import getpass
import time
import socket
import re
bin_ver = "1.99"
chip = "6233"
remote_bin_path = "/path/of/the/binary/release/v"+bin_ver+"/bin/"
print("path of binary on remote server is " + remote_bin_path)
os.getcwd()
os.chdir(remote_bin_path)
os.getcwd()
if (chip == "6233" or chip == "62334"):
sub_ver = "ab"
else:
sub_ver = "xy"
print("sub_ver is: " + sub_ver)
search_string = "*" + chip + "*" + sub_ver + "*"
print("search_string is " + search_string)
cmd = "ls -1 " + search_string
print("cmd is " + cmd)
required_file = os.popen(cmd).read()
print("INFO: required file is \n"+required_file)
Input dataset: sample file names present on remote server:
binary-v1.99-6233b0-ab-cde-1.2.3.bin
binary-v1.99-62334a0-xy-cde-1.2.3.bin
binary-v1.99-62334b0-ab-cde-1.2.3.bin
binary-v1.99-62334a0-xy-cde-1.2.3.bin
binary-v1.99-6278b0-ab-cde-1.2.3.bin
binary-v1.99-6277a0-ab-cde-1.2.3.bin
Problem with above implementation:
This code runs fine on the host server, but the issue is it returns 2 file names:
binary-v1.99-6233b0-ab-cde-1.2.3.bin
binary-v1.99-62334b0-ab-cde-1.2.3.bin
Since the input variable chip = "6233", i do not want the output corresponding to 62334. How to set the correct word boundary for 6233 in search_string variable?
Next Setps:
How can I run this script on the remote server and return the filename to the host server.
I am tasked with developing a mitmproxy script, something I have never done before.
I am trying to modify "Location" headers on the fly during http-flow on a reverse proxy instance in mitmproxy, these headers are often different but my main goal right now is to simply detect them in a static manner.
I am running mitmproxy as such:
mitmdump --set flowdetail:3 -s rproxy.py -p 8080 --mode reverse:http://www.com
^ That is not the real URL of question of course.
Say if I have the header being sent from the server
Location: http://www.com/example-location
But I want to change it to
Location: http://localhost:8080/example-location
I have tried every example I could find on google, including the ones on StackOverFlow.
Some things I have tried without any success.
from mitmproxy import http
import sys
def response(flow: http.HTTPFlow) -> None:
flow.request.host = "http://localhost:8080/example-location"
from mitmproxy import http
import sys
def response(flow):
flow.repsonse.headers['Location'] = ["http://localhost:8080/example-location"]
flow.repsonse.headers['Location'] = "http://localhost:8080/example-location" ## I tried this line separately without the above line
And
from mitmproxy import http
import sys
def response(flow: http.HTTPFlow) -> None:
flow.repsonse.headers['Location'] = ["http://localhost:8080/example-location"]
flow.repsonse.headers['Location'] = "http://localhost:8080/example-location" ## I tried this line separately without the above line
How do I do this correctly?
If it is not possible to manipulate this header (but why shouldn't it be) what are options do I have?
I'm looking for the fastest pinging method via python. I need to ping over 100,000 servers and my current procedure below takes approximately 85 minutes to complete. I've read small snippets about scapy, along with general ICMP and python ping. I need to know a definitive method, or at least a solid way to test, which is the fastest. I cannot test python - ping from work as it is not an approved package. I also tried a code snippet for scapy, but got an error:
OSError: Windows native L3 Raw sockets are only usable as administrator !
Install 'Winpcap/Npcap to workaround !
So I'm admittedly looking for code snippets I can test at home or ways around that error from more experienced persons
To prove I've tried, here are some related posts, as well as my current code
Current code:
import pandas as pd
import subprocess
import threading
raw_list = []
raw_list2 = []
def ping(host):
raw_list.append(host+ ' '+ str((subprocess.run('ping -n 3 -w 800 '+host).returncode)))
with open(r"FILEPATH", "r") as server_list_file:
hosts = server_list_file.read()
hosts_list = hosts.split('\n')
num_threads = 100
num_threads2 = 10
num_threads3 = 1
number = 0
while number<len(hosts_list):
print(number)
if len(hosts_list)>number+num_threads:
for i in range(num_threads):
t = threading.Thread(target=ping, args=(hosts_list[number+i],))
t.start()
t.join()
number = number + num_threads
elif len(hosts_list)>(number+num_threads2):
for i in range(num_threads2):
t = threading.Thread(target=ping, args=(hosts_list[number+i],))
t.start()
t.join()
number = number + num_threads2
elif len(hosts_list)>(number+num_threads3-1):
for i in range(num_threads3):
t = threading.Thread(target=ping, args=(hosts_list[number+i],))
t.start()
t.join()
number = number + num_threads3
else:
number = number+1
for x in range(len(raw_list)):
if(raw_list[x][-1] == '0'):
raw_list2.append(raw_list[x][0:-2])
to_csv_list = pd.DataFrame(raw_list2)
to_csv_list.to_csv('ServersCsv.csv', index = False, header = False)
to_csv_list.to_csv(r'ANOTHERFILEPATH', index = False, header = False)
subprocess.call(r'C:\ProgramData\Anaconda3\python.exe "A_PROGRAM_THAT_INSERTS_INTO_SQL"')
This does exactly what I need, however, it does not do it quickly enough.
I've tried the very small snippet:
from scapy.all import *
packets = IP(dst=["www.google.com", "www.google.fr"])/ICMP()
results = sr(packets)
resulting in gaierror: [Errno 11001] getaddrinfo failed
I've also tried:
TIMEOUT = 2
conf.verb = 0
packet = IP("ASERVERNAME", ttl=20)/ICMP()
reply = sr1(packet, timeout=TIMEOUT)
if not (reply is None):
print(reply.dst + "is online")
else:
print("Timeout waiting for %s") % packet[IP].dst
resulting in:
OSError: Windows native L3 Raw sockets are only usable as administrator !
Install Winpcap/Npcap to workaround !
A few links I looked at but could not garner a solid answer from:
Ping a site in Python?
Fastest way to ping a host in python?
This only solves the Python part. The comments are very right.
OSError: Windows native L3 Raw sockets are only usable as administrator ! Install Winpcap/Npcap to workaround !
I find this pretty damn explicit. If you follow's Scapy documentation for windows it says you need to install Npcap.https://nmap.org/npcap/
Other than that,
packets = IP(dst=["www.google.com", "www.google.fr"])/ICMP()
results = sr(packets)
Is likely the cleanest way to go. Works on my machine.. make sure you're using the latest development version from GitHub (unzip it and install it via python setup.py install).
If you are using the latest version, you might even want to turn on threaded=True in sr() to send and receive packets on two threads, as pointed out by the comments. You might also want to use prn and store=False to not store the answers (100k is a lot)
#!/usr/bin/python3
import cgi
import cgitb
import urllib.request
import os
import sys
def enco_print(string="", encoding = "utf8"):
sys.stdout.buffer.write(string.encode(encoding) + b"\n")
cgitb.enable()
form = cgi.FieldStorage(endcoding="utf8")
name_name= form.getvalue("name")
url_name = form.getvalue("url")
response = urllib.request.urlopen(str(url_name))
html = response.read().decode("utf8")
if not os.path.exists("gecrwalt"):
os.mkdir("gecrwalt")
with open("/gecrwalt/" + str(url_name) + ".html", "w", endcoding="utf8")
as f:
f.write(str(html))
When I try to run this script, I get 500 Status Error on my Website. I can´t see what´s wrong with this code.
I´m very thankful for help.
There are a few typos where you wrote endcoding instead of encoding.
The last segment here
with open("/gecrwalt/" + str(url_name) + ".html", "w", endcoding="utf8")
as f:
f.write(str(html))
has broken indentation, not sure if this was due to a copy-paste error here on Stackoverflow.
Another issue here is that (if I understood your code correlty) url_name will contain a complete URL like http://example.com, which will result in an error because that filename is invalid. You will have to come up with some schema to safely store these files, urlencode the URL or take a hash of the URL. Your save path also starts with /, not sure if intentational, starts at the file system root.
Changing the typo and the last bit has worked for me in a quick test:
with open("gecrwalt/something.html", "w", endcoding="utf8") as f:
f.write(str(html))
Debugging hint: I started a local Python webserver process with this command (from here)
python3 -m http.server --bind localhost --cgi 8000
Accessing http://localhost:8000/cgi-bin/filename.py will show you all errors that occur, not sure about the webserver you are currently using (there should be error logs somewhere I guess).
I'm in a pickle with writing a script that can SSH into device, run a command and parse that data out to a file. I've written this using Pyparsing and Exscript then I found out that the device I'm going to be using this on is using Python 2.4.4 and Debian 4.1.1 so the modules will not work on this. Now I am back to the drawing board trying to find out how to do this with NO modules. Anyone have any reference or point me in the right direction for this? Thank you in advance.
Here is my code:
from Exscript.util.interact import read_login
from Exscript.protocols import SSH2
import uuid
from pyparsing import *
import re
import yaml
account = read_login()
conn = SSH2()
conn.connect('172.0.0.1')
conn.login(account)
conn.execute('foobar')
data = conn.response
conn.send('exit\r')
conn.close()
###### PARSER ######
date_regex = re.compile(r'\d\d-\d\d-\d\d')
time_regex = re.compile(r'\d\d:\d\d:\d\d')
pairs = [{'category': 'General Information',
'kv': Group(Word(alphanums) + Word(alphanums))},
{'category': 'Last Reset:',
'kv': Group(Word(alphas, max=1) + Word(alphas)) + Literal(':').suppress()
+ Group(Regex(date_regex) + Regex(time_regex)
+ Optional(SkipTo(LineEnd())))
}
]
# build list of categories with associated parsing rules
categories = [Word("# ").suppress() + x['category']
+ OneOrMore(Group(x['kv']))
for x in pairs]
# account for thing you don't have specific rules for
categories.append(Word("#").suppress() + Optional(SkipTo(LineEnd())) +
Group(OneOrMore(Combine(Word(alphanums) + SkipTo(LineEnd()))))
)
# OR all the categories together
categories_ored = categories[0]
for c in categories[1:]:
categories_ored |= c
configDef = OneOrMore(categories_ored)
suppress_tokens = ["show all", "SSH>", "Active System Configuration"]
suppresses = [Literal(x).suppress() for x in suppress_tokens]
for s in suppresses:
configDef.ignore(s)
result = configDef.parseString(data)
for e in result:
print(e)
with open('/Users/MyMac/development/data.yml', 'w') as outfile:
outfile.write( yaml.dump(e))
UPDATE
I have followed the advice below and now have Pexpect installed and found a older version of Python-Pyparsing that I have also installed. So I'm on my way again to getting my scripts to work with modules. Thanks!
Looks like this is already solved, but...
As long as your SSH is configured for this host (or the host doesn't require you to log-in), you should be able to do the following.
import os
""" This will execute foobar on the remote host
and store the command output to a text file
on your machine."""
os.system("ssh 172.0.0.1 foobar > ~/data.txt")
""" Commence processing """
data = open("data.txt", mode='r')
# and so on and so on
You can also use the subprocess library, but os.system for these types of tasks is the simplest IMO.