I wrote a hiscore checker for a game that I play, basically you enter a list of usernames into the .txt file & it outputs the results in found.txt.
However if the page responds a 404 it throws an error instead of returning output as " 0 " & continuing with the list.
Example of script,
#!/usr/bin/python
import urllib2
def get_total(username):
try:
req = urllib2.Request('http://services.runescape.com/m=hiscore/index_lite.ws?player=' + username)
res = urllib2.urlopen(req).read()
parts = res.split(',')
return parts[1]
except urllib2.HTTPError, e:
if e.code == 404:
return "0"
except:
return "err"
filename = "check.txt"
accs = []
handler = open(filename)
for entry in handler.read().split('\n'):
if "No Displayname" not in entry:
accs.append(entry)
handler.close()
for account in accs:
display_name = account.split(':')[len(account.split(':')) - 1]
total = get_total(display_name)
if "err" not in total:
rStr = account + ' - ' + total
handler = open('tried.txt', 'a')
handler.write(rStr + '\n')
handler.close()
if total != "0" and total != "49":
handler = open('found.txt', 'a')
handler.write(rStr + '\n')
handler.close()
print rStr
else:
print "Error searching"
accs.append(account)
print "Done"
HTTPERROR exception that doesn't seem to be working,
except urllib2.HTTPError, e:
if e.code == 404:
return "0"
except:
return "err"
Error response shown below.
Now I understand the error shown doesn't seem to be related to a response of 404, however this only occurs with users that return a 404 response from the request, any other request works fine. So I can assume the issue is within the 404 response exception.
I believe the issue may lay in the fact that the 404 is a custom page which you get redirected too?
so the original page is " example.com/index.php " but the 404 is " example.com/error.php "?
Not sure how to fix.
For testing purposes, format to use is,
ID:USER:DISPLAY
which is placed into check.txt
It seems that total can end up being None. In that case you can't check that it has 'err' in it. To fix the crash, try changing that line to:
if total is not None and "err" not in total:
To be more specific, get_total is returning None, which means that either
parts[1] is None or
except urllib2.HTTPError, e: is executed but e.code is not 404.
In the latter case None is returned as the exception is caught but you're only dealing with the very specific 404 case and ignoring other cases.
Related
im working with OpenAI API and I want generate a request from a list, but when the request fails the script stops with Exit Code 1 and dont continue with the next in the list. This is the function called from my "For"
try:
response = openai.Completion.create(
engine=config['OPENAI']['engine'].strip(),
prompt=prompt,
temperature=float(config['OPENAI']['temperature'].strip()),
max_tokens=int(config['OPENAI']['max_tokens'].strip()),
top_p=float(config['OPENAI']['top_p'].strip()),
frequency_penalty=float(config['OPENAI']['frequency_penalty'].strip()),
presence_penalty=float(config['OPENAI']['presence_penalty'].strip()),
)
with open('./output/openai_log.txt', 'a') as f:
f.write(prompt + '\n')
f.write(json.dumps(response, indent=4,) + '\n')
f.write('-_' * 60 + '\n')
return response
except openai.APIError as e:
print(f'\nFailed to generate for: {prompt}', e)
return None
Ive tried with try/catch but fails
I've been having a lot of trouble multiprocessing - I've literally been trying for hours and can't get it right. Here's my code, commented the best I could do.
Linked all my code as I don't know what's causing it exactly.
Line 74 it says, on p.start()
The most relevant part of code is the bottom of the question.
Here are my imports
import urllib
import socket
import multiprocessing as mp
import queue
import requests
Header used for higher chance of success upon connecting to a website
headers={'User-agent' : 'Mozilla/5.0'}
Main function takes four parameters - queue, the URL List, the Output file, and the list of vulnerable URLs.
def mainFunction(q, URLList, Output, vulnURLS):
This list is used to check if the page source has any of the errors in the list after adding a string query to the end of the url (')
queries = ['SQL syntax', 'mysql_fetch', 'mysql_num_rows', 'mySQL Error', 'mySQL_connect()', 'UNION SELECT', 'MySQL server version']
This puts the URL in the correct format before testing for injection points.
URLReplace = [("['", ""),("']",""), ("\n", ""), ("https://","http://"), ("\s", "%20"), ("\s", "%20")]
URL = ''.join(str(URLList))
for URL in URLList:
if (z < len(URLReplace)):
URL = URL.replace(URLReplace[z])
z = z + 1
URL = (URL + "'")
This is the try request, where it attempts to connect and scrapes the HTML off of the webpage.
try:
req = requests.get(URL, timeout=2)
htmlObject = urllib.request.urlopen(URL)
This iterates through the list to check for any possible vulnerabilities. Also returns 404/400 messages.
if (y < len(queries)):
if queries[x] in htmlObject:
print ("\t [+] " + URL)
vulnURLS.append(URL)
Output.open()
for VURLS in vulnURLS:
Output.write(VURLS + '\n')
Output.close()
y = y + 1
else:
print ("\t [-] " + URL)
except urllib.error.HTTPError as e:
if e.code == 404:
print("\t [-] Page not found.")
if e.code == 400:
print ("\t [+] " + URL)
except urllib.error.URLError as e:
print("\t [-] URL Timed Out")
except socket.timeout as e:
print("\t [-] URL Timed Out")
except socket.error as e:
print("\t [-] Error in URL")
Here's the important part, where I use the Queue & multiprocessor.
if __name__=='__main__':
q = mp.Queue()
URLList = [i.strip().split() for i in open('sites.txt').readlines()]
Output = open('output.txt', 'r')
vulnURLS = []
p = mp.Process(target=mainFunction, args=(q, URLList, Output, vulnURLS))
p.start()
q.put(mainFunction(URLList))
q.close()
q.join_thread()
p.join()
Please help me out with this problem, I've been stuck on it for hours and am getting very frustrated that I cannot follow the solution. Every module I look at I follow to a T and get this same error.
I have tried multi-threading, but it is extremely slow and unstable when compared to multiprocessing.
Change to the following:
p = mp.Process(target=mainFunction, args=(q, Output))
p.start()
for url in URLList:
q.put(url)
I am lost here, I have an API that streams prices, I am trying to compare the second to last price with the last price, for instance, if x > y then do something. I cannot figure out how to compare the last to the second to the last price when the prices are streaming. Could someone please shed some light on how this may work? Thanks in advance!
my stream:
def stream_to_queue(self):
response = self.connect_to_stream()
if response.status_code != 200:
return
for line in response.iter_lines(1):
if line:
try:
msg = json.loads(line)
except Exception as e:
print "Caught exception when converting message into json\n" + str(e)
return
if msg.has_key("instrument") or msg.has_key("tick"):
price = msg["tick"]["ask"]
print price
This prints a price like 1.23004 and then continues to loop and print more prices. I have tried to save the current price in a variable outside the loop and then reference it when a new price comes in but it's not working..
my attempt:
def stream_to_queue(self):
response = self.connect_to_stream()
if response.status_code != 200:
return
oldLine = ''
for line in response.iter_lines(1):
if line:
try:
msg = json.loads(line)
except Exception as e:
print "Caught exception when converting message into json\n" + str(e)
return
if msg.has_key("instrument") or msg.has_key("tick"):
price = msg["tick"]["ask"]
oldLine = price
newLine = oldLine
if newLine > oldLine:
print newLine
Couple of things:
1- Your indentation is a bit off as the comparison should be done inside the 'for' loop. In your case, the comparison is only being made when the streaming is complete.
2- You are comparing oldLine with newLine which are equal, so nothing will happen. Instead you should compare newLine with price.
Consider the following code:
for line in response.iter_lines(1):
if line:
try:
msg = json.loads(line)
except Exception as e:
print "Caught exception when converting message into json\n" + str(e)
return
if msg.has_key("instrument") or msg.has_key("tick"):
price = msg["tick"]["ask"]
oldLine = price
newLine = oldLine
if newLine > price:
print newLine
I'm encountering this error message:
TypeError: add_header() takes exactly 3 arguments (2 given)
when using these parameters:
testService("SomeServiceName", "POST", "[redacted valid url]", ('Content-type','application/json'), [redacted valid json])
Normally this error means I'm not passing "self" as a parameter, but seeing as this method is not being called in a class, I'm not sure what to do. I've tried passing self in as a parameter in both the parameters and inside the method. And I've tried wrapping the header in brackets and parentheses. When I pass "self" I get the error message that self is undefined, and when I use the brackets instead of parentheses, I get the same error as above.
Anyone with magical Python debugging skills out there? Thanks so much for taking the time to check this out!
def testService(name, verb, url, header="", requestBody=""):
#Log out the name of the request we're testing
if (name is not None) or (name.strip() is not ""):
print "Checking " + name + "\n\n"
# Make URL with StoreNumber
if (url is not None) or (url is not ""):
testUrl = url
# If specified verb is GET
if verb.strip().upper() == "GET":
# Create request
req = urllib2.Request(testUrl)
print "Making request with URL: " + testUrl + "\n\n"
# Send request
try:
response = urllib2.urlopen(req)
# If service returns 200 Okay
print "Connection to " + name + " Service successful. Returned with code " + str(response.code) + "\n\n"
# Log response
print "Response: " + response.read() + "\n\n"
# Handle exceptions
# If HTTP Error
except HTTPError as e:
if hasattr(e, 'reason'):
print name + ' failed to reach a server.'
print 'Reason: ', e.reason
elif hasattr(e, 'code'):
print e.code
elif hasattr(e, 'message'):
print e.message
pass
# If URL was the problem
except URLError as e:
if hasattr(e, 'reason'):
print name + ' failed to reach a server.'
if str(e.reason) == "[Errno 11004] getaddrinfo failed":
print "[Errno 11004] getaddrinfo failed with bad url: " + testUrl + "\n\n"
else:
print 'Reason: ', e.reason
elif hasattr(e, 'code'):
print 'Error code: ', e.code
elif hasattr(e, 'message'):
print e.message
pass
# If specified verb was POST
elif verb.strip().upper() == "POST":
# Check for None requestBody
if (requestBody is not None) or (requestBody.strip() is not ""):
data = urllib.urlencode(requestBody)
# Create request
req = urllib2.Request(testUrl, data)
# Check for header
if (header is not None) or (header.strip() is not ""):
req.add_header(header)
# YO YO THE BELOW CODE IS INCOMPLETE PLEASE FINISH
# Log request with URL and Data
print "Making request with URL: " + testUrl + " and data: THIS PART IS UNFINISHED PLEASE FINISH ME \n\n"
try:
response = urllib2.urlopen(req)
# If service returns 200 Okay
print "Connection to " + name + " Service successful. Returned with code " + str(response.code) + "\n\n"
# Log response
print "Response: " + response.read() + "\n\n"
# Handle exceptions
# If HTTP Error
except HTTPError as e:
if hasattr(e, 'code'):
print e.code
elif hasattr(e, 'message'):
print e.message
elif hasattr(e, 'reason'):
print name + ' failed to reach a server.'
print 'Reason: ', e.reason
pass
except URLError as e:
if hasattr(e, 'reason'):
print name + ' failed to reach a server.'
if str(e.reason) == "[Errno 11004] getaddrinfo failed":
print "[Errno 11004] getaddrinfo failed with bad url: " + url + "\n\n"
else:
print 'Reason: ', e.reason
elif hasattr(e, 'code'):
print 'Error code: ', e.code
elif hasattr(e, 'message'):
print e.message
pass
# Header non-existent in testService call
else:
print "Service header not provided. Exiting program"
sys.exit()
# Requesty Body not present in testService call
else:
print "Service request body not provided in code. Exiting program"
sys.exit()
# If specified verb is not supported (Currently only GET and POST are supported)
else:
print name + " Service written with HTTP verb other than GET or POST. Exiting program"
sys.exit()
else:
print "Service url not provided in code. Exiting program"
sys.exit()
else:
print "Service name not provided in code. Exiting program"
sys.exit()
From the documentation, add_header takes two arguments. You are calling it with one argument, a tuple with two values.
What you should do:
req.add_header(key, value)
What you are currently doing because you are getting the header as a tuple:
req.add_header((key, value,)) # aka passing a tuple with both arguments to the key parameter
You need to unpack the tuple:
req.add_header(header[0], header[1])
Or even better, using the splat operator (*):
req.add_header(*header) # Does the same thing as above
Also, you are using an empty string as the default argument for header, when when it is supplied it is a tuple. You should probably change the default value to a tuple or None.
Your header is a 2-tuple:
('Content-Type', 'application/json')
You're trying to do this:
req.add_header('Content-Type', 'application/json')
But in reality you're doing this:
req.add_header(('Content-Type', 'application/json'))
Notice that you're only passing one argument - a tuple - instead of two, a key and a value.
To fix, unpack your header when you pass it with the * (informally, 'splat') operator:
req.add_header(*header)
Take a look at the documentation: http://docs.python.org/2/library/urllib2.html#urllib2.Request.add_header.
While the function expects a key and a value, you're passing only a single object. Since you're calling this on the req object, that is the implicit "self" that's being passed as well.
You could call this function in two ways:
req.add_header(key, value)
urllib2.Request.add_header(req, key, value) # explicitly passing the reference instead of self
I'm not sure whether you're expecting the string you pass to be treated as the key or the value, but adding another paramater (or making the header parameter take a dict and then splitting appropriately in a for loop) should solve the issue. For example (with irrelevant code removed):
def testService(name, verb, url, header=None, requestBody=""):
if header is None:
header = {}
for key, value in header.iteritems():
req.add_header(key, value)
I need to pass a query to a function. It should be really simple! here is the code:
def get_file_ID(q):
dump = drive_service.files().list(q=q, fields = 'items(mimeType,id,title,downloadUrl)').execute()
fileItems = dump['items']
for item in fileItems:
try:
if item['mimeType'] == "application/octet-stream":
return item['id']
except KeyError:
print "No item of the required type, or item has been deleted"
return None
def lets_go_get(file_Id):
f = drive_service.files().get(fileId=file_Id).execute()
resp, content = drive_service._http.request(f.get('id'))
if resp.status == 200:
#print 'Status: %s' % resp
return content
else:
print 'An error occurred: %s' % resp
return None
text = raw_input('Enter title search text: ')
query = "title contains '" + text +"'"
selectedFile = "'" + get_file_ID(query) +"'"
print lets_go_get(selectedFile)
I get the following error:
HttpError: <HttpError 404 when requesting https://www.googleapis.com/drive/v2/files/%270BxDfMkL6x0wjMS03Nz.....Tk%27?alt=json returned "File not found: '0BxDfMkL6x0wjMS03.....MTU5NjQ0ZjdhOTk'">
But when i substitute the last call to 'lets_go_get' with the DIRECT fileID like this:
print lets_go_get('0BxDfMkL6x0wjMS03.....MTU5NjQ0ZjdhOTk')
then it works.
I can't see any difference between the two. can anyone explain what I'm missing???
Many thanks.
Use downloadUrl to download the contents:
f = drive_service.files().get(fileId=file_Id).execute()
resp, content = drive_service._http.request(f.get('downloadUrl'))
Docs have a working sample on the "Python" tab: https://developers.google.com/drive/v2/reference/files/get