Downloading a webpage using urllib3 - python

I'm trying to write a program for an assignment that uses urllib3 to download a webpage and store it in a dictionary. (I'm using spyder 3.6)
The program is giving me an 'AttributeError' and I have no idea what I'm doing wrong. here is my code with step by step notes I wrote for the assignment.
#Downloading a webpage
import urllib3
import sys
#these import statements allow us to use 'modules' aka 'libraries' ....
#code written by others that we can use
urlToRead = 'http://www.google.com'
#This value won't actually get used, because of the way the while loop
#below is set up. But while loops often need a dummy value like this to
#work right the first time
crawledWebLinks = {}
#Initialize an empty dictionary, in which (key, value) pairs will correspond to (short, url) eg
#("Goolge" , "http://www.google.com")
#Ok, there is a while loop coming up
#Here ends the set up
while urlToRead != ' ':
#This is a condition that dictates that the while loop will keep checking
#as long as this condition is true the loop will continue, if false it will stop
try:
urlToRead = input("Please enter the next URL to crawl")
#the "try" prevents the program from crashing if there is an error
#if there is an error the program will be sent to the except block
if urlToRead == '':
print ("OK, exiting loop")
break
#if the user leaves the input blank it will break out of the loop
shortName = input("Please enter a short name for the URL " + urlToRead)
webFile = urllib3.urlopen(urlToRead).read()
#This line above uses a ready a readymade function in the urllib3 module to
#do something super - cool:
#IT takes a url, goes to the website for the url, downloads the
#contents (which are in the form of HTML) and returns them to be
#stored in a string variable (here called webFile)
crawledWebLinks[shortName] = webFile
#this line above place a key value pair (shortname, HTML for that url)
#in the dictionary
except:
#this bit of code - the indented lines following 'except:' will be
#excecuted if the code in the try block (the indented following lines
#the 'try:' above) throw and error
#this is an example of something known as exeption-handling
print ("*************\nUnexpected Error*****", sys.exc_info()[0])
#The snip 'sys.exc_info()[0]' return information about the last
#error that occurred -
#this code is made available through the sys library that we imported above
#Quite Magical :)
stopOrProceed = input("Hmm..stop or proceed? Enter 1 to stop, enter anything else to continue")
if stopOrProceed ==1 :
print ('OK...Stopping\n')
break
#this break will break out of the nearest loop - in this case,
#the while loop
else:
print ("Cool! Let's continue\n")
continue
# this continue will skip out of the current iteration of this
#loop and move to the next i.e. the loop will reset to the start
print (crawledWebLinks.keys())

Your issue is that you are trying to call urllib3.urlopen(), and urllib3 does not have a member urlopen Here is a working snippet. All that I did was replace urllib3 with urllib.request:
import urllib.request
import sys
urlToRead = 'http://www.google.com'
crawledWebLinks = {}
while urlToRead != ' ':
try:
urlToRead = input("Please enter the next URL to crawl: ")
if urlToRead == '':
print ("OK, exiting loop")
break
#if the user leaves the input blank it will break out of the loop
shortName = input("Please enter a short name for the URL " + urlToRead + ": ")
webFile = urllib.request.urlopen(urlToRead).read()
crawledWebLinks[shortName] = webFile
except:
print ("*************\nUnexpected Error*****", sys.exc_info()[0])
stopOrProceed = input("Hmm..stop or proceed? Enter 1 to stop, enter anything else to continue")
if stopOrProceed ==1 :
print ('OK...Stopping\n')
break
else:
print ("Cool! Let's continue\n")
continue
print (crawledWebLinks)
Another note, simply printing out the type of error in your except block is not very useful. I was able to debug your code in 30 seconds once I removed that and viewed the actual traceback.

Related

having trouble implementing the right word to a url link

i have a program that asks users which department they want to choose from, i made the question into a while loop that would keep asking the same question until they put a valid response in. and whenever they wanted to break the loop they would type the word "exit". once they type in a valid response it would add ask another question in another function that would ask for the class your want info on based on the department you're in, it would then lead it to a website where i can get the information i need. the issue im facing is that it doesn't take the valid response that was typed in, it would instead implement the word "exit" which was used to break out of the loop as it was the last input. i want it to take the input before that and not "EXIT"
here is the code
def get_departments():
umd_departments = requests.get("https://api.umd.io/v0/courses/departments")
umd_departments_list = umd_departments.json()
umd_departments_list2 = json.dumps(umd_departments_list, indent=1)
department_storage = [department['dept_id'] for department in umd_departments_list]#comprehensive for loop to put the department ID into a list
print(department_storage)
while True:
dept = input('what department are you in right now: ')
dept = dept.upper()
if dept == 'EXIT':
break
if dept not in department_storage:
print("not approriate response")
else:
try:
department_url = requests.get(f"https://api.umd.io/v0/courses?dept_id={dept}")
specific_major =department_url.json()
keep_keys = ["course_id"]
courses = [{k: json_dict[k] for k in keep_keys}
for json_dict in specific_major]
print(courses)
except Exception as e:
print(e)
return courses,dept
def get_classes(courses):
classes = [course['course_id'] for course in courses]
print(classes)
course_select = input('what class would you like to select').upper()
if course_select not in classes:
raise ValueError(" class does not exist")
else:
driver = webdriver.Chrome(ChromeDriverManager().install())
url = f"https://app.testudo.umd.edu/soc/202008/{dept}"
driver.get(url)
section_container = driver.find_element_by_id(f"{course_select}")
section_container.find_element_by_xpath(".//a[#class='toggle-sections-link']").click()# it would click on the show section button on the site that would reveal the sections
sleep(1)
section_info = section_container.find_element_by_xpath(".//div[#class='sections sixteen colgrid']").text
return section_info
let's say for example in the get departments function i type in INST and its considered a valid response, it would then ask for which class i want to choose from that department. it would then create a url that would get me the info i need like this:
https://app.testudo.umd.edu/soc/202008/INST
however i get this:
https://app.testudo.umd.edu/soc/202008/EXIT
the latter doesn't produce anything as it doesn't exist and it causes errors. is there a way to make it so that it doesn't make the "EXIT" input stored into a valuable and instead takes the valid response before it? greatly appreciated if anyone could help.
Looks like you just need another variable
userInput = input('what department are you in right now: ')
userInput = userInput.upper()
if userInput == 'EXIT':
break
else:
dept = userInput

Break outside the loop

I am new to Python. I am trying to run the following code. But every time I try to run it, the IDE says that the break is outside the loop
catname = []
print("Enter the name of the cats")
name = input()
if name == '':
break
catname = catname+[name]
print("The cat Names are :")
for catname in name:
print(name)
Can you please help me?
Thanks
You use break when you want to break free from a loop, to exit the loop, to jump to the nearest code after the loop.
Your code doesn't contain a loop, so nothing to break free from, hence the error.
I think you meant exit() instead of break
You use "break" just inside the loop ("for" or "while"), you are trying use brake inside the "if"
How about this:
if name != '':
catname = catname+[name]
print("The cat Names are :")
for catname in name:
print(name)
Your break statement is not in a loop, it's just inside an if statement.
But maybe you want to do something like the following.
If you want to let the user enter an random number of names and print the names out, when the user entered nothing, you can do the following:
# Here we declare the list in which we want to save the names
catnames = []
# start endless loop
while True:
# get the input (choose the line which fits your Python version)
# comment out the other or delete it
name = input("Enter the name of a cat\n") # input is for Python 3
# name = raw_input("Enter the name of a cat\n") # raw_input is for Python 2
# break loop if name is a empty string
if name == '':
break
# append name to the list catnames
catnames.append(name)
print("The cat names are :")
# print the names
for name in catnames:
print(name)
What you are looking for is exit().
However, your code has also other problems, here is a piece of code that does what you probably want (when prompted, enter the names separated by spaces, like: Cat1 Cat2):
name = raw_input("Enter the name of the cats: ")
if len(name) == 0:
exit()
print("\nThe cat Names are:")
for c_name in name.split():
print(c_name)
If this is the entirety of your code, then it's telling you exactly what the problem is:
catname = []
print("Enter the name of the cats")
name = input()
if name == '':
break
You have a break statement in the code that's not contained inside a loop. What do you expect the code above to do?

Undefined variable in ceaser cipher -Python

import sys
import string
array = []
while True:
input = raw_input("Please enter no more than 10 characters, one per line, and terminate the message by entering % ")
def main():
key = 3
message = input
cryptMessage = ""
for ch in message:
cryptMessage = cryptMessage + chr(ord(ch) + key)
if input == "%"
print array, len(array), "The coded message is:", cryptMessage
sys.exit(1) #This tells the program to exit
array.append(input)
main()
Basically I have everything working the way I want it to except for printing the user input text in encrypted form. It already prints in regular form, I want it to print in both regular and encrypted forms. It keeps saying that the cryptMessage variable in the print line is undefined. I thought I had defined it in the code above but apparently not. What am I missing?
I re-worked your code a little bit. The reason you were getting an undefined variable error is because you were defining the cryptMessage in your main() function, and it was not accessible outside of that.
import sys
# these can be declared up here
array = []
key = 3
# loop until the user tells us not to
while True:
# grab the input from the user
input = raw_input("Please enter no more than 10 characters, one per line, and terminate the message by entering % ")
# the string we'll fill
cryptMessage = ""
# go over the input and 'cypher' it
for ch in input:
cryptMessage += chr(ord(ch) + key)
# add this line of message to the array
array.append(cryptMessage)
# when the user tells us to stop
if input == "%":
# print and break out of the while loop
print "The coded message is:", ' '.join(array)
break
Output:
Please enter no more than 10 characters, one per line, and terminate the message by entering % tyler
Please enter no more than 10 characters, one per line, and terminate the message by entering % is
Please enter no more than 10 characters, one per line, and terminate the message by entering % my
Please enter no more than 10 characters, one per line, and terminate the message by entering % name
Please enter no more than 10 characters, one per line, and terminate the message by entering % %
The coded message is: w|ohu lv# p| qdph (

Python: Print just once in raw_function

result = raw_input("Enter results file: ")
while True:
try:
result = get_marks_from_file(result)
break
except IOError:
print 'Please supply correct file. '
Above is the raw_input function that I am using to bring a file. When I put correct file name, it works well. But when I put something else such as 'asdsada', the sentence 'please supply correct file. ' is printed and it doesn't stop. Is there any method for just one printing and return to the question?
Any advices or helps would be appreciated.
Move the raw_input inside your loop:
while True:
result = raw_input("Enter results file: ")
try:
result = get_marks_from_file(result)
break
except IOError:
print 'Please supply correct file. '

Having trouble parsing a txt file into a list full of zip codes in my zipcode lookup program

Hello everyone thanks for looking into my problem. What I am trying to do is write a "Structured" program in python that takes txt from a file and parses it into lists. Then after closing the file, I need to reference the user input (zipcode) in those lists and then print out the city and state according to the zipcode that they entered. My instructor is having us use structure by making several functions. I know there are probably lots of more efficient ways of doing this, but I must keep the structure thats in place.
EDIT
Here is my code(Current):
#-----------------------------------------------------------------------
# VARIABLE DEFINITIONS
eof = False
zipRecord = ""
zipFile = ""
zipCode = []
city = []
state = []
parsedList = []
#-----------------------------------------------------------------------
# CONSTANT DEFINITIONS
USERPROMPT = "\nEnter a zip code to find (Press Enter key alone to stop): "
#-----------------------------------------------------------------------
# FUNCTION DEFINITIONS
def startUp():
global zipFile
print "zipcode lookup program".upper()
zipFile = open("zipcodes.txt","r")
loadList()
def loadList():
while readRecord():
pass
processRecords()
def readRecord():
global eof, zipList, zipCode, city, state, parsedList
zipRecord = zipFile.readline()
if zipRecord == "":
eof = True
else:
parsedList = zipRecord.split(",")
zipCode.append(parsedList[0])
city.append(parsedList[1])
state.append(parsedList[2])
eof = False
return not eof
def processRecords():
userInput = raw_input(USERPROMPT)
if userInput:
print userInput
print zipCode
if userInput in zipCode:
index_ = zipcode.index(userInput)
print "The city is %s and the state is %s " % \
(city[index_], state[index_])
else:
print "\nThe zip code does not exist."
else:
print "Please enter a data"
def closeUp():
zipFile.close()
#-----------------------------------------------------------------------
# PROGRAM'S MAIN LOGIC
startUp()
closeUp()
raw_input("\nRun complete. Press the Enter key to exit.")
Here is a sample from the zipcode txt file:
00501,HOLTSVILLE,NY
I am definitely stuck at this point and would appreciate your help in this matter.
EDIT
Thanks for all the help everyone. I really do appreciate it. :)
why you fill the lists zipcode, city , state like that, i mean in each user entry we get the next line from the file
i think that you should do :
def loadList():
# Fill all the list first , make the readRecord() return eof (True or False).
while readRecord():
pass
# than process data (check for zip code) this will run it only one time
# but you can put it in a loop to repeat the action.
processRecords()
about your problem :
def processRecords():
userInput = raw_input(USERPROMPT)
# Check if a user has entered a text or not
if userInput:
# check the index from zipcode
if userInput in zipcode:
# the index of the zipcode in the zipcode list is the same
# to get related cities and states.
index_ = zipcode.index(userInput)
print "The city is %s and the state is %s " % \
(city[index_], state[index_])
else:
print "\nThe zip code does not exist."
else:
print "Please enter a data"
one of the beauties of Python is that it's interactive. if you take processRecords() out of loadList(), and then at the bottom of your program put:
if __name__ == '__main__':
processRecords()
Then, from the command prompt, type "python". You'll get the Python shell prompt, ">>>". There you type:
from zipcodes import * # this assumes your program is zipcodes.py
dir() # shows you what's defined
print zipCode # shows you what's in zipCode
that ought to help debugging.
Strings don't have an append method like lists do. What I think you're trying to do is append the strings zipCode, city, and state to parsedList. This is the code you'd use to do that:
parsedList.append(zipCode)
parsedList.append(city)
parsedList.append(state)
Or, even more compactly:
parsedList = [zipCode, city, state]
Let me know if you get another error message and I can offer more suggestions.

Categories