I am very very new to python but I have started an internship that requires me to do some python work, I am asked to create an application that can retrieve data from a webpage(IP address) and then compare those values to the correct values and then print out if it has passed or not. Please check this diagram I have made so that you guys can understand it better. Please take a look.
So far I have only written some python code to check if the website/ip address is up or not but I have no idea how to go further. could you guys please help me to execute the further steps with some examples maybe?
Here is a picture of the website. the values circled in red color need to be compared with the Auxiliary Values I hope this picture helps.
However, I could use http://192.168.100.2/globals.xml on this page to compare the values. Any help is much appreciated.
import requests
import urllib.request
import eventlet
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
eventlet.monkey_patch()
with eventlet.Timeout(10):
print(urllib.request.urlopen("http://192.168.100.5").getcode())
print("Website is UP")
eventlet.monkey_patch()
with eventlet.Timeout(10):
print(urllib.request.urlopen("http://10.10.10.2").getcode())
print("Website is UP")
You are off to a great start! Your next steps should be identifying unique traits about the elements that you want to scrape. Specifically, look for things like class or id names that are unique to only the data that you want to scrape.
You can also use tools like Selector Gadget (https://selectorgadget.com/) that can help automate the process. Unfortunately, since you are accessing local IP addresses, nobody here will be able to help you find these.
After you find the proper selectors, you can use BeautifulSoup to view the data. I'd recommend looking at the find and findall commands that BeautifulSoup has.
Related
I have seen a lot of posts on how to use requests.get(link).json(). I followed along and I am able to import the link but when I try to focus on a specific item with entry such as: optionchain['ask'], it gives me an error message.
I use data from this yahoo finance link: https://query2.finance.yahoo.com/v7/finance/options/amd
and would like to import as specific variables the different strike prices, ask and bid. Could anyone please help me with that? Thank you in advance
The JSON at the link you posted has multiple levels. To get the ask price you have to call
data['optionChain']['result'][0]['quote']['ask'] where data is the result from requests.get(link).json()
import requests
data = requests.get(r"https://query2.finance.yahoo.com/v7/finance/options/amd").json()
ask = data['optionChain']['result'][0]['quote']['ask']
I have started programming in python not too long ago and I came across with a problem for which I got stuck. I try to download (called export the button on the page) a csv file to my desktop from this page: http://kapar.mavir.hu/kapar/daily-publication.jsp?locale=hu_HU on a daily basis in python. My problem is that all the solution I found are for pages where the urls change after you push the button but in this case nothing happens with url after the push so the methods don't work as far as I'm concerned. How can I code this one down in python (python 2.7 are in use)?
Any help/suggestions are appreciated.
Thanks in advance.
Please provide the code you have so far. In the mean time;
find out what URL you need to fetch from if you haven't already (check html for this)
import urllib.request
import urllib.parse
data = urllib.parse.urlencode({...})
data = data.encode('ascii')
with urllib.request.urlopen("http://...", data) as fd:
print(fd.read().decode('utf-8'))
Also check this post out.
I need to make an app that uses Python to search for specific names on a website. For instance, I have to check if the string "Robert Paulson" is being used on a website. If it is, it returns True. Else, false. Also,is there any library that can help me make that?
Since you have not attempted to make your application first, then I am not going to post code for you. I will however, suggest using:
urllib2:
A robust module for interacting with webpages. i.e. pull back the html of a webpage.
BeautifulSoup (from bs4 import BeautifulSoup):
An awesome module to "regex" html to find what is is that you're looking for.
Good luck my friend!
You could do something similar to this other answer. You will just need the regex to find your string.
I have also used Selenium webdriver to solve some more complex webesite searching, although I think the link I provided would solve your problem more simply.
Beginning programmer here. Just completed the CS61A introduction Python class # UC Berkeley, and I was thinking of trying to implement a little program:
Basically, I want to be able to enter a band name, and have the program search www.setlist.fm, and return a bunch of setlists for recent concerts of that band. Sounds easy enough... I have a VERY basic idea with what to do with urllib and urlopen, but that's about it. Any pointers or guidance on how to get started?
Thanks!
Read about their API.
http://api.setlist.fm/docs/index.html
Read how to make HTTP GET requests using urllib2
http://www.voidspace.org.uk/python/articles/urllib2.shtml
I am a newbie trying to achive this simple task by using Scrapy with no luck so far. I am asking your advice about how to do this with Scrapy or with any other tool (with Python). Thank you.
I want to
start from a page that lists bios of attorneys whose last name start with A: initial_url = www.example.com/Attorneys/List.aspx?LastName=A
From LastName=A to extract links to actual bios: /BioLinks/
visit each of the /BioLinks/ to extract the school info for each attorney.
I am able to extract the /BioLinks/ and School information but I am unable to go from the initial url to the bio pages.
If you think this is the wrong way to go about this, then, how would you achieve this goal?
Many thanks.
Not sure I fully understand what you're asking, but maybe you need to get the absolute URL to each bio and retrieve the source code for that page:
import urllib2
bio_page = urllib.urlopen(bio_url).read()
Then use a regular expressions or other parsing to get the attorney's law school.