I'm trying to scrape data from this webpage and I have only used pandas once for scraping. I want to get the 7 Day High/ Low values. It is on the bottom of the page. What code should I write, and which modules should I use? Thanks for your time! This would help me so much!
https://coinmarketcap.com/currencies/bitcoin/historical-data/
Why not use pandas again and come back when you run into problems.
Related
I am trying to find/work out the code needed to get a table of sales data to regenerate at the beginning of each sales day using googlesheets and python3. I've been combing the web and youtube but not had any luck yet. Any advice would be really appreciated :)
Right now I'm trying to scrape the dividend yield from a chart using the following code.
df = pd.read_html('https://www.macrotrends.net/stocks/charts/BMO/Bank-of-Montreal/dividend-yield-history')
df = df[0].dropna()
But the code wont pick up the chart's data.
Any suggestions on pulling it from the website?
Here is the specific link I'm trying to use: https://www.macrotrends.net/stocks/charts/BMO/Bank-of-Montreal/dividend-yield-history
I've used the code for picking up the book values but the objects they're using for the dividends and book values must be different.
Maybe I could use Beautiful Soup?
Sadly that website is rendered dynamically, so there's nothing in the html pandas is getting to scrape from. (The chart is loaded after the page). Scraping manually isn't going to help you here, because the data isn't there. (It's fetched after the page is loaded.)
You can either find an api which provides the data (best, quite possible given the content), work out where the page is fetching its data from and see if you can get it directly (better if possible), or use something like selenium to control a real browser, render the page, get the html, and then use that.
I am new here and I am trying to scrape the nearest station and distance list from this link https://www.onthemarket.com/details/10405122/ I have been stuck here for a day. any help would be apreciated.
I have tried
response.xpath('//div[#class = "tab-content"]/span')
response.xpath('//section//span[#class="poi-name"]')
response.xpath('//section[#class="poi"]/div//text()').extract()
nothing seems to work.
please if you are able to get it please do explain why I failed that would be much apreciated.
The data is not in the downloaded html:
<ol class="tab-list"></ol><div class="tab-content"></div>
It probably receives the data in another call. Try not hurry up writing the scraper, invest some time to understand how this particular UI works. I would also suggest downloading data via curl or scrapy shell "your_url" (as in this case it will not be downloaded by browser, which renders the page and can trick you like right now).
I am very very new to python but I have started an internship that requires me to do some python work, I am asked to create an application that can retrieve data from a webpage(IP address) and then compare those values to the correct values and then print out if it has passed or not. Please check this diagram I have made so that you guys can understand it better. Please take a look.
So far I have only written some python code to check if the website/ip address is up or not but I have no idea how to go further. could you guys please help me to execute the further steps with some examples maybe?
Here is a picture of the website. the values circled in red color need to be compared with the Auxiliary Values I hope this picture helps.
However, I could use http://192.168.100.2/globals.xml on this page to compare the values. Any help is much appreciated.
import requests
import urllib.request
import eventlet
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
eventlet.monkey_patch()
with eventlet.Timeout(10):
print(urllib.request.urlopen("http://192.168.100.5").getcode())
print("Website is UP")
eventlet.monkey_patch()
with eventlet.Timeout(10):
print(urllib.request.urlopen("http://10.10.10.2").getcode())
print("Website is UP")
You are off to a great start! Your next steps should be identifying unique traits about the elements that you want to scrape. Specifically, look for things like class or id names that are unique to only the data that you want to scrape.
You can also use tools like Selector Gadget (https://selectorgadget.com/) that can help automate the process. Unfortunately, since you are accessing local IP addresses, nobody here will be able to help you find these.
After you find the proper selectors, you can use BeautifulSoup to view the data. I'd recommend looking at the find and findall commands that BeautifulSoup has.
I am trying to scrape data from Trip.com, specifically this page here . When you visit this page in browser it shows result of 20 hotels but as you scroll down more hotel details are loaded. Now what i want is to scrape data of first 50 hotels. But i am asked to not use Scrapy, Selenium. Any help is appreciated. Thanks in advance.
If you use the DevTools and look at the Network tab you can see that a request goes out to https://www.trip.com/restapi/soa2/16709/json/HotelSearch? This endpoint returns the results in a JSON format.
The next step would be to reverse engineer and copy the request into Python using urllib, which is built into Python. This step might require some experience and knowledge of how HTTP requests work.