Im building tools for a business I am starting, and one of the tools I need is a live reader of scrap prices to put into my website as well as deciphering overhead/profit. How would I take that information and then put it into a live data table?
Very amateur programmer, but better to do it myself.
You can use flask and Beautiful Soup. With the flask you will be able to save the data after obtaining the information from the site after manipulating the html with beautiful soup.
Related
I am going to make a simple website that displays data from a script I made with Beautiful Soup. I already have a working Python code that scrapes the data I need.
What I do not know how to do is drop this Python code into a website that scrapes data on a daily basis and displays it nicely. I am scraping daily stock prices and I want my site to store each day's price in a database and plot everything to date on a simple line graph and an accompanying table.
What keywords do I use in researching this? I've started to look into Django vs Flask. Am I on the right path?
For a research project, I am trying to implement a script that will go through this site and save the set of images from each species, with the file saved as "genus_species_index.jpeg". I have been looking at Beautiful soup tutorials as well. The main issue is that accessing each species page via script has proved to be quite difficult.
I would recommend looking at scrapy to solve your problem. Beautiful soup is a parser (that does a great job of what you are looking for) but does not handle the crawling. Generally when doing tasks like this you will first scrape the site and then parse it to extract the data and spiders like scrapy were invented for the first purpose. (here is a link for some context: https://www.scrapehero.com/a-beginners-guide-to-web-scraping-part-1-the-basics/)
I would like to download census data (year 2001 and 2011) from http://bhuvan5.nrsc.gov.in/bhuvan/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.MapPreviewPage as kml/kmz format for multiple states of India. I am thinking to automate the process using python as the data contains huge number of file. I am a beginner in this kind of programming. It would be great if any one help me or guide me regarding this.
This is an ugly target for a first scrapping project as it has a javascript paginated table.
As you'll need a javascript engine, Python's most friendly option is to use Selenium with python bindings to scrap your page.
Before using it, you should have read some basics on html DOM and xpaths.
I am trying to scrape data from the morningstar website below:
http://financials.morningstar.com/ratios/r.html?t=IBM®ion=USA&culture=en_US
I am currently trying to do just IBM but hope to eventually be able to type in the code of another company and do this same with that one. My code so far is below:
import requests, os, bs4, string
url = 'http://financials.morningstar.com/ratios/r.html?t=IBM®ion=USA&culture=en_US';
fin_tbl = ()
page = requests.get(url)
c = page.content
soup = bs4.BeautifulSoup(c, "html.parser")
summary = soup.find("div", {"class":"r_bodywrap"})
tables = summary.find_all('table')
print(tables[0])
The problem I am experiencing at the moment is unlike a simpler webpage I have scraped the program can't seem to locate any tables even though I can see them in the HTML for the page.
In researching this problem the closest stackoverflow question is below:
Python webscraping - NoneObeject Failure - broken HTML?
In that one they explained that Morningstar's tables are dynamically loaded and used some json code I am unfamiliar with and somehow generated a different weblink which managed to scrape the data but I don't understand where it came from?
It's a real problem scraping some modern web pages, particularly on pages generated by single-page applications (where the content is maintained by AJAX calls and DOM modification rather than delivered as ready-to-go HTML in a single server response).
The best way I have found to access such content is to use the Selenium web testing environment to have a browser load the page under the control of my program, then extract the page contents from Selenium for scraping. There are other environments that will execute the scripts and modify the DOM appropriately, but I haven't used any of them.
It's not as difficult as it sounds, but it will take you a little jiggering around to get there.
Web scraping can be greatly simplified when the site offers an API, be it officially supported or just an unofficial hack. Even the hack is better than trying to fiddle with the HTML which can change every day.
So a search for morningstar api might be fruitful. And, in fact, some friendly Gister has already worked this out for you.
Would the search be without result, a usually fruitful approach is to investigate what ajax calls the page is doing to retrieve data and then issue them directly. This can be achieved via the browser debuggers, tab "network" or so where each request can be investigated in detail in a very friendly UI.
I've found scraping dynamic sites to be a lot easier with JavaScript than with Python + Selenium. There is a great module for nodejs/phantomjs: ScraperJS. It is very easy to use: it injects jQuery into the scraped page and you can extract data with jQuery selectors.
I'm currently trying to get separate lists of prices to upload straight to my website: virtualcoincomparison.com
How do I do this? Is this even possible using Beautiful Soup? I'm new to python and Beautiful Soup so i'm not sure
There are many ways you could do this:
use SFTP or WebDAV to upload data files
create an admin page on your site (allow it to pull from your computer)
create an API, then use the API to submit the data
set up a timed script, so your site does the scraping on its own
?? probably about 30 other ways