Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
i would like to scrape a table from a website and display the scraped table afterwards in a gui created with python. The table (and the website) is in html. How would you make it? In case that is possible i would love to display the table in my gui afterwards formatted in the exact same way. "Hypothetical it would be enough for me to take a screenshot of the table and display this picture afterwards in the gui"
Ps: i thought using scrapy but i guess in that case i have to build the table from scratch again in my gui later?
Thanks for your ideas!
You don't really need scrapy if you just need to scrape a single page and don't need crawling. As for the toolkit BeautifulSoup + Requests is a popular choice.
As for the table content - you will most likely store in a list of lists (maybe some structure - it's up to you) and than populate table in your GUI application.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
I'm using beautiful soup to scrape a webpages.
I am trying to scrape data from this https://painel-covid19.saude.ma.gov.br/vacinas. But the problem is I am getting the tags in outputs empty. In the Inspect Element I can see the data, but in page source not. You can see the code is hidden in . How can I retrieve it using python? Someone can help me?
The issue isn't "not visible". The issue is that the data is being filled in by Javascript code. You won't see the data unless you are executing the Javascript on the page. You can do that with the selenium package, which runs a copy of Chrome to do the rendering.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I need to scrape historical market rates of freight from differing origins and destinations. Currently, I only have interactive graphs like this available to me:
Sample Graph
You have to click on the graph to get the numbers to appear (all of them appear at once).
I have some experience with HTML web scraping through the Scrapy library, but I was wondering if something like BeautifulSoup would be capable of handling this type of problem.
To put it shortly - yes but it depends.
Most javascript graphs work by embeding json data in <script> tags or making ajax request for it. So there is graph data in json format somewhere - you just need to find it.
To find it you should first open up page source and ctrl+f for some keypoints you see in the graph. In your case start with £407 - it's very likely it's in embeded json:
<script type="application/ld+json">
{'prices': ['£407',...]}
</script>
Alternative it could also be retrieved as AJAX request. For example take this craft.co case. When you load https://craft.co/netflix page it makes AJAX request for graph data:
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to create a web app for my company that functions like a tool suite for non-technical employees to use. I'm using
Python
Selenium WebDriver,
BeautifulSoup,
Srapy,
Django
Is it possible and is this the right approach?
"The idea is to create a variety of methods that can be run by
clicking a button and/or inserting content in inputs, and having the
tests / functions run and returning the specified results."
Maybe Flask can help you here. It has a nice functionality to route urls to specific functions or methods in your code. It provides a convenient way to bind actions to code. You can search the web for how you can leverage Flask to cater to your specific needs, but here I just wanted to convey that it would help.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am trying to write a program in Python, so that I can use a Raspberry Pi to take hardware input and then do some IoT stuff as a result.
I'm comfortable with all the hardware stuff, and am pretty sure that I'll be able to figure out how to do Facebook posting and tweeting, but I also want to submit data into a webpage.
I don't have control of the webpage, access to the code or anything like that and it's going to be nigh-on impossible to get the access to the code so I'm relying on inspect element here to get any data which I need. I'm also not supposed to post the URL of the said webpage publicly, as it needs a login which I am not at liberty to release.
I want to interact with several features on the webpage, namely:
A mouse-over drop-down menu
A text entering field
A few buttons
I think that I need to do something with 'event listeners', but I'm unsure how to go about this; I have quite a lot of Python experience but not much web development knowledge.
Thanks
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I made in python a webcrawler, and I want to make Django website to display the results of the crawling, can I do it? for example: the crawler enter to "https://stackoverflow.com/questions" and take the questins of the first page and display it on the website(maybe on the tamplate of Django).
EDIT: I just re-read your question and it looks like you're already doing the scraping? You should implement that functionality into a Django site and then yes of course you can display it.
Yes you can do it. A great combination for web scraping with django is using the python requests library (which I'm guessing you're using) andBeautiful Soup. I've used this combination in a few different projects and I've been happy with it.
Another popular option uses the python Scrapy library and combines it into a django app called django-dynamic-scraper.
Hopefully this points you in the right direction.