Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
So let's say I have this URL: https://www.python.org/
and I want to download the page's source into a .txt file named python_source.txt
how would I do that?
Use urllib2, Here's how it's done:
response = urllib2.urlopen(url)
content = response.read()
Now you can save the content in any text file.
The python package urllib does just this. The documentation gives a very clear example on what you want to do.
import urllib.request
local_filename, headers = urllib.request.urlretrieve('http://python.org/')
html = open(local_filename)
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed last year.
Improve this question
I'm using the below code to extract full HTML:
cont = await page1.content()
The website I intend to extract from is:
https://www.mohmal.com/en
which is a website to make temporary email accounts. The exact thing I want to do is reading the content of received emails, but by using the above code, I could not extract inner frame HTML where received emails contents placed within it. How can I do so?
Did you try using urllib?
You can use the urllib module to read html websites.
from urllib.request import urlopen
f = urlopen("https://www.google.com")
print(f.read())
f.close()
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
Given a FB url like:
https://www.facebook.com/photo.php?fbid[LONGUSERID]&set=a.313002535549859&type=3&theater
How could I extract the real photo URL using PHP or Python?
Normally the actual URL looks like this ( as seen in Chrome Network tab)
https://scontent.fbru1-1.fna.fbcdn.net/v/t31.0-1/cp0/p32x32/11942095_139657766378816_623531952343456734_o.jpg?_nc_cat=106&_nc_sid=0081f9&_nc_ohc=VpijQtyWbUQAX-fsPMj&_nc_ht=scontent.fbru1-1.fna&oh=eb4435eed183716c807b405d0d57c3a4&oe=5F674BAB
But is there a way to automate this extraction this with script? Any example would be appreciated.
The simplest example.
I just got an HTML page, divided the text by double quotes into lines. Then I checked to see if the JPG extension was on the line.
import requests
from html import unescape
from urllib.parse import unquote
url = "https://www.facebook.com/photo.php?fbid=445552432123146"
response = requests.get(url)
if response:
lines = response.text.split('\"')
for line in lines:
if ".jpg" in line:
print(unquote(unescape(line)))
else:
print("fail!")
With the help of Selenium you can already search for elements in HTML code correctly.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I get a link with requests.get and when I check history it's empty although the link redirects to another address when I open it with my browser. What is the problem?
import requests
r=requests.get('http://dir.iran.ir/home?p_p_id=webdirectorydisplay_WAR_webdirectoryportlet&p_p_lifecycle=0&p_p_state=exclusive&p_p_mode=view&_webdirectorydisplay_WAR_webdirectoryportlet_itemEntryId=14439&_webdirectorydisplay_WAR_webdirectoryportlet_cmd=redirectToLink')
result=r.history
but result equal with empty list
and final link is http://www.dps.ir/
You should check the result of that URL first.
>>> r.content
'<script type="text/javascript">window.location.href="http://www.dps.ir";</script> '
Requests library doesn't provide ability to execute Javascript, so that explains why there is no history.
PS: Btw you could give phantomjs a shot.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
I want to extract the HTML from a webpage:
import urllib2
req = urllib2.Request('https://www.example.com')
response = urllib2.urlopen(req)
fullhtml = response.read()
I tried with "ulrllib2" but since the page is built dynamically, the HTML content is empty.
Is there a way to wait for the javascript to load?
Take a look at this http://phantomjs.org/ . Most websites are javascript based and php or python can not execute them. I think this library will be the best you can get.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am new to Python and I want to use Reddit API to retrieve top 10 headline on the front page of Reddit using Python. I tried to read the API documentation but I am not able to understand how to proceed.
It would be great if someone can give me an example.
Thanks
Here's a quick example on how to download the json data you want. Basically, open the URL, download the data in JSON format, and use json.loads() to load it into a dictionary.
try:
from urllib.request import urlopen
except ImportError: # Python 2
from urllib2 import urlopen
import json
url = 'http://www.reddit.com/r/python/.json?limit=10'
jsonDownload = urlopen(url)
jsonData = json.loads(jsonDownload.read())
From there, you can print out 'jsonData', write it to a file, parse it, whatever.