I am using bs4 to extract text from a web document. But Its output is very strange. like
Ú©Ø¨Ú¾Û ÛÛ Ø¨Ø§Øª سÙجھ ÙÛÚº ÙÛ Ø§ÙØªÛ ØªÚ¾ÛÛ Ù¾Ú¾Ø± اÙÛØ³ØªÛ Ø§ÙÛØ³ØªÛ Ø¬Ø¨ Ú©ÚÚ¾ عÙ٠اÙÙÛ Ø´Ø±Ùع ÛÙØ¦Û ØªÙ Ø¨Ø§Øª Ú©ÚÚ¾ Ù¾ÙÛ Ù¾ÚÛÛÙÛک٠اب ÛÛ Ø¨Ø§Øª اÛØ³Û Ø³Ùجھ ÙÛÚº اÙØ¦Û Ú©Û Ø³ÙÚا اÙÙ¾ Ú©ÛÙÙÚ¯ÙÚº Ú©Û Ø¨Ú¾Û Ø´Ûئر کرÙÚºÛ ÚÙد رÙز Ùب٠ÙÛرا 8 Ù
I think it is some encoding. I am a new user of bs4. Please guide me how to decode it to show as urdu text.
Here is a document source whose title I want to extract
Follwoing code I am using to do it.
from bs4 import BeautifulSoup
import urllib2
import requests
url="http://blog.jang.com.pk/blog_details.asp?id=11058"
r = requests.get(url)
data = r.text
soup = BeautifulSoup(data,'lxml')
print str(soup.title)
Burhan Khalid's answer works, but because the original web page is encoded in utf-8:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
You should update the requests' response field to match the original page's encoding:
from bs4 import BeautifulSoup
import urllib2
import requests
url="http://blog.jang.com.pk/blog_details.asp?id=11058"
r = requests.get(url)
# Update encoding to match source
r.encoding = "utf-8"
data = r.text
soup = BeautifulSoup(data,'lxml')
print str(soup.title)
Now any field you access will have the correct encoding rather than having to set to Urdu on a per field basis.
If you simply try to print the string, you'll get garbage characters out:
>>> import requests
>>> from bs4 import BeautifulSoup as bs4
>>> r = requests.get('http://blog.jang.com.pk/blog_details.asp?id=11058')
>>> s = bs4(r.text, 'lxml')
>>> print s.title.text
Ú©ÚÚ¾ تÙØ¬Û Ø§Ø³ طر٠بھÛ!
You need to encode it properly, since the result is a unicode bytestring.
>>> print s.title.text.encode('iso-8859-1')
کچھ توجہ اس طرف بھی!
If it displays the glyphs correctly, but in the wrong order (ie, they are not right-to-left), then this is a problem with your operating system/terminal/shell/program you are using to run the application.
The above is from gnome-terminal, which doesn't support Arabic RTL properly.
If I run the same code in mlterm:
The white box is there because I am using an Arabic font, which doesn't have all the characters in the Urdu language.
I think what is happening is that there some badly formed Unicode in the website response:
----> 1 r.content.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd8 in position 1106: invalid continuation byte
Hence the text is being decoded using the ANSI codec which is obviously wrong. You can work around this issue by calling decode with the option errors='ignore' (we are using the content rather than text because this is the raw binary response from the website:
data = r.content.decode(errors='ignore')
soup = BeautifulSoup(data,'lxml')
print (str(soup.title))
<title>کچھ توجہ اس طرف بھی!</title>
Related
I am scraping an arabic website using BeautiifulSoup but the arabic characters returned are returned inverted and separate chars (pasting it here correctly encodes it so you have to trust me on that :).
The website charset is using UTF-8
<meta charset=UTF-8>
This is how I am parsing it:
url = 'https://new.kooora4live.net/matches-today-1/'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'lxml', from_encoding='utf-8')
Writing the requested HTML to a file with utf-8 encoding correctly formats it in the file so it seems as it's an issue with BeautifulSoup.
Any idea what am I doing wrong or how to fix it?
....
Update:
Encoding with utf-8-sig also doesn't work.
You need to set the page encoding to match its apparent encoding.
Try this:
import requests
from bs4 import BeautifulSoup
page = requests.get('https://new.kooora4live.net/matches-today-1/')
page.encoding = page.apparent_encoding
soup = BeautifulSoup(page.content, 'lxml').select("a")
print("\n".join(a.getText(strip=True) for a in soup))
This will print out:
الأخبار
أهم المباريات
جداول
ترتيب الفرق
ترتيب الهدافين
مباريات الأمس
مباريات اليوم
مباريات الغد
جمهورية التشيك7:00 PM0-0لم تبدأ بعدالدنماركبي ان ماكس 1احمد البلوشييورو 2020
اوكرانيا10:00 PM0-0لم تبدأ بعدإنجلترابي ان ماكس 1حسن العيدروسيورو 2020
and more ...
I’m trying to parse a page and I’m having some issue with special characters such as é è à, etc.
According to the Firefox page information tool, the page is encoded in UTF - 8
My code is the following :
import bs4
import requests
url = 'https://www.registreentreprises.gouv.qc.ca/RQEntrepriseGRExt/GR/GR99/GR99A2_05A_PIU_AfficherMessages_PC/ActiEcon.html'
page = requests.get(url)
cae_obj_soup = bs4.BeautifulSoup(page.text, 'lxml', from_encoding='utf-8')
list_all_domain = cae_obj_soup.find_all('th')
for element in list_all_domain:
print(element.get_text())
The output is :
Pêche et piégeage
Exploitation forestière
I tried changing the encoding with iso-8859-1 (French encoding) and some other encodings without success. I read several posts on parsing special characters, and they basically states that it’s an issue of selecting the right encoding. Is there a possibility that I can’t decode correctly the special characters on some specific webpage or am I doing something wrong ?
The requests library takes a strict approach to the decoding of web pages. On the other hand, BeautifulSoup has powerful tools for determining the encoding of text. So it's better to pass the raw response from the request to BeautifulSoup, and let BeautifulSoup try to determine the encoding.
>>> r = requests.get('https://www.registreentreprises.gouv.qc.ca/RQEntrepriseGRExt/GR/GR99/GR99A2_05A_PIU_AfficherMessages_PC/ActiEcon.html')
>>> soup = BeautifulSoup(r.content, 'lxml')
>>> list_all_domain = soup.find_all('th')
>>> [e.get_text() for e in list_all_domain]
['Agriculture', "Services relatifs à l'agriculture", 'Pêche et piégeage', ...]
import bs4 as bs
import urllib.request
sauce = urllib.request.urlopen('https://stats.swehockey.se/Teams/Info/TeamRoster/10333').read()
soup = bs.BeautifulSoup(sauce, 'lxml')
print(soup)
This code producing the following error
UnicodeEncodeError: 'ascii' codec can't encode character '\xbb' in position 1509: ordinal not in range(128)
I have tried several work arounds but they all have some drawback. After searching on stackoverflow, I found the solution of changing .stdout, like this:
import bs4 as bs
import urllib.request
import sys
import codecs
sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stderr = codecs.getwriter('utf-8')(sys.stderr)
sauce = urllib.request.urlopen('https://stats.swehockey.se/Teams/Info/TeamRoster/10333').read()
soup = bs.BeautifulSoup(sauce, 'lxml')
print(soup)
I no longer get the error, however, the output is no longer directed towards the terminal. I'm not sure why this is happening. Using the .prettify('utf-8') method also gets rid of the error and produces output, however, the resulting object is a string, and not a beautiful soup object, and thus has none of the associated bs methods (e.g. .find_all()). A similar problem arises with the .encode('utf-8') approach.
Also, I've noticed that in the output, there are many \r and \n characters still in the beautiful soup object instead of the pure html content.
I want a beautiful soup object without any of the \r or \n characters that I can print to the terminal.
In your code:
sauce = urllib.request.urlopen('https://stats.swehockey.se/Teams/Info/TeamRoster/10333').read()
soup = bs.BeautifulSoup(sauce, 'lxml')
sauce is of type bytes. When you pass that into bs.BeautifulSoup(), BeautifulSoup attempts to decode those bytes as an ascii string, and fails, because it is actually a utf-8 string -- according to both the Content-Type response header (text/html; charset=utf-8) and also the meta tag at the start of the html document (<meta charset="utf-8" />).
The first argument for bs.BeautifulSoup(), markup, takes a string or a file-like object representing markup to be parsed. You should explicitly decode those bytes as a utf-8 encoded string, and use that instead of the raw bytes, like so:
sauce = urllib.request.urlopen('https://stats.swehockey.se/Teams/Info/TeamRoster/10333').read().decode('utf-8')
soup = bs.BeautifulSoup(sauce, 'lxml')
Also, I've noticed that in the output, there are many \r and \n characters still in the beautiful soup object instead of the pure html content.
I want a beautiful soup object without any of the \r or \n characters that I can print to the terminal.
The \r and \n characters are just representations of the newline character. If you were to print these, or view them in a text editor, they would show up as actual newlines.
I'm trying to scrape websites using requests and BeautifulSoup4 packages.
>>>import requests
>>>from bs4 import BeautifulSoup
>>>r = requests.get('https://www.yellowpages.com/search?search_terms = coffee&geo_location_terms=Los+Angeles%2C+CA')
>>>r.content #shows source code (mess) bytes type
>>>soup = BeautifulSoup(r.content,'html.parser')
When I try to prettify and display the html code of the page with
print(soup.prettify())
I get the error
UnicodeEncodeError: 'charmap' codec can't decode the character '\u2013'
in position 44379: character maps to <undefined>
I also tried
>>>soupbytes = soup.prettify(encoding = 'utf-8') #this is bytes format
>>>soupstr = soupbytes.decode('utf-8') #this is str format
With the first one I don't get any problem to print (print(soupbytes)), but it doesn't print the text 'pretty', and it is bytes format. If I try to print the second one (print(soupstr)) I get the error again but I get the object in str type.
I have to say also that I don't get any error in the IDE (spyder). It's to say, if I run the next code in spyder:
import requests
from bs4 import BeautifulSoup
r = requests.get('https://www.yellowpages.com/search?
search_terms=coffee&geo_location_terms=Los+Angeles%2C+CA')
r.content #muestra html de la pagina
soup = BeautifulSoup(r.content,'html.parser')
print(soup.prettify())
I don't have any error and it is printed nicely.
Why is that difference? And how could I avoid the error in the terminal???
When I using lxml library in python to get data on a html page (Youtube video title), It not return text correctly It return a text Like this "à·à·à¶½à¶±à·à¶§à¶ºà¶±à"
Here my code,
page = requests.get("https://www.youtube.com/watch?v=MZMapfEg5g8")
source = html.fromstring(page.content)
links = source.xpath('//link[#type="text/xml+oembed"]')
for href in links:
return href.attrib['title']
Language I need is is in sinhala, and it's unicode.
Use:
source = html.fromstring(page.text)
With the requests library, page.text will be the decoded text. page.content is the raw byte content, and page.encoding is the encoding of the page.
In summary:
import requests
from lxml import html
page = requests.get("https://www.youtube.com/watch?v=MZMapfEg5g8")
source = html.fromstring(page.text)
links = source.xpath('//link[#type="text/xml+oembed"]')
for href in links:
print href.attrib['title']
Output (requires terminal/IDE that supports the characters/font):
පේස් බුක් මැසේජ් එක (Facebook Messege)
Apparently the title you get is a byte string, you need to .decode() it to turn into Unicode. The title in question, "පේස් බුක් මැසේජ් එක", when encoded as UTF-8 and then decoded / displayed as Latin-1, gives the output you complain about.
A simple .decode('utf-8') should fix the problem.