I am scraping a web page and it worked pretty good except a part where re.compile() returns empty [] while the text passed to it is present. Here is my scrape code
dob = soup.find(text = re.compile('Date of Birth')).findNext('td').text
print(dob)
father_name = soup.find(text = re.compile("Father's Name")).findNext('td').text
print(father_name)
mob_no_parent = soup.find(text = re.compile("Mobile Number")).findNext('td').text
print(mob_no_parent)
mob_no_student = soup.findAll(text = re.compile("Mobile Number(Student)"))
print(mob_no_student)
email = soup.find(text = re.compile("E - Mail Address")).findNext('td').text
print(email)
p_address = soup.find(text = re.compile("PermanentAddress")).findNext('td').text
print(p_address)
The above code works fine on every text except
mob_no_student = soup.findAll(text = re.compile("Mobile Number(Student)"))
print(mob_no_student)
The above one returns []
Here is my html code
<td align="left" width="50%" class="inner_padding_even"> Registration No </td>
<td align="left" width="50%" class="inner_padding_even">CPT0000</td>
</tr>
<tr>
<td align="left" width="50%" class="inner_padding_odd"> Name of Candidate</td>
<td align="left" width="50%" class="inner_padding_odd"><font face=arial size=2>KKKKKKK B.</font></td>
</tr>
<tr>
<td align="left" class="inner_padding_even"> Date of Birth</td>
<td align="left" class="inner_padding_even">16.11.1900</td>
</tr>
<tr>
<td align="left" class="inner_padding_even"> Father's Name</td>
<td align="left" class="inner_padding_even">BBBBBBBB.</td>
</tr>
<tr>
<td align="left" class="inner_padding_even"> Mobile Number</font>(Parent)</td>
<td align="left" class="inner_padding_even">99999999999</td>
</tr>
<tr>
<td align="left" class="inner_padding_odd"> Mobile Number(Student)</td>
<td align="left" class="inner_padding_odd">9999999999</td>
</tr>
<tr>
<td align="left" class="inner_padding_even"> E - Mail Address</td>
<td align="left" class="inner_padding_even">keyansgm#gmail.com</td>
</tr>
<tr>
<td width="50%" align="left" class="inner_padding_even"> Permanent Address</td>
<td width="50%" align="left" class="inner_padding_even">Blah blah</td>
</tr>
What am i missing here ?
In a regex, you need to escape brackets, If not it will refer to a group
Try this
mob_no_student = soup.findAll(text = re.compile("Mobile Number\(Student\)"))
Related
the web source is like this:
<div class="MT12">
<table class="tblchart" border="0" cellspacing="0" cellpadding="0">
<tr>
<th rowspan="2" width="100" align="left" valign="top">Date</th>
<th rowspan="2" width="100" style="text-align:right;" valign="top">Open</th>
<th rowspan="2" width="100" style="text-align:right;" valign="top">High</th>
<th rowspan="2" width="100" style="text-align:right;" valign="top">Low</th>
<th rowspan="2" width="100" style="text-align:right;" valign="top">Close</th>
<th colspan="2" style="text-align:center;" valign="top">- SPREAD -</th>
</tr>
<tr>
<th width="100" style="text-align:right;" valign="top">(High-Low)</th>
<th width="100" style="text-align:right;" valign="top" class="last">(Open-Close)</th>
</tr>
<tr>
<td align="left" valign="top">2019-12-24</td>
<td valign="top" style="text-align:right;">12269.25</td>
<td valign="top" class="b_12vv" style="text-align:right">12283.70</td>
<td valign="top" style="text-align:right;">12202.10</td>
<td valign="top" style="text-align:right;">12214.55</td>
<td valign="top" style="text-align:right;">81.60</td>
<td align="right" valign="top" class="last" style="text-align:right;">54.70</td>
</tr>
<tr>
<td align="left" valign="top">2019-12-23</td>
<td valign="top" style="text-align:right;">12235.45</td>
<td valign="top" class="b_12vv" style="text-align:right">12287.15</td>
<td valign="top" style="text-align:right;">12213.25</td>
<td valign="top" style="text-align:right;">12262.75</td>
<td valign="top" style="text-align:right;">73.90</td>
<td align="right" valign="top" class="last" style="text-align:right;">-27.30</td>
</tr>
<tr>
<td align="left" valign="top">2019-12-20</td>
<td valign="top" style="text-align:right;">12266.45</td>
<td valign="top" class="b_12vv" style="text-align:right">12293.90</td>
<td valign="top" style="text-align:right;">12252.75</td>
<td valign="top" style="text-align:right;">12271.80</td>
<td valign="top" style="text-align:right;">41.15</td>
<td align="right" valign="top" class="last" style="text-align:right;">-5.35</td>
</tr>
</table>
</div>
I want to get the following numbers for every date:
say for example I have to get the numbers 12269.25, 12283.70, 12202.10 and 12214.55 for a particular date (2019-12-24). Then proceed for the next date given.
I am facing difficulty because I need to select next 4 lines(whose xpath is not exatly related much as shown above) following each date in the page. The dates can range from single date to 100-200 dates.
Can anybody please help with webdriver code snippet for the same.
Thanks a lot
Can this meet your needs
from simplified_scrapy.simplified_doc import SimplifiedDoc
html = '''<div class="MT12">
<table class="tblchart" border="0" cellspacing="0" cellpadding="0">
<tr>
<th rowspan="2" width="100" align="left" valign="top">Date</th>
<th rowspan="2" width="100" style="text-align:right;" valign="top">Open</th>
<th rowspan="2" width="100" style="text-align:right;" valign="top">High</th>
<th rowspan="2" width="100" style="text-align:right;" valign="top">Low</th>
<th rowspan="2" width="100" style="text-align:right;" valign="top">Close</th>
<th colspan="2" style="text-align:center;" valign="top">- SPREAD -</th>
</tr>
<tr>
<th width="100" style="text-align:right;" valign="top">(High-Low)</th>
<th width="100" style="text-align:right;" valign="top" class="last">(Open-Close)</th>
</tr>
<tr>
<td align="left" valign="top">2019-12-24</td>
<td valign="top" style="text-align:right;">12269.25</td>
<td valign="top" class="b_12vv" style="text-align:right">12283.70</td>
<td valign="top" style="text-align:right;">12202.10</td>
<td valign="top" style="text-align:right;">12214.55</td>
<td valign="top" style="text-align:right;">81.60</td>
<td align="right" valign="top" class="last" style="text-align:right;">54.70</td>
</tr>
<tr>
<td align="left" valign="top">2019-12-23</td>
<td valign="top" style="text-align:right;">12235.45</td>
<td valign="top" class="b_12vv" style="text-align:right">12287.15</td>
<td valign="top" style="text-align:right;">12213.25</td>
<td valign="top" style="text-align:right;">12262.75</td>
<td valign="top" style="text-align:right;">73.90</td>
<td align="right" valign="top" class="last" style="text-align:right;">-27.30</td>
</tr>
<tr>
<td align="left" valign="top">2019-12-20</td>
<td valign="top" style="text-align:right;">12266.45</td>
<td valign="top" class="b_12vv" style="text-align:right">12293.90</td>
<td valign="top" style="text-align:right;">12252.75</td>
<td valign="top" style="text-align:right;">12271.80</td>
<td valign="top" style="text-align:right;">41.15</td>
<td align="right" valign="top" class="last" style="text-align:right;">-5.35</td>
</tr>
</table>
</div>'''
doc = SimplifiedDoc(html)
table = doc.getElement(tag='table',value='tblchart')
trs = table.trs.notContains('<th') # get tr
for tr in trs:
tds = tr.tds # get all td
data = [td.text for td in tds]
print (data[0],data[1],data[2],data[3],data[4])
I have a self-project to scrape data online using BeautifulSoup and Python, and I think historical stocks data would be a good one for me to practice. I looked at the source code here to analyze how I can use BeautifulSoup's select() or findall() to parse part of the data from the table. Here is the code I use, but it parsed things other than the table.
soup = bs4.BeautifulSoup(res.text, 'lxml')
table = soup.findAll( 'td', {'class':'yfnc_tabledata1'} )
print table
My Question: How to I parse only the 2 rows showing the 2 days of data from the table?
Here is the table that has 2 days of the historical data:
<table class="yfnc_datamodoutline1" width="100%" cellpadding="0" cellspacing="0" border="0">
<tr valign="top">
<td>
<table border="0" cellpadding="2" cellspacing="1" width="100%">
<tr>
<th scope="col" class="yfnc_tablehead1" align="right" width="16%">Date</th>
<th scope="col" class="yfnc_tablehead1" align="right" width="12%">Open</th>
<th scope="col" class="yfnc_tablehead1" align="right" width="12%">High</th>
<th scope="col" class="yfnc_tablehead1" align="right" width="12%">Low</th>
<th scope="col" class="yfnc_tablehead1" align="right" width="12%">close</th>
<th scope="col" class="yfnc_tablehead1" align="right" width="16%">Volume</th>
<th scope="col" class="yfnc_tablehead1" align="right" width="15%">Adj Close*</th>
</tr>
<tr>
<td class="yfnc_tabledata1" nowrap align="right">12 Aug 2016</td>
<td class="yfnc_tabledata1" align="right">107.78</td>
<td class="yfnc_tabledata1" align="right">108.44</td>
<td class="yfnc_tabledata1" align="right">107.78</td>
<td class="yfnc_tabledata1" align="right">108.18</td>
<td class="yfnc_tabledata1" align="right">18,612,300</td>
<td class="yfnc_tabledata1" align="right">108.18</td>
</tr>
<tr>
<td class="yfnc_tabledata1" nowrap align="right">11 Aug 2016</td>
<td class="yfnc_tabledata1" align="right">108.52</td>
<td class="yfnc_tabledata1" align="right">108.93</td>
<td class="yfnc_tabledata1" align="right">107.85</td>
<td class="yfnc_tabledata1" align="right">107.93</td>
<td class="yfnc_tabledata1" align="right">27,484,500</td>
<td class="yfnc_tabledata1" align="right">107.93</td>
</tr>
<tr>
<td class="yfnc_tabledata1" colspan="7" align="center">
* <small>Close price adjusted for dividends and splits.</small>
</td>
</tr>
</table>
</td>
</tr>
</table>
I only need the specific 2 rows of data from above:
<tr>
<td class="yfnc_tabledata1" nowrap align="right">12 Aug 2016</td>
<td class="yfnc_tabledata1" align="right">107.78</td>
<td class="yfnc_tabledata1" align="right">108.44</td>
<td class="yfnc_tabledata1" align="right">107.78</td>
<td class="yfnc_tabledata1" align="right">108.18</td>
<td class="yfnc_tabledata1" align="right">18,612,300</td>
<td class="yfnc_tabledata1" align="right">108.18</td>
</tr>
<tr>
<td class="yfnc_tabledata1" nowrap align="right">11 Aug 2016</td>
<td class="yfnc_tabledata1" align="right">108.52</td>
<td class="yfnc_tabledata1" align="right">108.93</td>
<td class="yfnc_tabledata1" align="right">107.85</td>
<td class="yfnc_tabledata1" align="right">107.93</td>
<td class="yfnc_tabledata1" align="right">27,484,500</td>
<td class="yfnc_tabledata1" align="right">107.93</td>
</tr>
You can select the all the rows from the nested table inside the yfnc_datamodoutline1 table and index the first two:
soup = BeautifulSoup(html)
table_rows = soup.select("table.yfnc_datamodoutline1 table tr + tr")
row1, row2 = table_rows[0:2]
print(row1)
print(row2)
Which would give you:
<tr>
<td align="right" class="yfnc_tabledata1" nowrap="">12 Aug 2016</td>
<td align="right" class="yfnc_tabledata1">107.78</td>
<td align="right" class="yfnc_tabledata1">108.44</td>
<td align="right" class="yfnc_tabledata1">107.78</td>
<td align="right" class="yfnc_tabledata1">108.18</td>
<td align="right" class="yfnc_tabledata1">18,612,300</td>
<td align="right" class="yfnc_tabledata1">108.18</td>
</tr>
<tr>
<td align="right" class="yfnc_tabledata1" nowrap="">11 Aug 2016</td>
<td align="right" class="yfnc_tabledata1">108.52</td>
<td align="right" class="yfnc_tabledata1">108.93</td>
<td align="right" class="yfnc_tabledata1">107.85</td>
<td align="right" class="yfnc_tabledata1">107.93</td>
<td align="right" class="yfnc_tabledata1">27,484,500</td>
<td align="right" class="yfnc_tabledata1">107.93</td>
</tr>
To get the td data just extract the text from each td:
print([td.text for td in row1.find_all("td")])
print([td.text for td in row2.find_all("td")])
Which would give you:
[u'12 Aug 2016', u'107.78', u'108.44', u'107.78', u'108.18', u'18,612,300', u'108.18']
[u'11 Aug 2016', u'108.52', u'108.93', u'107.85', u'107.93', u'27,484,500', u'107.93']
table.yfnc_datamodoutline1 table tr + tr selects all the rows inside the inner table skipping the first which is the header row.
I am working to parse a html table given below(its a section of complete html code) But the code is not working. Can some one please help me.There is an error saying "table has no attribute findall".
The code is:
import re
import HTMLParser
from urllib2 import urlopen
import urllib2
from bs4 import BeautifulSoup
url = 'http://164.100.47.132/LssNew/Members/Biography.aspx?mpsno=4064'
url_data = urlopen(url).read()
html_page = urllib2.urlopen(url)
soup = BeautifulSoup(html_page)
title = soup.title
final_tit = title.string
table = soup.find('table',id = "ctl00_ContPlaceHolderMain_Bioprofile1_Datagrid1")
tr = table.findall('tr')
for tr in table:
cols = tr.findAll('td')
for td in cols:
text = ''.join(td.find(text=True))
print text+"|",
print
<table style="WIDTH: 565px">
<tr>
<td vAlign="top" align="left"><img id="ctl00_ContPlaceHolderMain_Bioprofile1_Image1" src="http://164.100.47.132/mpimage/photo/4064.jpg" style="height:140px;border-width:0px;" /></td>
<td vAlign="top"><table cellspacing="0" rules="all" border="2" id="ctl00_ContPlaceHolderMain_Bioprofile1_Datagrid1" style="border-color:#FAE3C3;border-width:2px;border-style:Solid;width:433px;border-collapse:collapse;">
<tr>
<td>
<table align="center" height="30px">
<tr valign="top">
<td align="center" valign="top" class="gridheader1">Aaroon Rasheed,Shri J.M.</td>
</tr>
</table>
<table height="110px">
<tr>
<td align="left" class="darkerb" width="133px" valign="top">Constituency :</td>
<td align="left" valign="top" class="griditem2" width="300px">Theni (Tamil Nadu )</td>
</tr>
<tr>
<td align="left" width="133px" class="darkerb" valign="top">
Party Name :</td>
<td align="left" width="300px" valign="top" class="griditem2">Indian National Congress(INC)</td>
</tr>
<tr>
<td align="left" class="darkerb" valign="top" width="133px">
Email Address :
</td>
<td align="left" valign="top" class="griditem2" width="300px">jm.aaronrasheed#sansad.nic.in</td>
</tr>
</table>
</td>
</tr>
</table></td>
</tr>
</table>
The method is called find_all(), not findall:
tr = table.find_all('tr')
I'm trying to learn how to scrape the web, but I have some problem getting my code to work. The number I want to extract is 77.80 from the code below. The problem I have is to find something that is unique enough to find the information(the place). can you help me with the right code. Thanks in advance!
</td>
<td class="small"> </td>
<td align="center" nowrap valign="center" class="small">
<a alt="Utvald" class="small" href="javascript:QT('/se/skandia/funds/chosen.aspx?tab=5&cid=0P0000T35O&lang=SV&curiso=SEK&country=SE&clientattributes=8&lastpage=Sök fond&LastPageURL=/se/skandia/quickrank/index.aspx?tab=RSLTS|lang=SV|univ=SE1|country=SE|curiso=SEK|mec=|cat=-1|search=|sortby=Custom_4|sortorder=ASC|PageNo=1|Firstletter=','0P0000T35O','600')" onmouseout="status=''; return true"><img src="../read/im/sigillsvartsmall_FFFFFF.gif" border="0" alt="Utvald av Skandia" height="12" width="9"/></a>
</td>
<td class="small"> </td>
<td align="right" nowrap valign="top" class="small">
77.80
</td>
<td class="small"> </td>
<td align="right" nowrap valign="top" class="small">
<!--<img src="../read/im/valueSEK.gif" align="texttop" height="10" width="22">-->
SEK
</td>
<td class="small"> </td>
<td align="right" nowrap valign="top" class="small">
1.4
</td>
<td class="small"> </td>
<td align="right" nowrap valign="top" class="small">
0.5
</td>
<td class="small"> </td>
<td align="right" nowrap valign="top" class="small">
2.7
</td>
<td class="small"> </td>
<td align="right" nowrap valign="top" class="small">
6.6
</td>
Here is how to find text you wanted. This just looks for the first td that has class='small' and valign='top'.
soup = BeautifulSoup(s)
tds = soup.find_all('td', attrs={'class': 'small', 'valign': 'top'})
the_td = tds[0].text.strip()
I'm trying to parse specific content from an X number of HTML files to a single TXT file.
I have dirtily coded the following:
#!/usr/bin/python
import sys, mechanize, BeautifulSoup
def parsedata():
##do stuff
prvitekst = soup.find(text='Random Number')
prvikesh = prvitekst.findNextSiblings('td')
drugitekst = soup.find(text='Random Month/Yeare')
drugikesh = drugitekst.findNextSiblings('td')
trechitekst = soup.find(text='Small Random Number')
trechikesh = trechitekst.findNextSiblings('td')
content = prvikesh + ";" + drugikesh + ";" + trechikesh + ";"
writeFile(content);
def readFile(id):
fi = open('result/page-%s.html' % id, 'r');
def writeFile(content):
f = open('parsed.txt', 'a')
f.write(content,"\n")
f.close();
def main(start):
##initialize vars
id = int(start)
page = readFile(id)
soup = BeautifulSoup(page)
print soup.prettify()
readFile(id)
for id in range(1000000000):
parsedata()
id = id + 1
continue
main(sys.argv[1]);
While the HTML part im trying to scrape looks like this
<tr style="height:40px; background-color:#f0f0f0;"><td colspan="4" class="textLargeBold" style="border-bottom: solid 1px #c4c4c4;">Random Details</td></tr>
<tr class="text">
<td align="left" valign="top"><b>Type</b></td>
<td align="left" valign="top">Color</td>
<td align="left" valign="top"><b>Random Number</b></td>
<td align="left" valign="top">213523123123123</td>
</tr>
<tr class="text"
<td align="left" valign="top"><b>Random Month/Year</b></td>
<td align="left" valign="top">12/13</td>
<td align="left" valign="top"><b>Small Random Number</b></td>
<td align="left" valign="top">13233</td>
</tr>
I want the details that come after the first one. thus if I'm searching for Typem I want it to show me Color.
and ofcourse in the end I'd like the gotten contents to be parsed in a format similar to CSV.
Type;Random Number;Random Month/Year
it should parse
Color;213523123123123;12/13
ofcourse in the code i made already im not searching for type, but that can easily be changed.
EDIT : Fixed intendation
html='''
<tr style="height:40px; background-color:#f0f0f0;"><td colspan="4" class="textLargeBold" style="border-bottom: solid 1px #c4c4c4;">Random Details</td></tr> <tr class="text"> <td align="left" valign="top"><b>Type</b></td> <td align="left" valign="top">Color</td> <td align="left" valign="top"><b>Random Number</b></td> <td align="left" valign="top">213523123123123</td> </tr> <tr class="text" <td align="left" valign="top"><b>Random Month/Year</b></td> <td align="left" valign="top">12/13</td> <td align="left" valign="top"><b>Small Random Number</b></td> <td align="left" valign="top">13233</td> </tr>
'''
import htql;
a=[x for x in htql.HTQL(html, "<b sep>2-0 {name=<b>:tx; value=<td>1:tx } ")];
a