We Keep Coding

Element changes from display: none to display:block selenium python

I am trying to fill a form online using selenium and at some point I have to fill a date. I can't use send_keys() since it is not allowed by the page. Instead, when I click on the date field, it pops up a datepicker window that prompts to select the year, and I can do this successfully.
After picking the year, the previous window is removed and a new one that prompts to select the month is displayed. This is done by setting the style from display: none to display: block and to the previous year window the style is set from display: block to display: none.
The problem is that even if the new window is_displayed() and is_enabled() methods return True, the elements of the second window, when using is_displayed() on them returns False, even if the is_enabled() method returns True.
I think that I should refresh the dom elements of my driver, but driver.refresh() puts me back in step 0, where I have to pick the year again.
This is my code:
# Code for selecting year (Works)
dateWindow = driver.find_element_by_xpath('/html/body/div[9]/div[3]/table')
rows = dateWindow.find_elements_by_tag_name("tr")
rows[1].find_element_by_xpath('//span[text()="%s"]' % str_year).click()
# Code for selecting month (Does not work)
dateWindow = driver.find_element_by_xpath('/html/body/div[9]/div[2]/table')
rows = dateWindow.find_elements_by_tag_name("tr")
rows[1].find_element_by_xpath('//span[text()="%s"]' % str_month).click()
In the last line, I get this error:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
This is the html of the page before selecting the year:
<div class="datepicker-days" style="display: none;">
<table class=" table-condensed">
<thead>
<tr>
<th class="prev" style="visibility: visible;">«</th>
<th colspan="5" class="datepicker-switch">June 1993</th>
<th class="next" style="visibility: visible;">»</th>
</tr>
<tr>
<th class="dow">Su</th>
<th class="dow">Mo</th>
<th class="dow">Tu</th>
<th class="dow">We</th>
<th class="dow">Th</th>
<th class="dow">Fr</th>
<th class="dow">Sa</th>
</tr>
</thead>
<tbody>
<tr>
<td class="old day">30</td>
<td class="old day">31</td>
<td class="day">1</td>
<td class="day">2</td>
<td class="day">3</td>
<td class="day">4</td>
...
<td class="day">29</td>
<td class="day">30</td>
<td class="new day">1</td>
<td class="new day">2</td>
<td class="new day">3</td>
</tr>
<tr>
<td class="new day">4</td>
<td class="new day">5</td>
<td class="new day">6</td>
<td class="new day">7</td>
<td class="new day">8</td>
<td class="new day">9</td>
<td class="new day">10</td>
</tr>
</tbody>
<tfoot>
<tr>
<th colspan="7" class="today" style="display: none;">Today</th>
</tr>
<tr>
<th colspan="7" class="clear" style="display: none;">Clear</th>
</tr>
</tfoot>
</table>
</div>
<div class="datepicker-months" style="display: none;">
<table class="table-condensed">
<thead>
<tr>
<th class="prev" style="visibility: visible;">«</th>
<th colspan="5" class="datepicker-switch">1993</th>
<th class="next" style="visibility: visible;">»</th>
</tr>
</thead>
<tbody>
<tr>
<td colspan="7">
<span class="month">Jan</span>
<span class="month">Feb</span>
<span class="month">Mar</span>
<span class="month">Apr</span>
<span class="month">May</span>
<span class="month">Jun</span>
<span class="month">Jul</span>
<span class="month">Aug</span>
<span class="month">Sep</span>
<span class="month">Oct</span>
<span class="month">Nov</span>
<span class="month">Dec</span>
</td>
</tr>
</tbody>
<tfoot>
<tr>
<th colspan="7" class="today" style="display: none;">Today</th>
</tr>
<tr>
<th colspan="7" class="clear" style="display: none;">Clear</th>
</tr>
</tfoot>
</table>
</div>
<div class="datepicker-years" style="display: block;">
<table class="table-condensed">
<thead>
<tr>
<th class="prev" style="visibility: visible;">«</th>
<th colspan="5" class="datepicker-switch">1990-1999</th>
<th class="next" style="visibility: visible;">»</th>
</tr>
</thead>
<tbody>
<tr>
<td colspan="7">
<span class="year old">1989</span>
<span class="year">1990</span>
<span class="year">1991</span>
<span class="year">1992</span>
<span class="year">1993</span>
<span class="year active">1994</span>
<span class="year">1995</span>
<span class="year">1996</span>
<span class="year">1997</span>
<span class="year">1998</span>
<span class="year">1999</span>
<span class="year new">2000</span>
</td>
</tr>
</tbody>
<tfoot>
<tr>
<th colspan="7" class="today" style="display: none;">Today</th>
</tr>
<tr>
<th colspan="7" class="clear" style="display: none;">Clear</th>
</tr>
</tfoot>
</table>
</div>
This is the html of the page before selecting the month and after selecting the year:
<div class="datepicker-days" style="display: none;">
<table class=" table-condensed">
<thead>
<tr>
<th class="prev" style="visibility: visible;">«</th>
<th colspan="5" class="datepicker-switch">June 1993</th>
<th class="next" style="visibility: visible;">»</th>
</tr>
<tr>
<th class="dow">Su</th>
<th class="dow">Mo</th>
<th class="dow">Tu</th>
<th class="dow">We</th>
<th class="dow">Th</th>
<th class="dow">Fr</th>
<th class="dow">Sa</th>
</tr>
</thead>
<tbody>
<tr>
<td class="old day">30</td>
<td class="old day">31</td>
<td class="day">1</td>
<td class="day">2</td>
<td class="day">3</td>
<td class="day">4</td>
...
<td class="day">29</td>
<td class="day">30</td>
<td class="new day">1</td>
<td class="new day">2</td>
<td class="new day">3</td>
</tr>
<tr>
<td class="new day">4</td>
<td class="new day">5</td>
<td class="new day">6</td>
<td class="new day">7</td>
<td class="new day">8</td>
<td class="new day">9</td>
<td class="new day">10</td>
</tr>
</tbody>
<tfoot>
<tr>
<th colspan="7" class="today" style="display: none;">Today</th>
</tr>
<tr>
<th colspan="7" class="clear" style="display: none;">Clear</th>
</tr>
</tfoot>
</table>
</div>
<div class="datepicker-months" style="display: block;">
<table class="table-condensed">
<thead>
<tr>
<th class="prev" style="visibility: visible;">«</th>
<th colspan="5" class="datepicker-switch">1993</th>
<th class="next" style="visibility: visible;">»</th>
</tr>
</thead>
<tbody>
<tr>
<td colspan="7">
<span class="month">Jan</span>
<span class="month">Feb</span>
<span class="month">Mar</span>
<span class="month">Apr</span>
<span class="month">May</span>
<span class="month">Jun</span>
<span class="month">Jul</span>
<span class="month">Aug</span>
<span class="month">Sep</span>
<span class="month">Oct</span>
<span class="month">Nov</span>
<span class="month">Dec</span>
</td>
</tr>
</tbody>
<tfoot>
<tr>
<th colspan="7" class="today" style="display: none;">Today</th>
</tr>
<tr>
<th colspan="7" class="clear" style="display: none;">Clear</th>
</tr>
</tfoot>
</table>
</div>
<div class="datepicker-years" style="display: none;">
<table class="table-condensed">
<thead>
<tr>
<th class="prev" style="visibility: visible;">«</th>
<th colspan="5" class="datepicker-switch">1990-1999</th>
<th class="next" style="visibility: visible;">»</th>
</tr>
</thead>
<tbody>
<tr>
<td colspan="7">
<span class="year old">1989</span>
<span class="year">1990</span>
<span class="year">1991</span>
<span class="year">1992</span>
<span class="year">1993</span>
<span class="year active">1994</span>
<span class="year">1995</span>
<span class="year">1996</span>
<span class="year">1997</span>
<span class="year">1998</span>
<span class="year">1999</span>
<span class="year new">2000</span>
</td>
</tr>
</tbody>
<tfoot>
<tr>
<th colspan="7" class="today" style="display: none;">Today</th>
</tr>
<tr>
<th colspan="7" class="clear" style="display: none;">Clear</th>
</tr>
</tfoot>
</table>
</div>
Any ideas? Thanks in advance

The desired element is an dynamic element so while selecting the Month you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using XPATH:
dateWindow = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[9]/div[2]/table")))
rows = dateWindow.find_elements_by_tag_name("tr")
rows[1].find_element_by_xpath('//span[text()="%s"]' % str_month).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

change the color of the borders of my bootstrap table without css

i want to change the color of the borders of my table, i'm using the class .table-bordered the color of borders is grey/silver and i want to change it to black without using css, i used style="border-color:black;" but it doesn't change anything any ideas please ?!!
this is my XML file
<table class="table table-bordered" style="border-color:black;">
<tr class="border-black">
<th class="text-center">
<h3> Bulletin de paie </h3>
</th>
<th class="text-center" align="center">
Adresse
<br></br>
<span t-field="o.adresse"/>
</th>
<th class="text-center" align="center">
Date de paie
<br></br>
<span t-field="o.datedepaie"/>
</th>
</tr>
<tr bordercolor="black">
<th class="text-center">
<strong>Matricule</strong>
<br></br>
<span t-field="o.matricule"/>
</th>
<th class="text-center">
<strong>Nom et Prénom</strong>
<br></br>
<span t-field="o.name"/>
</th>
<th class="text-center">
<strong>CNSS</strong>
<br></br>
<span t-field="o.cnss"/>
</th>
</tr>
<tr bordercolor="black">
<th class="text-center">
<strong>Date de naissance</strong>
<br></br>
<span t-field="o.datenaissance"/>
</th>
<th class="text-center">
<strong>Date d'embauche</strong>
<br></br>
<span t-field="o.dateembauche"/>
</th>
<th class="text-center">
<strong>Fonction</strong>
<br></br>
<span t-field="o.fonction"/>
</th>
</tr>
</table>
<table class="table table-bordered">
<thead>
<tr>
<th>Libellé</th>
<th>Base</th>
<th>Taux (%)</th>
<th>Gains</th>
<th>Retenues</th>
</tr>
</thead>
<tbody class="invoice_tbody">
<tr t-foreach="o.salaire_id" t-as="l">
<th><span t-field="l.libelle"/></th>
<th><span t-field="l.base"/></th>
<th><span t-field="l.taux"/></th>
<th><span t-field="l.gains"/></th>
<th><span t-field="l.retenues"/></th>
</tr>
<tr>
<th colspan="3" class="text-right">
<strong>Total</strong>
</th>
<th class="text-center">
<span t-field="o.total"/>
</th>
<th class="text-center">
<span t-field="o.totall"/>
</th>
</tr>
<tr>
<th colspan="5" class="text-right">
<strong>Net à payer:</strong>
<span t-field="o.net"/>
<br></br>
</th>
</tr>
</tbody>
</table>
</div>
</div>
<div class="footer">

IIRC the border color is set on Rows, td's, and th's. It's out of the top of my head though.
You'd need to alter their border color, not the table's

You should post your CSS file. It would be helpful to see the existing styles. Also, line breaks don't require closing tags. I think your <br></br> are breaking the HTML.
You only need <br />

web scraping to get the required output format

i am trying to scrap azure sites to get the instance prices but got stuck with with the required output format can any one help me in getting the output..
here is the sample source page
<section class="section section-size3"> <div class="row column"> <h2>Virtual machines categories</h2> </div> <div class="row row-size2 column"> <a class="toggler toggler-pricing" data-event="pricingdetails-clicked-accordion" href="#"> <h3 class="heading">General Purpose</h3> <p class="sd-truncateText">Balanced CPU to memory ratio. Ideal for testing and development, small to medium databases and low to medium traffic web servers.</p> </a> <div class="toggled"> <div class="row row-size3 column"> <h3>A0-4 – Basic <a class="arrowLink text-heading5" href="/en-in/pricing/details/virtual-machines/series/#a-series">More information</a></h3> <p>A Basic is an economical option for development workloads, test servers, build servers, code repositories, low-traffic websites and web applications, micro services, early product experiments and small databases.</p> <table class="table-width-even"> <thead> <tr> <th>Instance</th> <th>Cores</th> <th>RAM</th> <th>Disk sizes</th> <th>Price</th> </tr> </thead> <tbody> <tr> <td> A0 </td> <td>1</td> <td>0.75 GB</td> <td><strong>20</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.018,"us-west":0.018,"us-north-central":0.018,"us-south-central":0.018,"europe-north":0.018,"europe-west":0.018,"asia-pacific-southeast":0.018,"asia-pacific-east":0.018,"japan-east":0.022,"japan-west":0.019,"brazil-south":0.022,"us-east-2":0.018,"us-central":0.018,"usgov-virginia":0.017,"usgov-iowa":0.017,"australia-east":0.024,"australia-southeast":0.024,"central-india":0.018,"south-india":0.016,"canada-east":0.02,"canada-central":0.022,"germany-central":0.021,"germany-northeast":0.021,"united-kingdom-west":0.02,"united-kingdom-south":0.02,"us-west-2":0.018,"us-west-central":0.018}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.018</span> </td> </tr> <tr> <td> A1 </td> <td>1</td> <td>1.75 GB</td> <td><strong>40</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.035,"us-west":0.034,"us-north-central":0.038,"us-south-central":0.035,"europe-north":0.028,"europe-west":0.027,"asia-pacific-southeast":0.037,"asia-pacific-east":0.046,"japan-east":0.04,"japan-west":0.044,"brazil-south":0.054,"us-east-2":0.024,"us-central":0.038,"usgov-virginia":0.04,"usgov-iowa":0.04,"australia-east":0.04,"australia-southeast":0.046,"central-india":0.038,"south-india":0.042,"canada-east":0.038,"canada-central":0.042,"germany-central":0.0322,"germany-northeast":0.0322,"united-kingdom-west":0.056,"united-kingdom-south":0.056,"us-west-2":0.024,"us-west-central":0.035}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.044</span> </td> </tr> <tr> <td> A2 </td> <td>2</td> <td>3.50 GB</td> <td><strong>60</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.079,"us-west":0.081,"us-north-central":0.085,"us-south-central":0.079,"europe-north":0.075,"europe-west":0.078,"asia-pacific-southeast":0.095,"asia-pacific-east":0.104,"japan-east":0.109,"japan-west":0.099,"brazil-south":0.112,"us-east-2":0.068,"us-central":0.085,"usgov-virginia":0.08,"usgov-iowa":0.08,"australia-east":0.098,"australia-southeast":0.104,"central-india":0.096,"south-india":0.094,"canada-east":0.087,"canada-central":0.095,"germany-central":0.0863,"germany-northeast":0.0863,"united-kingdom-west":0.112,"united-kingdom-south":0.112,"us-west-2":0.068,"us-west-central":0.079}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.088</span> </td> </tr> <tr> <td> A3 </td> <td>4</td> <td>7.00 GB</td> <td><strong>120</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.176,"us-west":0.188,"us-north-central":0.188,"us-south-central":0.176,"europe-north":0.188,"europe-west":0.204,"asia-pacific-southeast":0.232,"asia-pacific-east":0.232,"japan-east":0.276,"japan-west":0.2196,"brazil-south":0.232,"us-east-2":0.176,"us-central":0.188,"usgov-virginia":0.16,"usgov-iowa":0.16,"australia-east":0.232,"australia-southeast":0.232,"central-india":0.232,"south-india":0.209,"canada-east":0.194,"canada-central":0.211,"germany-central":0.216,"germany-northeast":0.216,"united-kingdom-west":0.224,"united-kingdom-south":0.224,"us-west-2":0.176,"us-west-central":0.176}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.176</span> </td> </tr> <tr> <td> A4 </td> <td>8</td> <td>14.00 GB</td> <td><strong>240</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.352,"us-west":0.376,"us-north-central":0.376,"us-south-central":0.352,"europe-north":0.376,"europe-west":0.408,"asia-pacific-southeast":0.464,"asia-pacific-east":0.464,"japan-east":0.552,"japan-west":0.4392,"brazil-south":0.464,"us-east-2":0.352,"us-central":0.376,"usgov-virginia":0.32,"usgov-iowa":0.32,"australia-east":0.464,"australia-southeast":0.464,"central-india":0.464,"south-india":0.418,"canada-east":0.387,"canada-central":0.422,"germany-central":0.432,"germany-northeast":0.432,"united-kingdom-west":0.449,"united-kingdom-south":0.449,"us-west-2":0.352,"us-west-central":0.352}}'
</div> <div class="row row-size3 column"> <h3>Av2 Standard <a class="arrowLink text-heading5" href="/en-in/pricing/details/virtual-machines/series/#a-series">More information</a></h3> <p>Av2 Standard is the latest generation of A series VMs with similar CPU performance and faster disk. These VMs are suitable for development workloads, build servers, code repositories, low-traffic websites and web applications, micro services, early product experiments and small databases. Like the prior A Standard generation, Av2 VMs will include load balancing and auto-scaling at no additional charge.</p> <table class="table-width-even"> <thead> <tr> <th>Instance</th> <th>Cores</th> <th>RAM</th> <th>Disk sizes</th> <th>Price</th> </tr> </thead> <tbody> <tr> <td> A1 v2 </td> <td>1</td> <td>2.00 GB</td> <td><strong>10</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.043,"us-west":0.043,"us-north-central":0.043,"us-south-central":0.043,"europe-north":0.041,"europe-west":0.041,"asia-pacific-southeast":0.045,"asia-pacific-east":0.05,"japan-east":0.054,"japan-west":0.054,"brazil-south":0.061,"us-east-2":0.043,"us-central":0.043,"usgov-virginia":0.048,"usgov-iowa":0.048,"australia-east":0.5,"australia-southeast":0.05,"central-india":0.047,"south-india":0.047,"canada-east":0.043,"canada-central":0.047,"germany-central":0.055,"germany-northeast":0.055,"united-kingdom-west":0.05,"united-kingdom-south":0.05,"us-west-2":0.036,"us-west-central":0.04}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.043</span> </td> </tr> <tr> <td> A2 v2 </td> <td>2</td> <td>4.00 GB</td> <td><strong>20</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.091,"us-west":0.091,"us-north-central":0.091,"us-south-central":0.091,"europe-north":0.087,"europe-west":0.087,"asia-pacific-southeast":0.095,"asia-pacific-east":0.106,"japan-east":0.113,"japan-west":0.113,"brazil-south":0.129,"us-east-2":0.091,"us-central":0.091,"usgov-virginia":0.097,"usgov-iowa":0.097,"australia-east":0.106,"australia-southeast":0.106,"central-india":0.098,"south-india":0.098,"canada-east":0.091,"canada-central":0.098,"germany-central":0.109,"germany-northeast":0.109,"united-kingdom-west":0.106,"united-kingdom-south":0.106,"us-west-2":0.076,"us-west-central":0.083}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.091</span> </td> </tr> <tr> <td> A4 v2 </td> <td>4</td> <td>8.00 GB</td> <td><strong>40</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.191,"us-west":0.191,"us-north-central":0.191,"us-south-central":0.191,"europe-north":0.183,"europe-west":0.183,"asia-pacific-southeast":0.198,"asia-pacific-east":0.222,"japan-east":0.238,"japan-west":0.238,"brazil-south":0.27,"us-east-2":0.191,"us-central":0.191,"usgov-virginia":0.194,"usgov-iowa":0.194,"australia-east":0.222,"australia-southeast":0.222,"central-india":0.206,"south-india":0.206,"canada-east":0.191,"canada-central":0.206,"germany-central":0.218,"germany-northeast":0.218,"united-kingdom-west":0.222,"united-kingdom-south":0.222,"us-west-2":0.159,"us-west-central":0.175}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.191</span> </td> </tr> <tr> <td> A8 v2 </td> <td>8</td> <td>16.00 GB</td> <td><strong>80</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.6,"us-west":0.4,"us-north-central":0.4,"us-south-central":0.4,"europe-north":0.383,"europe-west":0.383,"asia-pacific-southeast":0.417,"asia-pacific-east":0.467,"japan-east":0.5,"japan-west":0.5,"brazil-south":0.567,"us-east-2":0.4,"us-central":0.4,"usgov-virginia":0.387,"usgov-iowa":0.387,"australia-east":0.467,"australia-southeast":0.467,"central-india":0.433,"south-india":0.433,"canada-east":0.4,"canada-central":0.433,"germany-central":0.437,"germany-northeast":0.437,"united-kingdom-west":0.467,"united-kingdom-south":0.467,"us-west-2":0.333,"us-west-central":0.367}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.4</span> </td> </tr> <tr> <td> A2m v2 </td> <td>2</td> <td>16.00 GB</td> <td><strong>20</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.119,"us-west":0.149,"us-north-central":0.129,"us-south-central":0.124,"europe-north":0.139,"europe-west":0.124,"asia-pacific-southeast":0.134,"asia-pacific-east":0.178,"japan-east":0.153,"japan-west":0.178,"brazil-south":0.208,"us-east-2":0.119,"us-central":0.129,"usgov-virginia":0.176,"usgov-iowa":0.176,"australia-east":0.149,"australia-southeast":0.178,"central-india":0.158,"south-india":0.144,"canada-east":0.129,"canada-central":0.168,"germany-central":0.251,"germany-northeast":0.251,"united-kingdom-west":0.158,"united-kingdom-south":0.178,"us-west-2":0.099,"us-west-central":0.119}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.149</span> </td> </tr> <tr> <td> A4m v2 </td> <td>4</td> <td>32.00 GB</td> <td><strong>40</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.238,"us-west":0.297,"us-north-central":0.27,"us-south-central":0.248,"europe-north":0.291,"europe-west":0.26,"asia-pacific-southeast":0.281,"asia-pacific-east":0.374,"japan-east":0.322,"japan-west":0.374,"brazil-south":0.437,"us-east-2":0.249,"us-central":0.27,"usgov-virginia":0.353,"usgov-iowa":0.353,"australia-east":0.312,"australia-southeast":0.374,"central-india":0.333,"south-india":0.301,"canada-east":0.27,"canada-central":0.353,"germany-central":0.502,"germany-northeast":0.502,"united-kingdom-west":0.333,"united-kingdom-south":0.374,"us-west-2":0.208,"us-west-central":0.249}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.297</span> </td> </tr> <tr> <td> A8m v2 </td> <td>8</td> <td>64.00 GB</td> <td><strong>80</strong> GB</td> <td> <span class="price-data " data-amount='{"regional":{"us-east":0.475,"us-west":0.594,"us-north-central":0.568,"us-south-central":0.495,"europe-north":0.611,"europe-west":0.546,"asia-pacific-southeast":0.589,"asia-pacific-east":0.786,"japan-east":0.677,"japan-west":0.786,"brazil-south":0.917,"us-east-2":0.524,"us-central":0.568,"usgov-virginia":0.704,"usgov-iowa":0.704,"australia-east":0.655,"australia-southeast":0.786,"central-india":0.699,"south-india":0.633,"canada-east":0.568,"canada-central":0.742,"germany-central":1.004,"germany-northeast":1.004,"united-kingdom-west":0.699,"united-kingdom-south":0.786,"us-west-2":0.437,"us-west-central":0.524}}' data-hour-format="{0}/hr" data-month-format="{0}/mo" data-region-unavailable="N/A">$0.594</span> </td> </tr> </tbody> </table>
how can i get below output ...
General Purpose A0 us-east 0.018 General Purpose A0 us-west 0.018......... ...............................so on............................ General Purpose A1 us-east 0.018........ ...............................so on..................... Av2 Standard A0 us-west 0.19..................... ............................................................... ..........and so on...................................

Signing In and Scraping Webpages with Scrapy

Hoping some one is able to help me with this. I am using Scrapy to login and scrape data. This particular code works for one website. But I have another website that Scrapy cannot log on to because of the following issue:
There are no form elements on the page.
The username and password fields, and the submit button are not in a form element, instead they are in a table element, which makes it a bit confusing. How can I make Scrapy log into the webpage using the table/tr elements....and not the form element?
Any help would be appreciated!
class LoginSpider(BaseSpider):
name = 'project'
allowed_domains = ["domain.com"]
start_urls = ["theloginURL"]
#this function will look for the form element and login with the username and password
def parse(self, response):
return [FormRequest.from_response(response,
formdata={'user_name': ' username123', 'Password': ' psd123'},
formxpath='//*[#name="Form1"]',callback=self.after_login)]
This is the HTML code that is on the log in page, in case you guys need it:
<table height="260px" id="loginMainTable" width="100%" cellspacing="0" cellpadding="0">
<tbody><tr>
<td>
<table align="center" class="blueBorder" cellspacing="0" cellpadding="0">
<tbody><tr class="HeaderFooterHide">
<td id="companyTD" width="100%" colspan="3" style="position:relative;">
<span class="header1 header1pos">
Application and Network Access Portal
</span>
<table width="100%" cellpadding="0" cellspacing="0">
<tbody><tr>
<td width="32px">
<img src="/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/headertopl.gif" align="absmiddle">
</td>
<td style="background-image: url('/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/headertopm.gif'); background-repeat: repeat-x">
</td>
<td width="520px" style="background-image: url('/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/headertopr.gif');">
</td>
</tr>
</tbody></table>
</td>
</tr>
<tr class="HeaderFooterHide">
<td width="100%" colspan="3" style="position:relative;">
<span style="position:absolute;margin-left:20px;">
</span>
<table width="100%" cellpadding="0" cellspacing="0">
<tbody><tr>
<td width="30px">
<img src="/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/headerbottoml.gif" align="absmiddle">
</td>
<td style="background-image: url('/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/headerbottomm.gif'); background-repeat: repeat-x">
</td>
<td width="30px" style="background-image: url('/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/headerbottomr.gif');">
</td>
</tr>
</tbody></table>
</td>
</tr>
<tr>
<td class="contentleft">
</td>
<td align="center" class="internalTD">
<table width="100%" height="100%" cellspacing="0" cellpadding="0" align="center">
<form id="form1" name="form1" autocomplete="off" method="post" action="/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/Validate.asp" onsubmit="return(SubmitForm());"></form>
<tbody><tr>
<td valign="top" height="250px" align="center">
<table border="0" cellspacing="0" cellpadding="0" class="content">
<tbody><tr>
<td class="msgText">Log On</td>
</tr>
</tbody></table>
<table border="0" cellspacing="0" cellpadding="0" class="content">
<tbody><tr>
<td class="paramText">Username</td>
<td><input class="paramTextbox" type="text" id="user_name" name="user_name" maxlength="50" size="11"></td>
</tr>
<tr>
<td class="paramText">Password</td>
<td><input class="paramTextbox" type="password" id="password" name="password" maxlength="20" onkeypress="capsDetect(arguments[0]);" size="11"></td>
</tr>
<input type="hidden" id="repository" name="repository" value="ADLDS" size="11">
<tr height="0px">
<td colspan="2" id="capsLockTD" height="0px"> </td>
</tr>
<tr>
<td> </td>
<td class="EzBiz_Text1"><input name="chkUsername" type="CHECKBOX" onclick="saveUserName()" id="chkUsername">Remember my User ID</td>
</tr>
<tr>
<td colspan="2" align="right">
<input border="0" class="button" type="submit" id="submit_button" value="Log On">
</td>
<td></td>
</tr>
</tbody></table>
<div class="EzBiz_Loginbutton" style="float: left;">
<input border="0" class="EzBiz_Button" type="image" id="submit_button" src="/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/CustomUpdate/login_submit.jpg">
</div>
<div class="EzBiz_PasswordForget">
<img src="/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/CustomUpdate/arrow.gif">
Did you forget your password?
</div>
</td>
</tr>
<input type="hidden" name="site_name" id="site_name" value="ezbizportal">
<input type="hidden" name="secure" id="secure" value="1">
<input type="hidden" name="resource_id" id="resource_id" value="49203789E2E14C4A92CEC904C24909CE">
<input type="hidden" name="login_type" id="login_type" value="2">
<tr>
<td>
<table cellspacing="0" cellpadding="0" class="content" width="100%">
<tbody><tr>
<td id="openerExistsTD" class="notification">
For security purposes, when you finish working with this site do one of the following:<li>Click the Logoff button to log off from the site.</li><li>Close all browser windows (including applications that are open in other windows).</li>
</td>
</tr>
<!-- Windows XP Service Pack 2 / 2003 / Vista Message - Start -->
<tr>
<td class="notification">
This site is intended for authorized users only.<br>
If you experience access problems contact the site administrator.
</td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>
</td>
<td class="contentright">
</td>
</tr>
<tr class="HeaderFooterHide">
<td width="100%" colspan="3" style="position:relative;">
<span class="bottomText bottomTextPos">
© 2010 Microsoft Corporation. All rights reserved. Terms and Conditions.
</span>
<table width="100%" cellpadding="0" cellspacing="0">
<tbody><tr>
<td width="47px">
<img src="/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/footerbgl.gif" align="absmiddle">
</td>
<td style="background-image: url('/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/footerbgm.gif'); background-repeat: repeat-x">
</td>
<td width="47px" style="background-image: url('/uniquesig6d7a33b352f4c09846f8a6563bae192b/uniquesig0/InternalSite/images/footerbgr.gif');">
</td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>

A generic login module would be more useful. Try scrapy loginform library:
Install
pip install -i https://pypi.binstar.org/pypi/simple loginform
Code
from loginform import fill_login_form
import requests
url = "https://example.com/login"
r = requests.get(url)
result=fill_login_form(url, text, 'username', 'password')
About
loginform is a library for filling HTML login forms given the login
url, username & password. Which form and fields to fill are inferred
automatically.

Python 2.7.8 script to login to my netgear smart switch and do some configurations

I need help on writing a script to log my Netgear GS108Tv2 smart switch's web interface and to do some configurations on mac filtering on my school's window 7 pc. I tried to use the mechanize code and some examples i found on the net, but the web interface's code format seems a little different from the example in the mechanize folder i downloaded.
Below is the code i came up with after using the example. For now i would be very glad if i am able to just log in to the web interface as i believe that is pretty much all i can do. The rest might prove a little tough but i can work on them after i manage to log in.
I am very new to code writing and totally fresh to python. Hope i can get all the help i could. Thank you very much.
import mechanize
browser = mechanize.Browser()
browser.open("http://172.16.164.23/")
browser.select_form(nr = 0)
browser.form["pwd"] = "password"
browser.submit()
browser.open("http://172.16.164.23/base/web_main.html")
well that is pretty much all could come up with, there are no errors but it does not seem to work.
Below is the web interface's code.
<html><head>
<link rel="stylesheet" href="/base/style.css" type="text/css">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <!-- Style Sheet link and Meta data -->
<title>NETGEAR GS108T</title> <!-- Netgear Page Title -->
<script src="/base/js/tabs_Layer2.js" type="text/javascript"></script>
<script src="/base/js/rollover.js" type="text/javascript"></script>
<script src="/base/js/browser.js" type="text/javascript"></script>
<script language="javascript">
var a1, a2, a3, a4, a5, a6, a7, a8;
a1 = new Image(130,29);
a1.src = "/base/images/tab_Login_off.gif";
a2 = new Image(130,29);
a2.src = "/base/images/tab_Login_on.gif";
a3 = new Image(130,29);
a3.src = "/base/images/tab_Login_ro.gif";
a4 = new Image(130,29);
a4.src = "/base/images/tab_Help_off.gif";
a5 = new Image(130,29);
a5.src = "/base/images/tab_Help_on.gif";
a6 = new Image(130,29);
a6.src = "/base/images/tab_Help_ro.gif";
a7 = new Image(130,29);
a7.src = "/base/images/onlinehelp_off.gif";
a8 = new Image(130,29);
a8.src = "/base/images/onlinehelp_on.gif";
</script>
<script type="text/javascript" language="JavaScript">
//******************************************************************************
//* Purpose: Display an input error message.
//* Return: none
//******************************************************************************
function DisplayErrorMsg()
{
alert(document.forms[0].err_msg.value);
}
function CheckError()
{
if (document.forms[0].err_flag.value == 1)
{
DisplayErrorMsg();
}
document.forms[0].pwd.focus();
}
</script>
<script language="javascript">
<!--
if (top != self)
{
top.location.href = "/base/web_main.html";
}
if (self.opener != undefined)
{
self.close();
self.opener.location.reload();
}
-->
</script>
</head>
<body onload="CheckError()">
<form method="POST" action="/base/main_login.html">
<table class="tableStyle" height="100%">
<tbody><tr class="topAlign">
<td class="leftEdge"> </td>
<td valign="top">
<table class="tableStyle">
<tbody><tr>
<td class="leftInside"><img src="/base/images/clear.gif" width="8"></td>
<td colspan="2">
<table class="tableStyle">
<tbody><tr>
<td class="logoNetGear space50Percent topAlign"><img src="/base/images/clear.gif" width="149" height="62"></td>
<td class="gs108tImage spacer50Percent topAlign rightHAlign"><img src="/base/images/clear.gif" width="190" height="60"></td> <!-- Used to get the top right logo image with help URL -->
</tr>
<tr>
<td colspan="2" class="bottomAlign">
<table class="tableStyle">
<tbody><tr>
<td class="spacer20Percent bottomAlign"><table border="0" cellpadding="0" cellspacing="0">
<tbody><tr id="tabss" class="spacer100Percent">
<td class="navTopCell"><img src="/base/images/tab_Login_on.gif" border="0"></td>
<td class="navTopCell"><img src="/base/images/tab_Help_off.gif" border="0"></td>
</tr>
</tbody></table>
</td>
<td class="loginActionCell spacer80Percent rightHAlign padding5LeftRight"></td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>
</td>
<td class="rightHAlign"><img src="/base/images/clear.gif" width="2"></td>
</tr>
<tr class="background-blue">
<td><img src="/base/images/clear.gif" width="8"></td>
<td colspan="2" class="padding7Top spacer100Percent">
<div id="primaryNav">
<table border="0" cellpadding="0" cellspacing="0">
<tbody><tr>
<td id="cloneTd" class="navCell"> </td>
</tr>
<tr id="subTubs"></tr>
</tbody></table>
</div>
</td>
<td></td>
</tr>
</tbody></table>
</td>
<td class="rightEdge"> </td>
</tr>
<tr class="topAlign">
<td valign="top" class="leftBodyNotch topAlign"> </td>
<td>
<table class="tableStyle">
<tbody><tr class="topAlign">
<td class="leftNextBodyNotch"><img src="/base/images/clear.gif" width="11" height="16"></td>
<td class="middleBodyNotch spacer100Percent"> </td>
<td class="rightNextBodyNotch"><img src="/base/images/clear.gif" width="11"></td>
</tr>
</tbody></table>
</td>
<td class="rightBodyNotch"> </td>
</tr>
<tr height="100%">
<td rowspan="2" class="leftEdge"> </td>
<td valign="top">
<table class="tableStyle" height="100%">
<tbody><tr>
<td class="leftInside"><img src="/base/images/clear.gif" width="8"></td>
<td class="spacer100Percent loginTable topAlign">
<table class="loginBox">
<tbody><tr>
<td colspan="3">
<table class="tableStyle">
<tbody><tr>
<td colspan="2" class="subSectionTabTopLeft spacer80Percent font12BoldBlue">Login</td>
<td class="subSectionTabTopRight spacer20Percent"><img src="/base/images/help_icon.gif" width="12" height="12" title="Click for help"></td></tr><tr><td colspan="3" class="subSectionTabTopShadow"> </td>
</tr>
</tbody></table>
</td>
</tr>
<tr>
<td class="subSectionBodyDot"> </td>
<td class="paddingsubSectionBodyNone">
<table class="tableStyle">
<tbody><tr>
<td class="font10Bold padding4Top">Password</td>
<td class="padding4Top"><input class="input" type="PASSWORD" name="pwd" maxlength="20" value=""></td>
</tr>
<tr>
<td colspan="2" class="padding5TopBottom10Right"><input type="IMAGE" name="login" src="/base/images/login_on.gif"></td>
</tr>
</tbody></table>
</td>
<td class="subSectionBodyDotRight"> </td>
</tr>
<tr>
<td colspan="3" class="subSectionBottom"> </td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>
</td>
<td rowspan="2" class="rightEdge"> </td>
</tr>
<tr>
<td>
<table class="tableStyle">
<tbody><tr>
<td colspan="3" class="topBottomDivider" id="cloneTd"><img src="/base/images/clear.gif" height="3"></td>
</tr>
<tr>
<td colspan="3" class="footerBody">
<table class="tableStyle rightHAlign" align="right">
<tbody><tr>
<td id="ButtonsDiv"></td>
</tr>
</tbody></table>
</td>
</tr>
</tbody></table>
</td>
</tr>
<tr>
<td class="leftEdgeFooter"><img src="/base/images/clear.gif" width="11" height="9"></td>
<td>
<table class="tableStyle">
<tbody><tr>
<td class="leftBottomDivider"><img src="/base/images/clear.gif" width="11" height="9"></td>
<td class="middleBottomDivider spacer100Percent"><img src="/base/images/clear.gif" height="9"></td>
<td class="rightBottomDivider spacer1Percent"><img src="/base/images/clear.gif" height="9"></td>
</tr>
</tbody></table>
</td>
<td class="rightEdgeFooter"><img src="/base/images/clear.gif" width="11" height="9"></td>
</tr>
<tr>
<td class="leftCopyrightFooter"><img src="/base/images/clear.gif" width="11" height="9"></td>
<td class="middleCopyrightDivider">
<table class="blue10 tableStyle">
<tbody><tr class="topAlign">
<td>Copyright © 1996-2013 NETGEAR ®</td>
</tr>
</tbody></table>
</td>
<td class="rightCopyrightFooter"><img src="/base/images/clear.gif" width="11" height="9"></td>
</tr>
</tbody></table>
<input type="hidden" name="err_flag" size="16" maxlength="15" value="0">
<input type="hidden" name="err_msg" size="128" maxlength="512" value="">
</form>
</body></html>