I would like to retrieve some financial data in Bloomberg, however I found that it can realise that the execution is a bot/automation or something.
import ws_functions.config as ws_config
import ws_functions.cust_functions as ws_functions
import time
import sqlite3
from selenium.webdriver.common.keys import Keys
ws_functions.print_runtime()
bloomberg_frontpage_url = 'https://www.bloomberg.com/asia'
browser = ws_functions.get_ChromeDriver((""))
browser.implicitly_wait(10)
try:
browser.get(bloomberg_frontpage_url)
time.sleep(5)
browser.find_element_by_xpath('//*[#id="nav-bar-search-button"]').click() #Click "Search"
time.sleep(3)
browser.find_element_by_xpath('//*[#id="navi-search-input"]').send_keys('HSI') #Send 'HSI' to the searchbar
time.sleep(2)
browser.find_element_by_partial_link_text('HSI:IND').click()
May I know how I can resolve this?
The upon is my all codes so far, hope someone can advise me, thanks all :)
Below is the website popped to me:
I would like to scrape the below data:
Here is the direct URL to the Hang Seng Index :
https://www.bloomberg.com/quote/HSI:IND
Finally I have got a workaround to just simply call the Bloomberg API to retrieve that instead of using the mentioning solution, thanks.
Related
I'm trying to make a temporary email generator using 20-minute mail, but I can seem to print the text from my XPath. I started python 2 months ago and have been getting really good answers with my other questions. any response is appreciated.
code:
from selenium import webdriver
from time import sleep
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("http://www.20minutemail.com/")
sleep(1)
createMail = driver.find_element_by_xpath("//*[#id=\"headerwrap\"]/header/div[2]/div/div/input[2]")
createMail.click()
sleep(3)
email = driver.find_element_by_xpath("//*[#id=\"userTempMail\"]/text()")
print(email)
I've got similar problems when I tried to get some kind of attribute using xpath. I'm still not sure why. I worked arround it using the WebElement attribute. Try this:
email = driver.find_element_by_xpath("//*[#id=\"userTempMail\"]).text
Also, if you want to optimize your code you change sleep(time) for WebDriverWait(driver, time).until(some_condition). This'll stop halting your code as soon as some_condition is met. More on this here: https://selenium-python.readthedocs.io/waits.html#explicit-waits
I changed it to
email = driver.find_element_by_xpath("//*[#id=\"userTempMail\"]")
(taking the /text() out so it knows its just html)
then doing
print(email.text)
to get the inner text out.
There is a web site, I can get the data I need with Python / Selenium (I am new to Selenium and Python)
on the web page there are TABS, I can get the data on the first tab as that one is active by default, I cannot get data on the second TAB.
I attached an image: this shows the data in the overview TAB, I want to get the data in the Fundamental TAB as well. This web page is investing.com.
As for the code: (I did not use everything yet, some were added for future use)
from time import sleep, strftime
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import smtplib
from email.mime.multipart import MIMEMultipart
from bs4 import BeautifulSoup
url = 'https://www.investing.com/stock-screener/?
sp=country::6|sector::a|industry::a|equityType::a|exchange::a|last::1,1220|avg_volume::250000,15950000%3Ceq_market_cap;1'
chrome_path = 'E:\\BackUp\\IT\\__Programming\\Python\\_Scripts\\_Ati\\CSV\\chromedriver'
driver = webdriver.Chrome(chrome_path)
#driver = webdriver.Chrome()
driver.implicitly_wait(10)
driver.get(url)
my_name = driver.find_elements_by_xpath("//td[#data-column-name='name_trans']")
my_symbol = driver.find_elements_by_xpath("//td[#data-column-name='viewData.symbol']")
my_last = driver.find_elements_by_xpath("//td[#data-column-name='last']")
my_change = driver.find_elements_by_xpath("//td[#data-column-name='pair_change_percent']")
my_marketcap = driver.find_elements_by_xpath("//td[#data-column-name='eq_market_cap']")
my_volume = driver.find_elements_by_xpath("//td[#data-column-name='turnover_volume']")
The code Above all works.
The Xpath of the second tab does not work.
PE Ratio is in the second tab. (in the fundamentals)
I tried the three:
my_peratio = driver.find_elements_by_xpath("//*[#id="resultsTable"]/tbody/tr[1]/td[4]")
my_peratio = driver.find_elements_by_xpath("//*[#id='resultsTable']")
my_peratio = driver.find_elements_by_xpath("//td[#data-column-name='eq_pe_ratio']")
There are no error messages but the string 'my_peratio' han nothing in it. It is empty.
I really appreciate if you could direct me to the right direction.
Thanks a lot
Ati
enter image description here
Probably the data which is shown on the second tab is loaded dynamically.
In that case, you have to click on the second tab to show the data first.
driver.find_element_by_xpath("selector_for_second_tab").click()
After that it should be possible to get the data.
I have tried to log in a portal of Wifi automatically using python. However, find_element_by_X gives errors.
I am using Chrome as a browser.
from selenium import webdriver
import time
driver = webdriver.Chrome()
driver.get('https://hinet.hiroshima-u.ac.jp/loginweb.html')
time.sleep(2)
#driver.find_element_by_css_selector('a.button').click()
username = driver.find_element_by_css_selector("input")
username.clear
#Enter HiroshimaU ID
username.send_keys('input_username')
password = driver.find_element_by_name('pwd')
password.clear
password.send_keys('input_userpassword')
This code should work, but it just gives me errors:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"input"}
I have tried other methods, such as by_name or by_id. But none of them is working.
I am a very beginner so my question might not be clear, but I appreciate your help.
Edit(Oct 23, 2019):
I am sorry you cannot access the portal site.
I hope this screenshot may help.
Portal site
Input is not a valid cssSeletor.
You need top do something like below:
Example:
for https://www.google.com/ website,
cssSeletor of textFiled would be
input[name='q']
First, I couldn't get access to the site you mentioned, as it helps to see how the html formed.
Second based on that, I ask you to have a look at python selenium find_element_by_name
It shows how to access element by name as:
elem=browser.find_element_by_name("Email")
where you can see in the html the name tag of "Email"
I've successfully coded Python to open a website and then click a link. After that, I'd like to grab info from the site that is "active". But I'm getting the following error:
Search = Regex.search(res.text)
AttributeError: 'NoneType' object has no attribute 'text'
I think the problem is that I don't know how to "define" the clicked-into webpage as a variable. Here is the code that is relevant:
import re, requests, csv, pyperclip, logging
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('http://espn.go.com/golf/players')
find_player = browser.find_element_by_partial_link_text("Allenby, Robert")
res = find_player.click()
xRegex = re.compile(r'(1991)')
xSearch = xRegex.search(res.text)
output_player_name = xSearch.group(1)
This is my first Python coding experience and my first post to ask a question. Thanks in advance for any help.
PS I know that 1991 appears in the webpage. It's the year Robert Allenby turned pro.
find_element_by_partial_link_text() returns a anchor link and click is an event in browser which returns None. To get page content use browser.page_source and process the same as per requirement.
import re
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('http://espn.go.com/golf/players')
href_link = browser.find_element_by_partial_link_text("Allenby")
href_link.click()
res = browser.page_source
print res
xRegex = re.compile(r'(1991)')
xSearch = xRegex.search(res)
output_player_name = xSearch.group(1)
print output_player_name
Hope this helps :)
find_player is of type WebElement, res is type NoneType. The clicked-into webpage is held in the variable browser. You can use functions like find_element_by_partial_link_text to find what you need.
By the way, what are you trying to accomplish? It might help me answer your question better.
I want to download and save images using selenium in python2.7
I've tried:
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
url= "https://in.images.search.yahoo.com/images/view;_ylt=A2oKiHPRis1VplIALaEO9olQ;_ylu=X3oDMTIyN2I2OHZkBHNlYwNzcgRzbGsDaW1nBG9pZANjN2U1ZjU4NjAwMDQ1MDA0OGExZGMxY2Y0MzMyMDk0MwRncG9zAzEEaXQDYmluZw--?.origin=&back=https%3A%2F%2Fin.images.search.yahoo.com%2Fyhs%2Fsearch%3Fp%3D%2522Eiffel%2BGreens%2522%2BBalewadi%2509Pune%26n%3D60%26ei%3DUTF-8%26y%3DSearch%26type%3Dff.40.w81.hp.04-01.in.avg._.0715av%26fr%3Dyhs-avg-fh_lsonsw%26fr2%3Dsb-top-in.images.search.yahoo.com%26hsimp%3Dyhs-fh_lsonsw%26hspart%3Davg%26tab%3Dorganic%26ri%3D1&w=556&h=309&imgurl=www.propertyonepune.com%2Fimg%2Fgallery%2F0becda3e53f8db646a699e54b1333a4c.jpg&rurl=http%3A%2F%2Fwww.propertyonepune.com%2Fproperties%2F46%2FBalewadi&size=49.8KB&name=...+bungalows+by+Eiffel+Developers+%26+Realtors+Ltd.+at+%3Cb%3EBalewadi%3C%2Fb%3E%2C+%3Cb%3EPune%3C%2Fb%3E&p=%22Eiffel+Greens%22+Balewadi%09Pune&oid=c7e5f586000450048a1dc1cf43320943&fr2=sb-top-in.images.search.yahoo.com&fr=yhs-avg-fh_lsonsw&tt=...+bungalows+by+Eiffel+Developers+%26+Realtors+Ltd.+at+%3Cb%3EBalewadi%3C%2Fb%3E%2C+%3Cb%3EPune%3C%2Fb%3E&b=0&ni=21&no=1&ts=&tab=organic&sigr=11lu74lc1&sigb=17t67hvmu&sigi=1284god0v&sigt=12i2gtekb&sign=12i2gtekb&.crumb=wZ3uTmSmDfL&fr=yhs-avg-fh_lsonsw&fr2=sb-top-in.images.search.yahoo.com&hsimp=yhs-fh_lsonsw&hspart=avg&type=ff.40.w81.hp.04-01.in.avg._.0715av"
driver = webdriver.Firefox()
driver.get(url)
path = '//div[#class="iholder"]//img[#src]'
for k in driver.find_elements_by_xpath(path):
items = []
src = (k.get_attribute('src')).encode('utf8')
items.append(src)
print items
for lm in items:
driver.get(lm)
driver.sendKeys(Keys.Control + "s")
driver.send_keys(Keys.Enter)
It's giving me error:
Traceback (most recent call last):
File "C:/Users/Heypillow/Desktop/download.py", line 17, in <module>
driver.sendKeys(Keys.Control + "s")
AttributeError: 'WebDriver' object has no attribute 'sendKeys'
I've tried with:
driver.send_keys(Keys.CONTROL + "s")
Same error is showing
What should I do to save the images? Thanks in advance
Actually, op's first attempt is more correct than the selected answer. If you're not sending keys to an element for typing then you're sending them to the browser for shortcuts, etc.
ActionChains(driver).key_down(Keys.Control).send_keys("s").key_up(Keys.Control).perform()
It looks like you want to save the html for each picture, so you could use actions to get the context-menu of firefox -> "p" is shortcut for save page:
for lm in items:
driver.get(lm)
body = driver.find_element(By.tagName("body"));
ActionChains(driver).move_to_element(body).context_click(htmlElement).send_keys("p").send_keys(Keys.RETURN).perform();
I'm usually using Java, so there might be some typos in this python code of mine ;-)
The error you are getting is because .send_keys does not hang off of webdriver, it hangs off of webelement. You need to get a webelement first before trying to use .send_keys. For example,
for lm in items:
lm.sendKeys(Keys.Control + "s")
This isn't going to answer your main question but it does explain why you are getting the error message.
To answer your main question, google it and you will find many responses such as this one that already has answers.
The Accepted Answer has had one change since 2015.
Instead of
Keys.Control
It has now changed to
Keys.CONTROL
and the snippet changes to
ActionChains(browser).key_down(Keys.CONTROL).send_keys("s").key_up(Keys.CONTROL).perform()