I'm relatively new to web scraping so I'm not sure about which approach I should use to collect informations in a specific scenario in which the informations are stored on a map and displayed in popups, such as : https://utils.ocim.fr/cartocim2/
Basically :
the website shows a map,
contact informations are displayed in popups,
a popup will appear when clicking on a geo-tag button,
targeted informations are those lines stored in that popup
I was thinking of using selenium + xpath method but I'm unsure regarding the way to deal :
with this amount of buttons that have to be clicked on
with the popups.
Would you have any resources / tips to advise me to know where to start ?
With great difficulty
Here's a start but it gets a little more complicated as the markers start overlapping so clicking the elements fails, might need to add a step to zoom in etc
from selenium import webdriver
import requests
import pandas as pd
url_base = r'https://utils.ocim.fr/cartocim2/'
driver = webdriver.Chrome(r'C:\Users\username\Downloads\chromedriver_win32\chromedriver.exe')
driver.get(url_base) #open page
#find all the icons
links = driver.find_elements_by_css_selector('div.leaflet-pane.leaflet-marker-pane > img')
import time
output = [] #temp table to append into
for i in range(5): #chaneg to len(links) when done
links[i].click() #click on first icon
output.append(driver.find_elements_by_xpath('//*[#id="popup-header"]')[0].text) #get the text of the name
time.sleep(1) #sleep
driver.find_element_by_css_selector('#initmap').click() #reset the map - needed as without it the next icon might not be on the screen due to map relocation or popup overlap
time.sleep(1)
I want to use pywhatkit to search in same tab instead of opening in new tab.I am using following code:
import pywhatkit as kit
kit.search(query)
It open new search query in new tab instead of using current tab. if anyone knows a way to handle this problem please help.
Creator/Author of the pywhatkit here, the library uses webbrowser module to search on Google and at this moment, for searching in the same tab you have to manually update the source code of the library.
To do so, go to Lib\site-packages\pywhatkit\mainfunctions.py and replace line number 230 with web.open(link,new=2), setting new=2 opens the url in same tab. I haven't tested this so I can't assure you that this will work so better to try it by importing the webbrowser module first and doing the same then update the source code.
One workaround that I found was to alter the code from the library, and I didn't actually called the library function.
import time
import webbrowser as web
import pyautogui as pg
import keyboard as kb
# Open the tab and wait for it to load
web.open(f"https://web.whatsapp.com/")
time.sleep(10)
message = 'Line 1%0aLine 2'
phone_number = '+5563999999999'
# Click on Chrome address bar then wait 1 second and click again to edit
# To find coordinates you can run
# >>> import mouseinfo
# >>> mouseinfo.MouseInfoWindow()
pg.click(1123,62)
time.sleep(1)
pg.click(1123,62)
# Type in the message, press enter, then wait page to load (8 sec) and click
# enter again to send the message
# These times will depend on internet and computer
kb.write('/send?phone=' + str(phone_number) + '&text=' + str(message_alt))
pg.press('enter')
time.sleep(8)
pg.press('enter')
# This last pause is to run this in a loop
time.sleep(2)
I'm currently using chrome(selenium) to automate something for me. For some reason I cannot get selenium to open up another webpage.
def gotoInventory():
#WebDriverWait(driver,5)
inventoryPS = driver.switch_to.window(driver.window_handles[1])
inventoryPS
openInventoryTab = driver.execute_script("window.open('https://sellercentral.amazon.com/inventoryplanning/dashboard','new window')")
openInventoryTab
def loginProton():
#Need to create a new tab to open proton mail
ActionChains(driver).key_down(Keys.CONTROL).send_keys('t').key_up(Keys.CONTROL).perform()
#above code did not work
#also tried browser.find_element_by_tag_name('body').send_keys(Keys.CONTROL + 't')
openProtonTab = driver.execute_script("window.open('https://mail.protonmail.com/login','new window')")
openProtonTab
For some reason selenium always opens the webpage on the inventory tab which is not what I want. protonMail has to be opened on its own tab and not on the inventory tab.
Any help would be greatly apperciated
The issue happens due to using the same window name in both cases which is 'new window'. You may try using different window names in inventory and proton cases. Also note that window.open creates a browser window.
I am trying to capture all the visible content of a page as text. Let's say that one for example.
If I store the page source then I won't be capturing the comments section because it's loaded using javascript.
Is there a way to take HTML snapshots with selenium webdriver?
(Preferably expressed using the python wrapper)
Regardless of whether or not the HTML of the page is generated using JavaScript, you will still be able to capture it using driver.page_source.
I imagine the reason you haven't been able to capture the source of the comments section in your example is because it's contained in an iframe - In order to capture the html source for content within a frame/iframe you'll need to first switch focus to that particular frame followed by calling driver.page_source.
This code will take a screenshot of the entire page:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://dukescript.com/best/practices/2015/11/23/dynamic-templates.html')
driver.save_screenshot('screenshot.png')
driver.quit()
however, if you just want a screenshot of a specific element, you could use this:
def get_element_screenshot(element: WebElement) -> bytes:
driver = element._parent
ActionChains(driver).move_to_element(element).perform() # focus
src_base64 = driver.get_screenshot_as_base64()
scr_png = b64decode(src_base64)
scr_img = Image(blob=scr_png)
x = element.location["x"]
y = element.location["y"]
w = element.size["width"]
h = element.size["height"]
scr_img.crop(
left=math.floor(x),
top=math.floor(y),
width=math.ceil(w),
height=math.ceil(h))
return scr_img.make_blob()
Where the WebElement is the Element you're chasing. of course, this method requires you to import from base64 import b64decode and from wand.image import Image to handle the cropping.
How to upload a picture on a web application with the selenium testing tool? I am using python.
I tried many things, but nothing worked.
What I'm doing is this (make sure drv is an instance of webdriver):
drv.find_element_by_id("IdOfInputTypeFile").send_keys(os.getcwd()+"/image.png")
and then find your submit button and click it.
A very easy way to control components like windows file selector (or just your OS in general) is by using pyautogui. You can install pyautogui through pip
import pyautogui
... # set the webdriver etc.
...
...
element_present = EC.presence_of_element_located((By.XPATH, "//button[#title='Open file selector']")) # Example xpath
WebDriverWait(self.driver, 10).until(element_present).click() # This opens the windows file selector
pyautogui.write('C:/path_to_file')
pyautogui.press('enter')
I am using fine-uploader, running selenium tests with pytest and this worked for me:
elm = driver.find_element_by_xpath("//input[#type='file']")
elm.send_keys(os.getcwd() + "/tests/sample_files/Figure1.tif")
No form submission or Enter key is needed in my case.
I added an answer for anyone looking to use deal with the annoying msofiledialogs. This is working off of saravanan's proposed solution, but more fleshed out for Python.
I had a similar problem with a script I'm working on for a company on the side. I'm attempting to upload documents for a company's clients, but due to the way their site worked, I could not utilize send_keys to directly send the path, so I had to rely on msofiledialog.
You only need to install AutoIt
https://pypi.python.org/pypi/PyAutoIt/0.3 or just "pip install -U pyautoit" through the cmd screen
type "import autoit" on your script page
Type the following before the file dialog pops up in your script:
autoit.win_active("Open")
autoit.control_send("Open","Edit1",r"C:\Users\uu\Desktop\TestUpload.txt")
autoit.control_send("Open","Edit1","{ENTER}")
It will look for the open file dialog window and fill it out and press enter.
"Open" is the title of my file dialog screen. Put the title of yours in place of "Open". There are more creative ways to utilize AutoIt's functions, but this is an easy, straightforward way for beginners.
Edit: DO NOT. DO NOT use control_send on most things if you can avoid it. It has a well-known issue of sending erroneous text. In my case, the colon in my file path was being turned into a semi colon. If you need to send input keys, it should be fine, however if you need to send text, use control_set_text. It has the same syntax.
autoit.control_set_text("Open","Edit1",r"C:\Users\uu\Desktop\TestUpload.txt")
All these approach wont work with modern image uploaders in olx !!!
Alternative approach (only for windows )
1. Automation of windows can be done using Autoit
2. Install Autoit and SciTe Script editor
3. Autoit code :(imageupload.au3)
WinActivate("File Upload"); for chrome use "open" that is the name of the window that pops
send("D:\images\image1.png"); path of the file
Send("{ENTER}")
4. compile it to get an .exe file(imageupload.exe)
5. Now from python call the .exe file like
import os
import os.system('C:\images\imageupload.exe') #path of the .exe file
Upload input control opens a native dialog (it is done by browser) so clicking on the control or browse button via Selenium will just pop the dialog and the test will hang.
The workaround is to set the value of the upload input via JavaScript (in Java it is done via JavascriptExecutor) and then submit the form.
See this question for sample in C#, I am sure there's also a way to call JavaScript in Python but I never used Selenium Python bindings
Here is the code that i used:
Imagepath = "C:\User\Desktop\image.png"
driver.find_element_by_xpath('//html/body/input').send_keys(Imagepath)
driver.find_element_by_xpath('//html/body/button').click()
I accept the Answer by karloskar. Note It is not working for FireFox (59). And it is works with Chrome Driver only.
import win32com.client
shell = win32com.client.Dispatch("WScript.Shell")
shell.Sendkeys("C:\text.txt")
shell.Sendkeys("~")
Will resolve the issue
You can easily add this one line of code to solve the problem:
driver.find_element_by_xpath("your fullpath").send_keys("C://1.png(your file root")
But pay attention, sometimes maybe you put a wrong xpath in first field. follow below steps for reach to rightful xpath:
open the inspect and click exactly on the box which you want to
upload the file.
right click on the html code and select xpath full address from copy
sub menu.
paste the root in xpath field in the code.
full code to achieve file upload using autoit tool.
u can just copy paste this and u can run, it will work since it is a acti-time demo.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time
import os
def fileUploading():
driver = webdriver.Firefox()
driver.implicitly_wait(20)
wait = WebDriverWait(driver, 10)
driver.get("https://demo.actitime.com/login.do");
driver.find_element(By.ID,"username").send_keys("admin")
driver.find_element(By.NAME, "pwd").send_keys("manager")
driver.find_element(By.XPATH, "//div[.='Login ']").click()
wait.until(ec.element_to_be_clickable((By.XPATH, "(//div[#class='popup_menu_icon'])[3]")))
driver.find_element(By.XPATH, "(//div[#class='popup_menu_icon'])[3]").click()
wait.until(ec.element_to_be_clickable((By.XPATH, "//a[contains(text(),'Contact actiTIME Support')]")))
driver.find_element(By.XPATH, "//a[contains(text(),'Contact actiTIME Support')]").click()
wait.until(ec.element_to_be_clickable((By.XPATH,"//div[#class='dz-default dz-message']")))
driver.find_element(By.XPATH,"//div[#class='dz-default dz-message']").click()
os.system("C:\\Users\\mallikar\\Desktop\\screenUpload.exe")
time.sleep(2000)
fileUploading()
below is the content of autoit code:
WinWaitActive("File Upload")
Send("D:\SoftwareTestingMaterial\UploadFile.txt")
Send("{ENTER}")
download autoIt and autoIt SCITE editor tool.
once done install autoit and the open the scite editor and paste the above code and save it with .au3 extension and once saved, right click on the file and select complile script(x64), now .exe file is created.
now use the below code:
os.system("C:\\Users\\mallikar\\Desktop\\screenUpload.exe")
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.example.org")
def upload_file():
upload_file = driver.find_element_by_id("paste here id of file which
is
enter code here`shown in html code...")
upload_file.send_keys("copy the path of file from your pc with name and
paste here...")
if toast message is disappeared within seconds you can use Chropath extension in chrome to find its xpath
This is a pure python code.
Instead of using control_send, use control_set_text to resolve inconsistencies present in the string sent to the window. See this link for reference: https://www.autoitscript.com/forum/topic/85929-incorrect-string-being-sent-via-controlsend/
import autoit
autoit.win_wait_active("Open")
autoit.control_set_text("Open", "Edit1", imgPath)
autoit.send("{ENTER}")
Use PyAutoGui if sendkeys function is not working on buttons.
sample code:
import pyautogui
driver.find_element_by_xpath("Enter Xpath of File upload button").click()
time.sleep(4) #waiting for window popup to open
pyautogui.write(r"C:\Users\AmarKumar\FilesForUploading\image.jpg") #path of File
pyautogui.press('enter')
I have used below script format to upload the images. This may help you.
Imagepath=os.path.abspath('.\\folder1\\subfolder2\file1.jpg')
driver.find_element_by_id("Id of the element").clear()
driver.find_element_by_id("Id of the element").send_keys(Imagepath)
if you do not have ID of the object ,then you can use xpath or css selector accordingly.
Using splinter :
browser.attach_file('file_chooser_id',fully_qualified_file_path)
If you are using service you will get an exception
service = Service('driver_path')
service.start()
driver = webdriver.Remote(service.service_url)
choose_image = driver.find_element(By.ID, 'id')
choose_image.send_keys(os.getcwd()+'/image.jpg')
Exception :
selenium.common.exceptions.WebDriverException: Message: unknown command: unknown command: session/$sessionId/se/file
Working code (suggestion - use id of the element instead of other)
driver=webdriver.Chrome(executable_path=driver_path)
choose_image=driver.find_element(By.ID, 'id')
choose_image.send_keys(os.path.join(os.getcwd(), 'image.jpg'))
In case you want to upload multiple pictures, you can alter the single image answer as follows:
images = ["/path/to/image1.png", "/path/to/image2.png","/path/to/image3.png"]
drv.find_element_by_id("IdOfInputTypeFile").send_keys("\n".join(images))
Note that the input form should have the multiple attribute set to True.
I post a solution that i consider best. it is a modification on solution by #sherlock (see my comment on that post)
import autoit
autoit.win_wait_active("Open")
autoit.control_set_text("Open", "Edit1", imgPath)
autoit.control_click("Open", "Button1")
Install these three
sudo apt-get install python3-tk python3-dev
pip install tk
pip install PyAutoGUI
use these two line of code.
pyautogui.write('/home/image.jpg')
pyautogui.press('enter')