is it possible run functions in diferents tabs at the same time?(firefox)
let me explain
I am doing scraping...then I want to open a diferent tabs and to do scraping in these at the same time.
I used to do it using different windows (the easy way). But now the web I scraped dont let me to be loggin in diferentes windows. but if it is in the same windows with multiple tabs I can be logged in everyone.
or maybe there is other way: Is there a way to run two diferents script in the same windows?. For example, run the first script and later that the second script open a new tab in the windows opened for the first script?
Thank you for the help.
In short, no. In a single Selenium webdriver instance, you can only interact with a single window handle at any given time. This answer has more related details you may find pertinent.
With regard to running two different scripts on the same window in different tabs... it is (or at least was at some point in time) technically possible to do in at least Internet Explorer. Selenium maintainers decided that it would not be feasible to implement a general solution for this. While you may find a hack to do this, it almost certainly will be browser-specific and very fragile, therefore not recommended.
Your best bet will be to just have two separate instances.
Related
I have something in my clipboard, and I'd like to run a python script that invokes CTRL+V as if it was pressed on the keyboard, and pastes the clipboard's content to the current focused window (say chrome). Any idea how to do that?
You have an X-Y problem.
What you want to accomplish is programmatically take data from one program (where you hit cntrl-V) and place it into another arbitrary program (chrome).
There are two ways to do that:
First
You can either set the programs up to have a data exchange mechanism such as a system pipe, or a network connection. This requires some API for data exchange to be already included in the program or access to the source so you might add one. There are very specific channels for cross program data exchange and you wont do well to try to circumvent them. Program A cant just say
get_program_b().get_text_box().add(clip_board);
That would be a violation of Process Isolaton and an OS like windows is written expressly to make it impossible. Some programs are designed to take input from other programs.
popen.open('mysql -e "ISNERT INTO table (a) VALUES ('4')")
Chrome is not one of those programs, chrome avoids allowing programs from doing this because it would be a target for programs to do things like, get the saved password or credit card data out of chrome. Or use save password to login to someone account and buy things in someone elses name.
Second
You could try to spoof user input and input the data exactly like a user would so chrome wont know the difference. But spoofing a user is hard to do and intentionally so because it prevents malicious scripts from taking control of a computer and doing bad things. The makers of windows are accutely aware that spoofing input is a method to circumvent allowed data exchange channels. So the makers of windows made it hard to do. You need to have access to a lot of system assets that most programs wont be given. At a minimum a program has to run as admin on windows to accomplish this, then there are libs that will let you do it. Even then Im willing to bet there are easier way to get the job done. On a machine where you have access to anything and everything it is possible. If you don't have admin access, it should be downright impossible without knowing some unpatched exploit in the system.
Therefore
What you are trying to do goes against what the computer was designed to let you do. If we had more information on what you want to accomplish maybe some of the wonderful people here could help. Getting to the end result you want shouldnt be that hard. But you way of doing it is like trying to run across the ocean, when you just need a boat. As it is my answer is -- dont do it, that's not how windows was designed to work.
First of all - many thanks in advance. I really appreciate it all.
So I'm in need for crawling a small amount of urls rather constantly (around every hour) and get specific data
A PHP site will be updated with the crawled data, I cannot change that
I've read this solution: Best solution to host a crawler? which seems to be fine and has the upside of using cloud services if you want something to be scaled up.
I'm also aware of the existence of Scrapy
Now, I winder if there is a more complete solution to this matter without me having to set all these things up. It seems to me that it's not a very distinguish problem that I'm trying to solve and I'd like to save time and have some more complete solution or instructions.
I would contact the person in this thread to get more specific help, but I can't. (https://stackoverflow.com/users/2335675/marcus-lind)
Currently running Windows on my personal machine, trying to mess with Scrapy is not the easiest thing, with installation problems and stuff like that.
Do you think there is no way avoiding this specific work?
In case there isn't, how do I know if I should go with Python/Scrapy or Ruby On Rails, for example?
If the data you're trying to get are reasonably well structured, you could use a third party service like Kimono or import.io.
I find setting up a basic crawler in Python to be incredibly easy. After looking at a lot of them, including Scrapy (it didn't play well with my windows machine either due to the nightmare dependencies), I settled on using Selenium's python package driven by PhantomJS for headless browsing.
Defining your crawling function would probably only take a handful of lines of code. This is a little rudimentary but if you wanted to do it super simply as a straight python script you could even do something like this and just let it run while some condition is true or until you kill the script.
from selenium import webdriver
import time
crawler = webdriver.PhantomJS()
crawler.set_window_size(1024,768)
def crawl():
crawler.get('http://www.url.com/')
# Find your elements, get the contents, parse them using Selenium or BeautifulSoup
while True:
crawl()
time.sleep(3600)
Im trying to communicate with a windows application with python. Need to fill in text fields and retrieve results (which are also displayed in text fields).
Currently using PywinAuto, works perfectly but its too slow for my purpose. Filling in 6 textfields and pressing two buttons takes 2 to 3 seconds... Im looking for a way to speed this up.
What is the fastest way to control and retrieve data from a windows application, that is feasible for a beginner in Python?
Thanks in advance.
This is very difficult. PywinAuto is one of the best ways to handle this kind of problem, but you have to be very careful about which Windows application you are working with. This is because not every Windows application will "publish" it's controls in a reliable way for you to automate. This is particularly true of Mozilla Firefox. However, the Microsoft Office suite does consistently publish just about every control and button on each of its interfaces that I have ever seen. Thus, the real problem is not with PywinAuto, or even with Windows, it is with whoever wrote the application you are trying to automate and whether or not they reliably publish the interfaces you were trying to control.
The other question you have to ask yourself is how you are populating the text fields and what is actually taking the time. Filling in fields and buttons should take a fraction of a second if they are independently workable. Otherwise, there is probably something else going on that you should investigate.
Good luck. This is a really tough problem.
I have been using pywinauto for 1.5 years. And I have tried lots of different tools for UI automation. You know what, pywinauto not the slowest among them.
Ofcource some actions can take a long tome (seconds), but as a rule it is a high weith actions, such as count children, etc.
Please be sure you do not call findwindows method when it is not realy need.
I'm trying to find a way to dynamically decide which web browser will open the link I clicked.
There are a few sites that I visit that work best on Iexplore and others that I prefer to open with chrome. If I set my default browser to one of these, than I'll constantly find myself opening a site with one browser, than copying the url and opening it in a new one. This happens a lot when people send me links.
I've thought of making a python script as the default browser and making a function that decides which browser should open the page. I've tried setting the script as my default browser by changing some registry keys. It seemed to work but when I try to open a site (for example writing "http://stackoverflow.com" in the run window), the url doesn't show in sys.argv.
Is there another way of finding the arguments sent to the program?
The registry keys I changed are:
HKEY_CURRENT_USER\Software\Classes\http\shell\open\command
HKEY_CURRENT_USER\Software\Classes\https\shell\open\command
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\http\shell\open\command
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\https\shell\open\command
It seemed to work on windows XP but it doesn't work on 7 (the default browser is still the same...)
Have you considered using browsers extension that emulate IE rendering instead of a homegrown solution? I believe there is one called 'ie tab' for chrome/firefox. http://www.ietab.net/
You can try build something on top of existing software which automates browser-webpage interaction, have a look at Selenium, maybe you can tweak it somehow to suit your needs.
But beware, the problem you are trying to solve is fairly complex and complicated, for instance consider just this: how are you going to translate your own subjective experience of a website into code? There are some objective indices, some pages simply break, but many things, such as bad css styling are difficult to asses and quantify.
EDIT: here's a web testing framework in which you can generate your own tests in Python It's probably easier to start with then Selenium.
I have to automate multiple webpages using Selenium. The preferred method is - WebDriver with Python on Windows. Since there number of webpages to test is very very large, I am trying to figure out if I can make this process parallel. E.g. From command line, I execute
python script1.py
Say I have 100 such scripts and I want to execute them in batches of 5 each. Also one requirement is that is 1 out of 5 scripts completes, then the master starts 6th script to always keep 5 scripts in parallel.
I have searched in docs and some forums. But I could not find any help with this. I have done similar thing in the past, but that involved firing multiple browsers actually from code, so kind of different. This involves Python and Webdriver.
Any help is appreciated.
Thanks and Regards.
I wanted to do something similar where I wanted to run multiple testcases at once. I guess this can be achieved by using Selenium Grid .
I have no idea why this was downvoted? Anyways, I found a way to do this.
It can be done by importing subprocess module and then passing arguments to the call function as -
subprocess.call(["python", "d:/pysel/hello.py"])
subprocess.call(["python", "d:/pysel/goodbye.py"])
It is not exactly parallel. But may work for my situation.