I am trying to crontab a python/selenium script but obviously this won't work since Firefox requires a display, I went down the PyVirtualDisplay / xvfb route but this won't give me visual feedback, I don't want to run the script headless,I just want it to launch automatically like if I was entering the command line by hand in the terminal, just automatic . Would you know of a way to do it without going through the crontab/headless route ?
thank you ,
If you want to run your test in background but same time you can see browser when you want, try this:
Install CI and run it service by your user (try Jenkins, google will help you)
Write Ant Script, it will help to automate the run. (URL)
I'm using CI and Ant to run my test by schedule and when i want buy one click. Good luck maybe this will help you.
Related
I want to know is there a way to create a webdriver by script1.py , but close the webdriver by script2.py . I don't use the time.sleep() because I set these scripts to execute after few month. And I'm afraid that the scripts will delay due to the network crash. Any ideas are welcome.
Thanks!
The Python Selenium framework, SeleniumBase, comes with a special pytest command-line option called, --reuse-session, which tells all your tests to reuse the same browser session, even if all the tests live in different Python files. (More info on SeleniumBase command-line options here.) When using --reuse-session mode, the first test run will spin up the web browser, and then the last test run will close it. This should give you what you're looking for, where a different Python script closes the browser than the one that originally opened it.
Is there a way to run Selenium program outside of my PC? I'm happy with my code since its working as intended but running it over and over is annoying since its slowing my PC while opening and reloading browser. So my question is:
What should I use to run my script on something like host, Virtual Machines or something.
I'm pretty new to Selenium and Python so I dont know how to do it and i cant find enything useful on the internet that would help me.
I'm running Ubuntu server 16.04 and still getting to grips with it. I have a python script that runs in an endless loop, performing a task related to fetching data from an external source.
What I'm trying to do, is make this python script start after (or during) boot and then run in the background.
I've tried editing rc.local but the boot sequence just hangs since the script keeps running.
Any advice would be greatly appreciated.
As one of the comments mentions, you can use cronjobs to start scripts at certain times such as at startup(as you would like to do). It also would not halt execution like you mentioned with rc.local
The line that you need to add to the chronjob list is -
#reboot python /home/MyPythonScript.py
Here is are a couple of useful tutorials that show you how to do this: http://kvz.io/blog/2007/07/29/schedule-tasks-on-linux-using-crontab/
https://help.ubuntu.com/community/CronHowto
If you would like to do it with python itself there is this handy python library - https://pypi.python.org/pypi/python-crontab/
tmux is a great utility for background desktops. You can use it for this:
sudo apt get install tmux
Then add it to your rc.local:
/usr/bin/tmux new-session -d 'python /path/to/your/script'
After boot you can use it as follow:
tmux attach
And your console will be attached to the last desktop working at background.
I wrote a simple script that parses some stuff off the web and emails it to me. Very simple. But I'm starting to realize that implementing this is going to be far more difficult that it really should be.
All I really want to do is run this script once a day.
I have explored using Google App Engine, but it doesn't like smtplib using ssl to login to my gmail to send an email.
I am considering using Heroku, but that just seems like a lot of work for something so simple.
I tried using my raspberry pi, but I'm not sure the script is still running when I exit ssh. I looked into running the scrip on a cron job, but I'm not sure thats an elegant solution.
I looked into running an applescript from my calendar, but I'm not sure what happens if my computer is closed and/or offline.
My question is: Is there a simple, elegant, easy solution here?
When you start the script from your session (./script.py or python script.py) than it stops running when you disconnect. If you want to run the script this way for whatever reason, I would recommend using tmux .
If you're using Raspian or another Debian based distro for your Pi:
$ apt-get install tmux
$ tmux
# disconnect from your tmux session with pressing CTRL+B and (after that) D
# to reattach to your session later, use
$ tmux attach
I would recommend using cron. Just add a file like this in /etc/cron.d/, if you want to run it at a specific time (e.g. every day at 1am), like so:
$ echo "0 1 * * * python /path/to/your/script.py > /dev/null 2>&1" > /etc/cron.d/script-runner
# and don't forget to make it executable
$ chmod +x /etc/cron.d/script-runner
Wikipedia has a nice explanation of the format (and also of shortcuts like #hourly and #daily).
If you don't care when exactly it runs, you can just put your script into /etc/cron.daily/. Don't forget a chmod +x to make it executable.
If you don't want to run it on one of your machines, you can also get a shell on one of uberspaces servers. You can pay whatever you wan't (minimum 1 Eur/month) and you get a shell on a linux box with 10GB storage (the first month is free for testing, cancelation happens automatically when you don't pay, no strings attached). I'm sure there are a lot of other services like that, I just mention it, because it's a cheap one with nice support. Also you get a domain (..uberspace.de) and can send mail from the server (e.g. with mail). So no need to use a gmail account.
Edit: Overread the "python" part. Changed everything to .py. Either use #!/usr/bin/env python3 (or 2.7) in your script or start the script via python scriptname.py.
I have a Python script that uses Selenium WebDriver (with PyVirtualDisplay as the display) to log into Flickr.
http://pastebin.com/dqmf4Ecw (you’ll need to add your own Flickr credentials)
When I run it as myself on my Debian server, it works fine. (I’m a sudoer, but I don’t use sudo when running the script.)
When I run it as the user www-data (which is what it’ll be running as eventually, because I want to trigger it from a Django website), I get two problems, one small, one big:
(Small): the webdriver.Firefox() call takes 30–45 seconds to return, compared to 2 seconds when run as myself
(Big): the script fails to log into Flickr. In order to log in, I find the username and password fields on the Flickr signin page (http://www.flickr.com/signin/), and use element.send_keys() to enter the username and password. Although Selenium seems to find the elements (i.e. no NoSuchElementException is thrown), the values do not get entered in the fields when the script is run as www-data (according to the screenshots I take using browser.save_screenshot), unlike when the script is run as myself.
Why does send_keys() not work when the script is run as www-data? (And is it related to the browser taking much longer to start?)
Maybe you have something different in your environment.
Try copy by example your ~/.bashrc in /home/www-data
If it's not sufficient, run this command both as your current user & as www-data:
strace -tt -f -s 1000 -o /tmp/trace ./script.py
And paste it (filter out your logins/passwords) somewhere.
We will see what's happens.
Sometimes, Firefox performs some nasty plugin compatibility check during startup. As each user can have a different set of browser plugins, this could be responsible for the difference in startup times. You could try to sync your Firefox profiles between users.
Then, are you sure that Firefox as user www-data has proper network/internet access? Can you confirm that the Flickr site loads properly via SeleniumHQ? "The script fails to log into Flickr" is too unprecise. Some more details about why it fails might reveal the problem instantaneously.
Edit: Sorry, I just understood that there shouldn't be a difference in profiles, because Selenium creates one. Nevertheless, my second point might be useful, so I won't delete this answer.
Some more things to ponder about:
Could you spawn firefox manually from www-data account once and make sure that Firefox is not updating itself before every execution of the script? I once faced this problem with Selenium RC on Windows and had to let the update finish before starting the script with the updated binary.
As a workaround, I guess you could you try running the script as www-data user but connecting remotely to a webdriver server running in your login (aka "grid" mode). Would that work for you?
I would suggest getting the latest chrome from google and trying input.send_keys() in that browser instead.
Sometimes some features of webdriver get broken with new releases.. If you are bent on testing with firefox, you might have better luck with an older/newer version of selenium webdriver.
I remember having a similar issue regarding send_keys() on a mac.. My issue was that send_keys() did not work in certain modal windows after I updated selenium webdriver.. I fixed it by reverting to an older webdriver that I knew to work. However, I was using Ruby and not Python to drive webdriver.
sometimes, there might also be a problem with getting the correct ENV variables in your shell if you use it as a different user. I would suggest trying to troubleshoot and see if all the shell ENV variables are set properly under www-data.