I want to load a webpage in python and do something
import webbrowser
url = 'http://google.com'
webbrowser.open(url)
//do something
But I don't want the browser has been showed on my screen, I want to hide it so that I can do some other stuff,
please give me some advices,
thanks
In some cases, autoraise is working.
Example:
import webbrowser
webbrowser.open('www.yoursite.com', autoraise=False)
Related
I would like some advice on how to scrape data from this website.
I started with selenium, but got stuck at the beginning because, for example, I have no idea how to set the dates.
My code until now:
from bs4 import BeautifulSoup as soup
from openpyxl import load_workbook
from openpyxl.styles import PatternFill, Font
from selenium import webdriver
from selenium.webdriver.common.by import By
import datetime
import os
import time
import re
day = datetime.date.today().day
month = datetime.date.today().month
year = datetime.date.today().year
my_url = 'https://www.eex-transparency.com/homepage/power/germany/production/availability/non-usability-by-unit/non-usability-history'
cookieValue = '12-c12-cached|from:' +str(year)+ '-' +str(month)+ '-' +str(day-5)+ ','+'to:' +str(year)+ '-' +str(month)+ '-' + str(day) +',dateType:1,company:PreussenElektra,fuel:uranium,canceled:0,durationComparator:ge,durationValue:5,durationUnit:day'
#saving url
browser = webdriver.Chrome(executable_path=r"C:\Program Files (x86)\Google\Chrome\Application\chromedriver.exe")
my_url = 'https://www.eex-transparency.com/homepage/power/germany/production/availability/non-usability-by-unit'
browser.add_cookie({'name': 'tem', 'value': cookieValue})
browser.get(my_url)
my_url = 'https://www.eex-transparency.com/homepage/power/germany/production/availability/non-usability-by-unit/non-usability-history'
browser.get(my_url)
Obviously I am not asking for code, just some suggestions on how to continue with Selenium (how to set dates and other data) or any idea on how to scrape this website
Thanks in advance.
EDIT: I am trying to follow the cookie way. That is my updated code, I read that the cookie need to be created before loading the page and so I did, any idea why it is not working?
Best approach for you will be changing cookies, because every filter data is saved in cookie.
Check cookies in chrome ( f12 -> application -> cookies ) and play with filters. If you will change it in programmers tools you have to refresh website :)
Check this post on how to change cookies in selenium python.
To get values from website you have to use classic way like u did here, but you will have to use classes:
radio = browser.find_elements_by_class_name('aaaaaa')
You can always use xPath to search elements ( chrome will generate them for you ).
Is there any particular reason why you have decided to use selenium over other web scraping tools (scrapy, urllib, etc.)? I personally have not used Selenium but I have used some of the other tools. Below is an example of a script to just pull all the html from a page.
import urllib
import urllib2
from bs4 import BeautifulSoup as soup
link = "https://ubuntu.com"
page = urllib2.urlopen(link)
data = soup(page, 'html.parser')
print (data)
This is just a short script to pull all the HTML off a page. I believe BeautifulSoup has additional tools for inputting data into fields, but the exact method slips my mind right now, if I can find my notes on it I will edit this post. I remember it being very straightforward, though.
Best of luck!
Edit: here's a discussion web scraping tools from reddit a while back that I had saved https://www.reddit.com/r/Python/comments/1qnbq3/webscraping_selenium_vs_conventional_tools/
I install python spynner im tring to load my page "http://nexo.rf.gd/app.js"
Cant load this please help
I try with mechinize.Browser() but it can't load html
Then i try with selenium remote webdrivers and finally i try with spynner because this browser can able to run js but dont know how ?? Help me please
## with mechanize.Browser() ##
import mechanize
Br = mechanize.Browser()
Br.open('http://nexo.rf.gd/app.js')
Br.response().read()
Answer is HTML code witch says this site need js enable browser
answer will be 'import sqlite'`
Finally i found a way to open my page html code #witch is import sqlite3
Every browser fails but with dryscrape my problem solve thanks anyway
Python Webbrowser library has the new varibale for its open where, by default, it will open up a file in the same browser window. Is there a way to open up the same file in the same TAB. Like whatever the current page that is open on the browser, redirect that page the specified url.
current code is:
import webbrowser
url = "http://www.google.com"
webbrowser.open(url)
but this opens in a new tab but I want it to open in my current tab. Thanks in advance.
It should be as easy as adding new=0 like this:
webbrowser.open(url, new=0)
Display url using the default browser. If new is 0, the url is opened in the same browser window if possible.
according to:
https://docs.python.org/2/library/webbrowser.html
It is not possible with the Webbrowser module.
It is possible with Selenium module though
http://selenium-python.readthedocs.org/
download the module: https://pypi.python.org/pypi/sst/0.2.4
In the code:
from sst.actions import *
go_to('url')
import webbrowser
webbrowser.open(url)
I am using this to open url in browser. But it opens only in 'Mozilla' why?
Just look at the docs. It uses the default browser. Look at webbrowser.get() for instructions on using a different browser.
import urllib
fun open():
return urllib.urlopen('http://example.com')
But when example.com opens it does not render CSS or JavaScript. How can I open the webpage in a web browser?
#error(404)
def error404(error):
return webbrowser.open('http://example.com')
I am using bottle. Giving me the error:
TypeError("'bool' object is not iterable",)
with the webbrowser module
import webbrowser
webbrowser.open('http://example.com') # Go to example.com
import webbrowser
webbrowser.open(url, new=0, autoraise=True)
Display url using the default browser. If new is 0, the url is opened in the same browser window if possible. If new is 1, a new browser window is opened if possible. If new is 2, a new browser page (“tab”) is opened if possible. If autoraise is True, the window is raised
webbrowser.open_new(url)
Open url in a new window of the default browser
webbrowser.open_new_tab(url)
Open url in a new page (“tab”) of the default browser
On Windows
import os
os.system("start \"\" https://example.com")
On macOS
import os
os.system("open \"\" https://example.com")
On Linux
import os
os.system("xdg-open \"\" https://example.com")
Cross-Platform
import webbrowser
webbrowser.open('https://example.com')
You have to read the data too.
Check out : http://www.doughellmann.com/PyMOTW/urllib2/ to understand it.
response = urllib2.urlopen(..)
headers = response.info()
data = response.read()
Of course, what you want is to render it in browser and aaronasterling's answer is what you want.
You could also try:
import os
os.system("start \"\" http://example.com")
This, other than #aaronasterling ´s answer has the advantage that it opens the default web browser.
Be sure not to forget the "http://".
Here is another way to do it.
import webbrowser
webbrowser.open("foobar.com")
I think this is the easy way to open a URL using this function
webbrowser.open_new_tab(url)