We Keep Coding

Scraping SVG charts

I am trying to scrape the following svg's from the following link:
https://finance.yahoo.com/quote/AAPL/analysts?p=AAPL
The portion I am trying to scrape is as follows:
Images Here
I do not need the words of the chart (just the graphs themselves). However, I have never scraped an svg image before and i'm not sure if it is possible. I looked around but could not find any useful python packages to directly do this.
I know that I can take a screenshot of the image with python using selenium and then use PIL to crop it and save it as an svg, but I am wondering if there is a more direct way to grab these charts off the page. Any useful packages or implementations would be helpful. Thank you.
Edit: Got some down votes but not sure why Here is how I would implement it in my way..
import sys
import time
from PyQt4.QtCore import *
from PyQt4.QtGui import *
from PyQt4.QtWebKit import *
class Screenshot(QWebView):
def __init__(self):
self.app = QApplication(sys.argv)
QWebView.__init__(self)
self._loaded = False
self.loadFinished.connect(self._loadFinished)
def capture(self, url, output_file):
self.load(QUrl(url))
self.wait_load()
# set to webpage size
frame = self.page().mainFrame()
self.page().setViewportSize(frame.contentsSize())
# render image
image = QImage(self.page().viewportSize(), QImage.Format_ARGB32)
painter = QPainter(image)
frame.render(painter)
painter.end()
print 'saving', output_file
image.save(output_file)
def wait_load(self, delay=0):
# process app events until page loaded
while not self._loaded:
self.app.processEvents()
time.sleep(delay)
self._loaded = False
def _loadFinished(self, result):
self._loaded = True
s = Screenshot()
s.capture('https://finance.yahoo.com/quote/AAPL/analysts?p=AAPL', 'yhf.png')
I would then use the crop function in PIL to take the images out of the charts.

Using QWebView for web scraping seams weird to me, although I do realize that there is an advantage that it says to the server "I'm not a web scraper, I'm an embeded browser". Note that this approach is not bulletproof: your scraper can still be detected if it shows a behavior unusual for a human user.
This is how I would do it:
Id use requests to download the page (may be through a proxy that hides your real ip addres to combat ip-bans).
Then I'd parse the page using BeautifulSoup to get the url of the svg file you are trying to get.
Then I'd download the svg file and convert it into an image using something like this
If you want to continue using Qt instead, look for methods in the web view that allow inspecting DOM or extracting the resources the view downloaded.

PyQT4 QTextBrowser auto-scroll

I need a little help here.
I have this QTextBrowser where I redirect all stdout to it.
self.console_window = QtGui.QTextBrowser()
self.console_window.setReadOnly(True)
What I need now is to auto scroll to the bottom so I can see what is happening without the need of manually scroll to the bottom.
I tried this
scrollBar = self.console_window.verticalScrollBar()
scrollBar.setValue(scrollBar.maximum())
but is not working.
Any thoughts?
FIXED!!!
def handleOutput(self, text, stdout):
self.console_window.moveCursor(QtGui.QTextCursor.End)
self.console_window.ensureCursorVisible()
self.console_window.insertPlainText(text)
def Console_Window(self):
self.console_window = QtGui.QTextEdit()
self.console_window.setReadOnly(True)

just a quick update in Pyqt5.
i did it bit differently, as i've seen that a delay is needed:
self.scrollbar = self.log_output.verticalScrollBar() #the self.scrollbar is the same as your self.console_window
try:
time.sleep(0.1) #needed for the refresh
self.scrollbar.setValue(10000) #try input different high value
except:
pass #when it is not available

How to print image with pygtk

I have an application that show an image.
I want to be able to print the picture, but I can't figure how to send the image to the printer.
I ran into gtk.PrintOperation, but I seen nothing to tell what image file I want to print (so I have a blank page).
Does anyone knows how to set the data to print with pygtk and maybe gtk.PrintOperation?

In gtk.PrintOperation for painting in pages you must connect "draw-page" to a function that get PrintOperation object, a context as a parameter that you can draw page by cairo and pango libraries and page_number as third parameter, and draw pages by this function, similar below:
def print_page(print_dialog, context, n):
ctx = context.get_cairo_context()
img = cairo.ImageSurface.create_from_png("an-image.png")
cr.set_source_surface(img, 20, 20)
cr.paint()
print_dialog = gtk.PrintOperation()
print_dialog.set_default_page_setup(self.print_page_setup)
print_dialog.set_unit(gtk.Unit.POINTS)
print_dialog.set_n_pages(1)
print_dialog.set_export_filename("/path/to/export/file.pdf")
print_dialog.connect("draw-page", print_page)
win = gtk.Window()
win.connect('clicked', lambda widget: print_dialog.run())
win.set_size_request(200, 200)
win.show_all()
gtk.main()
Read more...

Thanks a lot for the hint!
I don't have only PNG images, so I had to adapt the code like this:
def print_page(self,print_dialog, context, n):
ctx = context.get_cairo_context()
gdkcr = gtk.gdk.CairoContext(ctx)
gdkcr.set_source_pixbuf(self.getPixbuf(), 0,0)
gdkcr.paint ()
def print_image(self):
print_dialog = gtk.PrintOperation()
print_dialog.set_n_pages(1)
print_dialog.connect("draw-page", self.print_page)
Thanks again .

QWebView in QDockWidget refuses to resize

This bad boy just does not want to change size to fill the dock widget area. I have tried all sorts of variations with QSizePolicy but nothing seems to work. The size of the QWebView always stays the same. Do I need to write a resize() callback?
Here's what I have right now:
self.helpwindow = QtGui.QDockWidget("Doc Browser")
self.helpwindow.setAllowedAreas(QtCore.Qt.RightDockWidgetArea)
self.helpwindow.setFeatures(QtGui.QDockWidget.DockWidgetClosable | QtGui.QDockWidget.DockWidgetFloatable)
helpAction = self.helpwindow.toggleViewAction()
helpAction.setText("&Help Browser")
helpAction.setShortcut(QtGui.QKeySequence("F1"))
helpMenu.addAction(helpAction)
self.addDockWidget(QtCore.Qt.RightDockWidgetArea, self.helpwindow)
helpbrowser = QtWebKit.QWebView(self.helpwindow)
indexpath = resource_filename(__name__,"help/index.html")
url = QtCore.QUrl("file://" + indexpath)
helpbrowser.load(url)
helpbrowser.show()
helpbrowser.updateGeometry()
helpbrowser.update()
helpbrowser.setMinimumWidth(400)
helpbrowser.setMinimumHeight( self.helpwindow.height())
sizepolicy = helpbrowser.sizePolicy()
sizepolicy.setVerticalPolicy(QtGui.QSizePolicy.MinimumExpanding)
sizepolicy.setHorizontalPolicy(QtGui.QSizePolicy.MinimumExpanding)
I'm using PyQT4 but C++ solutions gladly accepted.

Setting only paren't wont make QDockWidget manage child widget. You need to call:
helpwindow.setWidget(helpbrowser)

PyQt QWebKit frame bug?

I'm using Python, PyQt4, and QtWebKit to load a web page into a bare-bones browser to examine the data.
However, there is a small issue. I'm trying to get the contents and src of every iframe on the loaded page. I'm using webView.page().mainFrame().childFrames() to get the frames. To problem is, childFrames() loads the frames ONLY if they're visible by the browser. For example, when your browser is positioned at the top of the page, childFrames() will not load the iframes are at the footer of the page. Is there a way or setting I could tweak where I can get all ads? I've attached the source of my "browser". Try scrolling down when the page finishes it's loading. Watch the console and you will see that the iframes load dynamically. Please help.
from PyQt4 import QtGui, QtCore, QtWebKit
import sys
import unicodedata
class Sp():
def Main(self):
self.webView = QtWebKit.QWebView()
self.webView.load(QtCore.QUrl("http://www.msnbc.msn.com/id/41197838/ns/us_news-environment/"))
self.webView.show()
QtCore.QObject.connect(self.webView,QtCore.SIGNAL("loadFinished(bool)"),self.Load)
def Load(self):
frame = self.webView.page().mainFrame()
children = frame.childFrames()
fT = []
for x in children:
print "=========================================="
print unicodedata.normalize('NFKD', unicode(x.url().toString())).encode('ascii','ignore')
print "=========================================="
fT.append([unicode(x.url().toString()),unicode(x.toHtml()),[]])
for x in range(len(fT)):
f = children[x]
tl = []
for fx in f.childFrames():
print "___________________________________________"
print unicodedata.normalize('NFKD', unicode(fx.url().toString())).encode('ascii','ignore')
print "___________________________________________"
tl.append([unicode(fx.url().toString()),unicode(fx.toHtml()),[]])
fT[x][2] = tl
app = QtGui.QApplication(sys.argv)
s = Sp()
s.Main()
app.exec_()

Not sure why you're doing what you're doing, but if it's only loading what's visible, you can set the page viewport size to the content size and that should load everything:
def Load(self):
self.webView.page().setViewportSize(
self.webView.page().mainFrame().contentsSize())
However, this has a weird effect in the GUI so this solution may be unacceptable for what you are trying to do.