I will be using python. My plan is to make a program that searches a bunch of things, and sees how many search results google has for it. But I can only figure out how to get custom search engine to kind of work.
In python, how do I use the Google API to do a simple search using Google's main search engine? As I understand, the answer to this has changed within in the last few years as google has made a push to the google app engine.
Recently I was also looking for Google Search API and was misguided by a lot of outdated information. Here is what I found on Google Developers website: https://developers.google.com/api-client-library/python/apis/customsearch/v1
According to the docs your function will be something like
from googleapiclient.discovery import build
def google_results_count(query):
service = build("customsearch", "v1",
developerKey="[put your API key here]")
result = service.cse().list(
q=query,
cx='[put your CSE key here']
).execute()
return result["searchInformation"]["totalResults"]
print google_results_count('Python is awesome')
Unfortunately, using CSE API will give you different result count from the one you get using web search. In the example above I got 2 680 000 in Python and approx. 21 000 000 for the same query on Google.com
Here is an explanation why: https://support.google.com/customsearch/answer/70392?hl=en
Getting the API and CSE keys and all the limitations of CSE is a whole different story, I highly recommend you looking at this answer: https://stackoverflow.com/a/11206266/1704272 and the next one below for the alternatives.
Another approach is to parse the HTML response from Google.com which will give you the most complete results but it is not very reliable because Google changes the HTML markup. And more important this is against their TOS, more to read here: Is it ok to scrape data from Google results?
My conclusion.
You have three options:
Use Google CSE API (free). Use this, if you need to stay legal and you are sure you won't exceed the limit. Can not be used in public application.
Use paid API (Google or any other, less expensive). It is legal to use this for any public application but be ready to pay for that.
Scrape Google web page. This will give you the best results but I would use this option only for private needs.
Related
As the title states I'm looking for a way to get backlinks for a given url / website using Google APIs, since I already have an api key and I'd rather use it instead of relying on other services.
I already tested services like ahrefs, majestic, moz, serpstat etc and actually they can give me the infomation I need, but I was wondering if there was a way to do it with Google.
For what I've read during my past researches I saw that Google offered a way to do it, but then it became deprecated, so no more usable. Do they really took away this feature for good?
I've also noticed that Google offers a similar service with his Google Search Console, but it can just be used for your own website, I'd like to get those kind of information for a random given url.
Actually I will be using Python in my project, but I don't think there's a package able to deliver me these kind of data, or at least I looked for it and didn't find anything.
Any help would be appreciated.
Right now I am trying to figure out how to take a row of data, maybe like 50 entries max, and enter it individually into a search bar. But first I need to understand the beginning concepts so I want to do a practice program that could take info from an Excel sheet and enter into a Google search or YouTube, for example.
My problem is there seems to be no resource on how to do this for beginners. All posts I have read are either parts of the whole problem or not related to actually using a search bar but instead creating one. Even then every post I read has 100 plug-ins I could possibly add.
I'm just looking for a consistent explanation to where I can grasp how I can manipulate code in order to use a search bar function.
To perform a web search (Google, YouTube or whatever) from a program you need to execute the search, either by building up and calling an appropriate search URL or by making a call to an API provided by that site.
The article 'Python - Search for YouTube video' provides a code sample and explanation of how to generate and call a URL to perform a YouTube keyword search. You could do something similar for a Google search by analysing the URL from the result of a Google search, or try searching for 'Python submit google search url'.
The above approach is simplistic and relies on the URL structure for a particular site staying the same. A more complex, reliable and flexible approach is to use the API. For YouTube:
YouTube API - Python developers guide
YouTube API - Python code samples - Search by keyword
I am making a web application that will monitor the amount of members and discussions in each one of the groups listed here (http://www.codecademy.com/groups#web) and display that information in nice graphs.
However, as you have already seen, it looks like I need to create an account and login with it.
Having in mind that my project is using Python for the server side, how do I do it? Which API is easier? (Google, FB or twitter?)
I would really love if you could also provide some examples because I am really new at this (and at Python too).
The official wrapper around the Twitter API for Python is this one. I used it and it's very easy. You should first read this page and also register an application to get OAuth keys.
Example:
import twitter
# Remember to put these values
api = twitter.Api(consumer_key="",
consumer_secret="",
access_token_key="",
access_token_secret="")
# Get your timeline
print api.GetHomeTimeline()
Hope it helps.
I have a python program that takes the md5 & sha1 hash values of passwords and searches for them on the internet using Google's custom search api. The problem is that I'm getting 0 results(which means the hash probably isn't in a rainbow table) when I run the program. But when I searched using my browser, I get a whole bunch of results, in fact at least 10 pages of results.
Could the problem lie in the cx value I used? I picked it up from the sample program provided by google as I couldn't figure out how to get one for myself. Or does the custom search api give only selected results and it's futile trying to get more results from it?
I know it's pretty old post but it is still returned very high in google results so a little bit of clarification:
You can create your own CSE in here: https://www.google.com/cse/ .
API codes can be created using API console: https://cloud.google.com/ .
Using Google Custom Search you can search the whole Web: go to the system from point 1, from the menu on the left choose the CSE to edit, then in the Configuration -> Basics -> Sites select the option to search the whole Web and finally remove previously specified sites.
Still using CSE you might not get the same results as using live google as it does not include google features (real-time results, social features etc.) and once you specify more than 10 sites to look on it can actually use sub-index. More information can be found in here: https://support.google.com/customsearch/answer/70392?hl=en
The Google Custom Search API let's you search the Google indexes for a specific website only, and you will not find any results from anywhere else on the internet. The cx parameter tells Google what website you want to search.
From the Google Custom Search Engine page:
With Google Custom Search, add a search box to your homepage to help people find what they need on your website.
You could use the deprecated Google Web Search API (JavaScript API, should work until November 2013), or you'd have to scrape the HTML UI provided to your browser instead (also see What are the alternatives now that the Google web search API has been deprecated?).
I am trying to use the Google API Custom Search, and I don't have any clue where to start. It seems you have to create a "custom search engine" in order to parse search results, and you are limited to 100 per day.
What module should I use for this? I believe I start here: http://code.google.com/p/google-api-python-client/
I need an API key or something? Basically I want to be able to do this operation, and Google's documentation is confusing, or perhaps beyond my level.
Pseudocode:
from AwesomeGoogleModule import GoogleSearch
search = GoogleSearch("search term")
results = search.SearchResultsNumber
print results
Basically, that number you get of total results for a particular search term? I want to scrape it. I don't want to go via the front-end Google, because that's very easy to get blocked. I don't need to go beyond the 100 searches that the API allows. This will only be for 30-50 search terms, maybe 80-100 at MOST.
Sample code for Custom Search using the google-api-python-client library is here:
http://code.google.com/p/google-api-python-client/source/browse/#hg%2Fsamples%2Fcustomsearch
You will need to create your own API Key by visiting:
https://code.google.com/apis/console/
Create a project in the APIs Console, making sure to turn on the Custom Search API for that project, and then you will find the API Key at the bottom of the API Access tab.