Web Scraping Api with Python - python

I've already built a python script that scrapes some data from a website that require a login-in. My question is: How can i transform this script into an api? For example i send to the api username, password and data required, then it returns the data needed.

A web API is nothing but an HTTP layer over your custom logic so that requests can be served the HTTP way (GET PUT POST DELETE).
Now, the question is, how?
The easiest way is to use already available packages called "web frameworks" which python has in abundance.
The easiest one to probably implement would mostly be Flask.
For a more robust application, you can use django as well.

Related

Rest API Get Automation using Robotframework

I want to automate a rest api get method using robotframework
In robot we have a library called requests. Using this we can get the data and all. My question is that to automate that api is it OK just validate the status code 200?
Or do we need to validate the entire json data?
If we must to validate the data then how to validate?
If we have one value we can validate that but if we have multiple how to validate??
If you are still having issues this package may help you:
https://github.com/Accruent/robotframework-zoomba
For APIs it uses the requests library but with some extended methods for making calls easier and validation very simple. You can take a look at the example robot tests for help.

How I can get user input from browser using python

I am in the middle of my personal website development and I am using python to create a "Comment section" which my visitors could leave comments at there in public (which means, everybody can see it, so don't worry about the user name registration things). I already set up the sql database to store those data but only thing I haven't figured out yet was how to get the user input (their comments) from the browser. So, is there any modules in python could do that? (Like, the "Charfield" things in django, but unfortunately I don't use django)
For that you would need a web framework like Bottle or Flask. Bottle is a simple WSGI based web framework for Python.
Using either of these you may write simple REST based APIs, one for set and other for get. The "set" one could accept data from your client side and store it on your database where as your "get" api should return the data by reading it from your DB.
Hope it helps.

API endpoint confusion

Disclaimer: I am new to working with APIs
I am working on leveraging gimbals API and am trying to figure out what exactly end points are? I realize that they link to a server, but how exactly are they used in development?
Are the endpoints used to link to specific sources of data?
I am using python(django) and it would be great to understand exactly how access and or change information on gimbals end.
PS- When looking at the gimbal api, I noticed that they have a REST api and some other mobile stuff going on. If I am building a web platform, I would only be interested in the REST API portion correct?
An endpoint in a RESTful API is generally just a URL. The URL represents some kind of resource. If it was an order processing API, the resources would be things like customers, orders, etc.
You interact with these resources by making HTTP requests of various sorts; GET if you want to know the content of a resource, POST if you want to change something, and so on. Have a look at this for basic information on REST and web APIs.
You don't need Django to interact with a RESTful API that someone else provides. All you really need is python's urllib, and maybe the json module, if they're sending the data in JSON. The REST API they provide is probably the main thing they want developers using, but if they have multiple APIs then it's hard to say which one is right for you without understanding the application better.

Distinguishing between GET and POST data in CherryPy?

I've been deciding between Python web frameworks for a project of mine and I've really liked how lightweight, flexible, and concise CherryPy is compared to others. The only problem I'm having is I can't find any documentation on how to distinguish between data sent via GET and via POST.
For example, I don't want users to be able to provide their login credentials through a GET request (http://example.com/login?username=user&password=pass) but, according to CherryPy's tutorial, all data is sent as method parameters, no matter what HTTP method they're sent as. Is there some way to say I only want the POST data or do I have to use MethodDispatcher?
Thanks!
See the docs.
A string containing the HTTP method, such as "GET" or "POST". Set
in the "run" phase.
looks like checking cherrypy.request.method is what you want to do.

How to use python for a webservice

I am really new to python, just played around with the scrapy framework that is used to crawl websites and extract data.
My question is, how to I pass parameters to a python script that is hosted somewhere online.
E.g. I make following request mysite.net/rest/index.py
Now I want to pass some parameters similar to php like *.php?id=...
Yes that would work. Although you would need to write handlers for extracting the url parameters in index.py. Try import cgi module for this in python.
Please note that there are several robust python based web frameworks available (aka Django, Pylons etc.) which automatically parses your url & forms a dictionary of all it's parameters, plus they do much more like session management, user authentication etc. I would highly recommend you use them for faster code turn-around and less maintenance hassles.

Categories