The API for the website Urban dictionary is a URL that takes you to a page that dumps out the json, see example here: http://api.urbandictionary.com/v0/define?term=test
Is there a simple way to grab all the text on that page? Do I still need to use some type of HTML parser?
You could use a command line tool such as curl
curl http://api.urbandictionary.com/v0/define?term=test
For a python specific solution you could try a library such as requests.
pip install requests
import requests
data = requests.get(http://api.urbandictionary.com/v0/define?term=test)
Related
I want to test my different env. like DEV,TEST,STAGE,PRODUCTION. The API call for the env. are different. For instance, http://dev.myclient.com, http://stage.myclient.com etc.
So, I want to write the test cases that will go to my specific URL make a search of specific thing and whatever the response come for example. I search for apples and I got 500 results related to that so, I want that result to print and save into text or Json. and same applies for the all different env.
then I will compare all the environments by one each other when I have a raw response data.
Any ideas how I can do that? In python specifically.
Thanks In advance!!
You could use the requests library for sending HTTP requests.This isn't a built-in library, so you should use pip install requests to install the library.
Here's an example:
import requests
url = "http://dev.myclient.com/"
response = requests.get(url).json()
print(response)
The console should now print the JSON response of the request you just sent. For this to work the URL provided should return a JSON string response.
I am automating the functionality which is written in Nodejs,and gives a graphical view when the web page is called. I need to retrieve the contents of webpage in a file. All this code will be written in python. How can I call the web page api in the python code so that I get all the contents in a file.
You can use Python Requests Module:
import requests
response = requests.get('https://example.com')
print(response.text)
To learn more
If I were trying to google something, how would I send the data I want to search to be searched? I know you can add it to the url, but I do not want to do this.
Using the requests library. You would use the .post method in the same way as the .get method. Passing in the data as a dictionary to the data parameter of the function.
The quickstart docs describe it here http://requests.readthedocs.org/en/latest/user/quickstart/#more-complicated-post-requests
If using urllib or urllib2 the data parameter of the urlopen function will POST the data to the page rather than GET it.
see the docs here http://docs.python.org/library/urllib.html#urllib.urlopen
https://stackoverflow.com/a/64983/468251 - Hello, I have question about this code, how made that working with remote website url, and how got value = fooId['value'] from all inputs, no only from first?
When you parse url on the internet, you need to find a way to download the page content html first. There are great libraries, like requests, which is said to be best for python. Say you want to parse https://stackoverflow.com/
import requests
response = requests.get("https://stackoverflow.com/")
page_html = response.text
The page_html is the page html in python string, then you can treat it like a local html file, and preform any kind of parsing on them.
As for getting all the occurrence of a pattern, you can do soup.findAll('input',name='fooId',type='hidden'), instead of just soup.find(). The soup.findAll will return a list of all occurrence.
The example use a local file. If you want to use a remote site, you need to download the file from the server and parse the html.
You can look at request or urllib2 for this.
I hope it helps
i have to retrieve some text from a website called morningstar.com . To access that data i have to log in. Once i log in and provide the url of the web page , i get the HTML text of a normal user (not logged in).As a result am not able to accees that information . ANy solutions ?
BeautifulSoup is for parsing html once you've already fetched it. You can fetch the html using any standard url fetching library. I prefer curl, as you tagged your post, python's built-in urllib2 also works well.
If you're saying that after logging in the response html is the same as for those who are not logged in, I'm gonna guess that your login is failing for some reason. If you are using urllib2, are are you making sure to store the cookie properly after your first login and then passing this cookie to urllib2 when you are sending the request for the data?
It would help if you posted the code you are using to make the two requests (the initial login, and the attempt to fetch the data).