I am trying to access an API and it is mentioned that it gives in HTML. I went through these answers
(Get html using Python requests?) but I am not getting my results. I just wanted to make sure I am doing it correctly as I am getting error like this ("'{"request":{"category_id":"717234","command":"category"},"data":{"error":"invalid or missing api_key'" Is this API not working ? Is there any way to get HTML data and convert them to CSV or excel?
Here is the code which I am using.
import requests
URL = "https://api.eia.gov/category?api_key=YOUR_API_KEY_HERE&category_id=717234"
r = requests.get(url = URL)
r.text[:100]
you are using a invalid api the link to your html page is not working :
import requests
URL = "https://api.eia.gov/category?api_key=YOUR_API_KEY_HERE&category_id=717234"
headers = {'Accept-Encoding': 'identity'}
r = requests.get(URL, headers=headers)
print(r.text[:100])
output:
{"request":{"category_id":"717234","command":"category"},"data":{"error":"invalid or missing api_key
i try to change the link of the link with that one given in the answer that you put the link and i get a result :
import requests
URL = "http://www.wrcc.dri.edu/WRCCWrappers.py?sodxtrmts+028815+por+por+pcpn+none+mave+5+01+F"
headers = {'Accept-Encoding': 'identity'}
r = requests.get(URL, headers=headers)
print(r.text[:100])
output:
<!DOCTYPE html>
<HTML>
<HEAD><TITLE>Average of Precipitation, Station id: 028815</TITLE></HEAD>
<BO
as a solution you can use an external api the devoloper mode of that api : https://www.eia.gov//developer// or check this link to get a key :https://www.eia.gov/opendata/
Its not an error.
seems to me like youre missing your API KEY.
this is what wrote in the link you put:
{"request":{"category_id":"717234","command":"category"},"data":{"error":"invalid or missing api_key. For key registration, documentation, and examples see https:\/\/www.eia.gov\/developer\/"}}
Related
I am setting up code to check the reputation of any URL E.g. http://go.mobisla.com/ on Website "https://www.virustotal.com/gui/home/url"
First, the very basic thing I am doing is to extract all the Website contents using BeautifulSoup but seems the information I am looking for is in shadow-root(open) -- div.detections and span.individual-detection.
Example Copied Element from Webpage results:
No engines detected this URL
I am new to Python, wondering if you can share the best way to extract the information
Tried requests.get() function but it doesn't give the required information
import requests
import os,sys
from bs4 import BeautifulSoup
import pandas as pd
url_check = "deloplen.com:443"
url = "https://www.virustotal.com/gui/home/url"
req = requests.get(url + url_str)
html = req.text
soup = BeautifulSoup(html, 'html.parser')
print(soup.prettify())
Expect to see "2 engines detected this URL" along with Detection Example: Dr. Web Malicious
If you use their website, it'll only return a loading screen for VirusTotal, as this isn't the proper way.
What Shows Up:
Instead, what you're supposed to do is use their public API to make requests. However, you'll have to make an account to obtain a Public API Key.
You can use this code which is able to retrieve JSON info about the link. However, you'll have to fill in the API KEY with yours.
import requests, json
user_api_key = "<api key>"
resource = "deloplen.com:443"
# feel free to remove this, just makes it look nicer
def pp_json(json_thing, sort=True, indents=4):
if type(json_thing) is str:
print(json.dumps(json.loads(json_thing), sort_keys=sort, indent=indents))
else:
print(json.dumps(json_thing, sort_keys=sort, indent=indents))
return None
response = requests.get("https://www.virustotal.com/vtapi/v2/url/report?apikey=" + user_api_key + "&resource=" + resource)
json_response = response.json()
pretty_json = pp_json(json_response)
print(pretty_json)
If you want to learn more about the API, you can use their documentation.
I'm trying to get the flt information and prices through https://www.easyjet.com by using requests module.
Through browser when I filled the form easyjet.com and click on submit, it is internally fetching the data using following call:
https://www.easyjet.com/ejavailability/api/v15/availability/query?AdditionalSeats=0&AdultSeats=1&ArrivalIata=%23PARIS&ChildSeats=0&DepartureIata=%23LONDON&IncludeAdminFees=true&IncludeFlexiFares=false&IncludeLowestFareSeats=true&IncludePrices=true&Infants=0&IsTransfer=false&LanguageCode=EN&MaxDepartureDate=2018-02-23&MinDepartureDate=2018-02-23
when I'm trying to mimic the same by using following code, I'm not getting the response. I'm pretty new to this domain. Can anyone help to understand what is going wrong?
here is my code
import requests
url = 'https://www.easyjet.com/en/'
url1 = 'https://www.easyjet.com/ejavailability/api/v15/availability/query?AdditionalSeats=0&AdultSeats=1&ArrivalIata=%23PARIS&ChildSeats=0&DepartureIata=%23LONDON&IncludeAdminFees=true&IncludeFlexiFares=false&IncludeLowestFareSeats=true&IncludePrices=true&Infants=0&IsTransfer=false&LanguageCode=EN&MaxDepartureDate=2018-02-23&MinDepartureDate=2018-02-21'
http = requests.Session()
response = http.get(url, verify=False)
response1 = http.get(url1, verify=False)
print(response1.text)
I'm building a Python web scraper (personal use) and am running into some trouble retrieving a JSON file. I was able to find the request URL I need, but when I run my script (I'm using Requests) the URL returns HTML instead of the JSON shown in the Chrome Developer Tools console. Here's my current script:
import requests
import json
url = 'https://nytimes.wd5.myworkdayjobs.com/Video?clientRequestID=1f1a6071627946499b4b09fd0f668ef0'
r = requests.get(url)
print(r.text)
Completely new to Python, so any push in the right direction is greatly appreciated. Thanks!
Looks like that website returns the response depending on the accept headers provided by the request. So try:
import requests
import json
url = 'https://nytimes.wd5.myworkdayjobs.com/Video?clientRequestID=1f1a6071627946499b4b09fd0f668ef0'
r = requests.get(url, headers={'accept': 'application/json'})
print(r.json())
You can have a look at the full api for further reference: http://docs.python-requests.org/en/latest/api/.
import requests
url = "https://stackoverflow.com/"
payload = {"q": "python"}
s = requests.session()
r = s.post(url, data=payload)
print r.text
I wish to use a post request in order to obtain the subsequent webpage. However, the above code prints the source code of the home page and not the the next page. Can someone tell me what I should do to obtain the source code of the next page? I have searched through many questions on StackOverflow related to this and haven't found a solution.
Thanks in advance.
I am trying to use the requests function in python to post the text content of a text file to a website, submit the text for analysis on said website, and pull the results back in to python. I have read through a number of responses here and on other websites, but have not yet figured out how to correctly modify the code to a new website.
I'm familiar with beautiful soup so pulling in webpage content and removing HTML isn't an issue, its the submitting the data that I don't understand.
My code currently is:
import requests
fileName = "texttoAnalyze.txt"
fileHandle = open(fileName, 'rU');
url_text = fileHandle.read()
url = "http://www.webpagefx.com/tools/read-able/"
payload = {'value':url_text}
r = requests.post(url, payload)
print r.text
This code comes back with the html of the website, but hasn't recognized the fact that I'm trying to a submit a form.
Any help is appreciated. Thanks so much.
You need to send the same request the website is sending, usually you can get these with web debugging tools (like chrome/firefox developer tools).
In this case the url the request is being sent to is: http://www.webpagefx.com/tools/read-able/check.php
With the following params: tab=Test+by+Direct+Link&directInput=SOME_RANDOM_TEXT
So your code should look like this:
url = "http://www.webpagefx.com/tools/read-able/check.php"
payload = {'directInput':url_text, 'tab': 'Test by Direct Link'}
r = requests.post(url, data=payload)
print r.text
Good luck!
There are two post parameters, tab and directInput:
import requests
post = "http://www.webpagefx.com/tools/read-able/check.php"
with open("in.txt") as f:
data = {"tab":"Test by Direct Link",
"directInput":f.read()}
r = requests.post(post, data=data)
print(r.content)