I have a curl POST to do for elasticsearch:
curl -XPOST "http://localhost:9200/index/name" --data-binary "#file.json"
How can I do this in a python shell? Basically because i need to loop over many json files. I want to be able to do this in a for loop.
import glob
import os
import requests
def index_data(path):
item = []
for filename in glob.glob(path):
item.append(filename[55:81]+'.json')
return item
def send_post(url, datafiles):
r = requests.post(url, data=file(datafiles,'rb').read())
data = r.text
return data
def main():
url = 'http://localhost:9200/index/name'
metpath = r'C:\pathtofiledirectory\*.json'
jsonfiles = index_data(metpath)
send_post(url, jsonfiles)
if __name__ == "__main__":
main()
I fixed to do this, but is giving me a TypeError:
TypeError: coercing to Unicode: need string or buffer, list found
You can use requests http client :
import requests
files = ['file.json', 'file1.json', 'file2.json', 'file3.json', 'file4.json']
for item in files:
req = requests.post('http://localhost:9200/index/name',data=file(item,'rb').read())
print req.text
From your edit, you would need :
for item in jsonfiles:
send_post(url, item)
Use the requests library.
import requests
r = requests.post(url, data=data)
It is as simple as that.
Related
I want to add a referer while retrieving data from the web but this is not working on my python2 referer request.add_header('Referer', 'https://www.python.org').
My Url.txt content
https://www.python.org/about/
https://stackoverflow.com/questions
https://docs.python.org/2.7/
These are my codes
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
import urllib2
import threading
import time
import requests
max_thread = 5
urllist = open("Url.txt").readlines()
def url_connect(url):
try :
request = urllib2.Request(url)
request.add_header('Referer', 'https://www.python.org')
request.add_header('User-agent', 'Mozilla/5.0')
goo = re.findall('<title>(.*?)</title>', urllib2.urlopen(url.replace(' ','')).read())[0]
print '\n' + goo.decode("utf-8")
with open('SaveMyDataFile.txt', 'ab') as f:
f.write(goo + "\n")
except Exception as Errors:
pass
for i in urllist:
i = i.strip()
if i.startswith("http"):
while threading.activeCount() >= max_thread:
time.sleep(0.1)
threading.Thread(target=url_connect, args=(i,)).start()
Looks to me the problem is in your call to urlopen. You call it with the url and not with the request.
From https://docs.python.org/2/library/urllib2.html#urllib2.urlopen
Open the URL url, which can be either a string or a Request object.
You need to pass urllib.urlopen() the Request object that you just built--you're currently not doing anything with that.
I have around 100 machines running Mersive Solstice, which is a wireless display tool. I'm trying to gather a few important pieces of information, in particular the fulfillment ID for the license for each installed instance.
Using the Solstice OpenControl API, found here, I whipped up a python script to grab everything I needed using a json GET. However, even when using the example GET from the documentation,
import requests
import json
url = ‘http://ip-of-machine/api/stats’
r = requests.get(url)
jsonStats = json.loads(r.text)
usersConnected = jsonStats.m_statistics.m_connectedUsers
I encounter:
Traceback (most recent call last):
File "C:/Python27/test.py", line 7, in <module>
usersConnected = jsonStats.m_statistics.m_connectedUsers
AttributeError: 'dict' object has no attribute 'm_statistics'
Which is very confusing. I've found plenty of similar questions on SO regarding this problem, but not one that's been specifically regarding wrong GET requests from the API Reference guide.
Additionally, here is my script:
import requests
import json
from time import sleep
url = 'test'
f = open("ip.txt", "r")
while(url != ""):
url = f.readline()
url = url.rstrip('\n')
print(url)
try:
r = requests.get(url)
except:
sleep(5)
jsonConfig = json.loads(r.text)
displayName = jsonConfig.m_displayInformation.m_displayName
hostName = jsonConfig.m_displayInformation.m_hostName
ipv4 = jsonConfig.m_displayInformation.m_ipv4
fulfillmentId = jsonConfig.m_licenseCuration.fulfillmentId
r.close()
f.close
I import the URL's from a text document for easy keeping. I'm able to make the connection to the /api/config JSON, and when the URL is put into a browser it does spit out the JSON records:
Json uses "Dicts" which are a type of array. You are just using them in the wrong way. I recommend reading Python Data Structures.
Json.Loads()
Returns a dictionary not a object. Do:
dict['key']['key']
Here is how your code should look:
import requests
import json
from time import sleep
url = 'test'
f = open("ip.txt", "r")
while(url != ""):
url = f.readline()
url = url.rstrip('\n')
print(url)
try:
response = requests.get(url)
json_object = json.loads(response .text)
displayName = json_object['m_displayInformation']['m_displayName']
hostName = json_object['m_displayInformation']['m_hostName']
ipv4 = json_object['m_displayInformation']['m_ipv4']
fulfillmentId = json_object['m_licenseCuration']['fulfillmentId']
except:
pass
response .close()
f.close()
I hope this was helpful!
This question already has answers here:
HTTP requests and JSON parsing in Python [duplicate]
(8 answers)
Closed 8 months ago.
How would I parse a json api response with python?
I currently have this:
import urllib.request
import json
url = 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty'
def response(url):
with urllib.request.urlopen(url) as response:
return response.read()
res = response(url)
print(json.loads(res))
I'm getting this error:
TypeError: the JSON object must be str, not 'bytes'
What is the pythonic way to deal with json apis?
Version 1: (do a pip install requests before running the script)
import requests
r = requests.get(url='https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
print(r.json())
Version 2: (do a pip install wget before running the script)
import wget
fs = wget.download(url='https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
with open(fs, 'r') as f:
content = f.read()
print(content)
you can use standard library python3:
import urllib.request
import json
url = 'http://www.reddit.com/r/all/top/.json'
req = urllib.request.Request(url)
##parsing response
r = urllib.request.urlopen(req).read()
cont = json.loads(r.decode('utf-8'))
counter = 0
##parcing json
for item in cont['data']['children']:
counter += 1
print("Title:", item['data']['title'], "\nComments:", item['data']['num_comments'])
print("----")
##print formated
#print (json.dumps(cont, indent=4, sort_keys=True))
print("Number of titles: ", counter)
output will be like this one:
...
Title: Maybe we shouldn't let grandma decide things anymore.
Comments: 2018
----
Title: Carrie Fisher and Her Stunt Double Sunbathing on the Set of Return of The Jedi, 1982
Comments: 880
----
Title: fidget spinner
Comments: 1537
----
Number of titles: 25
I would usually use the requests package with the json package. The following code should be suitable for your needs:
import requests
import json
url = 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty'
r = requests.get(url)
print(json.loads(r.content))
Output
[11008076,
11006915,
11008202,
....,
10997668,
10999859,
11001695]
The only thing missing in the original question is a call to the decode method on the response object (and even then, not for every python3 version). It's a shame no one pointed that out and everyone jumped on a third party library.
Using only the standard library, for the simplest of use cases :
import json
from urllib.request import urlopen
def get(url, object_hook=None):
with urlopen(url) as resource: # 'with' is important to close the resource after use
return json.load(resource, object_hook=object_hook)
Simple use case :
data = get('http://url') # '{ "id": 1, "$key": 13213654 }'
print(data['id']) # 1
print(data['$key']) # 13213654
Or if you prefer, but riskier :
from types import SimpleNamespace
data = get('http://url', lambda o: SimpleNamespace(**o)) # '{ "id": 1, "$key": 13213654 }'
print(data.id) # 1
print(data.$key) # invalid syntax
# though you can still do
print(data.__dict__['$key'])
With Python 3
import requests
import json
url = 'http://IP-Address:8088/ws/v1/cluster/scheduler'
r = requests.get(url)
data = json.loads(r.content.decode())
I'm trying to write a simple script in python using urllib2 and json where I print the json to the console.
Currently I'm having trouble getting the auth_signature value. I already have the url variable setup with the appropriate keys except for the auth_signature. How do I go about this?
Here is what I have:
import json
import urllib2
import oauth2
timestamp = oauth2.generate_timestamp
nonce = oauth2.generate_nonce
url = "http://api.yelp.com/v2/search?term=food&location=Seattle&callback=callbackYelpAuth&oauth_consumer_key=XXX&oauth_consumer_secret=XXX&oauth_token=XXX&oauth_signature_method=HMAC-SHA1&oauth_timestamp=" + str(timestamp) + "&oauth_nonce=" + str(nonce) + "&oauth_signature=" + str(????)
json_obj = urllib2.urlopen(url)
data = json.load(json_obj)
print data
Following code is working fine. Hope it will help you.
import urlparse
url = "http://api.yelp.com/v2/search?term=food&location=Seattle&callback=callbackYelpAuth&oauth_consumer_key=XXX&oauth_consumer_secret=XXX&oauth_token=XXX&oauth_signature_method=HMAC-SHA1&oauth_timestamp=temstamp_value&oauth_nonce=nonce_value&oauth_signature=signature_value"
parsed = urlparse.urlparse(url)
params = urlparse.parse_qsl(parsed.query)
for x,y in params:
print "Parameter = "+x,", Value = "+y
I am using JSON library and trying to import a page feed to an CSV file. Tried many a ways to get the result however every time code execute it Gives JSON not serialzable. No Facebook use auth code which I have and used it so connection string will change however if you use a page which has public privacy you will still be able to get the result from below code.
following is the code
import urllib3
import json
import requests
#from pprint import pprint
import csv
from urllib.request import urlopen
page_id = "abcd" # username or id
api_endpoint = "https://graph.facebook.com"
fb_graph_url = api_endpoint+"/"+page_id
try:
#api_request = urllib3.Requests(fb_graph_url)
#http = urllib3.PoolManager()
#api_response = http.request('GET', fb_graph_url)
api_response = requests.get(fb_graph_url)
try:
#print (list.sort(json.loads(api_response.read())))
obj = open('data', 'w')
# write(json_dat)
f = api_response.content
obj.write(json.dumps(f))
obj.close()
except Exception as ee:
print(ee)
except Exception as e:
print( e)
Tried many approach but not successful. hope some one can help
api_response.content is the text content of the API, not a Python object so you won't be able to dump it.
Try either:
f = api_response.content
obj.write(f)
Or
f = api_response.json()
obj.write(json.dumps(f))
requests.get(fb_graph_url).content
is probably a string. Using json.dumps on it won't work. This function expects a list or a dictionary as the argument.
If the request already returns JSON, just write it to the file.