I'm trying to write a web service in Python (fairly new to it). I have acces to an API that wants an url in a specific format:
http://api.company-x.com/api/publickey/string/0/json
It is not a problem to perform a GET-request one by one but I would like to do it in a batch. So I have a text-file with strings in it. For example:
string1,
string2,
string3,
I would like to write a Python-script that iterates through that file, makes it in the specific format, performs the requests and writes the responses of the batch to a new text-file. I've read the docs of requests and it mentioned adding parameters to your url but it doesn't do it in the specific format I need for this API.
My basic code so far without the loop looks like this:
import requests
r = requests.get('http://api.company-x.com/api/publickey/string/0/json')
print(r.url)
data = r.text
text_file = open("file.txt", "w")
text_file.write(data)
text_file.close()
First open the file that has the strings,
import requests
with open(filename) as file:
data = file.read()
split_data = data.split(',')
Then iterate through the list,
for string in split_data:
r = requests.get(string)
(...your code...)
Is this what you wanted?
I've played around some more and this is what I wanted:
#requests to talk easily with API's
import requests
#to use strip to remove spaces in textfiles.
import sys
#two variables to squeeze a string between these two so it will become a full uri
part1 = 'http://api.companyx.com/api/productkey/'
part2 = '/precision/format'
#open the outputfile before the for loop
text_file = open("uri.txt", "w")
#open the file which contains the strings
with open('strings.txt', 'r') as f:
for i in f:
uri = part1 + i.strip(' \n\t') + part2
print uri
text_file.write(uri)
text_file.write("\n")
text_file.close()
#open a new file textfile for saving the responses from the api
text_file = open("responses.txt", "w")
#send every uri to the api and write the respsones to a textfile
with open('uri.txt', 'r') as f2:
for i in f2:
uri = i.strip(' \n\t')
batch = requests.get(i)
data = batch.text
print data
text_file.write(data)
text_file.write('\n')
text_file.close()
Related
I am running this program to basically get the page source code of a website I put in. It saves it to a file and what I want is it to look for a specific string which is basically # for the emails. However, I can't get it to work.
import requests
import re
url = 'https://www.youtube.com/watch?v=GdKEdN66jUc&app=desktop'
data = requests.get(url)
# dump resulting text to file
with open("data6.txt", "w") as out_f:
out_f.write(data.text)
with open("data6.txt", "r") as f:
searchlines = f.readlines()
for i, line in enumerate(searchlines):
if "#" in line:
for l in searchlines[i:i+3]: print((l))
You can use the regex method findall to find all email addresses in your text content, and use file.read() instead of file.readlines(). To get all content together rather than split into separate lines.
For example:
import re
with open("data6.txt", "r") as file:
content = file.read()
emails = re.findall(r"[\w\.]+#[\w\.]+", content)
Maybe cast to a set for uniqueness afterwards, and then save to a file however you like.
I am trying to convert the cloud-init logs to json, so that the filebeat can pick it up and send it to the Kibana. I want to do this by using a shell script or python script. Is there any script that converts such logs to json?
My python script is below
import json
import subprocess
filename = "/home/umesh/Downloads/scripts/cloud-init.log"
def convert_to_json_log(line):
""" convert each line to json format """
log = {}
log['msg'] = line
log['logger-name'] = 'cloud-init'
log['ServiceName'] = 'Contentprocessing'
return json.dumps(log)
def log_as_json(filename):
f = subprocess.Popen(['cat','-F',filename],
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
while True:
line = f.stdout.readline()
log = convert_to_json_log(line)
print log
with open("/home/umesh/Downloads/outputs/cloud-init-json.log", 'a') as new:
new.write(log + '\n')
log_as_json(filename)
The scripts returns a file with json format, but the msg filed returns empty string. I want to convert each line of the log as message string.
Firstly, try reading the raw log file using python inbuilt functions rather than running os commands using subprocess, because:
It will be more portable (work across OS'es)
Faster and less prone to errors
Re-writing your log_as_json function as follows worked for me:
inputfile = "cloud-init.log"
outputfile = "cloud-init-json.log"
def log_as_json(filename):
# Open cloud-init log file for reading
with open(inputfile, 'r') as log:
# Open the output file to append json entries
with open(outputfile, 'a') as jsonlog:
# Read line by line
for line in log.readlines():
# Convert to json and write to file
jsonlog.write(convert_to_json(line)+"\n")
After taking some time on preparing the customised script finally i made the below script. It might be helpful to many others.
import json
def convert_to_json_log(line):
""" convert each line to json format """
log = {}
log['msg'] = json.dumps(line)
log['logger-name'] = 'cloud-init'
log['serviceName'] = 'content-processing'
return json.dumps(log)
# Open the file with read only permit
f = open('/var/log/cloud-init.log', "r")
# use readlines to read all lines in the file
# The variable "lines" is a list containing all lines in the file
lines = f.readlines()
# close the file after reading the lines.
f.close()
jsonData = ''
for line in lines:
jsonLine = convert_to_json_log(line)
jsonData = jsonData + "\n" + jsonLine;
with open("/var/log/cloud-init/cloud-init-json.log", 'w') as new:
new.write(jsonData)
I am trying to fetch data from the internet to save it to a csv file.
I am facing a problem with writing to the csv file, the library leaves an empty row in the file
The data is random.org integers in text/plain format.
I'm using urllib.request to fetch the data and I am using this code to get the data and decode it
req = urllib.request.Request(url, data=None, headers={
'User-Agent': '(some text)'})
with urllib.request.urlopen(req) as response:
html = response.read()
encoding = response.headers.get_content_charset('utf-8')
data = html.decode(encoding)
I am using this line of code to open the csv file :csvfile = open('data.csv', "a")
Writing to the file:
writer = csv.writer(csvfile, lineterminator = '\n')
writer.writerows(data)
and of course I close the file at the end
Things I tried and didn't help :
Using (lineterminator = '\n') when writing
Using (newline = "") when opening the file
Defining a "delimiter" and a "quotechar" and "quoting"
Updated Answer
If when you create the list that is being written to the data list, if you add it as a list so that data becomes a list of lists, then set your line delimiter to be '\n', it should work. Below is the working code I used to test.
import csv
import random
csvfile = open('csvTest.csv', 'a')
data = []
for x in range(5):
data.append([str(random.randint(0, 100))])
writer = csv.writer(csvfile, lineterminator = '\n')
writer.writerows(data)
csvfile.close()
and it outputs
I have a long list of .json files that I need to download to my computer. I want to download them as .json files (so no parsing or anything like that at this point).
I have some code that works for small files, but it is pretty buggy. Also it doesn't handle multiple links well.
Appreciate any advice to fix up this code:
import os
filename = 'test.json'
path = "C:/Users//Master"
fullpath = os.path.join(path, filename)
import urllib2
url = 'https://www.premierlife.com/secure/index.json'
response = urllib2.urlopen(url)
webContent = response.read()
f = open(fullpath, 'w')
f.write(webContent)
f.close
It's creating a blank file because the f.close at the end should be f.close().
I took your code and made a little function and then called it on a little loop to go through a .txt file with the list of urls called "list_of_urls.txt" having 1 url per line (you can change the delimiter in the split function if you want to format it differently).
def save_json(url):
import os
filename = url.replace('/','').replace(':','')
# this replaces / and : in urls
path = "C:/Users/Master"
fullpath = os.path.join(path, filename)
import urllib2
response = urllib2.urlopen(url)
webContent = response.read()
f = open(fullpath, 'w')
f.write(webContent)
f.close()
And then the loop:
f = open('list_of_urls.txt')
p = f.read()
url_list = p.split('\n') #here's where \n is the line break delimiter that can be changed
for url in url_list:
save_json(url)
I have this script which abstract the json objects from the webpage. The json objects are converted into dictionary. Now I need to write those dictionaries in a file. Here's my code:
#!/usr/bin/python
import requests
r = requests.get('https://github.com/timeline.json')
for item in r.json or []:
print item['repository']['name']
There are ten lines in a file. I need to write the dictionary in that file which consist of ten lines..How do I do that? Thanks.
To address the original question, something like:
with open("pathtomyfile", "w") as f:
for item in r.json or []:
try:
f.write(item['repository']['name'] + "\n")
except KeyError: # you might have to adjust what you are writing accordingly
pass # or sth ..
note that not every item will be a repository, there are also gist events (etc?).
Better, would be to just save the json to file.
#!/usr/bin/python
import json
import requests
r = requests.get('https://github.com/timeline.json')
with open("yourfilepath.json", "w") as f:
f.write(json.dumps(r.json))
then, you can open it:
with open("yourfilepath.json", "r") as f:
obj = json.loads(f.read())