I'm trying to make a json get request on multiple urls.
(I'm new to programming as you will see below.)
My goal is to run this python script with multiple links. The only thing needs to be changed is the "b" part of the link, so it can run multiple searches with different numbers (strings), preferably using a list of numbers and exporting all the results to a csv list.
My question is: How should I change the code so I can use a list of numbers for "b" and get all the requests? (like a loop)
Please find below my code and thank you for your time.
import requests
import json
a = 'https://www.g2a.com/lucene/search/filter?jsoncallback=jQuery111002521088376353553_1491736907010&skip=28837%2C28838%2C28847%2C28849%2C28852%2C28856%2C28857%2C28858%2C28859%2C28860%2C28861%2C28862%2C28863%2C28867%2C28868%2C28869%2C29472%2C29473%2C29474%2C29475%2C29476%2C29482%2C29486%2C33104&minPrice=0.00&maxPrice=640.00&cn=&kr=&stock=all&event=&platform=0&search=&genre=0&cat=0&sortOrder=popularity+desc&start=0&rows=12&steam_app_id='
b = '515220'
c = '&steam_category=&steam_prod_type=&includeOutOfStock=false&includeFreeGames=false&_=1491736907012'
d = a + b + c
r = requests.get(d)
json_object = json.loads('{"data":%s}}' % (response.content.decode("utf-8").replace("jQuery111002521088376353553_1491736907010(", "")[:-2].replace("\'", "")))
for game in json_object["data"]["docs"]:
print ("Name: %s (Price: %s)" % (game["name"], game["minPrice"]))
You can loop over the list of b values like below,
l = [1,2,3,4]
for b in l:
d = a + b + c
r = requests.get(d)
json_object = json.loads('{"data":%s}}' % (response.content.decode("utf-8").replace("jQuery111002521088376353553_1491736907010(", "")[:-2].replace("\'", "")))
for game in json_object["data"]["docs"]:
print ("Name: %s (Price: %s)" % (game["name"], game["minPrice"]))
Related
I've been using the following code to get the limits:
compute_limits = novaClient.limits.get().absolute
for s in compute_limits:
print s.name + " = " + str(s.value)
However I only want specific values from the limits.get(), namely totalRAMUsed and maxTotalRAMSize. There seems to be very little on the internet about using the Python API (mainly all about the CLI). Is there a way to get these specific values to avoid displaying all the limit values?
You can display only one specific value:
compute_limits = novaClient.limits.get().absolute
for s in compute_limits:
if s.name == 'totalRAMUsed':
print s.name + " = " + str(s.value)
break
compute_limits is generator, you can't receive only one specific value by limit name. But you can convert compute_limits to dict. For example:
compute_limits = novaClient.limits.get().absolute
l = list(compute_limits)
limits = dict(map(lambda x: (x.name, x.value), l))
print limits['totalRAMUsed']
I would heavily need your help to simplify a code that allows me to have a data analysis on salesforce leads.
I have the following dataframes, which as you can see are split due to python limit on managing more than 550 objects in a single list.
Iterlist = list()
for x in range(0, int(len(List)/550)+1):
m = List[550*x: 550*x+550]
Iterlist.append(m)
Iterlist0= pd.DataFrame(Iterlist[0])
Iterlist1= pd.DataFrame(Iterlist[1])
Iterlist2= pd.DataFrame(Iterlist[2])
...and so on until the initial longer list is split
...
converted in the following lists for formatting reasons:
A= Iterlist0["Id"].tolist()
mylistA = "".join(str(x)+ "','" for x in A)
mylistA = mylistA[:-2]
mylistA0 = "('" + mylistA + ")"
B = Iterlist1["Id"].tolist()
mylistB = "".join(str(x)+ "','" for x in B)
mylistB = mylistB[:-2]
mylistB1 = "('" + mylistB + ")"
C = Iterlist2["Id"].tolist()
mylistC = "".join(str(x)+ "','" for x in C)
mylistC = mylistC[:-2]
mylistC2 = "('" + mylistC + ")"
and so on...
...
I want to create a loop that allows me to query from salesforce each of the lists using the following code template for example:
queryA='SELECT '+cols[1]+', '+cols[2]+', '+cols[3]+', '+cols[4]+', '+cols[5]+', '+cols[6]+', '+cols[7]+', '+cols[8]+' FROM LeadHistory WHERE LeadId IN '+mylistA0
and then finally:
sf = Salesforce(username='xxx', password='xxx', security_token='xxx')
leadhistory = sf.query_all(queryA)
I don´t want to write over and over numerous dataframes with specific names, lists and queries in order to get to the result. I would like to have a line for each of the codes written above, and let python automatically update the naming according to the number of 550 elements list.
I am new to this programming language and any tip would help me a lot. I think is possible to simplify it a lot but no idea how can be done.
Thanks in advance!
I am trying to use JSON to search through googlemapapi. So, I give location "Plymouth" - in googlemapapi it is showing 6 resultset but when I try to parse in Json, I am getting length of only 2. I tried with multiple cities too, but all I am getting is resultset of 2 rather.
What is wrong below?
import urllib.request as UR
import urllib.parse as URP
import json
url = "http://maps.googleapis.com/maps/api/geocode/json?address=Plymouth&sensor=false"
uh = UR.urlopen(url)
data = uh.read()
count = 0
js1 = json.loads(data.decode('utf-8') )
print ("Length: ", len(js1))
for result in js1:
location = js1["results"][count]["formatted_address"]
lat = js1["results"][count]["geometry"]["location"]["lat"]
lng = js1["results"][count]["geometry"]["location"]["lng"]
count = count + 1
print ('lat',lat,'lng',lng)
print (location)
Simply replace for result in js1: with for result in js1['results']:
By the way, as posted in a comment in the question, no need to use a counter. You can rewrite your for loop as:
for result in js1['results']:
location = result["formatted_address"]
lat = result["geometry"]["location"]["lat"]
lng = result["geometry"]["location"]["lng"]
print('lat',lat,'lng',lng)
print(location)
If you look at the json that comes in, you'll see that its a single dict with two items ("results" and "status"). Add print('result:', result) to the top of your for loop and it will print result: status and result: results because all you are iterating the the keys of that outer dict. That's a general debugging trick in python... if you aren't getting the stuff you want, put in a print statement to see what you got.
The results (not surprisingly) and in a list under js1["results"]. In your for loop, you ignore the variable you are iterating and go back to the original js1 for its data. This is unnecessary and in your case, it hid the error. Had you tried to reference cities off of result you would gotten an error and it may have been easier to see that result was "status", not the array you were after.
Now a few tweaks fix the problem
import urllib.request as UR
import urllib.parse as URP
import json
url = "http://maps.googleapis.com/maps/api/geocode/json?address=Plymouth&sensor=false"
uh = UR.urlopen(url)
data = uh.read()
count = 0
js1 = json.loads(data.decode('utf-8') )
print ("Length: ", len(js1))
for result in js1["results"]:
location = result["formatted_address"]
lat = result["geometry"]["location"]["lat"]
lng = result["geometry"]["location"]["lng"]
count = count + 1
print ('lat',lat,'lng',lng)
print (location)
I've only started python recently but am stuck on a problem.
# function that tells how to read the urls and how to process the data the
# way I need it.
def htmlreader(i):
# makes variable websites because it is used in a loop.
pricedata = urllib2.urlopen(
"http://website.com/" + (",".join(priceids.split(",")[i:i + 200]))).read()
# here my information processing begins but that is fine.
pricewebstring = pricedata.split("},{")
# results in [[1234,2345,3456],[3456,4567,5678]] for example.
array1 = [re.findall(r"\d+", a) for a in pricewebstring]
# writes obtained array to my text file
itemtxt2.write(str(array1) + '\n')
i = 0
while i <= totalitemnumber:
htmlreader(i)
i = i + 200
See the comments in the script as well.
This is in a loop and will each time give me an array (defined by array1).
Because I print this to a txt file it results in a txt file with separate arrays.
I need one big array so it needs to merge the results of htmlreader(i).
So my output is something like:
[[1234,2345,3456],[3456,4567,5678]]
[[6789,4567,2345],[3565,1234,2345]]
But I want:
[[1234,2345,3456],[3456,4567,5678],[6789,4567,2345],[3565,1234,2345]]
Any ideas how I can approach this?
Since you want to gather all the elements in a single list, you can simply gather them in another list, by flattening it like this
def htmlreader(i, result):
...
result.extend([re.findall(r"\d+", a) for a in pricewebstring])
i, result = 0, []
while i <= totalitemnumber:
htmlreader(i, result)
i = i + 200
itemtxt2.write(str(result) + '\n')
In this case, the result created by re.findall (a list) is added to the result list. Finally, you are writing the entire list as a whole to the file.
If the above shown method is confusing, then change it like this
def htmlreader(i):
...
return [re.findall(r"\d+", a) for a in pricewebstring]
i, result = 0, []
while i <= totalitemnumber:
result.extend(htmlreader(i))
i = i + 200
I have some code that is using the 'gdshortener' module to produce a shortened version of my source URLS:
import simplejson
import httplib2
import twitter
import gdshortener
from random import randint
print("Python will now attempt to submit tweets to twitter...")
try:
api = twitter.Api(consumer_key='',
consumer_secret='',
access_token_key='',
access_token_secret='')
b = 0
for a in range(0, 1): #only range 0-1 for this question, actually 1-21
b = b + 1
a = randint(1,60000000)
randint
print ("a = ", a)
aa = str(a)
s1 = gdshortener.ISGDShortener()
print s1.shorten(url = 'http://audiotechracy.blogspot.co.uk/2014/03/reviewing-synapse-antidote-rack.html', custom_url = aa)
ss1 = str(s1)
status = api.PostUpdate("The new Synapse Antidote Rack Extension:" + ss1 + " #propellerhead #synapse")
print("Tweets submitted successfully!")
except Exception,e:
print str(e)
print("Twitter submissions have failed!!!")
I am using a random number generator to produce six digit numbers which are then fed to the custom_url parameter of this module. That works fine and I get a series of pseudo-random numbers. However when I try and concatenate my tweet string, my dynamic short URL and some hashtags I get an error that I cannot concatenate string and integer values.
So I have then created the variable 'ss1' which is the string of 's1', however this now produces a tweet like this:
The new Synapse Antidote Rack Extension:<gdshortener.gdshortener.ISGDShortener object at 0x000000000542AA20> #propellerhead #synapse
How can i get it so that the tweet produced is:
The new Synapse Antidote Rack Extension: http://is.gd/58077181 #propellerhead #synapse
Thanks
Checked the module and discovered that it returns a tuple. See the following to extract the proper URL.
Code:
import gdshortener
s1 = gdshortener.ISGDShortener()
x1 = s1.shorten(url='http://audiotechracy.blogspot.co.uk/2014/03/reviewing-synapse-antidote-rack.html')[0]
print x1
Result:
http://is.gd/KKxmFd
[Finished in 0.8s]
Notice how I added a [0] at the end of the shorten. This is because shorten returns a tuple, which we can index similar to a list.
Hope this helps.