We Keep Coding

Scraping values from a request response

Unfortunately i can provide only the output of the request and not the full code since it contains quite private infos, basically when printing the request as text file i get a json one, something like that:
{"paymentResource":{"paymentToken":"PAYID-MEJ------","intent":"authorize","redirectUrl":"https://www.paypal.com/checkoutnow?nolegacy=1\u0026token=EC-5JS-----2S","authenticateUrl":null}}
How can i scrape that Paypal url? tried by doing this but it didn't worked (ppstep2 is the name of the request):
content = ppstep2.json()
pp = content["redirectUrl"]
I only get this error while doing it:
pp = content["redirectUrl"]
KeyError: 'redirectUrl'

Your variable content is a dictionary.
To get the value for "redirectUrl" you can do this:
pp = content['paymentResource']['redirectUrl']
The key error was caused by not including ['paymentResource']
I would recommend reviewing python dictionaries and the .get() method as well.
https://docs.python.org/3/library/stdtypes.html?highlight=dictionary%20get#dict.get

Try adding a print(json.dumps(content, indent=4)) before you try to access it and look at the output. You might spot why then.
redirectUrl isn't part of content. It's in the content['paymentResource'] dictionary in that response content.
using content['paymentResource']['redirectUrl'] should work.
Edit: If you want to try and get a value without ending up with an exception, try using .get():
# This will result in a KeyError as you experienced:
pp = content["redirectUrl"]
# This will instead set pp to None if 'redirectUrl' doesn't exist as a Key
pp = content.get("redirectUrl", None)

Python Zeep - List Object has no Attribute _xsd_name

I'm trying to call a SOAP endpoint using the Zeep library.
The endpoint has the following signature:
ExecuteStaticSQLQuery(queryType: ns0:StaticQueries, xmlParameters: {_value_1: ANY}, iRowsAffected: xsd:int) -> ExecuteStaticSQLQueryResult: xsd:int, iRowsAffected: xsd:int
I built a function that contains the following logic:
INSERT_AUTO_FORWARD_IMPORT = 'INSERT_AUTO_FORWARD_IMPORT_1'
query_parameter = self.client.get_type('ns0:QueryParameter')
db_type = self.client.get_type('ns0:DbType')
table_parameter = query_parameter(ParameterType=db_type('String'),
ParameterName='#Auto_Forward_Table_Name',
ParameterValue=auto_forward_table)
any_object = AnyObject(query_parameter, table_parameter)
xml_parameters = {
'_value_1': any_object
}
soap_envelope = self.service.ExecuteStaticSQLQuery(queryType=INSERT_AUTO_FORWARD_IMPORT,
xmlParameters=xml_parameters,
iRowsAffected=rows_affected)
When I call the function I get the following error:
'list' object has no attribute '_xsd_name'
I suspect the issue is coming from the any_object I build, because I can send a request to the endpoint without the xmlParameters argument and I get a response back.
Based on the Zeep Any Objects Docs, I would think I'm correctly constructing the AnyObject and implementing the call to the SOAP endpoint.
I'm not really sure where to go from here.

I'm not sure you are correctly creating the AnyObject.
According to the Zeep docs you link to, you should be calling get_element, not get_type, to construct the AnyObject.
In other words, try replacing the line
query_parameter = self.client.get_type('ns0:QueryParameter')
with
query_parameter = self.client.get_element('ns0:QueryParameter')

rets-python with flask cannot return as json

I am doing a project which has to do with rets, somehow my collaborator has problem using rets-client for js during installation.
Anyways, so I decided to find python instead which I found rets-python from https://github.com/opendoor-labs/rets
I am able to get data returned from rets, somehow I wanted to make an api so I installed flask (new with flask)
However I tried to return the data into JSON I just kept on getting errors
I this is my code
import json
from flask import Flask, Response, jsonify
app = Flask(__name__)
#app.route('/')
def mls():
from rets.client import RetsClient
client = RetsClient(
login_url='xxxxxxxxxxxx',
username='xxxxxxxxxxxx',
password='xxxxxxxxxxxxx',
# Ensure that you are using the right auth_type for this particular MLS
# auth_type='basic',
# Alternatively authenticate using user agent password
user_agent='xxxxxxxxxxxxxxxx',
user_agent_password='xxxxxxxxxxxxxxxx'
)
resource = client.get_resource('Property')
print(resource)
resource.key_field
print(resource.key_field)
resource_class = resource.get_class('RD_1')
print(resource_class)
search_result = resource_class.search(query='(L_Status=1_0)', limit=2)
I will write my returns here as I have tried few different returns
# return jsonify(results=search_result.data)
# TypeError: <Record: Property:RD_1:259957852> is not JSON serializable
return Response(json.dumps(search_result.data), mimetype='application/json')
# TypeError: <Record: Property:RD_1:259957852> is not JSON serializable
return json.dumps(search_result.data)
# TypeError: <Record: Property:RD_1:259957852> is not JSON serializable
I event tried making the results into dict such as dict(search_result.data) which gives me errors like TypeError: cannot convert dictionary update sequence element #0 to a sequence
if I try to get the print(type(search_result.data[0].data)) this is the return <class 'collections.OrderedDict'> I tried to jsonify, json.dump and also with Response with just this one record with or without dict() I get this error TypeError: Decimal('1152') is not JSON serializable
also print(type(search_result.data)) gives tuple
Anyone able to give me a hand on this?
Thanks in advance for any help and advises.

JSON looking string (or "\" in output) of Flask Restful/Pymongo JSON output

First, reference links to other questions I read through. They might be clues to what I am experiencing, although I am not understanding enough yet to see the solution to my problem.
How can I use Python to transform MongoDB's bsondump into JSON?
Unable to deserialize PyMongo ObjectId from JSON
I've got a Flask Restful API I'm working on. An excerpt of my code is as follows:
class DeviceAPI(Resource):
def get(self, deviceID):
# do the query
deviceCollection = db['device']
device = deviceCollection.find_one({'deviceID': deviceID})
print device #1
print ''
print json_util.dumps(device) #2
print ''
s = json_util.dumps(device)
print s #3
print ''
results = {}
results['device'] = s
print results #4
# respond
return results #5
At Print #1, I get the following, and I understand and expect this.
{u'deviceID': u'ABC123', u'_id': ObjectId('....')}
At Print #2 and #3 are identical outputs as expected, and again I understand and expect this (I think).
{"deviceID": "ABC123", "_id": {"$oid": "...."}}
Print #4 has an added key in the dictionary. However, it looks like the value of the key:value is a string where it should be the dictionary as in #2 and #3.
{'device': '{"deviceID": "ABC123", "_id": {"$oid": "...."}}'}
The returned result, #5, according to CURL is along the lines of the following. There are the added / in the there. I suspect because of #4 value looking like a string and that continues in #5 as well.
{"device": "{\"deviceID\": \"ABC123\", \"_id\": {\"$oid\": \"....\"}}"}
I'm trying to get a pure JSON output, not a string representation of the device document. #2 and #3 looked like JSON, but in #4 became a string. Why? And how to do this correctly?

I believe it's because json_utils.dumps is converting your device variable into a string when you should be just returning a complete json object. You essentially end up returning something that resembles this:
return {"device": "a string that resembles json"}
Instead, modify your code to look like this:
class DeviceAPI(Resource):
def get(self, deviceID):
# do the query
deviceCollection = db['device']
device = deviceCollection.find_one({'deviceID': deviceID})
results = {'device': device}
return results
Now, we're returning json that looks more like this:
return {"device": {"deviceID": "ABC123", "_id": {"$oid": "...."}}}
However, it looks like the recommended way to return json in flask is to actually use the flask.jsonify method so that Flask will return a proper response object:
from flask import jsonify
class DeviceAPI(Resource):
def get(self, deviceID):
# do the query
deviceCollection = db['device']
device = deviceCollection.find_one({'deviceID': deviceID})
return jsonify(device=device)

Michael0x2a helped to clear some fog in my mind, and after more experimenting and thinking this through, the following is working for me.
results['device'] = json.loads(json_util.dumps(device))
In my experimenting I was using json_util.dumps and json_util.loads. But I didn't recognize that while json_util.dumps was converting the BSON like item to JSON string, that json_util.loads was converting it directly back. Therefore, another function was needed to take the string like JSON output of json_util.dumps and make it into a dictionary JSON like object. Thus a combo was needed. This is using "import json" and "from bson import json_util" (of Pymongo).

Thank you for your reply
I had the same problem as you, and it was fixed just because you post back the solution, thanks million! Follow below my code
from flask_restplus import Namespace, Resource, fields
from .. import mongo
import json
from bson import json_util
api = Namespace('inventory', description='Store management related operations')
#api.route('/items')
class Inventory(Resource):
def get(self):
inventory_collection = mongo.db.inventory
resp = inventory_collection.find({})
a = json.loads(json_util.dumps(resp))
return {'result': a }

jquery: post with json will actually post array

I have a python as CGI and the POST from jquery will transform json object to array, so when I see the POST from jquery, I actually see:
login_user[username]=dfdsfdsf&login_user[password]=dsfsdf
(the [ and ] already escaped)
My question is how I can convert this string back to JSON in python? Or, how can I convert this string to python array/dict structure so that I can process it easier?
[edit]
My jquery is posting:
{'login_user': {'username':username, 'password':password}}

If what you want to accomplish is to send structured data from the browser and then unpack it in your Python backend and keep the same structure, I suggest the following:
Create JavaScript objects in the browser to hold your data:
var d = {}
d['login_user'] = { 'username': 'foo', 'password': 'bar' }
Serialize to JSON, with https://github.com/douglascrockford/JSON-js
POST to your backend doing something like this:
$.post(url, {'data': encoded_json_data}, ...)
In your Python code, parse the JSON, POST in my example is where you get your POST data in your CGI script:
data = json.loads(POST['data'])
data['login_user']

import re
thestring = "login_user[username]=dfdsfdsf&login_user[password]=dafef"
pattern = re.compile(r'^login_user\[username\]=(.*)&login_user\[password\]=(.*)')
match = pattern.search(thestring)
print match.groups()
Output:
>>> ('dfdsfdsf', 'dafef')
Thus,
lp = match.groups()
print "{'login_user': {'username':"+lp[0]+", 'password':"+lp[1]+"}}"
shall bear: >>> {'login_user': {'username':dfdsfdsf, 'password':dafef}}

>>> import json
>>> data = {'login_user':{'username':'dfdsfdsf', 'password':'dsfsdf'}}
>>> json.dumps(data)
'{"login_user": {"username": "dfdsfdsf", "password": "dsfsdf"}}'
I suspect that data would already be contained in a GET var if that's coming from the URL...