We Keep Coding

Post to Google Analytics, works from Node, fails from Python.... why?

I'm trying to post events to Google Analytics. It works fine when I do it using the NodeJS code below, but fails when I use the Python code below. Both do return a HTTP 200 and even when posting to the debug URL (https://www.google-analytics.com/debug/collect) Google Analytics returns success details in both cases (see valid: true in the response below). The problem is that when posting from NodeJS the result shows up in the GA website, when posting from Python it never shows up. I did compare the requests for both and have not been able to spot a difference.
{
"hitParsingResult": [ {
"valid": true,
"parserMessage": [ ],
"hit": "/debug/collect?v=1\u0026t=event\u0026tid=XXXXXXX\u0026cid=YYYYYYu0026ec=Slack\u0026ea=SlashCommand\u0026el=whowasat-curl\u0026an=staging.Whereis-Everybody?\u0026aid=staging.whereis-everybody.com"
} ],
"parserMessage": [ {
"messageType": "INFO",
"description": "Found 1 hit in the request."
} ]
}
The NodeJS code is (result does show up in Google Analytics):
'use strict';
var request = require('request');
require('request-debug')(request);
function postEventToGA(category, action, label) {
var options = {
v: '1',
t: 'event',
tid: process.env.GOOGLEANALYTICS_TID,
cid: process.env.GOOGLEANALYTICS_CID,
ec: category,
ea: action,
el: label,
an: process.env.STAGE_INFIX + "appname",
aid: process.env.STAGE_INFIX + "appname"
};
console.log("payload: " + JSON.stringify(options))
request.post({ url: 'https://www.google-analytics.com/collect', form: options }, function (err, response, body) {
console.log(request)
if (err) {
console.log("Failed to post event to Google Analytics, error: " + err);
} else {
if (200 != response.statusCode) {
console.log("Failed to post event to Google Analytics, response code: " + response.statusCode + " error: " + err);
}
}
});
}
postEventToGA("some-category", "some-action", "some-label")
And the Python code is (result does not show up in Google Analytics):
import json
import logging
import os
import requests
LOGGER = logging.getLogger()
LOGGER.setLevel(logging.INFO)
GOOGLEANALYTICS_TID = os.environ["GOOGLEANALYTICS_TID"]
GOOGLEANALYTICS_CID = os.environ["GOOGLEANALYTICS_CID"]
STAGE_INFIX = os.environ["STAGE_INFIX"]
def post_event(category, action, label):
payload = {
"v": "1",
"t": "event",
"tid": GOOGLEANALYTICS_TID,
"cid": GOOGLEANALYTICS_CID,
"ec": category,
"ea": action,
"el": label,
"an": STAGE_INFIX + "appname,
"aid": STAGE_INFIX + "appname",
}
response = requests.post("https://www.google-analytics.com/collect", payload)
print(response.request.method)
print(response.request.path_url)
print(response.request.url)
print(response.request.body)
print(response.request.headers)
print(response.status_code)
print(response.text)
if response.status_code != 200:
LOGGER.warning(
"Got non 200 response code (%s) while posting to GA.", response.status_code
)
post_event("some-category", "some-action", "some-label")
Any idea why the NodeJS post will show up in Google Analytics and the Python post does not?
(while both return a HTTP200)

Did some more testing and discovered that the user agent HTTP header was causing the problem. When I set it to an empty string in the Python code it works. Like this:
headers = {"User-Agent": ""}
response = requests.post(
"https://www.google-analytics.com/collect", payload, headers=headers
)
The documentation at https://developers.google.com/analytics/devguides/collection/protocol/v1/reference does state that the user agent is used, but does not clearly state what the requirements are. "python-requests/2.22.0" (default value by python-requests lib) is apparently not accepted.

How do a make a API request in Swift like the working Python request I'm currently using?

I'm looking to convert this Python request to a Swift script.
Here is my working python script that returns the accessToken!
#!/usr/bin/python
import requests
import json
#MAKE THE REQUEST
URL = "http://this/is/the/url"
headers = {
'Accept': "application/json",
"Accept-Language": "en_US"
}
data = {
"grant_type": "password",
"username" : "GROUP\SITE\USERNAME",
"password" : "somepassword"
}
r = requests.get(url = URL, params = headers, data = data)
data = r.json()
accessToken = data['access_token']
print(accessToken)
When I run the Swift Playground for the code below nothing is returned!
It seems the script exits at guard let data = data else { return }
How could I get the same results as the Python Script above.
I've tried implementing URLComponents using this tutorial...
import UIKit
var url = "http://just/the/url"
extension Dictionary {
func percentEncoded() -> Data? {
return map { key, value in
let escapedKey = "\(key)"
let escapedValue = "\(value)"
print(escapedKey + "=" + escapedValue)
return escapedKey + "=" + escapedValue
}
.joined(separator: "&")
.data(using: .utf8)
}
}
extension CharacterSet {
static let urlQueryValueAllowed: CharacterSet = {
let generalDelimitersToEncode = ":#[]#" // does not include "?" or "/" due to RFC 3986 - Section 3.4
let subDelimitersToEncode = "$&'()*+,;="
var allowed = CharacterSet.urlQueryAllowed
allowed.remove(charactersIn: "\(generalDelimitersToEncode)\(subDelimitersToEncode)")
return allowed
}()
}
var request = URLRequest(url: URL(string:url)!)
request.httpMethod = "GET"
let parameters: [String: String] = [
"grant_type":"password",
"username":"GROUP\\SITE\\USER",
"password":"somePassword"
]
request.httpBody = parameters.percentEncoded()
request.setValue("application/x-www-form-urlencoded", forHTTPHeaderField: "Content-Type")
request.setValue("application/XML", forHTTPHeaderField: "Accept")
let config = URLSessionConfiguration.default
URLSession(configuration: config).dataTask(with: request) { (data, response, err) in
guard let data = data else { return }
print(data)
guard let dataAsString = String(data: data, encoding: .utf8)else {return}
print(dataAsString)
guard let httpResponse = response as? HTTPURLResponse,
(200...299).contains(httpResponse.statusCode) else {
print("Bad Credentials")
return
}
//HTTP Status Code!
print("HTTP RESPONSE:"+"\(httpResponse.statusCode)")
//
}.resume()

If I remember correctly, starting in iOS 13, you cant have httpBody for a GET call, so you'll either need to switch to a POST/PUT or add the params into the url string (See below)
You also had different Accept headers in your python vs. swift. One was xml the other was json.
var urlComponents = URLComponents(string: "http://this/is/the/url")
urlComponents?.queryItems = [
URLQueryItem(name: "grant_type", value: "password"),
URLQueryItem(name: "username", value: "username"),
URLQueryItem(name: "password", value: "somepassword")
]
guard let url = urlComponents?.url else { return } // You can print url here to see how it looks
var request = URLRequest(url: url)
request.httpMethod = "GET"
request.setValue("application/json", forHTTPHeaderField: "Accept")
request.setValue("en_US", forHTTPHeaderField: "Accept-Language")
let task = URLSession.shared.dataTask(with: request) { data, response, error in
guard let data = data,
let response = response as? HTTPURLResponse,
error == nil else {
print("error", error ?? "Unknown error")
return
}
print(response)
guard (200 ... 299) ~= response.statusCode else {
print("response = \(response)")
return
}
let responseString = String(data: data, encoding: .utf8)
print(responseString)
}
task.resume()

The problem was the following...
request.httpMethod = "GET"
I had to change the get to "POST" and now I have the token!!!!
I was confused because the python script used GET. I had a bash script that that used curl to get the token displayed the logged post.
In short my above Swift Playground now works by changing the request.httpMethod to "POST". THANKS FOR ALL THE HELP

Python Hug REST API consumed in .NET, JSON looks weird

When consuming a Hug REST endpoint from .net JSON has embedded characters. A complete failing example posted below. Any help greatly appreciated.
Python
#hug.post('/test')
def test(response, body=None):
input = body.get('input')
print('INSIDE TEST ' + input)
if input:
dict = {"lastname":"Jordan"}
dict["firstname"] = input
return json.dumps(dict, sort_keys=True, default=str)
.NET (can only use .net 3.5)
private static object GetParsedData(string data)
{
var posturl = "http://localhost:8000/test";
try
{
using (var client = new WebClient())
{
// upload values is the POST verb
var values = new NameValueCollection()
{
{ "input", data },
};
var response = client.UploadValues(posturl, values);
var responseString = Encoding.UTF8.GetString(response);
var settings = new JsonSerializerSettings
{
NullValueHandling = NullValueHandling.Ignore,
MissingMemberHandling = MissingMemberHandling.Ignore
};
JObject rss = JObject.Parse(responseString);
Console.WriteLine((string)rss["lastname"]);
}
}
catch (WebException ex)
{
if (ex.Response is HttpWebResponse)
{
var code = ((HttpWebResponse)ex.Response).StatusCode;
var desc = ((HttpWebResponse)ex.Response).StatusDescription;
}
//_logger.Error(ex.Message);
}
return false;
}
responseString looks like this:
"\"{\\\"firstname\\\": \\\"Mike\\\", \\\"lastname\\\": \\\"Jordan\\\"}\""
JObject.Parse throws error:
Newtonsoft.Json.JsonReaderException:
'Error reading JObject from JsonReader. Current JsonReader item is not an object: String. Path '', line 1, position 53.
Workaround - If I do something horrible like this to responseString JObject parses correctly:
str = str.Replace("\\", "");
str = str.Substring(1, len - 2);
Whats going on?

The default hug output format is json; it is not necessary to call json.dumps on return values, hug will do this automatically.

Post GraphQL mutation with Python Requests

I'm having trouble posting mutations with GraphQL and Python Requests.
My function looks like:
def create(request):
access_token = 'REDACTED'
headers = {
"X-Shopify-Storefront-Access-Token": access_token
}
mutation = """
{
checkoutCreate(input: {
lineItems: [{ variantId: "Z2lkOi8vc2hvcGlmeS9Qcm9kdWN0VmFyaWFudC80", quantity: 1 }]
}) {
checkout {
id
webUrl
lineItems(first: 5) {
edges {
node {
title
quantity
}
}
}
}
}
}
"""
data = (requests.post('https://catsinuniform.myshopify.com/api/graphql', json={'mutation': mutation}, headers=headers).json())
print(data)
return render(request, 'Stock/create.html', { 'create': data })
I'm getting errors saying I have a bad request "bad_request - Parameter Missing or Invalid" in my json response.

Even though you're sending a mutation, your request body should still include a query property, the value of which should be the string representing your operation. It's a bit confusing, but informally both queries and mutations are called "queries" (you're still "querying" the server either way). Change your request to:
requests.post('https://catsinuniform.myshopify.com/api/graphql', json={'query': mutation}, headers=headers)

In AngualrJS using $http, how to get the value of a variable set in Python CGI?

I'm making a POST request from AngularJS to Python.
I started with an JavaScript example. It works properly returning all the values.
However, when I try to do it from AngularJS I'm not able to read the value of the variable posted.
JAVASCRIP EXAMPLE THAT WORKS PROPERLY (I'm able to get the value (Mike) back of Name):
JS code
<script language="Javascript">
function asyncChange()
{
var request;
if (window.XMLHttpRequest) {
request = new window.XMLHttpRequest();
} else {
// Versiones antiguas de Internet Explorer.
request = new window.ActiveXObject("Microsoft.XMLHTTP");
}
request.open("POST","nctest.py" , true);
request.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
request.send("Name=Mike");
request.onreadystatechange = function()
{
if (request.readyState == 4 && request.status == 200)
{
document.getElementById("myLabel").innerHTML = "Hello " + request.responseText + "!";
}
}
}
</script>
nctest.py
#!/usr/bin/python
import cgi
input = cgi.FieldStorage()
print "Content-type: text/html\n\n"
print "input[Pe].value: "
print input["Pe"].value
ANGULARJS DOESN'T WORK PROPERLY (I'm not able to get the value (Mike) back of Name):
Angularjs code:
(function(){
'use strict'
var sectest= {
controller:sectestCtrl,
templateUrl:'app/components/component_test/test.html',
}
angular
.module('myapp')
.component('secTest',sectest);
function sectestCtrl($http){
var prac= this;
prac.method = 'POST';
prac.url = 'nctest.py';
prac.data = {Name : 'Mike'};
prac.data_answer
prac.headers = { 'Content-Type': 'application/x-www-form-urlencoded' };
prac.sendHTML = send;
function send(){
prac.code = null;
prac.response = null;
$http({method: prac.method, headers: prac.headers, url: prac.url, data: $.param(prac.data)}).
then(function(response) {
prac.status = response.status;
prac.data_answer = response.data;
console.log("OK prac.data_answer: ", prac.data_answer)
}, function(response) {
prac.data_answer = response.data || 'Request failed';
prac.status = response.status;
});
};
}
})();
nctest.py code
#!/usr/bin/python
import json
import cgi
input = cgi.FieldStorage()
print "Content-type: text/html\n\n"
print input["Name"].value
The problem is that prac.data_answer prints blank value.
I have already try with different headers for both angularjs and python codes but none seems to work:
prac.headers = { 'Content-Type': 'application/json' };
prac.headers = { 'Content-Type': 'application/x-www-form-urlencoded' };
prac.headers = { 'Content-Type': 'text/html\n\n' };
Many thanks.

There are 2 separate issues you're trying to address. Server (CGI) & client(angularjs). First check to see that you are receiving the data over the network - using Chrome developer tools, under the Network tab. If so, there's no need to change the Content-Type to json, since angular by default assumes all http data is in json format.
I don't think you need all those attributes for a post request. Seems like an overkiller when it can be simpler. Try this:
$http.post(url, data).then(function(response){
console.log(response.data);
});