We Keep Coding

Download file from GitLab using Python HTTP requests

I'd like to download a YAML file from Gitlab using Python requests. I'm almost there, but I havent quite got to the cigar stage.
I am doing the following :-
GITLAB_FILE ="https://my_url/api/v4/projects/my_id/repository/files/path_and_filename.yaml/raw?ref=master&private_token=mytoken"
g=requests.request("GET", GITLAB_FILE, verify=False)
print(g.json())
Now, it all works in as much as I can get to the file ok, but when it comes to accessing the data the print(g.json()) throws an error, but then continues to print out the file contents as I'd hoped. The error is :-
Traceback (most recent call last):
File "/home/myproj/edm/lib/python3.7/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./http_test.py", line 73, in <module>
print (g.json())
File "/home/myproj/edm/lib/python3.7/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value]
Once this is printed, it then proceeds to print out the contents of my yaml file correctly.
I suspect its something to do with the
print(g.json())
expecting a json format, and encountering a yaml file?
Any pointers as to how I can get the file contents error free would be helpful.

I've answered my own question.
Quite simply instead of
print(g.json())
I used
print(g.text)
Which did exactly what it said on the tin.

Problem with loading a json file for geo_data in python

I am currently trying to use folium library in python to create webmaps. I have a file world.json which contains geo_data. I have provided a link to the file at the end of this post. I tried the following code:
data = [json.loads(line) for line in open('world.json', 'r')]
and received the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <listcomp>
File "C:\Users\name\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\Users\name\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\name\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
How can I load this file?
What I want to achieve is essentially obtain the population data and create a Choropleth and overlay it on my webmap.
Edit: Forgot the link:
https://1drv.ms/u/s!Army95vqcKXpaooVAZU_g-VCAVw?e=vwTknq
Edit: Previous link to skydrive stopped working due to "high traffic". Below is link to dropbox, hopefully this works:
https://www.dropbox.com/s/gmm8db0g03rc7cv/world.json?dl=0

Good news/bad news:
It turns out that this file was encoded in a locale that we are not accustomed to, and json/ascii cannot make sense of some of the character encoding. I tried this, and it seems to be working for me -- with a major caveat:
with open("world.json", "r") as fh:
contents = fh.read()
asciiContents = contents.encode("ascii", errors="ignore")
data = json.loads(asciiContents)
The major caveat is that only 3 countries come through with no encoding errors:
>>> len(data["features"])
3
Maybe there another source for this data that is closer to a native english locale, or maybe someone else can provide wisdom in encoding foreign data in a more friendly way...

The open command will return a file handle, not string lines. I would do:
with open('world.json', 'r') as fh:
data = json.load(fh)
data will then be your contents converted to python (list or dictionary, etc)

Character Encoding error with Groupme bot using Groupy library

I am trying to get started writing a Groupme bot using Groupy.
There have been a few related posts to this one, but they seemed a bit more technically challenging of issues, the solutions didn't seem efficient, and none of them solved the issues I'm having.
Here is the code I have so far:
import groupy
token = "{My Token}"
from groupy.client import Client
client = Client.from_token(token)
groups = list(client.groups.list_all())
for group in groups:
print(group)
Firstly, I don't believe this is related but, in eclipse it is giving a warning that groupy is an unused import, even though it seems to me that that is not so.
More importantly, the strange part about the issues I'm having is that the error messages change without any edits to the above code.
Here is an error message that I've gotten:
In each of these error messages, I've edited the beginning of the file locations... because I'm paranoid
Traceback (most recent call last):
File "\Projects\Eclipse\eclipse-workspace\Gort's Cousin\groupme_bot.py", line 15, in <module>
<Group(name='One of my group's names')>
<Group(name='Another one of my group's names')>
print(group.name)
File "\Python\Python37-32\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 27-28: character maps to <undefined>
So, this time the system actually returns a couple groups I am in. But then calls an exception because of a unicode error of sorts.
Without changing any code, I also get this error:
Traceback (most recent call last):
File "\Python\Python37-32\lib\site-packages\groupy\session.py", line 49, in data
return self.json()['response']
File "\Python\Python37-32\lib\site-packages\requests\models.py", line 808, in json
return complexjson.loads(self.text, **kwargs)
File "\Python\Python37-32\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "\Python\Python37-32\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 60865 (char 60864)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "\Eclipse\eclipse-workspace\Gort's Cousin\groupme_bot.py", line 10, in <module>
groups = list(client.groups.list_all())
File "\Python\Python37-32\lib\site-packages\groupy\pagers.py", line 61, in autopage
self.items = self.fetch_next()
File "\Python\Python37-32\lib\site-packages\groupy\pagers.py", line 51, in fetch_next
return self.fetch()
File "\Python\Python37-32\lib\site-packages\groupy\pagers.py", line 42, in fetch
return self.endpoint(**self.params)
File "\Python\Python37-32\lib\site-packages\groupy\api\groups.py", line 21, in _raw_list
return [Group(self, **group) for group in response.data]
File "\Python\Python37-32\lib\site-packages\groupy\session.py", line 51, in data
raise exceptions.InvalidJsonError(self._resp) from e
groupy.exceptions.InvalidJsonError: The JSON was incomplete/invalid
I am most likely incorrect, but it seems to me that there is some issue with the way groupy is aggregating the groups, and/or how I've set up my project to handle those calls.
Any help with this issue would be awesome.
I can also add some code from the groupy library if that becomes necessary.
TLDR: Getting a "UnicodeEncodeError" error using Groupy to write a groupme bot in Python

Solved:
It was an issue where group names with emojis were not playing nice with the Cp1252 encoding I have as a global setting in eclipse. I changed the run configuration of the file to encode to UTF-16BE. The emojis don't print correctly to the console, but all of the groups are printed with no errors.

Passing json text as command line argument

I am trying to pass the following JSON text into my python code.
{"platform": "android", "version": "6.0.1"}
My code is as follows.
import sys
import json
data = json.loads(sys.argv[1])
print(str(data))
When running the following on Windows 10 PowerShell,
python jsonTest.py '{"platform": "android", "version": "6.0.1"}'
I get the following:
Traceback (most recent call last):
File "jsonTest.py", line 3, in <module>
data = json.loads(sys.argv[1])
File "C:\Users\Rishabh Bhatnagar\AppData\Local\Programs\Python\Python36-
32\lib\json\__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Users\Rishabh Bhatnagar\AppData\Local\Programs\Python\Python36-
32\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Rishabh Bhatnagar\AppData\Local\Programs\Python\Python36-
32\lib\json\decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double
quotes: line 1 column 2 (char 1)
As far as I know, I take my code, and pass the JSON text properly. I can't figure out what I'm doing wrong. I know the JSON text is valid (checked with https://jsonlint.com/). Thanks.

So I figured it out.
sys.argv[1]
The above line was taking my Json text below and taking out the quotes from it.
{"platform": "android", "version": "6.0.1"}
into
{platform: android, version: 6.0.1}
My workaround is to run it as follows.
Python jsonTest.py '{\"platform\": \"android\", \"version\": \"6.0.1\"}'
I will try to find a better way, but for today, I'm done.

import sys
import json
data = json.loads(sys.argv[1].replace("'", '"'))
print(str(data))
This seems to work for me, python 3.6 when calling with python jsonTest.py "{'platform': 'android', 'version': '6.0.1'}"

Trouble parsing JSON object in Python

I am trying to parse some text files containing JSON objects in Python using the json.load() method. It's working for one set of them, but for this one it will not:
{
"mapinfolist":{
"mapinfo":[
{"sku":"00028-0059","price":"38.35","percent":"50","basepercent":"50","exact":0,"match":0,"roundup":0}
,{"sku":"77826-7230","price":"4.18","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-1310","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-2020","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-3360","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-4060","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-4510","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-7230","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
],
"count":2
}
}
It is in a file called 'map.txt' - I open it using open('map.txt') and then call json.load(). When I run my test program (test.py), the following error trace is generated:
Traceback (most recent call last):
File "test.py", line 28, in <module>
main()
File "test.py", line 23, in main
map_list = json.load(f1)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
The JSON object is valid - when I put it into https://www.jsoneditoronline.org/ it is parsed and displayed correctly, so I am having trouble identifying what could be stopping it from working when I try to do it in Python. Any advice would be much appreciated. Thanks!
EDIT: Here's my code.
import json
def main():
with open('map.txt') as f1:
map_list = json.load(f1)
Trying map_list = json.loads(f1.read()) also does not work and gives me an almost identical error trace.
EDIT - RESOLVED:
I just copied and pasted FROM map.txt into a new TextEdit file map2.txt and used the new file instead, and it works now. I copied directly from the old file and made no changes - the only difference is that it is a different file. I can't make heads or tails of why that would be - any ideas? I would like to understand what may have happened so I can avoid the problem in the future.

Does the following solution work for you?
import json
f = open("map.txt")
map = json.loads(f.read())
Python Docs

maybe try to read all the file to string and then use json.loads
def yourfunc():
file = open('map.txt')
json_string = file.read()
map = json.loads(json_string)