I am trying to use the API from https://neweden-dev.com/ZKillboard_API in python.
My code works up to this point:
import xml.etree.ElementTree as ET
import urllib2
import gzip
import zlib
url = 'https://zkillboard.com/api/kills/characterID/95089021/xml/'
request = urllib2.Request(url)
request.add_header('Accept-encoding', 'gzip,deflate')
data = urllib2.urlopen(request)
After this point, I get very confused.
contents = data.read()
f = open("export.xml","w")
f.write(contents)
f.close()
Above was my first step. Opening the file made me realize that it was compressed (duh).
So from this point I tried two things:
f = open("test.xml.gz", "w")
f.write(data.read())
f.close()
g = gzip.open("test.xml.gz", "rb")
file_content = g.read()
g.close()
which gives me the error:
Traceback (most recent call last):
File "C:\Users\[filepath]", line 19, in <module>
file_content = g.read()
File "C:\Python27\lib\gzip.py", line 254, in read
self._read(readsize)
File "C:\Python27\lib\gzip.py", line 312, in _read
uncompress = self.decompress.decompress(buf)
error: Error -3 while decompressing: invalid distance too far back
My next step was to use zlib:
x = zlib.decompress(data.read())
which gives me the error of:
Traceback (most recent call last):
File "C:\Users\[filepath]", line 15, in <module>
x = zlib.decompress(data.read())
error: Error -3 while decompressing data: incorrect header check
I know I am missing something super simple here, any thoughts? Thanks.
Related
(I'm doing all this in python 3.10.4 using pycryptodome)
I'm trying to do this process:
Get a hash of a file
Save that hash somewhere
Load that hash and perform RSA signing using a private key
I'm having a problem in step 3 where to save the hash, I have to save it as a string which doesn't work in Step 3.
I've tried using pickle but I'm getting
"ctypes objects containing pointers cannot be pickled"
Code generating the hash:
sha256 = SHA256.new()
with open(fileDir, 'rb') as f:
while True:
data = f.read(BUF_SIZE)
if not data:
break
sha256.update(data)
Code to perform the signing:
get_file(fileName + '.hash', directory)
with open(currentDir + '/client_files/downloaded/' + fileName + '.hash', 'r') as f:
hash_data = f.read()
with open(currentDir + '/client_files/private_key.pem', 'rb') as f:
private_key = RSA.importKey(f.read())
print(private_key)
signer = PKCS1_v1_5.new(private_key)
signature = signer.sign(hash_data)
The error I'm getting:
Traceback (most recent call last):
File "c:\Users\User\Documents\Coding\VSCode Projects\practiceGround\sec_cloud_project\client\client.py", line 168, in <module>
main()
File "c:\Users\User\Documents\Coding\VSCode Projects\practiceGround\sec_cloud_project\client\client.py", line 163, in main
sign(fileName, 'worker_test_files')
File "c:\Users\User\Documents\Coding\VSCode Projects\practiceGround\sec_cloud_project\client\client.py", line 120, in sign
signature = signer.sign(hash_data)
File "C:\Users\User\anaconda3\envs\nscc_project\lib\site-packages\Crypto\Signature\pkcs1_15.py", line 77, in sign
em = _EMSA_PKCS1_V1_5_ENCODE(msg_hash, k)
File "C:\Users\User\anaconda3\envs\nscc_project\lib\site-packages\Crypto\Signature\pkcs1_15.py", line 191, in _EMSA_PKCS1_V1_5_ENCODE
digestAlgo = DerSequence([ DerObjectId(msg_hash.oid).encode() ])
AttributeError: 'str' object has no attribute 'oid'
Note that I'm currently saving the original hash as a string to a text file. If I try to use pickle to save the object as a whole I get this error
with open(currentDir + '/worker_files/sha256.pickle', 'wb') as f:
pickle.dump(sha256, f)
Traceback (most recent call last):
File "c:\Users\User\Documents\Coding\VSCode Projects\practiceGround\sec_cloud_project\worker\worker.py", line 188, in <module>
main()
File "c:\Users\User\Documents\Coding\VSCode Projects\practiceGround\sec_cloud_project\worker\worker.py", line 179, in main
hash_file(fileName, 'worker_test_files')
File "c:\Users\User\Documents\Coding\VSCode Projects\practiceGround\sec_cloud_project\worker\worker.py", line 55, in hash_file
pickle.dump(sha256, f)
ValueError: ctypes objects containing pointers cannot be pickled
Thanks to #Topaco. Changing to using Cyptography for both hashing and signing seemed to work.
Hashing with Cryptography, dumping to a file with pickle, then load and sign with Cryptography again.
I followed Kodi simple video plugin tutorial which works as expected with JSON string embedded into main.py file.
The tutorial refers that JSON string can be obtained by other means from other media (file, internet).
My attempt to read utf-8 JSON file into the string was unsuccessful so far.
Initially I tried the following approach
import json
import codecs
fname = 'c:/temp/iptv/video_data.json'
with open(fname, encoding='utf-8') as f:
data = f.read()
VIDEO = json.loads(data)
What produced the following error in log file
2020-05-09 11:11:29.327 T:19428 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
Error Type: <type 'exceptions.TypeError'>
Error Contents: 'encoding' is an invalid keyword argument for this function
Traceback (most recent call last):
File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.example\main.py", line 27, in <module>
with open(fname, encoding='utf-8') as f:
TypeError: 'encoding' is an invalid keyword argument for this function
-->End of Python script error report<--
I investigated the problem and found that Kodi 18.4 uses python27.dll and I assume that this library is somehow accessed from Kodi.
I substituted the code above with following code snippet into main.py file
import json
import codecs
fname = 'c:/temp/iptv/video_data.json'
with open(fname, 'rb') as f:
bytes = f.read()
data = bytes.decode('utf-8')
VIDEOS = json.loads(data)
On plugin's run it produces following error record in log file
2020-05-09 16:54:17.024 T:5700 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
Error Type: <type 'exceptions.UnicodeEncodeError'>
Error Contents: 'ascii' codec can't encode characters in position 1-4: ordinal not in range(128)
Traceback (most recent call last):
File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.example\main.py", line 239, in <module>
router(sys.argv[2][1:])
File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.example\main.py", line 233, in router
list_categories()
File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.animatron\main.py", line 137, in list_categories
url = get_url(action='listing', category=category)
File "C:\Users\Alex Fox\AppData\Roaming\Kodi\addons\plugin.video.animatron\main.py", line 66, in get_url
return '{0}?{1}'.format(_url, urlencode(kwargs))
File "C:\bin\Portable\Kodi\system\python\Lib\urllib.py", line 1343, in urlencode
v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-4: ordinal not in range(128)
-->End of Python script error report<--
While at the same time following code outside KODI (tested with Python 3.8.2)
import json
import codecs
fname = 'c:/temp/iptv/animation.json'
with open(fname, 'rb') as f:
bytes = f.read()
str = bytes.decode('utf-8')
VIDEOS = json.loads(str)
print(json.dumps(VIDEOS, indent=4))
for key in VIDEOS.keys():
print(key)
outputs JSON dump and VIDEO dictionary keys properly.
What is a proper way to read utf-8 encoded JSON file/internet into a string in Kodi?
Solution for obtaining data from web server:
Download and install from zip file script.modules.requtests-2.22.0.zip in Kodi.
Then add into addon.xml file <import addon="script.module.requests" version="2.22.0"/>
<requires>
<import addon="xbmc.python" version="2.25.0"/>
<import addon="script.module.requests" version="2.22.0"/>
</requires>
Now in plugin's main.py file insert following code snippet
import json
import requests
url = 'http://iptv.server.com/series.json'
VIDEOS = {}
html = requests.get(url)
DATA = json.loads(html.content)
for title in DATA.keys():
title_utf8 = title.encode('utf-8')
VIDEOS[title_utf8] = []
for episode in DATA[title]:
episode['name'] = episode['name'].encode('utf-8')
episode['genre'] = episode['genre'].encode('utf-8')
VIDEOS[title_utf8].append(episode)
Launch Kodi and test the plugin.
Next code snippet is for reading file from local storage
import json
fname = 'c:/temp/iptv/animation.json';
VIDEOS = {}
with open(fname,'r') as f:
str = f.read()
DATA = json.loads(str)
for title in DATA.keys():
title_utf8 = title.encode('utf-8')
VIDEOS[title_utf8] = []
for episode in DATA[title]:
episode['name'] = episode['name'].encode('utf-8')
episode['genre'] = episode['genre'].encode('utf-8')
VIDEOS[title_utf8].append(episode)
In python 3.6.8 I am trying to download a 'file' from a URL and process it directly, without creating a local file. I have tried the following code
import io
import requests
url = "https://raw.githubusercontent.com/enzoftware/random/master/README.md"
response = requests.get(url, stream=True)
with io.BytesIO(response.text) as f:
print(f.readlines())
but I get an error
Traceback (most recent call last):
File "tester.py", line 7, in <module>
with io.BytesIO(response.text) as f:
TypeError: a bytes-like object is required, not 'str'
How to do it right?
assuming you just want to read it line by line rather than considering any document (html) structure it may have you can just do
import requests
url = "https://raw.githubusercontent.com/enzoftware/random/master/README.md"
response = requests.get(url, stream=True)
for line in response.text.splitlines():
print (line)
I've just rebuilt my Raspberry Pi and hence installed the latest version of the Dropbox API and now my program doesn't work. I think this is due to point 1 in these breaking changes: https://github.com/dropbox/dropbox-sdk-python/releases/tag/v7.1.0. I'm sure this question from SO (Dropbox API v2 - trying to upload file with files_upload() - throws TypeError) solves my problem... but as a newbie, I can't figure out how to actually implement it - and anyway, I'm already using f.read()... can anyone help?
This is my code:
def DropboxUpload(file):
sourcefile = "/home/pi/Documents/iot_pm2/dropbox_transfer/" + filename
targetfile = "/" + filename
dbx = dropbox.Dropbox(cfg.dropboxtoken)
f = open(sourcefile, "r")
filecontents = f.read()
try:
dbx.files_upload(filecontents, targetfile, mode=dropbox.files.WriteMode.overwrite)
except dropbox.exceptions.ApiError as err:
print(err)
f.close()
And this is the error:
Traceback (most recent call last):
File "/home/pi/Documents/iot_pm2/dropbox_uploader.py", line 20, in <module>
DropboxUpload(filename)
File "/home/pi/Documents/iot_pm2/dropbox_uploader.py", line 12, in DropboxUpload
dbx.files_upload(filecontents, targetfile, mode=dropbox.files.WriteMode.overwrite)
File "/usr/local/lib/python3.5/dist-packages/dropbox/base.py", line 2125, in files_upload
f,
File "/usr/local/lib/python3.5/dist-packages/dropbox/dropbox.py", line 272, in request
timeout=timeout)
File "/usr/local/lib/python3.5/dist-packages/dropbox/dropbox.py", line 363, in request_json_string_with_retry
timeout=timeout)
File "/usr/local/lib/python3.5/dist-packages/dropbox/dropbox.py", line 407, in request_json_string
type(request_binary))
TypeError: expected request_binary as binary type, got <class 'str'>
Thanks in advance.
You need to supply bytes, but you're supplying str.
You can get bytes by changing the file mode to binary. I.e., instead of:
f = open(sourcefile, "r")
do:
f = open(sourcefile, "rb")
I am trying to get an image from a website and I don't know what i am doing wrong.
Here is my code:
import httplib2
h = httplib2.Http('.cache')
response, content = h.request('http://1.bp.blogspot.com/-KSBzFF0bIyU/TtFyj-KgC0I/AAAAAAAABEo/o43IoY_9Bec/s1600/praia-de-ponta-verde-maceio.jpg')
print(response.status)
with open('maceio.jpg', 'wb') as f:
print(content, file = f)
--------------------------------------------------------------------------------
200
Traceback (most recent call last):
File "/home/matheus/workspace/Get Link/new_method_v2.py", line 12, in <module>
print(content, file = f)
TypeError: 'str' does not support the buffer interface
The error is caused by the following line:
print(content, file = f)
print implicitely converts the bytes object named content to a string (str object), which cannot be written to a file in binary mode, since Python does not know which character encoding to use.
Why are you taking the print detour at all? Just write the contents to the file using the file.write() method:
with open('maceio.jpg', 'wb') as f:
f.write(content)