We Keep Coding

Python NetBox REST API error when uploading json to NetBox

I'm trying to modify a device in NetBox using subprocess, curl and the NetBox REST API through python.
The issue seems to be with special characters and JSON.
Note: I'm not using requests because we do everything in curl. We are also bypassing security by using the "--insecure" option for curl because this is just a proof-of-concept right now.
The script we are using has two functions:
get: - retrieves all the info for all devices in NetBox and create a file for each device containing it's current 'comments' data. This works just fine.
patch: - grabs the contents of the appropriate device and uploads to NetBox, which should be visible in it's web GUI. This is the function with the issue.
For example, if the file contained
abc
I should be able to change the contents of the file to
now it's more than abc
and then upload the change to NetBox. Looking at the web GUI, I should see the change.
The actual file I am trying to upload to the device comments contains this gibberish, which I typed in manually:
anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?
My upload_comments function:
Note: the {deviceid} in the url comes from a different section of code and is not shown here and is working fine.
def upload_comments(filename):
with open(filename, 'r') as f:
new_comment = f.read().strip()
print('new_comment: ', new_comment) # verify we have content
patch_command_line = f'''curl
--insecure
--verbose
--request PATCH
--header "Content-Type: application/json"
--header "Accept: application/json"
--header "Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
--user "xxxx:xxxx"
--url "https://xxx.xxx.xxx.xxx/api/dcim/devices/{deviceid}/"
--data-binary '{{ "comments": "{new_comment}" }}' '''
print(patch_command_line) # verify curl command line
try:
r = subprocess.run(patch_command_line, capture_output=True, shell=True, check=True, timeout=10)
# print("stdout: \n")
print(r.stdout.decode('utf8'))
print(r.stderr.decode('utf8'))
except Exception as e:
print(e)
^
This is the result of running that function:
new_comment: anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?
curl --insecure --verbose --request PATCH --header "Content-Type: application/json" --header "Accept: application/json" --header "Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --user "xxxx:xxxx!" --url "https://xxx.xxx.xxx.xxx/api/dcim/devices/2/" --data-binary '{ "comments": "anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?" }'
{"detail":"JSON parse error - Invalid control character at: line 1 column 53 (char 52)"}
* Trying xxx.xxx.xxx.xxx:443...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\* Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} \[5 bytes data\]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} \[512 bytes data\]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ \[122 bytes data\]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ \[21 bytes data\]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ \[892 bytes data\]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ \[264 bytes data\]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ \[52 bytes data\]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} \[1 bytes data\]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} \[52 bytes data\]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* start date: Nov 11 22:51:31 2022 GMT
* expire date: Nov 11 22:51:31 2023 GMT
* issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* SSL certificate verify result: self signed certificate (18), continuing anyway.
} \[5 bytes data\]
> PATCH /api/dcim/devices/2/ HTTP/1.1
> Host: xxx.xxx.xxx.xxx
> User-Agent: curl/7.68.0
> Content-Type: application/json
> Accept: application/json
> Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Content-Length: 108
} \[108 bytes data\]
* upload completely sent off: 108 out of 108 bytes
{ \[5 bytes data\]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ \[265 bytes data\]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ \[265 bytes data\]
* old SSL session ID is stale, removing
{ \[5 bytes data\]
* Mark bundle as not supporting multiuse
\< HTTP/1.1 400 Bad Request
\< Date: Tue, 22 Nov 2022 17:03:08 GMT
\< Server: gunicorn
\< Content-Type: application/json
\< Vary: Accept,Cookie,Origin
\< Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
\< API-Version: 3.3
\< X-Content-Type-Options: nosniff
\< Referrer-Policy: same-origin
\< Cross-Origin-Opener-Policy: same-origin
\< X-Frame-Options: SAMEORIGIN
\< Content-Length: 88
\< Connection: close
\<
{ \[5 bytes data\]
100 196 100 88 100 108 245 300 --:--:-- --:--:-- --:--:-- 545
* Closing connection 0
} \[5 bytes data\]
* TLSv1.3 (OUT), TLS alert, close notify (256):
} \[2 bytes data\]
Running curl from the command line gives me the correct result as long as there are no special characters in the value section of comments, otherwise I get errors like this:
{"detail":"JSON parse error - Unterminated string starting at: line 1 column 14 (char 13)"}\* Could not resolve host: november
* Closing connection 1
curl: (6) Could not resolve host: november
curl: (3) unmatched close brace/bracket in URL position 5:
11th}'
^

Python Curl To Elastic Search Issue

I am trying to use curl in python to push this command in Elastic search so that index patterns get created after the code finishes
import pycurl
import urllib
import urllib2
apiURL = 'http://localhost:9200/.kibana/index-pattern/james+_alerts* -d'
c = pycurl.Curl()
c.setopt(c.URL, apiURL)
c.setopt(c.POSTFIELDS, '{"title" : james+"_alerts*", "timeFieldName": "timeStamp"}')
c.setopt(c.VERBOSE, True)
c.perform()
c.close()
The output being returned is: -
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> POST /.kibana/index-pattern/james+_alerts* -d HTTP/1.1
Host: localhost:9200
User-Agent: PycURL/7.43.0 libcurl/7.52.1 GnuTLS/3.5.6 zlib/1.2.11 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) librtmp/2.3
Accept: */*
Content-Length: 56
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 56 out of 56 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
< content-type: application/json; charset=UTF-8
< content-length: 207
<
* Curl_http_done: called premature == 0
* Closing connection 0
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"invalid version format: -D HTTP/1.1"}],"type":"illegal_argument_exception","reason":"invalid version format: -D HTTP/1.1"},"status":400}
What could potentially be the issue and fix for this?

You have extra '-d' in apiURL. I guess that it came from copy&paste from command line. If not, then you must encode url (urllib.urlencode)

Calling a Flask REST service method in different OS with curl

I wrote the following POST method for my REST API, which was built using Flask. The method receives one parameter, which is a radio station url.
#app.route('/todo/api/v1.0/predvajaj', methods=['POST'])
def create_task():
print "Expression value: " + str(not request.json or not 'title' in request.json)
if not request.json or not 'title' in request.json:
abort(400)
link=request.json['title']
print "Link value: " + link
cmd = "pkill sox"
os.system(cmd)
time.sleep(2)
#link = "http://www.radiostationurl.m3u"
cmd = "sox -t mp3 " + link + " -t wav -r 22050 -c 1 - | sudo ../pifm - 90.5 &"
os.system(cmd)
return jsonify({'status': "ok"}), 201
The API runs on a Raspberry Pi with the ip address: 192.168.0.200. I tried testing the method locally (on the Pi), using the curl tool. This worked fine:
curl -i -H "Content-Type: application/json" -X POST -d '{"title":"http://www.radiostationurl.m3u"}' http://192.168.0.200:5000/todo/api/v1.0/predvajaj
Then I tried testing testing it with a computer (running Windows) in the same LAN with the same command and tool, but I get the following error:
HTTP/1.0 400 BAD REQUEST
Content-Type: text/html
Content-Length: 192
Server: Werkzeug/0.10.4 Python/2.7.3
Date: Wed, 05 Aug 2015 11:06:05 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
Webserver output (two requests - one from the pi and from the other pc):
Expression value: False
192.168.0.200 - - [05/Aug/2015 11:05:56] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 201 -
sox WARN wav: Length in output .wav header will be wrong since can't seek to fix it
192.168.0.103 - - [05/Aug/2015 11:06:05] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 400
So the problem is in the evaluation of the if expression. Can anybody tell me why is it failing to evaluate?
EDIT: Tried running curl with the -v switch as #meuh suggested. The content-length is different.
Pi:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* connected
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 192.168.0.200:5000
> Accept: */*
> Content-Type: application/json
> Content-Length: 51
>
* upload completely sent off: 51 out of 51 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.0, assume close after body
< HTTP/1.0 201 CREATED
< Content-Type: application/json
< Content-Length: 27
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:49:01 GMT
<
{
"status": "predvajam"
* Closing connection #0
}
Windows:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> Host: 192.168.0.200:5000
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 49
>
* upload completely sent off: 49 out of 49 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Content-Type: text/html
< Content-Length: 192
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:50:51 GMT
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
* Closing connection 0

I overlooked a note about this issue, on the site with the tutorial, which says:
Note: if you are on Windows and use the Cygwin version of curl from bash then the above command will work just fine. However, if you are using the native version of curl from the regular command prompt there is a little dance that needs to be done to send double quotes inside the body of a request. Essentially on Windows you have to use double quotes to enclose the body of the request, and then inside it you escape a double quote by writing three of them in sequence.
The correct command, in my case, is:
curl -i -H "Content-Type: application/json" -X POST -d "{"""title""":"""http://www.radiostationurl.m3u"""}" http://192.168.0.200:5000/todo/api/v1.0/predvajaj

python numpy list convert to flask jsonify failed

I use flask and it is so such:
global eventlist
NWEVENT = numpy.dtype({'names': ['call_event', 'ext_id', 'visitor_id','vfrom', 'to', 'caller_id', 'duration','create_time'],
'formats': ['S20','S20','S20','S20','S20','S20', 'i','S30']}, align = True)
and append any data into eventlist,then I want send eventlist to called:
#app.route('/api/v1.0/events', methods=['GET'])
#auth.login_required
def get_events():
return jsonify({'call_events':eventlist.tolist()})
when the eventlist is zero,and it return:
curl -u ok:python -i -H "Content-Type: application/json" -X GET http://192.168.1.140:9033/api/v1.0/events
HTTP/1.0 201 CREATED
Content-Type: application/json
Content-Length: 23
Server: Werkzeug/0.9.6 Python/2.7.8
Date: Thu, 05 Feb 2015 08:18:13 GMT
{
"call_events": []
}
but when it has any data,it return:
curl -u ok:python -i -H "Content-Type: application/json" -X GET http://192.168.1.140:9033/api/v1.0/events
Server: Werkzeug/0.9.6 Python/2.7.8
Date: Thu, 05 Feb 2015 08:18:13 GMT
{
HTTP/1.0 500 INTERNAL SERVER ERROR
Content-Type: text/html
Content-Length: 291
Server: Werkzeug/0.9.6 Python/2.7.8
Date: Thu, 05 Feb 2015 08:48:46 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
this use :
mylist = []
#print jsonify({'call_events':eventlist})
for event in eventlist:
myevent = {
'call_event':event['call_event'][0],
'ext_id':event['ext_id'][0],
'visitor_id':event['visitor_id'][0],
'vfrom':event['vfrom'][0],
'to':event['to'][0],
'caller_id':event['caller_id'][0],
'duration':event['duration'][0]
}
mylist.append(myevent)

Why can't I POST to Django with pyCurl?

I've hit something truly strange with a pyCurl script hitting a local Django-Tastypie REST webserver.
Issuing HTTP PUT requests to the server succeeds when I use everything but pycurl (including curl), and fails with error 400 in pycurl.
After much googling and experimentation, I'm stumped. What's wrong here?
Curl call that works:
curl --verbose -X PUT -H 'Content-Type: application/json' -d '{"first_name": "Gaius","id": 1,"last_name": "Balthazar","login": "gbalthazar"}' http://localhost:8000/api/person/1/
PyCurl call that DOESN'T work (error 400):
import pycurl
import StringIO
curl = pycurl.Curl()
url = 'http://localhost:8000/api/person/1/'
curl.setopt(pycurl.URL,url)
curl.setopt(pycurl.VERBOSE, 1)
body = '{"first_name": "Gaius","id": 1,"last_name": "Baltar","login": "gbaltar"}'
curl.setopt(pycurl.READFUNCTION, StringIO.StringIO(body).read)
curl.setopt(pycurl.UPLOAD, 1)
curl.setopt(pycurl.HTTPHEADER,['Content-Type: application/json','Expect:'])
curl.setopt(curl.TIMEOUT, 5)
curl.perform()
(I've tried removing the Expects header as well, I see the header set to 100-Continue in the pycurl call, but same result.)
Unfortunately this project really does need pycurl's low-level access to HTTP timing stats to measure performance, so I can't do it with another HTTP/REST library.
Output of Curl Call:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
> User-Agent: curl/7.27.0
> Host: localhost:8000
> Accept: */*
> Content-Type: application/json
> Content-Length: 78
>
* upload completely sent off: 78 out of 78 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 05 Jun 2014 23:45:26 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< Vary: Accept
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"first_name": "Gaius", "id": 1, "last_name": "Balthazar", "login": "gbalthazar", "pk": "1", "resource_uri": "/api/person/1/"}
Output of PyCurl Verbose Call:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
User-Agent: PycURL/7.27.0
Host: localhost:8000
Accept: */*
Transfer-Encoding: chunked
Content-Type: application/json
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Date: Thu, 05 Jun 2014 23:44:25 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"error": ""}
What am I missing here?

Found the answer:
It needs length of the request body to handle correctly
For POST:
curl.setopt(pycurl.POSTFIELDSIZE, len(body))
For PUT:
curl.setopt(pycurl.INFILESIZE, len(body))
(Yes, it's a different option for different HTTP calls... that's libcurl for you)
Not completely sure what triggers this behaviour, but the above fixes it and the tests work now.
EDIT: Adding verbose pycurl output from this:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
User-Agent: PycURL/7.27.0
Host: localhost:8000
Accept: */*
Content-Type: application/json
Content-Length: 72
* We are completely uploaded and fine
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Fri, 06 Jun 2014 17:41:38 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< Vary: Accept
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"first_name": "Gaius", "id": 1, "last_name": "Baltar", "login": "gbaltar", "pk": "1", "resource_uri": "/api/person/1/"}