I'm trying to modify a device in NetBox using subprocess, curl and the NetBox REST API through python.
The issue seems to be with special characters and JSON.
Note: I'm not using requests because we do everything in curl. We are also bypassing security by using the "--insecure" option for curl because this is just a proof-of-concept right now.
The script we are using has two functions:
get: - retrieves all the info for all devices in NetBox and create a file for each device containing it's current 'comments' data. This works just fine.
patch: - grabs the contents of the appropriate device and uploads to NetBox, which should be visible in it's web GUI. This is the function with the issue.
For example, if the file contained
abc
I should be able to change the contents of the file to
now it's more than abc
and then upload the change to NetBox. Looking at the web GUI, I should see the change.
The actual file I am trying to upload to the device comments contains this gibberish, which I typed in manually:
anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?
My upload_comments function:
Note: the {deviceid} in the url comes from a different section of code and is not shown here and is working fine.
def upload_comments(filename):
with open(filename, 'r') as f:
new_comment = f.read().strip()
print('new_comment: ', new_comment) # verify we have content
patch_command_line = f'''curl
--insecure
--verbose
--request PATCH
--header "Content-Type: application/json"
--header "Accept: application/json"
--header "Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
--user "xxxx:xxxx"
--url "https://xxx.xxx.xxx.xxx/api/dcim/devices/{deviceid}/"
--data-binary '{{ "comments": "{new_comment}" }}' '''
print(patch_command_line) # verify curl command line
try:
r = subprocess.run(patch_command_line, capture_output=True, shell=True, check=True, timeout=10)
# print("stdout: \n")
print(r.stdout.decode('utf8'))
print(r.stderr.decode('utf8'))
except Exception as e:
print(e)
^
This is the result of running that function:
new_comment: anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?
curl --insecure --verbose --request PATCH --header "Content-Type: application/json" --header "Accept: application/json" --header "Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --user "xxxx:xxxx!" --url "https://xxx.xxx.xxx.xxx/api/dcim/devices/2/" --data-binary '{ "comments": "anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?" }'
{"detail":"JSON parse error - Invalid control character at: line 1 column 53 (char 52)"}
* Trying xxx.xxx.xxx.xxx:443...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\* Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} \[5 bytes data\]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} \[512 bytes data\]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ \[122 bytes data\]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ \[21 bytes data\]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ \[892 bytes data\]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ \[264 bytes data\]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ \[52 bytes data\]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} \[1 bytes data\]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} \[52 bytes data\]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* start date: Nov 11 22:51:31 2022 GMT
* expire date: Nov 11 22:51:31 2023 GMT
* issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* SSL certificate verify result: self signed certificate (18), continuing anyway.
} \[5 bytes data\]
> PATCH /api/dcim/devices/2/ HTTP/1.1
> Host: xxx.xxx.xxx.xxx
> User-Agent: curl/7.68.0
> Content-Type: application/json
> Accept: application/json
> Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Content-Length: 108
} \[108 bytes data\]
* upload completely sent off: 108 out of 108 bytes
{ \[5 bytes data\]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ \[265 bytes data\]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ \[265 bytes data\]
* old SSL session ID is stale, removing
{ \[5 bytes data\]
* Mark bundle as not supporting multiuse
\< HTTP/1.1 400 Bad Request
\< Date: Tue, 22 Nov 2022 17:03:08 GMT
\< Server: gunicorn
\< Content-Type: application/json
\< Vary: Accept,Cookie,Origin
\< Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
\< API-Version: 3.3
\< X-Content-Type-Options: nosniff
\< Referrer-Policy: same-origin
\< Cross-Origin-Opener-Policy: same-origin
\< X-Frame-Options: SAMEORIGIN
\< Content-Length: 88
\< Connection: close
\<
{ \[5 bytes data\]
100 196 100 88 100 108 245 300 --:--:-- --:--:-- --:--:-- 545
* Closing connection 0
} \[5 bytes data\]
* TLSv1.3 (OUT), TLS alert, close notify (256):
} \[2 bytes data\]
Running curl from the command line gives me the correct result as long as there are no special characters in the value section of comments, otherwise I get errors like this:
{"detail":"JSON parse error - Unterminated string starting at: line 1 column 14 (char 13)"}\* Could not resolve host: november
* Closing connection 1
curl: (6) Could not resolve host: november
curl: (3) unmatched close brace/bracket in URL position 5:
11th}'
^
Related
I am trying to execute this curl command using python. It retrieves an output like below.
* Rebuilt URL to: <dns>
* Trying <ip>...
* TCP_NODELAY set
* Connected to escortpersonaladz.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=webdisk.escortpersonaladz.com
* start date: May 3 02:00:53 2020 GMT
* expire date: Aug 1 02:00:53 2020 GMT
* subjectAltName: host "escortpersonaladz.com" matched cert's "escortpersonaladz.com"
How can I extract that particular line * expire date: Aug 1 02:00:53 2020 GMT from the above output if it is exist?
import re
curl_output = '''
* Rebuilt URL to: <dns>
* Trying <ip>...
* TCP_NODELAY set
* Connected to escortpersonaladz.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=webdisk.escortpersonaladz.com
* start date: May 3 02:00:53 2020 GMT
* expire date: Aug 1 02:00:53 2020 GMT
* subjectAltName: host "escortpersonaladz.com" matched cert's "escortpersonaladz.com"
'''
match = re.search(r"(\*\s*expire date(.+?))\s*\*", curl_output)
if match:
desired = match.group(1)
print(desired)
#* expire date: Aug 1 02:00:53 2020 GMT
else:
print("Not found")
The regex matches the string you are looking for by looking between two *s that contain the expire date. It also accounts for possible spaces before and after. If a match is found, desired string lies in the first group of the match object. In case it is not found, re.search will return None, so we check and act accordingly.
This question already has answers here:
Requests SSL connection timeout
(1 answer)
Strange CURL issue with a particular website SSL certificate
(1 answer)
Golang Http Get Request very slow
(2 answers)
Scraping attempts getting 403 error
(1 answer)
Closed 3 years ago.
I need to retrive this site (www.extra.com.br) in a Python project.
Neither
page = requests.get("https://www.extra.com.br/")
nor
page = requests.get("http://www.extra.com.br/")
works, the script keeps stuck on this line.
So I went to try using curl to retrieve the page and check it off-line.
curl http://www.extra.com.br also does not work. This is the output using -v.
STATE: INIT => CONNECT handle 0x6dfd90; line 1392 (connection #-5000)
Rebuilt URL to: https://www.extra.com.br/
Added connection 0. The cache now contains 1 members
STATE: CONNECT => WAITRESOLVE handle 0x6dfd90; line 1428 (connection #0) % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0* Trying 23.74.196.194...
TCP_NODELAY set
STATE: WAITRESOLVE => WAITCONNECT handle 0x6dfd90; line 1509 (connection #0)
Connected to www.extra.com.br (23.74.196.194) port 443 (#0)
STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x6dfd90; line 1561 (connection #0) 0 0 0 0 0 0 0 0 --:--:--
--:--:-- --:--:-- 0* Marked for [keep alive]: HTTP default
ALPN, offering h2
ALPN, offering http/1.1
Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
successfully set certificate verify locations:
CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt CApath: none
TLSv1.2 (OUT), TLS header, Certificate Status (22): } [5 bytes data]
TLSv1.2 (OUT), TLS handshake, Client hello (1): } [512 bytes data]
STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x6dfd90; line 1575 (connection #0) { [5 bytes data]
TLSv1.2 (IN), TLS handshake, Server hello (2): { [108 bytes data]
TLSv1.2 (IN), TLS handshake, Certificate (11): { [2799 bytes data]
TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data]
TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data]
TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data]
TLSv1.2 (OUT), TLS change cipher, Client hello (1): } [1 bytes data]
TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data]
TLSv1.2 (IN), TLS change cipher, Client hello (1): { [1 bytes data]
TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data]
SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
ALPN, server accepted to use http/1.1
Server certificate:
subject: C=BR; ST=Sao Paulo; L=Sao Paulo; O=CNOVA Comercio Eletronico S.A.; OU=TI; CN=*.extra.com.br
start date: Jul 17 00:00:00 2018 GMT
expire date: Jul 17 12:00:00 2019 GMT
subjectAltName: host "www.extra.com.br" matched cert's "*.extra.com.br"
issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
SSL certificate verify ok.
STATE: PROTOCONNECT => DO handle 0x6dfd90; line 1596 (connection #0) } [5 bytes data]
GET / HTTP/1.1
Host: www.extra.com.br
User-Agent: curl/7.58.0
Accept: /
STATE: DO => DO_DONE handle 0x6dfd90; line 1658 (connection #0)
STATE: DO_DONE => WAITPERFORM handle 0x6dfd90; line 1783 (connection #0)
STATE: WAITPERFORM => PERFORM handle 0x6dfd90; line 1799 (connection #0) 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054
Marked for [closure]: Transfer returned error
multi_done
stopped the pause stream! 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0
Closing connection 0
The cache now contains 0 members } [5 bytes data] curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054
I am running a Python3 app via Gunicorn with two-way SSL configured. This requires a local cert/key to verify the app as well as a ca_certs file to verify the client. Gunicorn relies on the standard ssl module in Python, particularly the wrap_socket method.
The service starts and responds to curl requests fine when I use self-signed certificates for both server and client authentication. However when I use certificates signed by another CA, I get an error SSL_ERROR_UNKNOWN_CA_ALERT.
A working setup, with self-signed certs:
# Server cert
openssl req \
-newkey rsa:2048 -nodes -keyout domain.key \
-x509 -days 365 -out domain.crt
# Client (CA) cert
openssl req \
-newkey rsa:2048 -nodes -keyout twoway.key \
-x509 -days 365 -out twoway.crt
With Gunicorn configured as follows:
keyfile = domain.key
certfile = domain.crt
cert_reqs = ssl.CERT_REQUIRED
ca_certs=twoway.crt
And curling as follows:
curl -vk --key twoway.key --cert twoway.crt https://my.service
Produces a successful response:
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* start date: Dec 7 18:35:54 2018 GMT
* expire date: Dec 7 18:35:54 2019 GMT
* issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /manage/info HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: gunicorn/19.9.0
< Date: Tue, 11 Dec 2018 18:26:19 GMT
< Connection: keep-alive
< Content-Type: application/json
< Content-Length: 73
A failing setup, with a different series of certs:
With Gunicorn configured as follows:
keyfile = my_service_key.key
certfile = my_service_cert.crt
cert_reqs = ssl.CERT_REQUIRED
ca_certs = my_trusted_clients.crt
And curling as follows:
curl -vk --key my_trusted_key.key --cert my_trusted_clients.crt https://my.service
Produces an error:
About to connect() to localhost port 5000 (#0)
Initializing NSS with certpath: sql/etc/pki/nssdb
warning: ignoring value of ssl.verifyhost
skipping SSL peer certificate verification
NSS: client certificate from file
subject: CN=mycn,OU=abc,O=def,...
NSS error -12195
Closing connection #0
SSL connect error
curl: (35) SSL connect error
Any thoughts on whether I am configuring this the wrong way? And why self-signed certs are working but other certs are not?
Note this config worked previously when using Stunnel, where I set the verify level to 4 ("Ignore the chain and only verify the peer certificate."). If there is something similar in Python I believe that would get me in the right direction.
I don't think that this is be possible as of Gunicorn 19.9.
As well as having the complete certificate chain on the server, in order to validate the client/peer certificate, I believe you will need to be able to configure the SSLContext, and especially be able to set ssl.CERT_REQUIRED when in server-mode.
Gunicorn 19.9 (and master at the time of writing) does not currently use the SSLContext-based wrapper on the connection, so this is not possible, see https://github.com/benoitc/gunicorn/issues/1140 .
I am trying to use curl in python to push this command in Elastic search so that index patterns get created after the code finishes
import pycurl
import urllib
import urllib2
apiURL = 'http://localhost:9200/.kibana/index-pattern/james+_alerts* -d'
c = pycurl.Curl()
c.setopt(c.URL, apiURL)
c.setopt(c.POSTFIELDS, '{"title" : james+"_alerts*", "timeFieldName": "timeStamp"}')
c.setopt(c.VERBOSE, True)
c.perform()
c.close()
The output being returned is: -
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> POST /.kibana/index-pattern/james+_alerts* -d HTTP/1.1
Host: localhost:9200
User-Agent: PycURL/7.43.0 libcurl/7.52.1 GnuTLS/3.5.6 zlib/1.2.11 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) librtmp/2.3
Accept: */*
Content-Length: 56
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 56 out of 56 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
< content-type: application/json; charset=UTF-8
< content-length: 207
<
* Curl_http_done: called premature == 0
* Closing connection 0
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"invalid version format: -D HTTP/1.1"}],"type":"illegal_argument_exception","reason":"invalid version format: -D HTTP/1.1"},"status":400}
What could potentially be the issue and fix for this?
You have extra '-d' in apiURL. I guess that it came from copy&paste from command line. If not, then you must encode url (urllib.urlencode)
I wrote the following POST method for my REST API, which was built using Flask. The method receives one parameter, which is a radio station url.
#app.route('/todo/api/v1.0/predvajaj', methods=['POST'])
def create_task():
print "Expression value: " + str(not request.json or not 'title' in request.json)
if not request.json or not 'title' in request.json:
abort(400)
link=request.json['title']
print "Link value: " + link
cmd = "pkill sox"
os.system(cmd)
time.sleep(2)
#link = "http://www.radiostationurl.m3u"
cmd = "sox -t mp3 " + link + " -t wav -r 22050 -c 1 - | sudo ../pifm - 90.5 &"
os.system(cmd)
return jsonify({'status': "ok"}), 201
The API runs on a Raspberry Pi with the ip address: 192.168.0.200. I tried testing the method locally (on the Pi), using the curl tool. This worked fine:
curl -i -H "Content-Type: application/json" -X POST -d '{"title":"http://www.radiostationurl.m3u"}' http://192.168.0.200:5000/todo/api/v1.0/predvajaj
Then I tried testing testing it with a computer (running Windows) in the same LAN with the same command and tool, but I get the following error:
HTTP/1.0 400 BAD REQUEST
Content-Type: text/html
Content-Length: 192
Server: Werkzeug/0.10.4 Python/2.7.3
Date: Wed, 05 Aug 2015 11:06:05 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
Webserver output (two requests - one from the pi and from the other pc):
Expression value: False
192.168.0.200 - - [05/Aug/2015 11:05:56] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 201 -
sox WARN wav: Length in output .wav header will be wrong since can't seek to fix it
192.168.0.103 - - [05/Aug/2015 11:06:05] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 400
So the problem is in the evaluation of the if expression. Can anybody tell me why is it failing to evaluate?
EDIT: Tried running curl with the -v switch as #meuh suggested. The content-length is different.
Pi:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* connected
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 192.168.0.200:5000
> Accept: */*
> Content-Type: application/json
> Content-Length: 51
>
* upload completely sent off: 51 out of 51 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.0, assume close after body
< HTTP/1.0 201 CREATED
< Content-Type: application/json
< Content-Length: 27
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:49:01 GMT
<
{
"status": "predvajam"
* Closing connection #0
}
Windows:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> Host: 192.168.0.200:5000
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 49
>
* upload completely sent off: 49 out of 49 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Content-Type: text/html
< Content-Length: 192
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:50:51 GMT
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
* Closing connection 0
I overlooked a note about this issue, on the site with the tutorial, which says:
Note: if you are on Windows and use the Cygwin version of curl from bash then the above command will work just fine. However, if you are using the native version of curl from the regular command prompt there is a little dance that needs to be done to send double quotes inside the body of a request. Essentially on Windows you have to use double quotes to enclose the body of the request, and then inside it you escape a double quote by writing three of them in sequence.
The correct command, in my case, is:
curl -i -H "Content-Type: application/json" -X POST -d "{"""title""":"""http://www.radiostationurl.m3u"""}" http://192.168.0.200:5000/todo/api/v1.0/predvajaj