We Keep Coding

Python NetBox REST API error when uploading json to NetBox

I'm trying to modify a device in NetBox using subprocess, curl and the NetBox REST API through python.
The issue seems to be with special characters and JSON.
Note: I'm not using requests because we do everything in curl. We are also bypassing security by using the "--insecure" option for curl because this is just a proof-of-concept right now.
The script we are using has two functions:
get: - retrieves all the info for all devices in NetBox and create a file for each device containing it's current 'comments' data. This works just fine.
patch: - grabs the contents of the appropriate device and uploads to NetBox, which should be visible in it's web GUI. This is the function with the issue.
For example, if the file contained
abc
I should be able to change the contents of the file to
now it's more than abc
and then upload the change to NetBox. Looking at the web GUI, I should see the change.
The actual file I am trying to upload to the device comments contains this gibberish, which I typed in manually:
anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?
My upload_comments function:
Note: the {deviceid} in the url comes from a different section of code and is not shown here and is working fine.
def upload_comments(filename):
with open(filename, 'r') as f:
new_comment = f.read().strip()
print('new_comment: ', new_comment) # verify we have content
patch_command_line = f'''curl
--insecure
--verbose
--request PATCH
--header "Content-Type: application/json"
--header "Accept: application/json"
--header "Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
--user "xxxx:xxxx"
--url "https://xxx.xxx.xxx.xxx/api/dcim/devices/{deviceid}/"
--data-binary '{{ "comments": "{new_comment}" }}' '''
print(patch_command_line) # verify curl command line
try:
r = subprocess.run(patch_command_line, capture_output=True, shell=True, check=True, timeout=10)
# print("stdout: \n")
print(r.stdout.decode('utf8'))
print(r.stderr.decode('utf8'))
except Exception as e:
print(e)
^
This is the result of running that function:
new_comment: anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?
curl --insecure --verbose --request PATCH --header "Content-Type: application/json" --header "Accept: application/json" --header "Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --user "xxxx:xxxx!" --url "https://xxx.xxx.xxx.xxx/api/dcim/devices/2/" --data-binary '{ "comments": "anotyher device needing spellchecking
!##$%^&\*()\_+-=\[\]\\ \[\]\\ /.,,./
did yuou see that?" }'
{"detail":"JSON parse error - Invalid control character at: line 1 column 53 (char 52)"}
* Trying xxx.xxx.xxx.xxx:443...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\* Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} \[5 bytes data\]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} \[512 bytes data\]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ \[122 bytes data\]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ \[21 bytes data\]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ \[892 bytes data\]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ \[264 bytes data\]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ \[52 bytes data\]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} \[1 bytes data\]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} \[52 bytes data\]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* start date: Nov 11 22:51:31 2022 GMT
* expire date: Nov 11 22:51:31 2023 GMT
* issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* SSL certificate verify result: self signed certificate (18), continuing anyway.
} \[5 bytes data\]
> PATCH /api/dcim/devices/2/ HTTP/1.1
> Host: xxx.xxx.xxx.xxx
> User-Agent: curl/7.68.0
> Content-Type: application/json
> Accept: application/json
> Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Content-Length: 108
} \[108 bytes data\]
* upload completely sent off: 108 out of 108 bytes
{ \[5 bytes data\]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ \[265 bytes data\]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ \[265 bytes data\]
* old SSL session ID is stale, removing
{ \[5 bytes data\]
* Mark bundle as not supporting multiuse
\< HTTP/1.1 400 Bad Request
\< Date: Tue, 22 Nov 2022 17:03:08 GMT
\< Server: gunicorn
\< Content-Type: application/json
\< Vary: Accept,Cookie,Origin
\< Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
\< API-Version: 3.3
\< X-Content-Type-Options: nosniff
\< Referrer-Policy: same-origin
\< Cross-Origin-Opener-Policy: same-origin
\< X-Frame-Options: SAMEORIGIN
\< Content-Length: 88
\< Connection: close
\<
{ \[5 bytes data\]
100 196 100 88 100 108 245 300 --:--:-- --:--:-- --:--:-- 545
* Closing connection 0
} \[5 bytes data\]
* TLSv1.3 (OUT), TLS alert, close notify (256):
} \[2 bytes data\]
Running curl from the command line gives me the correct result as long as there are no special characters in the value section of comments, otherwise I get errors like this:
{"detail":"JSON parse error - Unterminated string starting at: line 1 column 14 (char 13)"}\* Could not resolve host: november
* Closing connection 1
curl: (6) Could not resolve host: november
curl: (3) unmatched close brace/bracket in URL position 5:
11th}'
^

Issue with request(get)

I have an issue with sending a request to the website. I am using the "hentai" library to make a useful telegram bot with nhentai API on Python (don't judge please). Why do I get this error on Linux (Ubuntu)?
:~$ curl -v https://nhentai.net
* Expire in 0 ms for 6 (transfer 0x55c3a8516530)
...
* Expire in 9 ms for 1 (transfer 0x55c3a8516530)
* Trying 104.27.195.88...
* TCP_NODELAY set
* Expire in 149985 ms for 3 (transfer 0x55c3a8516530)
* Expire in 200 ms for 4 (transfer 0x55c3a8516530)
* Connected to nhentai.net (104.27.195.88) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to nhentai.net:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to nhentai.net:443

My guess is that there is something in the middle, like a local antivirus or parental control software or some company/school middlebox/firewall which does not like the name of the website you are trying to access.
What you see is typical for DPI solutions which allow the initial TCP connect but then cause a connection close once they detect the domain name you are connecting to (which is contained in the TLS handshake ClientHello)

I tried with curl and python's request library on the site you mentioned and pulled the desired content. Make sure that the curl on your linux machine is working properly. If you want, you can try the simple code below.
import requests
myrequest = requests.get("https://nhentai.net/")
print(myrequest.text)

curl does not retrieve url. Error (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054 [duplicate]

This question already has answers here:
Requests SSL connection timeout
(1 answer)
Strange CURL issue with a particular website SSL certificate
(1 answer)
Golang Http Get Request very slow
(2 answers)
Scraping attempts getting 403 error
(1 answer)
Closed 3 years ago.
I need to retrive this site (www.extra.com.br) in a Python project.
Neither
page = requests.get("https://www.extra.com.br/")
nor
page = requests.get("http://www.extra.com.br/")
works, the script keeps stuck on this line.
So I went to try using curl to retrieve the page and check it off-line.
curl http://www.extra.com.br also does not work. This is the output using -v.
STATE: INIT => CONNECT handle 0x6dfd90; line 1392 (connection #-5000)
Rebuilt URL to: https://www.extra.com.br/
Added connection 0. The cache now contains 1 members
STATE: CONNECT => WAITRESOLVE handle 0x6dfd90; line 1428 (connection #0) % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0* Trying 23.74.196.194...
TCP_NODELAY set
STATE: WAITRESOLVE => WAITCONNECT handle 0x6dfd90; line 1509 (connection #0)
Connected to www.extra.com.br (23.74.196.194) port 443 (#0)
STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x6dfd90; line 1561 (connection #0) 0 0 0 0 0 0 0 0 --:--:--
--:--:-- --:--:-- 0* Marked for [keep alive]: HTTP default
ALPN, offering h2
ALPN, offering http/1.1
Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
successfully set certificate verify locations:
CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt CApath: none
TLSv1.2 (OUT), TLS header, Certificate Status (22): } [5 bytes data]
TLSv1.2 (OUT), TLS handshake, Client hello (1): } [512 bytes data]
STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x6dfd90; line 1575 (connection #0) { [5 bytes data]
TLSv1.2 (IN), TLS handshake, Server hello (2): { [108 bytes data]
TLSv1.2 (IN), TLS handshake, Certificate (11): { [2799 bytes data]
TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data]
TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data]
TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data]
TLSv1.2 (OUT), TLS change cipher, Client hello (1): } [1 bytes data]
TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data]
TLSv1.2 (IN), TLS change cipher, Client hello (1): { [1 bytes data]
TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data]
SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
ALPN, server accepted to use http/1.1
Server certificate:
subject: C=BR; ST=Sao Paulo; L=Sao Paulo; O=CNOVA Comercio Eletronico S.A.; OU=TI; CN=*.extra.com.br
start date: Jul 17 00:00:00 2018 GMT
expire date: Jul 17 12:00:00 2019 GMT
subjectAltName: host "www.extra.com.br" matched cert's "*.extra.com.br"
issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
SSL certificate verify ok.
STATE: PROTOCONNECT => DO handle 0x6dfd90; line 1596 (connection #0) } [5 bytes data]
GET / HTTP/1.1
Host: www.extra.com.br
User-Agent: curl/7.58.0
Accept: /
STATE: DO => DO_DONE handle 0x6dfd90; line 1658 (connection #0)
STATE: DO_DONE => WAITPERFORM handle 0x6dfd90; line 1783 (connection #0)
STATE: WAITPERFORM => PERFORM handle 0x6dfd90; line 1799 (connection #0) 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054
Marked for [closure]: Transfer returned error
multi_done
stopped the pause stream! 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0
Closing connection 0
The cache now contains 0 members } [5 bytes data] curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054

Gunicorn Two-way SSL Error "SSL_ERROR_UNKNOWN_CA_ALERT"

I am running a Python3 app via Gunicorn with two-way SSL configured. This requires a local cert/key to verify the app as well as a ca_certs file to verify the client. Gunicorn relies on the standard ssl module in Python, particularly the wrap_socket method.
The service starts and responds to curl requests fine when I use self-signed certificates for both server and client authentication. However when I use certificates signed by another CA, I get an error SSL_ERROR_UNKNOWN_CA_ALERT.
A working setup, with self-signed certs:
# Server cert
openssl req \
-newkey rsa:2048 -nodes -keyout domain.key \
-x509 -days 365 -out domain.crt
# Client (CA) cert
openssl req \
-newkey rsa:2048 -nodes -keyout twoway.key \
-x509 -days 365 -out twoway.crt
With Gunicorn configured as follows:
keyfile = domain.key
certfile = domain.crt
cert_reqs = ssl.CERT_REQUIRED
ca_certs=twoway.crt
And curling as follows:
curl -vk --key twoway.key --cert twoway.crt https://my.service
Produces a successful response:
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* start date: Dec 7 18:35:54 2018 GMT
* expire date: Dec 7 18:35:54 2019 GMT
* issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /manage/info HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: gunicorn/19.9.0
< Date: Tue, 11 Dec 2018 18:26:19 GMT
< Connection: keep-alive
< Content-Type: application/json
< Content-Length: 73
A failing setup, with a different series of certs:
With Gunicorn configured as follows:
keyfile = my_service_key.key
certfile = my_service_cert.crt
cert_reqs = ssl.CERT_REQUIRED
ca_certs = my_trusted_clients.crt
And curling as follows:
curl -vk --key my_trusted_key.key --cert my_trusted_clients.crt https://my.service
Produces an error:
About to connect() to localhost port 5000 (#0)
Initializing NSS with certpath: sql/etc/pki/nssdb
warning: ignoring value of ssl.verifyhost
skipping SSL peer certificate verification
NSS: client certificate from file
subject: CN=mycn,OU=abc,O=def,...
NSS error -12195
Closing connection #0
SSL connect error
curl: (35) SSL connect error
Any thoughts on whether I am configuring this the wrong way? And why self-signed certs are working but other certs are not?
Note this config worked previously when using Stunnel, where I set the verify level to 4 ("Ignore the chain and only verify the peer certificate."). If there is something similar in Python I believe that would get me in the right direction.

I don't think that this is be possible as of Gunicorn 19.9.
As well as having the complete certificate chain on the server, in order to validate the client/peer certificate, I believe you will need to be able to configure the SSLContext, and especially be able to set ssl.CERT_REQUIRED when in server-mode.
Gunicorn 19.9 (and master at the time of writing) does not currently use the SSLContext-based wrapper on the connection, so this is not possible, see https://github.com/benoitc/gunicorn/issues/1140 .

FTPS upload raises SSLError after successful upload

I have a small code that uploads file to FTPS server (no, SFTP is not an option :( ). However after successful upload SSLError is raised.
from ftplib import FTP_TLS, parse227, parse229
import io
import os
import socket
import six
def ensure_binary(value):
if isinstance(value, six.text_type):
value = value.encode(encoding='utf-8')
return value
class FTP_TLS_Host(FTP_TLS):
"""FTP_TLS class that ignores host from FTP server."""
def makepasv(self):
"""Set passive command, but ignore host from ftp response."""
if self.af == socket.AF_INET:
__, port = parse227(self.sendcmd('PASV'))
host = self.host
else:
host, port = parse229(self.sendcmd('EPSV'), self.sock.getpeername())
return host, port
class FTPSClient(object):
def __init__(self, username, password, hostname, port=990):
self.username = username
self.password = password
self.hostname = hostname
self.port = port
def get_conn(self, ftp_class=FTP_TLS_Host):
conn = ftp_class(timeout=10)
conn.connect(self.hostname, self.port)
conn.login(self.username, self.password)
conn.prot_p()
return conn
def upload(self, content, filename_server):
ftps = self.get_conn()
binary = io.BytesIO(ensure_binary(content))
ftps.storbinary('STOR %s' % filename_server, binary)
ftps_client = FTPSClient(
username=os.environ['FTPS_USERNAME'],
password=os.environ['FTPS_PASSWORD'],
hostname=os.environ['FTPS_HOST'],
port=989,
)
ftps_client.upload('content', 'test.txt')
This is traceback:
File "/test.py", line 54, in <module>
ftps_client.upload('content', 'test.txt')
File "test.py", line 45, in upload
ftps.storbinary('STOR %s' % filename_server, binary)
File "/2.7/lib/python2.7/ftplib.py", line 769, in storbinary
conn.unwrap()
File "/2.7/lib/python2.7/ssl.py", line 823, in unwrap
s = self._sslobj.shutdown()
ssl.SSLError: ('The read operation timed out',)
How do I upload there without getting SSLError?

Reproduction
Make the port 21, target file name dir/test.txt. Put your code in client.py.
openssl req -new -x509 -subj "/CN=ftps-test" -nodes \
-out vsftpd.crt -keyout vsftpd.key
docker run --name=ftps-test -d -e FTP_USER=user -e FTP_PASSWORD=pass \
-v `pwd`/vsftpd.crt:/etc/ssl/certs/vsftpd.crt:ro \
-v `pwd`/vsftpd.key:/etc/ssl/private/vsftpd.key:ro \
panubo/vsftpd vsftpd /etc/vsftpd_ssl.conf
docker exec ftps-test \
bash -c 'mkdir /srv/dir && chown ftp /srv/dir'
export FTPS_USERNAME=user
export FTPS_PASSWORD=pass
export FTPS_HOST=$(docker inspect \
--format '{{ .NetworkSettings.IPAddress }}' ftps-test)
echo "ftps-test $FTPS_HOST" > hosts
export HOSTALIASES=`pwd`/hosts
curl -v --ssl-reqd ftp://ftps-test/ --cacert vsftpd.crt --user user:pass
# Python 2 works as well
python3 client.py
docker exec ftps-test cat /srv/dir/test.txt
docker stop ftps-test && docker rm -v ftps-test
Here is the output:
Generating a 2048 bit RSA private key
..............+++
.................................................................+++
writing new private key to 'vsftpd.key'
-----
fe3ae61591747e3e6acb1da6c14d3c66489f7ce5eed2b836c0463d3342b2a739
* Trying 172.17.0.2...
* Connected to ftps-test (172.17.0.2) port 21 (#0)
< 220 (vsFTPd 3.0.2)
> AUTH SSL
< 234 Proceed with negotiation.
* found 1 certificates in vsftpd.crt
* found 604 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: ftps-test (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=ftps-test
* start date: Fri, 24 Nov 2017 21:39:54 GMT
* expire date: Sun, 24 Dec 2017 21:39:54 GMT
* issuer: CN=ftps-test
* compression: NULL
* ALPN, server did not agree to a protocol
> USER user
< 331 Please specify the password.
> PASS pass
< 230 Login successful.
> PBSZ 0
< 200 PBSZ set to 0.
> PROT P
< 200 PROT now Private.
> PWD
< 257 "/"
* Entry path is '/'
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Entering Extended Passive Mode (|||4560|).
* Trying 172.17.0.2...
* Connecting to 172.17.0.2 (172.17.0.2) port 4560
* Connected to ftps-test (172.17.0.2) port 21 (#0)
> TYPE A
< 200 Switching to ASCII mode.
> LIST
< 150 Here comes the directory listing.
* Maxdownload = -1
* Doing the SSL/TLS handshake on the data stream
* found 1 certificates in vsftpd.crt
* found 604 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL re-using session ID
* SSL connection using TLS1.2 / RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: ftps-test (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=ftps-test
* start date: Fri, 24 Nov 2017 21:39:54 GMT
* expire date: Sun, 24 Dec 2017 21:39:54 GMT
* issuer: CN=ftps-test
* compression: NULL
* ALPN, server did not agree to a protocol
drwxr-xr-x 2 ftp ftp 4096 Nov 24 21:39 dir
* Remembering we are in dir ""
< 226 Directory send OK.
* Connection #0 to host ftps-test left intact
contentftps-test
ftps-test
Conclusion
The code, as such, is okay. The issue likely lies in your server's configuration.