We Keep Coding

Python in AWS environment - Can't make pip install works

I'm trying to use pip install <package> but it always returns a timeout error as follows:
I'm using Python 3.8 and pip 20.2.1 in Windows Server 2019.
I assume that it's a problem related to pip can't reach pypl.org site.
I checked proxy configuration and it is as follows, i.e., there's no proxy configuration:
What I think is strange is that I can access the site using Chrome and download it manually. I can't find anything that is blocking pip access.
What can that be?
Thanks!

For someone that is facing the same problem, I needed to set a proxy in AWS instance. Now its working like a charm.
I also created two global environment variables
HTTP_PROXY = http://proxyIP:proxyPORT
HTTPS_PROXY = https://proxyIP:proxyPORT
Hope this is usefull for someone else.
Thanks!

SleekXMPP: "Certificate has expired."

I'm currently trying to use the sleekxmpp module in Python3.5 to connect to jabber.at, an XMPP service. Jabber.at's SSL certificates are issued by Let's Encrypt.
The problem I'm having is, when I try to log in to my jabber.at account using sleekxmpp, I get the following error:
INFO Negotiating TLS
INFO Using SSL version: TLSv1
ERROR Certificate has expired.
I've tried to resolve this issue on Ubuntu 16.04, Manjaro GNOME 17, and Windows 10, with a fresh installation of Python, sleekxmpp, and all of sleekxmpp's dependencies, on each of those machines. It didn't make a difference.
On each of my machines, I have the Let's Encrypt Authority X3 certificate installed, yet sleekxmpp still thinks the certificate is expired, even though the certificate is valid until 2022. Frustratingly, I'm able to log in to my XMPP account using Pidgin, and the certificate automatically saved by Pidgin (which you can find under ~/.purple/ in Linux) is exactly the same as the one I downloaded directly from Let's Encrypt.
I've tried manually changing the ssl version in sleekxmpp.xmlstream.xmlstream.py:125 to ssl.PROTOCOL_SSLv23, but that didn't work, either. I've also tried manually specifying the path to the Let's Encrypt .crt file in sleekxmpp.xmlstream.xmlstream.py:140, but I still have the same issue.
I should note that I do have dnspython, pyasn1, and pyasn1_modules installed on each machine. While I can successfully log into my jabber.at account by either uninstalling pyasn1, or explicitly ignoring certificates by setting sleekxmpp.xmlstream.xmlstream.py:140 to None, that isn't a solution in my case - I must encrypt my connection.
Is there anything I can do? I've been at this for days now, and I haven't been able to find any helpful solutions on Google.
I appreciate any help!

After over a month, I was able to find a solution to my own problem.
It turns out that this issue is caused by a regression in SleekXMPP, introduced in version 1.3.3:
GitHub: Regression from 1.3.1
It was only recently addressed, but as of now, there is no official release with the fix. The short-term solution to this problem is to uninstall SleekXMPP, and re-install SleekXMPP, specifying version 1.3.1:
sudo pip3 uninstall sleekxmpp
sudo pip3 install sleekxmpp==1.3.1
Reverting to version 1.3.1 solves this issue for me. Perhaps this will no longer be an issue in SleekXMPP 1.3.4.

How to add a custom CA Root certificate to the CA Store used by pip in Windows?

I just installed Python3 from python.org and am having trouble installing packages with pip. By design, there is a man-in-the-middle packet inspection appliance on the network here that inspects all packets (ssl included) by resigning all ssl connections with its own certificate. Part of the GPO pushes the custom root certificate into the Windows Keystore.
When using Java, if I need to access any external https sites, I need to manually update the cacerts in the JVM to trust the Self-Signed CA certificate.
How do I accomplish that for python? Right now, when I try to install packages using pip, understandably, I get wonderful [SSL: CERTIFICATE_VERIFY_FAILED] errors.
I realize I can ignore them using the --trusted-host parameter, but I don't want to do that for every package I'm trying to install.
Is there a way to update the CA Certificate store that python uses?

Self-Signed Certificate Authorities pip / conda
After extensively documenting a similar problem with Git (How can I make git accept a self signed certificate?), here we are again behind a corporate firewall with a proxy giving us a MitM "attack" that we should trust and:
NEVER disable all SSL verification!
This creates a bad security culture. Don't be that person.
UPDATE 2022 Python 3.10
Can experimentally use the system truststore.
https://pip.pypa.io/en/latest/user_guide/#using-system-trust-stores-for-verifying-https
# Requires Python 3.10 or later
python --version
Python 3.10.4
# Install the 'truststore' package from PyPI
python -m pip install truststore
# Use '--use-feature=truststore' flag to enable
python -m pip install SomePackage --use-feature=truststore
tl;dr: Configuring your own trust store CA bundle
pip config set global.cert path/to/ca-bundle.crt
pip config list
conda config --set ssl_verify path/to/ca-bundle.crt
conda config --show ssl_verify
# Bonus while we are here...
git config --global http.sslVerify true
git config --global http.sslCAInfo path/to/ca-bundle.crt
But where do we get ca-bundle.crt?
Get an up to date CA Bundle
cURL publishes an extract of the Certificate Authorities bundled with Mozilla Firefox
https://curl.haxx.se/docs/caextract.html
Direct Download
SHA256
I recommend you open up this cacert.pem file in a text editor as we will need to add our self-signed CA to this file.
Certificates are a document complying with X.509 but they can be encoded to disk a few ways. The below article is a good read but the short version is that we are dealing with the base64 encoding which is often called PEM in the file extensions. You will see it has the format:
----BEGIN CERTIFICATE----
....
base64 encoded binary data
....
----END CERTIFICATE----
https://support.ssl.com/Knowledgebase/Article/View/19/0/der-vs-crt-vs-cer-vs-pem-certificates-and-how-to-convert-them
Getting our Self Signed Certificate
Below are a few options on how to get our self signed certificate:
Via OpenSSL CLI
Via Browser
Via Python Scripting
Get our Self-Signed Certificate by OpenSSL CLI
https://unix.stackexchange.com/questions/451207/how-to-trust-self-signed-certificate-in-curl-command-line/468360#468360
echo quit | openssl s_client -showcerts -servername "curl.haxx.se" -connect curl.haxx.se:443 > cacert.pem
Get our Self-Signed Certificate Authority via Browser
Acquiring your CA: https://stackoverflow.com/a/50486128/622276
http://blog.majcica.com/2016/12/27/installing-self-signed-certificates-into-git-cert-store/
Thanks to this answer and the linked blog, it shows steps (on Windows) how to view the certificate and then copy to file using the base64 PEM encoding option.
Copy the contents of this exported file and paste it at the end of your cacerts.pem file.
For consistency rename this file cacerts.pem --> ca-bundle.crt and place it somewhere easy like:
# Windows
%USERPROFILE%\certs\ca-bundle.crt
# Linux/macOS
$HOME/certs/cabundle.crt
Get our Self-Signed Certificate Authority via Python
Thanks to all the brilliant answers in:
How to get response SSL certificate from requests in python?
I have put together the following to attempt to take it a step further.
https://github.com/neozenith/get-ca-py
Finally
Set the configuration in pip and conda so that it knows where this CA store resides with our extra self-signed CA.
# Windows
pip config set global.cert %USERPROFILE%\certs\ca-bundle.crt
conda config --set ssl_verify %USERPROFILE%\certs\ca-bundle.crt
OR
# Linux / macOS
pip config set global.cert $HOME/certs/ca-bundle.crt
conda config --set ssl_verify $HOME/certs/ca-bundle.crt
THEN
pip config list
conda config --show ssl_verify
# Hot tip: use -v to show where your pip config file is...
pip config list -v
# Example output for macOS and homebrew installed python
For variant 'global', will try loading '/Library/Application Support/pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/pip.conf'
Troubleshooting
Based on a great comment below
I've tried this and still get a SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')) error. Any suggestions?
here is a troubleshooting guide:
This is the normal error message when the certificates authorities are not yet correctly setup.
It could be a variety of factors to check:
The paths to your ca-bundle.crt have the right path separators for your OS (it has stung me),
you may not have the latest CAs to verify normal certificates,
you may not have added your CA in the right encoding.
Python is effectively doing those 3 steps:
Find my CA store,
read all entries,
look up this certificate against my trust store.
If any of those fail you get this error message from experience.
Check this answer linked from below to display and check your ssl_cert_dir using:
python -c "import ssl; print(ssl.get_default_verify_paths())"
References
Pip SSL: https://pip.pypa.io/en/stable/user_guide/#configuration
Conda SSL: https://stackoverflow.com/a/35804869/622276
Acquiring your CA: https://stackoverflow.com/a/50486128/622276
http://blog.majcica.com/2016/12/27/installing-self-signed-certificates-into-git-cert-store/
Using Python to automatically grab your Peer CA: How to get response SSL certificate from requests in python?

Run: python -c "import ssl; print(ssl.get_default_verify_paths())" to check the current paths which are used to verify the certificate. Add your company's root certificate to one of those.
The path openssl_capath_env points to the environment variable: SSL_CERT_DIR.
If SSL_CERT_DIR doesn't exist, you will need to create it and point it to a valid folder within your filesystem. You can then add your certificate to this folder to use it.

Alternative solution on Windows is to install python-certifi-win32 that will allow Python to use Windows Certificate Store.
pip install python-certifi-win32

Not best answer but you can reuse an already created ca bundle using --cert option of pip, for instance:
pip install SQLAlchemy==1.1.15 --cert="C:\Users\myUser\certificates\my_ca-bundle.crt"

On Windows, I solved it by creating a pip.ini file in %APPDATA%\pip\
e.g. C:\Users\asmith\AppData\Roaming\pip\pip.ini
In the pip.ini I put the path to my certificate:
[global]
cert=C:\Users\asmith\SSL\teco-ca.crt
https://pip.pypa.io/en/stable/user_guide/#configuration has more information about the configuration file.

I think nt86's solution is the most appropriate because it leverages the underlying Windows infrastructure (certificate store).
But it doesn't explain how to install python-certifi-win32 to start with since pip is non functional.
The trick is to use --trustedhost to install python-certifi-win32 and then after that, pip will automatically use the windows certificate store to load the certificate used by the proxy.
So in a nutshell, you should do:
pip install python-certifi-win32 -trustedhost pypi.org
and after that you should be good to go

In my scenario, on Ubuntu using python 3.8, I wanted to see what I could change in my local system (rather than code or a installing a new package) that would allow me to utilize my custom CA and my self-signed certs.
So I just traced where python was looking:
strace python my_prog 2> out.txt
Then grepped the output for pem (you may need to look for "crt" "cert")
grep -i pem out.txt
Even though certifi wasn't ever explicitely installed, the file it was looking for was:
/path/to/virtualenv/for/project/lib/python3.8/site-packages/certifi/cacert.pem
So I renamed the old file and linked it to my system's ca bundle (e.g., /etc/ssl/certs/ca-certificates.crt)
And now it worked.

Open Anaconda Navigator.
Go to File\Preferences.
Enable SSL verification Disable (not recommended)
or Enable and indicate SSL certificate path(Optional)
Update a package to a specific version:
Select Install on Top-Right
Select package click on tick
Mark for update
Mark for specific version installation
Click Apply

__init__() got an unexpected keyword argument 'mime' in python/django

The first line of this code in .py file returns an error.
mime = magic.Magic(mime=True)
content_type = mime.from_buffer((data).read(1024))
request.session['content_type'] = content_type
if content_type == 'application/pdf' or content_type == 'application/msword':
request.session['upload_status'] = "Content type is valid according to (MAGIC)"
The error message is
__init__() got an unexpected keyword argument 'mime'
I'm using Django 1.4.1 and Python 2.7.3. I have Magic installed. No clue whats going wrong- any help is appreciated!

Just a stab in the dark but The documentation would suggest that you shouldn't be instantiating the Magic class directly.
import magic
magic.from_buffer(open("testdata/test.pdf").read(1024))

The Magic object's constructor does not accept an argument named 'mime'. I would suggest looking at the doc string with help(magic.Magic); it may give you a clue.

As per suggestions above, if connection to JIRA server fails with error:
The error message is __init__() got an unexpected keyword argument 'mime'
Then go edit
/usr/lib/python2.7/site-packages/jira/client.py
Replace
self._magic = magic.Magic(mime=True)
with
self._magic = magic
Then run code like this:
from jira.client import JIRA
import magic
...
jira = JIRA(options={'server':'https://jira.server.goes.here'}, basic_auth=(options.username, options.password))
I'm using python 2.7.3 with jira-python (http://jira-python.readthedocs.org/en/latest/)

You most likely have another version of magic. If I remember correctly from the last time I looked this stuff up, there's a different version that comes with Ubuntu (maybe, there's also something called filemagic). The same thing is the case on the out-of-the-box version of Cygwin. That was my case.
I had reinstalled Cygwin and ran into this same issue - two versions of magic / python-magic / filemagic for Python. I looked back here on SO for the solution, but didn't find it quickly. Luckily, I kept notes from before, and the solution that worked for me was:
$ sudo pip3 uninstall filemagic
$ sudo pip3 install python-magic
OR, what worked better on my Cygwin installation:
$ python -m pip uninstall filemagic
$ python -m pip install python-magic
That fixed the problem for me.
As I searched for more on this problem, I came across a very similar solution here (archived), on github. There is an additional step.
uninstall filemagic:
sudo pip3 uninstall filemagic
uninstall python-magic:
sudo pip3 uninstall python-magic
(Possible addition)
sudo apt-get uninstall python-magic
install python-magic:
sudo pip3 install python-magic
With a quick search, I couldn't find details of the two versions of magic. I just found some comments on threads saying, "You must have the other version of magic", or "You must have a different version of magic".
Edit
I've updated this answer with the magic sources. From this, it seems that you might have to do another command before installing python-magic with pip
sudo apt-get uninstall python-magic
as noted above.
Edit
I found the details about the different versions of magic that are floating around.
I have the following three links ( first , second , third )
Basically, there are two versions going around. The one which has the most support is on PYPI and (it seems to me) is used much more often. It is v.0.4.15 currently (2020-02-19), and its github page is the first link. At that page, you can read the following:
Name Conflict
There are, sadly, two libraries which use the module name magic. Both have been around for quite a while. If you are using this module and get an error using a method like open, your code is expecting the other one. Hopefully one day these will be reconciled.
The second version, which seems to cause the most problems, is currently (2020-02-19) v5.25. According to the third link and my own research, this one gets installed when one uses sudo apt-get install python-magic on certain versions of Ubuntu. (Look here for some possible details from Ubuntu.)
The best explanation of all this is found in the second link. Props to #mhawke for explaining things so well.
Here are archived versions of the links above: archived first, archived second, archived third, archived Ubuntu information.

PIP and SlikSVN under a proxy

I am trying to checkout a subversion repository using PIP, since I am under a proxy I am calling PIP using the --proxy parameter:
pip install svn+http://django-compress.googlecode.com/svn/trunk/ --proxy=myproxy:8080
While PIP itself works fine under the proxy, it seen that its not passing the proxy parameter to the SVN client:
Downloading/unpacking svn+http://django-compress.googlecode.com/svn/trunk/
Checking out http://django-compress.googlecode.com/svn/trunk/ to c:\users\canassa\appdata\local\temp\pip-x_w9ct-build
svn: OPTIONS of 'http://django-compress.googlecode.com/svn/trunk': could not connect to server (http://django-compress.googlecode.com)
Complete output from command "C:\Program Files\SlikSvn\bin\svn.exe" checkout -q http://django-compress.googlecode.com/svn/trunk/ c:\users\canassa\appdata\local\temp\pip-x_w9ct-build:
----------------------------------------
Command "C:\Program Files\SlikSvn\bin\svn.exe" checkout -q http://django-compress.googlecode.com/svn/trunk/ c:\users\canassa\appdata\local\temp\pip-x_w9ct-build failed with error code 1
Storing complete log in C:\Users\canassa\AppData\Roaming\pip\pip.log
To make things worst I am under a Windows environment and using the SilkSVN client. I also have to switch off the proxy a few times every day since its a notebook and I have to connect it with the wireless, which don't uses a proxy.
My question is if there is a way to handle the SlikSvn proxy under PIP and Windows. Preferably making easy to switch it off when I need.

It is an open bug[1] and there is not solution to this problem right now.
If you find an answer, please, comment in pip's issue tracker.
[1] - https://github.com/pypa/pip/issues/227