We Keep Coding

Mac OS 11.4 Big Sur - fixing python cryptography package in /System/Library/Frameworks/Python.framework/Versions/2.7

I bought a MacBook Pro 13 M1 (16 GB RAM, 1 TB SSD) and I'm trying to get it setup for work and part of that is using python 2 (yes I know it's end of life, I still need to use it till we finish porting to go). I have some python packages I'd like to remove from my system python2 (I'm sure I must've run sudo pip install on these at some point, which I know was stupid, but what is done is done) and now that I'm on Big Sur I can't remove these packages. I tried disabling SIP until I can get them removed (csrutil disable from recovery), but now I'm getting errors that I can't remove files from a read only filesystem
sudo pip uninstall cryptography   2  14:34
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
WARNING: The directory '/Users/darren/Library/Caches/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Found existing installation: cryptography 3.3.2
Uninstalling cryptography-3.3.2:
Would remove:
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography-3.3.2-py2.7.egg-info
Proceed (y/n)? y
ERROR: Exception:
Traceback (most recent call last):
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
status = self.run(options, args)
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/commands/uninstall.py", line 90, in run
auto_confirm=options.yes, verbose=self.verbosity > 0,
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/req/req_install.py", line 686, in uninstall
uninstalled_pathset.remove(auto_confirm, verbose)
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/req/req_uninstall.py", line 403, in remove
moved.stash(path)
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/req/req_uninstall.py", line 292, in stash
renames(path, new_path)
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/utils/misc.py", line 355, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 323, in move
rmtree(src)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 270, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 275, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 273, in rmtree
os.remove(fullname)
OSError: [Errno 30] Read-only file system: '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/_der.py'
but I can't remount root as read/write and get this error when I try:
sudo mount -uw /
mount_apfs: volume could not be mounted: Permission denied
mount: / failed with 66
How do I remove these files from /System/Library if it's read only? Do I have to format my system and start from scratch now that mac os put these files in a read only filesystem or is there a way to remove them?
Edit:
So I have broken packages in /System/Library that are preventing fabric from running so I can't get our stuff up and running. The main one being the cryptography package. So I either need to fix these packages so fabric can run or remove them or something.
The other part that makes this such a pain is that I'm trying to run python 2 on an M1 Mac which means either using the built in python or using rosetta with an x86 python2 as I have found no method for installing arm python 2 on an M1 Mac (there isn't an installer from python.org or brew since python2 was end of life when the M1 Macs shipped).
Edit2:
Maybe these packages that I'm having issues with did ship with Mac OS as the package's folder's created timestamp is "Jan 1 01:00:00 2020" just like every other file & folder in that same folder. If so I guess I need to fix them somehow. When I try to run fab from the command line I get this error in the cryptography package:
fab
/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
from cryptography.hazmat.backends import default_backend
Traceback (most recent call last):
File "/Users/darren/Library/Python/2.7/bin/fab", line 5, in <module>
from fabric.main import program
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/fabric/__init__.py", line 3, in <module>
from .connection import Config, Connection
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/fabric/connection.py", line 16, in <module>
from paramiko.agent import AgentRequestHandler
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/__init__.py", line 22, in <module>
from paramiko.transport import SecurityOptions, Transport
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py", line 129, in <module>
class Transport(threading.Thread, ClosingContextManager):
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py", line 190, in Transport
if KexCurve25519.is_available():
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/kex_curve25519.py", line 30, in is_available
X25519PrivateKey.generate()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/primitives/asymmetric/x25519.py", line 39, in generate
from cryptography.hazmat.backends.openssl.backend import backend
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
from cryptography.hazmat.backends.openssl.backend import backend
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/backends/openssl/backend.py", line 117, in <module>
from cryptography.hazmat.bindings.openssl import binding
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: dlopen(/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/bindings/_openssl.so, 2): Symbol not found: _DTLS_client_method
Referenced from: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/bindings/_openssl.so
Expected in: flat namespace
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py", line 120, in _join_lingering_threads
for thr in _active_threads:
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py", line 120, in _join_lingering_threads
for thr in _active_threads:
TypeError: 'NoneType' object is not iterable
Has anyone else had this problem or have a fix for it?

I was able to install python2 for Apple Silicon/M1 from nix and then I had to manually install pip which let me install cryptography & cffi (I had libffi installed from brew). Here are the steps I used, I'm not sure if they're all necessary, but it's what I did:
Install Nix
see https://nixos.org/manual/nix/stable/#sect-macos-installation for more details, but if you have a mac with a T2 chip or M1 this will still be encrypted
sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume --daemon
Install python2 from nix
nix-env -i python-2.7.18
verify that you're using this python in a new terminal session (you'll need to start a new shell to get the updated path)
run which python
if it doesn't return something like /Users/user/.nix-profile/bin/python you'll want to add ~/.nix-profile/bin to the front of your path, start a new shell, and run which python again
Install pip
See https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py for more details, but if you curl the get-pip.py in this url and run it with python 2 it'll tell you to instead download and use the get-pip.py from https://bootstrap.pypa.io/pip/2.7/get-pip.py
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py -o get-pip.py
python ./get-pip.py
Use brew to install libiff and openssl#1.1
I had done this step previously and I'm not 100% sure it's necessary, but I believe it is
brew install libiff openssl#1.1
Add the following to your .bashrc, .zshrc, .profile, or wherever you set your environment variables:
export LDFLAGS="-L$(brew --prefix openssl#1.1)/lib"
export CFLAGS="-I$(brew --prefix openssl#1.1)/include"
use pip to install cryptography
pip install cryptography
After this I had a functional python with cryptography installed and I was then able to install fabric (via pip install fabric==1.14.1 since 1.14.1 is the version I need for this old project, use what you need)
Edit:
Thanks to #CharlesDuffy for all of his help and as he correctly mentions in the comments bellow, you could (and probably should) install all of the dependencies and cryptography from nix instead of using a combination of brew and nix. I had brew and the dependancies installed before I had Nix installed so I haven't tested it out, but it should work.

error with installing virtualenv with python 3

ive updated to python3 and downloaded virtualenv using:
sudo /usr/bin/easy_install virtualenv
when i go to start the virtualenv i got the following error message :
virtualenv project1
Traceback (most recent call last):
File "/usr/local/bin/virtualenv", line 6, in <module>
from pkg_resources import load_entry_point
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 3241, in <module>
#_call_aside
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 3225, in _call_aside
f(*args, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 585, in _build_master
return cls._build_from_requirements(__requires__)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'zipp>=0.4' distribution was not found and is required by importlib-resources
i looked around and realised the that 'zipp' had not been installed so i installed that also. when i went to run the virtualenv again i got the same error message again as above. and for some reason it keeps referencing python 2.7 even though ive upgraded to python3.

please try this ..
you just follow Exactly the bellow Items :
Open your Command Prompt AS ADMINISTRATOR -> right click on CMD and RUN AS ADMINISTRATOR
go to which Folder to you want to create your VirtualENV : For example : > Cd C:\Users...\Documents\python\src
type it : PIP install Virtualenv
make sure your internet connection is Ok
4.type : python -m venv env
and then you can create your project inside this env folder..
all Comment code should be write in Command Prompt AS Administrator

i ended up using the built in python 3 venv package. it works just as well and it was straight forward and simple. thank you for all the help
start the virtual environment: python3 -m venv project1
activate the virtual environment: source project1/bin/activate
deactivate the virtual environment: deactivate

If you're using Python 3.3 or later, you can use the built-in venv [https://docs.python.org/3/library/venv.html] module. This will reduce the number of dependencies, making installation and troubleshooting a bit easier.
To double-check that you are invoking the correct Python interpreter, run python3 -V or python -V (depending on how it's aliased).

Just try out.
virtualenv venv
It should create a venv directory in same directory if python 3.x is being used.

Python 2.7 - JavaError when using grammar-check 1.3.1 library

I am trying to use the grammar-check 1.3.1 library for Python
I have installed it using the pip commands
$ pip install --upgrade 3to2
$ pip install --upgrade language-check
I also unzipped the LanguageTool file from this link and moved it to C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check
Then I tried to run a test program
import grammar_check as gc
with open("file.txt") as f:
tool = gc.LanguageTool('en-US')
matches = tool.check(f.read())
print len(matches)
But I got this error
File "lang-grammar-checker.py", line 22, in main
tool = gc.LanguageTool(predicted)
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 190, in __init__
self._start_server_on_free_port()
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 318, in _start_server_on_free_port
cls._start_server()
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 330, in _start_server
server_cmd = get_server_cmd(cls._port)
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 541, in get_server_cmd
java_path, jar_path = get_jar_info()
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 554, in get_jar_info
raise JavaError(u"can't find Java")
grammar_check.JavaError: can't find Java
I also have Java8 installed in my computer. How can I solve this error?

The problem is that your system can't see your java. You can check if your system "knows" java by typing:
java -version
in command prompt (without going to your java location!).
You can do it adding to your environment variable PATH your java development kit location.
How do I set or change the PATH system variable?

Failed to install pip for pypy on Ubuntu

I cannot install pip for pypy 4.0.1 on Ubuntu 15.10.
I downloaded pypy 4.0.1 from http://pypy.org/download.html. Created a symlink for it. And used wget to download get-pip.py from https://bootstrap.pypa.io/get-pip.py.
But when I ran sudo pypy get-pip.py, I got exception:
Collecting pip
Exception:
Traceback (most recent call last):
File "/tmp/tmps2kjCI/pip.zip/pip/basecommand.py", line 211, in main
status = self.run(options, args)
File "/tmp/tmps2kjCI/pip.zip/pip/commands/install.py", line 294, in run
requirement_set.prepare_files(finder)
File "/tmp/tmps2kjCI/pip.zip/pip/req/req_set.py", line 334, in prepare_files
functools.partial(self._prepare_file, finder))
File "/tmp/tmps2kjCI/pip.zip/pip/req/req_set.py", line 321, in _walk_req_to_install
more_reqs = handler(req_to_install)
File "/opt/pypy-4.0.1-linux64/lib_pypy/_functools.py", line 42, in __call__
return self._func(*(self._args + fargs), **fkeywords)
File "/tmp/tmps2kjCI/pip.zip/pip/req/req_set.py", line 491, in _prepare_file
session=self.session)
File "/tmp/tmps2kjCI/pip.zip/pip/download.py", line 825, in unpack_url
session,
File "/tmp/tmps2kjCI/pip.zip/pip/download.py", line 673, in unpack_http_url
from_path, content_type = _download_http_url(link, session, temp_dir)
File "/tmp/tmps2kjCI/pip.zip/pip/download.py", line 885, in _download_http_url
with open(file_path, 'wb') as content_file:
TypeError: expected string, got NoneType object
I could run sudo python get-pip.py without problem but failed with pypy. I also tried Ubuntu 12.04 and 14.04 but no luck.
I have Python 2.7.10 installed and I'm using a Linode machine. But changed to a DigitalOcean one didn't help.

How about installing PyPy from the Ubuntu archives sudo apt install pypy. Or if you want the latest version, from the PyPy PPA
Then if you create a virtualenv, it will have pip inside it:
$ sudo apt-get install virtualenv
$ virtualenv -p pypy ve
$ source ./ve/bin/activate
$ pip install ...

I had the same issue, and the solution is easy, yo have to run this command:
export LC_ALL=C.UTF-8
And then run the pip installation.
You should add this line to your .bashrc or .bash_profile in the server, even you can add it to /etc/environment to have it in a global way. (If someone has a better solution, please tell me)
Edit (2016-10):
Above is the global and lazy solution for any other distribution.
If you're in ubuntu, you need to reconfigure the locales and you can do it with:
sudo locale-gen "en_US.UTF-8"
sudo dpkg-reconfigure locales

PyCharm's Python console fails when PyPy and iPython co-exist in a virtualenv created project

Some version info:
OS: Mac OS X 10.8.4
Python: 2.7.2 (that came with the Mac OS X)
PyPy: 2.0.2
iPython: 0.13.2
PyCharm: 2.5.1
PyCharm's Python console fails when PyPy and iPython co-exist in a virtualenv created project.
I created three virtual environment projects. I did not activate any of the virtual environments. So, the system python still exists as it is.
Env - 1:
$ virtualenv -p /path/to/system/installed/python /path/to/sample_1/virtualenv/project
$ /path/to/sample_1/virtualenv/project/bin/pip install ipython
Now, when I run the python console in PyCharm it works fine. (Interpreter in PyCharm is set to /path/to/sample_1/virtualenv/project/bin/python)
Env - 2:
$ virtualenv -p /path/to/pypy /path/to/sample_2/virtualenv/project
Now, when I run the python console in PyCharm it works fine. (Interpreter in PyCharm is set to /path/to/sample_2/virtualenv/project/bin/python)
Env - 3:
$ virtualenv -p /path/to/pypy /path/to/sample_3/virtualenv/project
$ /path/to/sample_3/virtualenv/project/bin/pip install ipython
Now, when I run the python console in PyCharm it fails with the following error. (Interpreter in PyCharm is set to /path/to/sample_3/virtualenv/project/bin/python)
/path/to/sample_3/virtualenv/project/bin/pypy -u /Applications/PyCharm.app/helpers/pydev/pydevconsole.py 60355 60356
PyDev console: using IPython 0.13.2
Traceback (most recent call last):
File "app_main.py", line 72, in run_toplevel
File "/Applications/PyCharm.app/helpers/pydev/pydevconsole.py", line 361, in <module>
StartServer(pydev_localhost.get_localhost(), int(port), int(client_port))
File "/Applications/PyCharm.app/helpers/pydev/pydevconsole.py", line 288, in StartServer
interpreter = InterpreterInterface(host, client_port, threading.currentThread())
File "/Applications/PyCharm.app/helpers/pydev/pydev_ipython_console.py", line 37, in __init__
self.interpreter = PyDevFrontEnd()
File "/Applications/PyCharm.app/helpers/pydev/pydev_ipython_console_011.py", line 35, in __init__
shell = TerminalInteractiveShell.instance()
File "/path/to/sample_3/virtualenv/project/site-packages/IPython/config/configurable.py", line 318, in instance
inst = cls(*args, **kwargs)
File "/path/to/sample_3/virtualenv/project/site-packages/IPython/frontend/terminal/interactiveshell.py", line 360, in __init__
user_module=user_module, custom_exceptions=custom_exceptions
File "/path/to/sample_3/virtualenv/project/site-packages/IPython/core/interactiveshell.py", line 454, in __init__
self.init_readline()
File "/path/to/sample_3/virtualenv/project/site-packages/IPython/core/interactiveshell.py", line 1843, in init_readline
self.refill_readline_hist()
File "/path/to/sample_3/virtualenv/project/site-packages/IPython/core/interactiveshell.py", line 1851, in refill_readline_hist
self.readline.clear_history()
File "/tmp/pypy-2.0.2/lib_pypy/pyrepl/readline.py", line 284, in clear_history
del self.get_reader().history[:]
File "/tmp/pypy-2.0.2/lib_pypy/pyrepl/readline.py", line 188, in get_reader
console = UnixConsole(self.f_in, self.f_out, encoding=ENCODING)
File "/tmp/pypy-2.0.2/lib_pypy/pyrepl/unix_console.py", line 98, in __init__
curses.setupterm(term, self.output_fd)
error: setupterm: could not find terminfo database
Process finished with exit code 1
Please be mindful that bin/python in any of the virtual environments points to the installed interpreter. For example, /path/to/sample_3/virtualenv/project/bin/python points to pypy and /path/to/sample_1/virtualenv/project/bin/python points to system installed python.
Seems like this is related to this. But, I am not sure.
Please help.

Looking at the actual Exception message, this may be a potential solution:
https://support.enthought.com/entries/22438744-Linux-IPython-in-terminal-error-setupterm-could-not-find-terminfo-database-
Meanwhile, you should be able to work around the problem by setting
export TERM="xterm-256color" in the shell from which you execute ipython,
or in your~/.bashrc file, which you then source.