We Keep Coding

Running pip install twice to see changes ("developer mode") -- second install fails but first works

I am wondering how to use pip to develop a Python package which is going through many revisions rapidly. My work flow is to write C++ code, compile and install with pip install and test my code.
Then, I would like to change some underlying C++ code, recompile and reinstall with pip, test the new feature, change something else, go back etc. until my package is ready.
Why did pip install ./cmake_example work well the first time but when making changes to the code, reinstalling with recompiling produced an error? I just re-ran the command pip install ./cmake_example.
I changed a single line of C++ code in an innocuous way (adding +1 in the 'add' function just to see if I can change code and recompile) and the code compiled fine in my IDE without pip.
My basic idea was to use pip following this method to avoid having to hackishly insert my shared object into some python directory each time I make a change.
I used the cmake_example from pybind here and followed the steps and did pip install ./cmake_example and it worked very well. I ran the example fine in a Python console.
Then, I changed some code (just added +1 to the adding function), so nothing substantial and wanted to re-install the package.
I then got this error:
Building wheels for collected packages: cmake-example
Building wheel for cmake-example (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for cmake-example (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [65 lines of output]
running bdist_wheel
running build
running build_ext
CMake Error at CMakeLists.txt:2 (project):
Running
'/tmp/pip-build-env-q_8_pyjk/overlay/bin/ninja' '--version'
failed with:
No such file or directory
and
ERROR: Could not build wheels for cmake-example, which is required to
install pyproject.toml-based projects
I have tried pip uninstall cmake_example and then reinstall but to no avail. The upgrade function did not work either. Does pip change something in the project folder?
PS: The makers of pybind11 also provide a scikit example here where removing the cache for repeated builds is not necessary. The cmake_example seems to be intended for legacy projects that do not use the cmake extension scikit. So if the structure of the cmake_example is essential to the project, removal of the cache is the only way to go.

I found that deleting the build directory inside the cmake_example directory resolved the problem and pip install ./cmake_example worked again as it did the first time. You can combine the two commands:
rm -rf ./cmake_example/build && pip install ./cmake_example
Looking a little closer, (for me) it was sufficient to delete cmake_example/temp.linux-x86_64-3.8/CMakeCache.txt. I suspect the lines
//Program used to build from build.ninja files.
CMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-yqopewjn/overlay/bin/ninja
inside CMakeCache.txt mean that cmake caches the path to the ninja executable, but the second time around the temporary path is different, so it can no longer be found where cmake expects it.

Python/PyCharm, new to Python package and syntax issues [duplicate]

This question already has answers here:
How do I use installed packages in PyCharm?
(14 answers)
Closed 1 year ago.
I am extremely new to python and have been learning basics via DataCamp.
I downloaded Python 3 and PyCharm to use it with and all was going okay with me replicating what i was learning in DataCamp into PyCharm.
I am now trying to install packages into PyCharm and the standard way (demonstrated in Data Camp) isnt working
I have opened a new .pynfile and typed 'pip install Scrapy' but everything is underlined red (below). When i hover over the 'install' part it says that the 'Statement seems to have no effect'
pip install Scrapy with red squiggly lines
Am i doing this wrong.
Secondly, I have tried to install the Scrapy package by going to settings and have faced a bunch of different errors. i finally downloaded most of what i think i needed, including the 'twisted' package (i had to download this manually). But PyCharm again isnt letting me install this package.
I read online it needed to be downloaded using this method python -m pip install C:\Users%USER%\Downloads\Twisted-17.9.0-cp36-cp36m-win32.whl but again this didnt work. I even opened up the Python.exe and ran this, replacing the file name with the correct path and it errored saying invalid syntax.
Please help !
Updated to add pic of cmd error
pic of error in cmd]2
Updated to add PyCharm Error
command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit code 2
----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\%USER%\AppData\Local\Programs\Python\Python39\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\%USER%\\AppData\\Local\\Temp\\pip-install-20efmwoj\\twisted_efc7d2a70fcd40639034d265eeb0e045\\setup.py'"'"'; __file__='"'"'C:\\Users\\%USER%\\AppData\\Local\\Temp\\pip-install-20efmwoj\\twisted_efc7d2a70fcd40639034d265eeb0e045\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\%USER%\AppData\Local\Temp\pip-record-abp6_zhs\install-record.txt' --single-version-externally-managed --user --prefix= --compile --install-headers 'C:\Users\%USER%\AppData\Roaming\Python\Python39\Include\Twisted' Check the logs for full command output.

PyCharm uses his own package manager. Open File->Settings->Project->Python Interpreter and search the "+" button on the right. In the new window that just appeared, you can search and install any package you want.

PIP is Python's package installer. You can use PIP by running it from the command line (CMD, so outside of PyCharm).
Opening the CMD:
Windows button + R (on windows)
Enter cmd and then press enter. Now you can execute the following command:
pip install scrapy
which will install the scrapy package
Probably PyCharm can immediately use it, but restarting PyCharm performs magic!

As I stated in a comment, you need to add python to PATH. You can do this following this tutorial: https://www.makeuseof.com/python-windows-path/#:~:text=How%20to%20Manually%20Add%20Python%20to%20the%20Windows,the%20PATH%20With%20the%20System%20Variables%20Option.%20. Then run pip install scrapy in the command prompt and that will work. Make sure you restart command prompt.

Installing PyAudio on Windows

Installing PyAudio on Windows 10 error
I want to install PyAudio
I have Windows 10, Python 3.7, pip 8.1
I'm installing it in Command Prompt with command pip install PyAudio
But there's an error error: command 'D:\\VisualStudio2017\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x86\\cl.exe' failed with exit status 2
Command "c:\python37\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\USER\\AppData\\Local\\Temp\\pip-install-is1aobj6\\pyaudio\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\USER\AppData\Local\Temp\pip-record-50bh4kz0\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\USER\AppData\Local\Temp\pip-install-is1aobj6\pyaudio\
Here's a
screenshot
Thanks for reading my question. I hope you can help :D

Well, I've had just this problem when I wanted to install pylisten (which has PyAudio as a dependency).
There are two problems here - first, your MSVC can't find portaudio.h and after that there will be a problem with the min macro when building. But first things first.
Get portaudio
This is a C dependency for PyAudio, so you need to have include files, in particular portaudio.h like you see on the screenshot. It's possible to get binaries for Windows, but if your using Anaconda, the simplest way to do it is to install it using conda:
(base) C:\> conda search portaudio
Loading channels: done
# Name Version Build Channel
portaudio 19.6.0 h0c8e037_3 pkgs/main
portaudio 19.6.0 hfa6e2cd_3 pkgs/main
(base) C:\> conda install portaudio
(...)
Next, we would like to know where the files from portaudio library actually went. There is no conda package --list command... (see here: https://github.com/conda/conda/pull/7457), so we need to do some magic found elsewhere on SO:
(base) C:\>dir %CONDA_PREFIX%\conda-meta\portaudio*
(...)
26.01.2020 13:17 5 577 portaudio-19.6.0-hfa6e2cd_3.json
We need to check this JSON and find the files property:
"extracted_package_dir": "C:\\Users\\(...)\\Anaconda3\\pkgs\\portaudio-19.6.0-hfa6e2cd_3",
"features": "",
"files": [
"Library/bin/portaudio.dll",
"Library/include/pa_asio.h",
"Library/include/pa_jack.h",
"Library/include/pa_linux_alsa.h",
"Library/include/pa_mac_core.h",
"Library/include/pa_win_ds.h",
"Library/include/pa_win_wasapi.h",
"Library/include/pa_win_waveformat.h",
"Library/include/pa_win_wdmks.h",
"Library/include/pa_win_wmme.h",
"Library/include/portaudio.h",
"Library/lib/portaudio.lib",
"Library/lib/portaudio_static.lib",
".nonadmin"
],
The extracted_package_dir (username removed above) actually amounts to:
%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3.
So, for this specific build, the necessary include directories (-I) are:
%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3\Library\include
And link directories (-L) are:
%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3\Library\lib
This will be useful below.
Simple way, which doesn't work
Now if your Conda environment has MSVC 14.0 for building native extensions you should be fine with:
(base) C:\> pip install --global-option=build_ext --global-option="-I%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3\Library\include" --global-option="-L%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3\Library\lib" pyaudio
Now, it will find portaudio.h!
But alas, at the time of writing, this fails with a compile error. I still put this here, because maybe the incompatibility with MSVC will be fixed.
The hard, but successful, way
We need to hack the sources of PyAudio. This can be done like so:
(base) C:\>pip download pyaudio
This will download a file like PyAudio-0.2.11.tar.gz (version number may differ in the future). You need to unpack it, using tar. I have it in C:\Windows\system32 - maybe because of WSL. You can also use 7-zip to unpack it. Anyway:
(base) C:\>tar xf PyAudio-0.2.11.tar.gz
(... unpacked ...)
(base) C:\>cd PyAudio-0.2.11
(base) C:\PyAudio-0.2.11>
Now to get the same result as before, you need to pass the special arguments to setup.py (in a different way than it's done for pip, of course):
(base) C:\PyAudio-0.2.11>python setup.py build_ext --include-dirs="%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3\Library\include" --library-dirs="%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3\Library\lib" install
(compile errors: min macro redefinition, __typeof__ undefined)
Now, edit the file src\_portaudiomodule.c, and delete/comment out these lines (right at the top, line ~39):
#define min(a, b) \
({ \
__typeof__(a) _a = (a); \
__typeof__(b) _b = (b); \
_a < _b ? _a : _b; \
})
This trivial macro is the culprit. If you want to submit a fix to Mr Pham (http://people.csail.mit.edu/hubert/pyaudio/), don't delete it, but use something like
#ifndef _MSC_VER ... #endif.
Save the file and run this (again):
(base) C:\PyAudio-0.2.11>python setup.py build_ext --include-dirs="%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3\Library\include" --library-dirs="%CONDA_PREFIX%\pkgs\portaudio-19.6.0-hfa6e2cd_3\Library\lib" install
Everything should be fine now.
EDIT: TL;DR easy way
After checking once again, I saw pyaudio in https://repo.anaconda.com/pkgs/main/win-64. This means, if you're using Anaconda you can just do:
conda install pyaudio
I'm leaving the rest of this answer for educational purposes.

how to install spacy for python 3.4, windows 7?

I am trying to install the spacy python library for python 3.4 on windows 7. i keep getting install errors (the important-looking error messages are below).
I installed it for 2.7, but I can't seem to install it for 3.4.
I tried doing:
pip3.4 install -U spacy
But it does not seem to work. Does anybody have any ideas? I'm trying to avoid crazy system surgery. There must be some simple way to do this.
Warning: The nvcc binary could not be located in your $PATH. For GPU capability, either add it to your path, or set $CUDA_HOME
warning: no files found matching '*.cpp' under directory 'include'
warning: no files found matching 'buildbot.json'
***********************************************************
WARNING: One of the following dependencies is unresolved:
ctypes >=1.0.1
readline >=1.7.1
***********************************************************
error: Unable to find vcvarsall.bat
Cleaning up...
Command C:\Python34\python.exe -c "import setuptools, tokenize;file='C:\Users\joe\AppData\Local\Temp\pip_build_joe\spacy\setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record C:\Users\joe\AppData\Local\Temp\pip-sxqy1vof-record\install-record.txt --single-version-externally-managed --compile failed with error code 1 in C:\Users\joe\AppData\Local\Temp\pip_build_joe\spacy
Storing debug log for failure in C:\Users\joe\pip\pip.log

In order for python to detect it you must place it in the following directory.
C:\Python27\
or where ever python is located.

Force `setup.py` to use setuptools

I'm using this code:
code = 'import setuptools;__file__={0!r};execfile(__file__)'.format(os.path.join(path, 'setup.py'))
args = ['install', '--single-version-externally-managed']
subprocess.check_call([sys.executable, '-c', code, args])
To execute a setup.py and install the package. The problem occurs when setup.py uses distutils instead of setuptools: --single-version-externally-managed is not recognized by distutils.
How can I force setup.py to use setuptools?

What you have written is basically what pip does. Based on the code you wrote, you will be using setuptools' setup function because you've imported from setuptools. Setuptools paves over Distutils' setup function in its __init__.py. Therefore, it doesn't mater if the setup.py script imports distutils or not. Setuptools will always win...
If for some reason you still have issues while running your command. Try compiling the file before execution. exec(compile(...)) rather than execfile(...)
In response to #jknair answer... I'd also discourage the use of ez_setup.py, because it's code duplication, has unexpected behavior and is often excluded during package distribution (which makes it hard for tools like pip to run the setup.py without an ImportError).