The problem
I am having trouble getting my docs to build successfully when clicking "Build" on the readthedocs.io web interface, but it builds just fine on my local machine. To test that it is an environment issue, I created a virtual environment:
conda create virtualenv -n venv
conda env export -n venv
source activate venv
Then I installed my requirements.txt file as:
pip install -r requirements.txt
and then ran
make clean html
In the virtual env and on buildthedocs online, I get the error:
Notebook error:
PandocMissing in ex_degassing.ipynb:
Pandoc wasn't found.
Please check that pandoc is installed:
http://pandoc.org/installing.html
make: *** [html] Error 2
I have searched and searched for a solution, but my best guess is that pandoc is not being installed via pip even though it is in the requirements.txt file. I tried also telling it to build from source by replacing pandoc with git+git://github.com/jgm/pandoc#egg=pandoc in my requirements.txt file, but this did not work (see below for what my files look like). I can easily install pandoc on my local machine, but it fails to install via the requirements.txt file in my virtual environment or on readthedocs.
Some of my files
Here is my requirements.txt file:
sphinx>=1.4
sphinx_rtd_theme
ipykernel
nbsphinx
pandas
pandoc
numpy
matplotlib
ipynb
scipy
I read through all of the readthedocs documentation and, ironically, found the documentation for building woefully inadequate for someone of my limited coding ability. I tried making both readthedocs.yml and environment.yaml files, but I have no idea if they are doing anything or if they are properly written. For completeness, here are those files:
readthedocs.yml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/conf.py
python:
version: 3.7
install:
- requirements: docs/requirements.txt
environment.yaml
channels:
- conda-forge
dependencies:
- python =3.7
- pandoc
In summary
I have created beautiful documentation with readthedocs and sphinx, but I cannot get it to build on the web. How can I either: a) solve the issue of installing pandoc (is that the issue??) or b) remove the need for pandoc (e.g., is there a way to include jupyter notebook files in my docs without needing pandoc? How to do this?)?
Thanks so much for anyone's advice!
It's currently not very well explained on the PyPI page for pandocs, but pandoc is actually written in Haskell, not Python. The pandoc package for python which you can install with pip install pandoc doesn't come with an executable copy of pandoc, it is just a wrapper to provide bindings to pandoc within python. It does this by making shell calls to your system copy of pandoc, and so you need to separately install a copy of the actual pandoc executable to your system in order for the python bindings to have something to bind to!
For example, on Debian/Ubuntu, you will need to install a system copy of pandoc using apt in addition to installing the python bindings with pip.
sudo apt install pandoc
pip install pandoc
Though you can do this on your local system, this isn't a viable solution for getting it to work on readthedocs.
In contrast to this, conda tries to make packages work irrespective of things installed on the rest of your system. Hence the conda pandoc package ships with an actual copy of pandoc, which it uses instead of trying to use your system copy of pandoc. If you're using conda, there is only one command you need to run to get pandoc working within python:
conda install pandoc
If you're using a conda environment for your readthedocs specification (as is the situation for OP), you can just add pandoc to the conda section instead of the pip section of the env.
The pandoc package on PyPI which you can install with pip isn't actually an official pandoc package for python. There's also pypandoc and py-pandoc, which are equally unofficial.
Supposedly py-pandoc is a one-stop shop for installing pandoc through a python interface. I think you just have to do pip install py-pandoc or conda install pandoc and it works immediately, though I'm not sure due to the lack of documentation.
For myself, I'm using Sphinx to build documentation. My personal solution to the problem of installing pandoc when building python documentation (including on readthedocs) is to add the following code block to conf.py.
from inspect import getsourcefile
# Get path to directory containing this file, conf.py.
DOCS_DIRECTORY = os.path.dirname(os.path.abspath(getsourcefile(lambda: 0)))
def ensure_pandoc_installed(_):
import pypandoc
# Download pandoc if necessary. If pandoc is already installed and on
# the PATH, the installed version will be used. Otherwise, we will
# download a copy of pandoc into docs/bin/ and add that to our PATH.
pandoc_dir = os.path.join(DOCS_DIRECTORY, "bin")
# Add dir containing pandoc binary to the PATH environment variable
if pandoc_dir not in os.environ["PATH"].split(os.pathsep):
os.environ["PATH"] += os.pathsep + pandoc_dir
pypandoc.ensure_pandoc_installed(
quiet=True,
targetfolder=pandoc_dir,
delete_installer=True,
)
def setup(app):
app.connect("builder-inited", ensure_pandoc_installed)
with pypandoc appearing in the requirements.txt file
This tells Sphinx to check to see if pandoc is already on the system path, and if it isn't available then it will download and install a copy to the docs/bin/pandoc directory of the repository and add it to PATH.
I faced the same issue.
Notebook error:
PandocMissing in getting_started.ipynb:
Pandoc wasn't found.
Please check that pandoc is installed:
https://pandoc.org/installing.html
It seems there is a difference whether you install pandoc via pip or if you use conda.
Try the following:
pip uninstall pandoc
conda install pandoc
Using the conda version solved the issue on my machine.
On a different front, if you are using poetry, you may also come across the same error message, i.e. PandocMissing.
Windows:
Install Pandoc via its Windows installer: https://pandoc.org/installing.html.
Linux (tested with Ubuntu 2204):
sudo apt install pandoc
first
pip uninstall pandoc
next,
conda install pandoc
just do it in Anaconda prompt, (restart kernel if necessary), and done!
This can be done by the non-Conda, pip fanboys.
I finally hit the wall and relented to allow Miniconda to play a small role in managing packages for my python3.1 on a Windows PC. I could not get Jupyter-book to convert a markdown file to a mySt markdown file. And the issue was with pandoc.
But I will always remain determined to continue controlling my python's packages myself using pip. I will not give control to Anaconda, and I will continue to control python's packages myself.
With that long qualifier out of the way, I will explain what I did.
I uninstalled pandocs using pip:
pip uninstall pandocs
And then I installed miniConda: https://docs.conda.io/en/latest/miniconda.html
There was a Windows Installer for my version of python, python3.10. It downloads into the Windows download folder.
Years ago, I fought Anaconda for control over my python and will not allow MiniConda any similar control. During the installation of Miniconda, I unselected two default checkboxes which would have given Miniconda control over vsCode, etc... And also did not accept the default installation destination and instead installed Miniconda to a new folder:
C:\miniConda
Then needed to add a path to Miniconda by adding a new system variable. That was done by Editing System Properties - Environmental Variables.
C:\MiniConda\Scripts\conda.exe
Close the cmd prompt and then open again. At the folder holding the .md file, open the cmd prompt. The smallish sized cmd prompt is preferred over the large Terminal window. Run conda and install pandocs.
conda install pandoc
Finally, convert the .md to myst.
jupyter-book myst init myBook.md --kernel ir
It works. Miniconda is kept under control and the pandoc package worked. And the process created a mySt markdown file.
And for those who don't read instructions carefully, they will miss this mention that I'm using the r kernel rather than the python kernel.
Well, I got it working on my system:
download and install pandoc from https://pandoc.org/installing.html
start a new session (cmd or power shell)
call sphinx make again
Simple as that. Hope you can get it working!
Related
I need to install python3.7-dev and libspatialindex-dev before setting up another private package in my Windows computer.
I have the instructions how to install in Linux (for example, see below):
sudo apt-get install -y libspatialindex-dev
There are alternative methods that I have found with conda in Windows (see below):
conda install -c conda-forge libspatialindex=1.9.3
I know that you can only get Linux commands like "sudo apt" via WSL or Chocolatey or like a linux virtual machine etc. (In Git for Windows Bash, how to install "get-apt" and "sudo" and all those basic commands?, https://superuser.com/questions/947220/how-to-install-packages-apt-get-install-in-windows) but I have also read that Windows doesn't require these '*.dev' packages because it comes with the alternative installation method as you see above. (How to get python-dev for windows?)
Can someone explain to me whether I should set up WSL, Chocolatey etc. or whether the alternative method is okay? If the alternative method isn't right, which out of linux package installing software should I select as the easiest to set up? Thanks in advance.
Setting up WSL will most likely not get you to be able to set up a build workflow on windows.
Getting the dev version of a library means having the necessary header files additionally to the compiled libraries needed for linking to these libraries. To achieve this will be different for the specific library you are looking for.
Specific to your question however, since you mentioned that you have conda, the python headers should already reside in the \include folder in your anaconda folder and python.dll should be in the main folder.
The command
conda install -c conda-forge libspatialindex=1.9.3
installs the headers you need under Library\include in your anaconda folder. So by setting the include dirs in your build pipeline accordingly, you should be able to compile your program
When creating an executable file using PyInstaller, how can I bundle a specific Python interpreter and some library from specific versions?
I want to create an executable file from my .py script. My __main__.py downloads some data from a website and transform them, then saves them in a folder as excel files.
And here are requirements.
Use Python 3.6
For this, I found some related posts but never clear to me. My main.py needs Python3.6 to be executed, and the users of this .exe file won't necessarily have Python3.6, so I want to bundle this Python in the .exe file as well. But is just a single command python3 -m pyinstaller __main__.py really enough? I'm not confident and I want to know what exactly I need to do.
Use specific versions of libraries such as pandas==0.23, beautifulsoup4==4.9.1 etc.
I think I need to work with pure in .spec file as the documentation says "pure: pure python modules needed by the scripts". But I cannot find any documentation about what exactly I need to do make it include pandas==0.23, beautifulsoup4==4.9.1 etc.
Any help would be appreciated!
I do this using virtual environments: if you have an environment with the desired Python version and package versions, you can pip install PyInstaller there and build the .EXE from that environment.
The Conda version would be:
Create the environment:
conda create --name my_env_name python=3.6
Activate it:
conda activate my_env_name
Install your desired package versions and PyInstaller:
pip install pandas==0.23
pip install beautifulsoup4==4.9.1
pip install pyinstaller
# or to get the developer version, which fixes some issues I have run into
# pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
And then build your EXE:
python pyinstaller myscript.py
Having this virtual environment can be a nice control as well for writing/testing your program, as it ensures no other dependencies are needed. Because of this, I tend to only install the necessary packages for the script into the environment.
However, I am not sure how one would do this with venv, if you do not have Anaconda. But maybe this will still point you in a direction (or someone else can elaborate? also see this post).
At work we have python installed, but no additional modules. I want to import some scipy modules but I have no access to the python directory for installation.
Similar questions have been asked on StackOverflow, but the answers always assumed easy install, pip or virtualenv were installed. At my workplace, none of these packages are installed. It's just the plain python installation and nothing else.
Is there still an option for me for installing modules in my local folder and calling them from python? If so, how do I go about it?
Not exactly installing modules on your local folder, but a solution nonetheless:
I used to work for a company that used windows and didn't have admin access, so I ended up using Portable python.
It seems portable python is no longer mantained, but you can see some other portable python solutions on their site, most of which you can run straight from your usb.
You can download pip from here http://pip.readthedocs.org/en/stable/installing/ and install it without root privileges by typing:
python get-pip.py --user
This will install to directory with prefix $HOME/.local so the pip executable will be in the directory $HOME/.local/bin/pip, for your convenience you can add this directory to $PATH by adding to end of .bashrc file this string
export PATH=$HOME/.local/bin/:$PATH
After this you can install any packages by typing
pip install package --user
Or you can alternatively compile the python distribution from source code and install to your home directory to directory $HOME/.local or $HOME/opt or any subfolder of $HOME you prefer, let's call this path $PREFIX. For doing this you have to download python source code from official site, unpack it and then run
./configure --prefix=$PREFIX --enable-shared
make install
And then add python binary to $PATH, python libraries to $LD_LIBRARY_PATH, by adding to the end of $HOME/.bashrc file whit strings
export PATH=$PREFIX/bin:$PATH
export LD_LIBRARY_PATH=$PREFIX/lib
and when after restarting bash you can also run
python get-pip.py
and pip and will be installed automatically to your $PREFIX directory. And all other packages those you will install with pip will be automatically installed also to $PREFIX directory. This way is more involved, but it allows you to have the last version of python.
I have scikit-learn 0.16.1 installed on Ubuntu 14.04 and am working through the tutorial. SKL was installed with all default configuration. The tutorial states
The source of this tutorial can be found within your scikit-learn folder:
scikit-learn/doc/tutorial/text_analytics/
I've used find on my entire drive and there is no "tutorial" folder. Not anywhere. Anybody know where these files are really installed?
Finding the package contents
Where the packages are installed depends on how you installed scikit-learn
If you used Ubuntu's package system via
sudo apt-get install python-sklearn python-sklearn-doc
(you often need the doc package to get the accompanying documentation), then the tutorial is simply missing. The doc/-folder it is not contained in the python-sklearn-doc-package. See the bug report.
You can find out the contents of the package via
dpkg-query --listfiles python-sklearn-doc
If you used the Python Package Index to install it via
pip install --user --install-option="--prefix=" -U scikit-learn
, then the installation should be at $HOME/.local/lib/python2.7/site-packages/sklearn. (as also of pip show -f scikit-learn) But a
find . | grep -i tutorial
did not find any tutorial/-folder.
If you installed it from source, consider reinstalling via pip, as the warning states that
Warning
Packages installed with the python setup.py install command cannot be
uninstalled nor upgraded by pip later. To properly uninstall
scikit-learn in that case it is necessary to delete the sklearn folder
from your Python site-packages directory.
Solution
A solution would be to use the source. Either download the master file or do it via git:
git clone https://github.com/scikit-learn/scikit-learn.git
The git archive is more than 60 MiB, so you might want to prefer the master zip.
We use Sphinx for documentation within our Linux/OSX python environment, but we have someone on Windows who would like to be able to get an environment in which they can edit and build the Sphinx documentation.
Is it possible to set up Sphinx on windows? I'm guessing the hard part is installing a python environment and all the dependencies, but I have 0 experience with python on windows so somebody might be able to point me to a straightforward way to proceed.
For installing python on windows download Python. Run the executable and you are all set.
If you have pip installed then you can open command prompt and just type
pip install -U Sphinx
If you don't have pip installed then you first need to install it by using the following at the command prompt
python -m pip install -U pip
You can use pip-win now to set up sphinx in a virtualenv on windows easily.
First, download and install Python. If you want to use the latexpdf builder you will also need to install either TeX Live or MiKTeX for windows.
Next, run the pip-win.exe. It will automatically pull the required setuptools required and install pip and virtualenv on your system. When loaded it will pull up the Python you just installed, but if you installed it in a location other than default directory you can easily enter the absolute path or browse for it.
Next, in the Command: line you can create a new virtualenv by inputing:
venv -c C:\foo\sphinx-venv
This will open a command prompt with you inside your virtualenv. Following this you can install sphinx with the pip command:
<sphinx-venv> C:\foo\> pip install sphinx
Now sphinx is installed on your windows machine in a virtual enviroment. You can use deactivate to escape your new environment.
When ever you wish to enter your virtualenv again to build more sphinx projects you can either open pip-win and run:
venv C:\foo\sphinx-venv
Or you can create a custom python file to activate your sphinx environment and build your sphinx file such as:
import os
os.system("cmd /c \"CALL C:\\foo\\sphinx-venv\\Scripts\\activate.bat && make html\"")
By far the simplest way to use Python on Windows is WinPython, which is a portable distribution that is as simple to use as any other portable app. It'll give you an icon for a WinPython command prompt from which you can use pip, etc., as if on any other (e.g. linux) system, or you can download packages and install them using WinPython's control panel.