Potentially devious behaviour of pygments´ mapping.py / Editing a pygments lexer - python

I'm using Windows 7, latest windows update etc.
Recently I decided to start using the minted package in LaTeX. For this, I was redirected to install pygments, which I installed through pip:
I'll run you through quickly what I did (to make sure I didn't do anything wrong):
Installed python 3.4 from this page.
easy_install pip in the windows command prompt.
pip install pygments in command prompt.
I've added the %PYTHONPATH% as well as %PYTHONPATH%\Scripts (which is where pygments operates from I've been led to believe), to the system PATH environment.
I'm typing source code from R into LaTeX, but the releases I could find do not offer the capital R syntax. If I use LaTeX code as such:
\documentclass{article}
\usepackage{minted}
\begin{document}
\begin{minted}{R} % Capital R won't function as intended with recent release
c(1,2,5,7,8)
\end{minted}
\end{document}
It won't recognise the capital R argument in minted.
Now, the lower case \begin{minted}{r} works just fine, so I've looked around on how to fix this, and found this answer.
Here it is suggested to adjust the math.py lexer (at %PYTHONPATH%\Lib\site-packages\pygments\lexers) to add an alias 'R' to current aliases. It also suggests to run _mapping.py in the same directory (%PYTHONPATH%\Lib\site-packages\pygments\lexers). Running _mapping.py has two effects:
It completely empties the file, the file size is reduced to 0 bytes and editing it with IDLE confirms an empty file.
It also yields this error message.
Not running _mapping.py won't apply my new alias to pygments. Running it, breaks pygments entirely. The error code is shown below.
I ran the LaTeX code again (with changes and _mapping.py also run), with pdfLaTeX, with arguments: $synctexoption, --enable-write18, -interaction=nonstopmode and $fullname (also in this order), I get the following error in LaTeX:
LaTeX Error: File `temp.out.pyg' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: out.pyg)
Enter file name:
! Emergency stop.
<read *>
l.64 \end{minted}^^M
After running _mapping.py on a clean installation even (no changes made to any lexer or anything), I also run into the same problem. I'm uncertain whether the initial changes I made, changed registry values and whatnot, but I'm having serious issues applying this simple change to the pygments lexer.

Related

Installed Python 3.10.2 and now my Pip installs are not being found

Everything in my code was working perfectly fine up until I decided I wanted to utilize match-case in Python. Find out its only a thing in 3.10+, so I quickly install it and change it to be the interpreter in command palette.
Then I try to run my code same as before, and I'm not sure what changed but my Keyboard import is giving me 'Import "keyboard" could not be resolved'. Issues. The same issue was actually present as well with the 'from nis import match' module.
I installed keyboard initially using 'pip install keyboard' when running my 3.9 version, and legit everything was fine. This all started after I installed 3.10 (which I did from Pythons website yes); and I did add to PATH, but I dont think that would have any impact on my imports in VScode.
Confused as heck right now, please look at my screenshots for clearest explanation of what I'm facing.
Import could not be resolved
.
Module Not Found Error
.
pip3.10 show keyboard
.
My interpreter list from command palette
Any and all help appreciated, I'm extremely confused and think I've tried it all now
New Python versions use new site-packages folders. You need to reinstall everything. This is why poetry, pipenv, or requirements.txt are used
I think the linter-like extension/whatever Pylance is is the problem. Why? Because as we've tried, pip install keyboard and pip install --upgrade keyboard just confirm that the requirement is satisfied and the latest version of keyboard is already installed.
What I did was change Pylance's settings. Yes, it's not an actual error (with the code), but Pylance's problem. It just didn't see enough files in the keyboard package to satisfy its hunger, or Microsoft was just too lazy to pass more checks and update it accordingly, because however it happened, when I tried changing the setting to display an "error" at "missing imports" for fun, it didn't show errors or the default "warning" but "none", because I had run the file before I changed it back to display an error (maybe Pylance observed that when the file was run, there was no ModuleNotFoundError, and so stopped showing the warning).
I better shut now, because the setting I changed/overrode by adding it in settings.json is:
"python.analysis.diagnosticSeverityOverrides": {"reportMissingImports":"none"}
which is in:
{
...,
...,
...,
"python.analysis.diagnosticSeverityOverrides": {"reportMissingImports":"none"}
}
Here is a list of keys like "reportMissingImports" whose values you can change. These are the allowed values for the keys in python.analysis.diagnosticSeverityOverrides:
error (red squiggle)
warning (yellow squiggle)
information (blue squiggle)
none (disables the rule)
You would want to change their values only if you want to change their behaviour.
You can find settings.json by its path or by going to the settings GUI and clicking on any "Edit in settings.json" link-like button. You can also make settings.json open instead of the GUI by default.
Now, you won't see any more warnings when you import modules and can code without being anxious about the stupid warning.
Wow, So as it turns out, a solution I tried previously, to no avail is now working to solve this bizarre issue.
Simply put, I once again went to Command Palette (Ctrl+Shft+P), and looked through my interpreters I had to see what could be wrong.
Decided to click on the 'recommended' option I assumed I had been running this whole time. Turns out I was using this 'third' option as shown in the screenshot (the one not highlighted ofcourse); and it's the reason my module installs were being found on my machine, but not by the interpreter; as such giving me errors and not running the module for the program.
Simple error, but thanks to those who did help.

Programmatically check if wheel is compatible with Python installation

Is it possible to programmatically check if a wheel (whl) is compatible with the chosen Python installation before attempting to install?
I'm making an automated packages installer (packages needed for my Python project to work), and I need to only attempt to install compatible pkgs, so if there are errors, I know they are only from the compatible modules and I should see what happened (not errors also from incompatible pkgs, which I wouldn't care). Example: I'd have wheels for Python 3.5 and 3.7, and in a 3.5 installation, 3.7 wheels could not be tried to be installed.
I've tried pkginfo (https://pypi.org/project/pkginfo/), but on wheel.supported_platforms, it returns an empty array and I can't do anything with that (a wheel with "any" or with "win32" on their name in the platform part, returned an empty array, so I can't use that, it seems).
Also tried the output from python -m pip debug --verbose, but the following appears:
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without no
tice.
This makes the command not possible to use, even though bellow that it prints the "Compatible tags", which more or less I could use to determine if a wheel is supported or not from its name. Example of those "Compatible tags" in a Python array:
['cp39-cp39-win_amd64', 'cp39-abi3-win_amd64', 'cp39-none-win_amd64', 'cp38-abi3-win_amd64', 'cp37-abi3-win_amd64', 'cp36-abi3-win_amd64', 'cp35-abi3-win_amd64', 'cp34-abi3-win_amd64', 'cp
33-abi3-win_amd64', 'cp32-abi3-win_amd64', 'py39-none-win_amd64', 'py3-none-win_amd64', 'py38-none-win_amd64', 'py37-none-win_amd64', 'py36-none-win_amd64', 'py35-none-win_amd64', 'py34-no
ne-win_amd64', 'py33-none-win_amd64', 'py32-none-win_amd64', 'py31-none-win_amd64', 'py30-none-win_amd64', 'cp39-none-any', 'py39-none-any', 'py3-none-any', 'py38-none-any', 'py37-none-any
', 'py36-none-any', 'py35-none-any', 'py34-none-any', 'py33-none-any', 'py32-none-any', 'py31-none-any', 'py30-none-any']
With, for example, "pyHook-1.5.1-cp36-cp36m-win32.whl", I could check the name and see if it's compatible or not (except because of the warning above...).
Any other ideas?
Thanks in advance for any help!
EDIT: I could go manually and pull things from the name and hard-code the some possibilities I see on documentation, like "win32" and "win_amd64" (as I did before), but then I'd need to know exactly all the possibilities that the parts of the name can have (I saw a cool expression on the documentation: "e.g." - which means there are more than the mentioned things) and have a lot of work on that. I was hoping there was already someone that had made such thing (maybe even Python itself has some way in any of its internal packages).
You can do this using packaging.
pip install packaging
An example code to get the tags similar to how you got from pip would be:
from packaging.tags import sys_tags
tags = sys_tags()
print([str(tag) for tag in tags])
# ['cp39-cp39-manylinux_2_33_x86_64', 'cp39-cp39-manylinux_2_32_x86_64', 'cp39-cp39-manylinux_2_31_x86_64', ..... , 'py31-none-any', 'py30-none-any']
Of course, you can do much more things programmatically with the above variable tags:
>>> tags = sys_tags()
>>> for tag in list(tags)[:3]:
... print(tag.interpreter, tag.abi, tag.platform)
...
cp39 cp39 manylinux_2_33_x86_64
cp39 cp39 manylinux_2_32_x86_64
cp39 cp39 manylinux_2_31_x86_64
For more in-depth documentation, check: https://packaging.pypa.io/en/latest/tags.html#packaging.tags.sys_tags

How to ensure that README.rst is valid?

There are two version of my little tool:
https://pypi.python.org/pypi/tbzuploader/2017.11.0
https://pypi.python.org/pypi/tbzuploader/2017.12.0 Bug: The pypi page looks ugly.
In the last update a change in README.rst cases a warning:
user#host> rst2html.py README.rst > /tmp/foo.html
README.rst:18: (WARNING/2) Inline emphasis start-string without end-string.
README.rst:18: (WARNING/2) Inline emphasis start-string without end-string.
Now the pypi page looks ugly :-(
I use this recipe to do CI, bumpversion, upload to pypi: https://github.com/guettli/github-travis-bumpversion-pypi
How could I ensure that no broken README.rst gets released any more? With other words I want to avoid that the pypi page looks ugly.
Dear detail lovers: Please don't look into the current particular error in the README.rst. That's is not the question :-)
Update
As of Sep 21, 2018, the Python Packaging Authority recommends an alternative command twine check. To install twine:
pip install twine
twine check dist/*
Note that twine requires readme_renderer. You could still use readme_renderer, and you only need to install twine if you want its other features, which is a good idea anyway if you are releasing to PyPI.
From the official Python packaging docs, Uploading your Project to PyPI:
Tip: The reStructuredText parser used on PyPI is not Sphinx! Furthermore, to ensure safety of all users, certain kinds of URLs and directives are forbidden or stripped out (e.g., the .. raw:: directive). Before trying to upload your distribution, you should check to see if your brief / long descriptions provided in setup.py are valid. You can do this by following the instructions for the pypa/readme_renderer tool.
And from that tool's README.rst:
To check your long description's locally simply install the readme_renderer library using:
$ pip install readme_renderer
$ python setup.py check -r -s
Preamble
I had a readme which would not render on PyPi, other than the first element on the page (an image). I ran the file against multiple validators, and tested it against other renders. It worked perfectly fine everywhere else! So, after a long, nasty fight with it, and numerous version bumps so I could test a PyPi revision, I tried reducing the file to a bare minimum, from which I'd build it back up. It turned out that the first line was always processed, and then nothing else was...
Solution
Discovering this clue regarding the first line, I then had an epiphany... All I had to do was change the line endings in the file! I was editing the file in Windows, with Windows line endings being tacked on implicitly. I changed that to Unix style and (poof!) PyPi fully rendered the doc!
Rant...
I've encountered such things in the past, but I took it for granted that PyPi would handle cross platform issues like this. I mean one of the key features of Python is being cross platform! Am I the first person working in Windows to encounter this?! I don't appreciate the hours of time this wasted.
You could try if rstcheck catches the type of error in your readme. If it does, run it after pytest in your script section. (and add it in your requirements ofc).

eclipse pydev - how to install python modules

Just working my way through a (very good) book call Test Driven Development using Python.
This makes use of Python3.4 by the way. By the way, I am running in a Windows 7 OS.
I've got all the stuff working using a simple text editor and running from the command line... in the course of which in particular I used "pip install" to install Django and Selenium, as per book's instructions.
This created folders "selenium" and "django" under ...\Python34\Lib\site-packages\ ... so I added these to the PythonPath for my Eclipse/PyDev project.
With the correct interpreter selected I then tried to run a file which runs fine on the command line: "> python3 functional_tests.py"... but I get
File "D:\apps\Python34\lib\site-packages\django\http\__init__.py", line 1, in <module>
from django.http.cookie import SimpleCookie, parse_cookie
File "D:\apps\Python34\lib\site-packages\django\http\cookie.py", line 5, in <module>
from django.utils.six.moves import http_cookies
ImportError: cannot import name 'http_cookies'
... to me this looks like a dependency thing... as though "pip install" handles dependency matters in a way just including a single folder doesn't.
Question boils down to this: what's the "proper" way to install a python module using PyDev?
several days later
wow... nothing? Nothing! I suppose this must mean that you either have to add dependencies manually or use something like Ant, Maven or Gradle within Eclipse itself. These latter are not my strong areas, even outside an IDE. Would still be nice to have an answer from a PyDev expert!
Well, pip install should work for PyDev (it should automatically recognize the dependency)...
I.e.: in your use case, the only folder that should be in the PYTHONPATH is D:\apps\Python34\lib\site-packages (and pip should install packages to that folder -- make sure you don't add extra folders for "D:\apps\Python34\lib\site-packages\django" nor anything else inside the site-packages to the PYTHONPATH).
If it's still not working, please check if the module django.utils.six.moves.http_cookies is indeed where you expect it to be. Also, you can print the PYTHONPATH being used in runtime with:
import sys
print('\n'.join(sorted(sys.path)))
To check if that's really what you expect.

ipython up and down arrow strange behaviour

In my installation of ipython I have this strange problem where I cannot reliably move through command history with up and down arrows... a lot of the time it just doesn't work (nothing happens on the key press). Also sometimes writing normal characters at the end of the command just doesn't work.
My system: Mac OSX Lion
I have readline installed...
thank you for the help!
david
Make sure you installed readline before ipython.
sudo pip uninstall ipython
sudo pip install readline ipython
(I know this question is a few months old, but for future reference)
I had to install readline with easy_install readline and that fixed it.
Using pip install readline did not work for me, and ipython gave a warning:
******************************************************************************
libedit detected - readline will not be well behaved, including but not limited to:
* crashes on tab completion
* incorrect history navigation
* corrupting long-lines
* failure to wrap or indent lines properly
It is highly recommended that you install readline, which is easy_installable:
easy_install readline
Note that `pip install readline` generally DOES NOT WORK, because
it installs to site-packages, which come *after* lib-dynload in sys.path,
where readline is located. It must be `easy_install readline`, or to a custom
location on your PYTHONPATH (even --user comes after lib-dyload).
******************************************************************************
Following trouble in iPython and up-&-down arrows to access history, and browsing this post, a simple solution (turn off "Scroll lock") turned out to work for me.
This is an intentional feature of IPython. If you type "abc" and then hit the up arrow, it's going to scroll only through lines that start with "abc". If you hit lift/right while you're scrolling, it triggers the same behavior. The entire contents of the current line are interpreted as your search prefix, any only lines starting with all that will show up on further up/down keypresses.
You can change this behavior in your PYTHONSTARTUP file. I have the following lines:
import readline
# Prevent ctrl-p/ctrl-n/Up/Down from doing prefix searching
readline.parse_and_bind('"\\C-p": previous-history')
readline.parse_and_bind('"\\C-n": next-history')
readline.parse_and_bind('"\\e[A": previous-history')
readline.parse_and_bind('"\\e[B": next-history')
If you're curious, here are the bindings in IPython's source code that we're overriding.
Unrelated, but I also like to to override readline's default ctrl-w:
# Ctrl-W behavior more like Vim
readline.parse_and_bind('"\\C-w": backward-kill-word')

Categories