TLDR
I'm noticing a significant difference in the information presented by the official python docs compared to what I'm seeing in the PyCharm hover-over / quickdocs. I'm hoping someone can point me to where I can find the source of this quickdoc information such that I can use it outside of PyCharm as a general reference.
For example in the python docs for os.makedir I see:
os.makedirs(name, mode=0o777, exist_ok=False)
Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory.
The mode parameter is passed to mkdir() for creating the leaf directory; see the mkdir() description for how it is interpreted. To set the file permission bits of any newly created parent directories you can set the umask before invoking makedirs(). The file permission bits of existing parent directories are not changed.
If exist_ok is False (the default), an FileExistsError is raised if the target directory already exists.
Note
makedirs() will become confused if the path elements to create include pardir (eg. “..” on UNIX systems).
This function handles UNC paths correctly.
Raises an auditing event os.mkdir with arguments path, mode, dir_fd.
New in version 3.2: The exist_ok parameter.
Changed in version 3.4.1: Before Python 3.4.1, if exist_ok was True and the directory existed, makedirs() would still raise an error if mode did not match the mode of the existing directory. Since this behavior was impossible to implement safely, it was removed in Python 3.4.1. See bpo-21082.
Changed in version 3.6: Accepts a path-like object.
Changed in version 3.7: The mode argument no longer affects the file permission bits of newly created intermediate-level directories.
But in the quickdocs I see:
os def makedirs(name: str | bytes | PathLike[str] | PathLike[bytes],
mode: int = ...,
exist_ok: bool = ...) -> None
makedirs(name [, mode=0o777][, exist_ok=False]) Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist. If the target directory already exists, raise an OSError if exist_ok is False. Otherwise no exception is raised. This is recursive.
Where is this quickdoc type hinting information coming from and where can I find a complete reference with all these type hints such that I can reference it outside of PyCharm?
Background
Coming mainly from a strongly typed language like Java, I struggle to make constructive use of the python documentation with regards to function input parameter types. I am hoping someone can elucidate a standard process for resolving ambiguity compared to my current trail+[lots of]errors approach.
For example, the os.makedir function's first parameter is name.
os.makedirs(name, mode=0o777, exist_ok=False)
It is not apparent to me what sorts of things I can pass as name here. Could it be:
A str? If so, how should I create this? Via a string literal, double quoted string? Does this accept / separators or \ separators or system dependent?
A pathlib.Path?
Anything pathlike?
[Note the above are rhetorical questions and not the focus of this post]. These are all informed guesses, but if I were completely new to python and was trying to use this documentation to make some directories, I see two options:
Read the source code via some IDE or other indexing
Guess until I get it right
The first is fine for easier to understand functions like makedirs but for more complicated functions this would require gaining expertise in a library that I don't necessarily want to reuse and just want to try out. I simply don't have enough time to become an expert in everything I encounter. This seems quite inefficient.
The second also seems to be quite inefficient, with the added demerit of not knowing how to write robust code to check for inappropriate inputs.
Now I don't want to bash the python docs, as they are LEAPS and BOUNDS better than a fair few other languages I've used, but is this dilemma just a case of unfinished/poor documentation, or is there a standard way of knowing/understanding what input parameters like name in this case should be that I haven't outlined above?
To be fair, this may not be the optimal example, as if you look towards the end of the doc for makedirs you can see it does state:
Changed in version 3.6: Accepts a path-like object.
but this is not specifically referring to name. Yes, in this example it may seem rather obvious it is referring to name, but with the advent of type-hinting, why are the docs not type hinted like the quickdocs from PyCharm? Is this something planned for the future, or is it too large a can of worms to try to hint all possibilities in a flexible language like python?
Just as a comparison, take a look at Java's java.io.file.mkdirs where the various constructors definitely tell you all the options for specifying the path of the file:
File(File parent, String child)
// Creates a new File instance from a parent abstract pathname and a child pathname string.
File(String pathname)
// Creates a new File instance by converting the given pathname string into an abstract pathname.
File(String parent, String child)
// Creates a new File instance from a parent pathname string and a child pathname string.
File(URI uri)
// Creates a new File instance by converting the given file: URI into an abstract pathname.
Just reading this I already know exactly how to make a File object and create directories without running/testing anything. With the quickdoc in PyCharm I can do the same, so where is this type hint information in the official docs?
Related
folks.
I've been studying yocto building process and I noticed the usage of the following structure:
PN = "${#bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
I know that ${} means variable expansion and grep command showed that the function "vars_from_file" is located at bitbake/lib/bb/parse/__init__.py.
I would like to understand how this variable expansion works, so I explored bitbake files and found out that:
oe-init-build-env calls oe-buildenv-internal, and the last one sets PYTHONPATH to bitbake/lib.
I'm infering that bitbake uses PYTHONPATH to search for the function vars_from_file
What I have not understood are:
the meaning of the symbol "#" in the variable expansion;
if bitbake uses PYTHONPATH to search for the function, why did not pass the absolute path ${#bb.parse.__init__.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'} instead of "${#bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
Is this type of variable expansion applied only in bitbake?. I searched in gnu website, and there is not the application of "#" in the beginning of the structures.
Can someone help me understand them?
The # symbol, is used bit bitbake for inline python variable expansion, and it basically states that you'd be calling a function and expanding its result, usually assigning it to a variable, in your case:
PN = "${#bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
Its assigning to PN (package name) the result of the function vars_from_file coming from the bitbake python module parse, located at bitbake/lib/bb/parse/
Like any other Python module, it automatically loads init.py and has access to those functions when imported.
Whats special about these functions is that they've got access to bitbake's data dictionary, called "d".
AFAIC this IS specific to bitbake.
I'm trying to use the typing module to document my Python package, and I have a number of situations where several different types are allowable for a function parameter. For instance, you can either pass a number, an Envelope object (one of the classes in my package), or a list of numbers from which an Envelope is constructed, or a list of lists of numbers from which an envelope is constructed. So I make an alias type as follows:
NumberOrEnvelope = Union[Sequence[Real], Sequence[Sequence[Real]], Real, Envelope]
Then I write the function:
def example_function(parameter: NumberOrEnvelope):
...
And that looks great to me. However, when I create the documentation using Sphinx, I end up with this horrifically unreadable function signature:
example_function(parameter: Union[Sequence[numbers.Real], Sequence[Sequence[numbers.Real]], numbers.Real, expenvelope.envelope.Envelope])
Same thing also with the hints that pop up when I start to try to use the function in PyCharm.
Is there some way I can have it just leave it as "NumberOrEnvelope". Ideally that would also link in the documentation to a clarification of what "NumberOrEnvelope" is, though even if it didn't it would be way better than what's appearing now.
I had the same issue and used https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_type_aliases, introduced in version 3.3.
In your sphinx conf.py, insert this section. It does not seem to make much sense at the first sight, but does the trick:
autodoc_type_aliases = dict(NumberOrEnvelope='NumberOrEnvelope')
Warning: It only works in modules that start with from __future__ import annotation
Note: If there is a target in the documentation, type references even have a hyperlink to the definition. I have classes, documented elsewhere with autoclass, which are used as types of function parameters, and the docs show the nice names of the types with links.
Support for this appears to be in the works.
See Issue #6518.
That issue can be closed by the recent updates to Pull Request #8007 (under review).
If you want the fix ASAP, you can perhaps try using that build.
EDIT: This doesn't quite work, sadly.
Turns out after a little more searching, I found what I was looking for. Instead of:
NumberOrEnvelope = Union[Sequence[Real], Sequence[Sequence[Real]], Real, Envelope]
I found that you can create your own compound type that does the same thing:
NumberOrEnvelope = TypeVar("NumberOrEnvelope", Sequence[Real], Sequence[Sequence[Real]], Real, Envelope)
This displays in documentation as "NumberOrEnvelope", just as I wanted.
I want to select 5 images with Python so that I can use these imges in my python program. I tried to do this with QFileDialog() (PyQt5) but I only succeed to single select a file.
And how to select a folder is also not really comprehensive.
I just want to select 5 images and select a folder so that I can save files in that folder. But it seems to be not so easy to do that.
I really like Python because its so easy but PyQt5 makes me everytime I use it just aggressive, all other libraries are just nice and easy to understand.
Maybe there is a good alternative to pyqt? tkinter maybe?
thanks.
In order to select a folder you can use this code:
widget = QWidget()
dialog = QFileDialog(
widget, "Select Directory of the Desired Files", os.path.curdir
)
dialog.setFileMode(QFileDialog.DirectoryOnly)
Qt supplies a bunch of static methods to get standardized file dialogs, two of them already satisfy your needs: getOpenFileNames() (stress on the final "s") and getExistingDirectory().
The first will return a list of absolute paths of selected file[s], the last will return the selected directory.
I know that reading the official documentation might be a bit overwhelming if you don't know anything about C++ (they are explained in detail, though), but they're not as hard as one could think.
Every function is listed in a very simple way:
returned_type : function_name(arguments) [const -> you can usually ignore this]
The returned_type is the type of the value the function is expected to return. In "c++ slang", void is the same as return (or return None or no return at all, as Python implicitly returns None if no other value/object is returned at the end of a function), if the type is a QString it's automatically converted to a Python str, while qreal is the same as Python's floats,. This is very important for "private" functions (methods), which are internally used by Qt: if you are subclassing and want to override a private method of a Qt class, you have to return the type Qt expects. You could theoretically ignore the returned_type for public functions if you know what you're doing, but it's usually better to stick with the original type.
There are some small "exceptions" that require some consideration. In some cases Qt expects some argument that will be modified within the function and would normally return whether the function was successful or not, while in Python it might return the reference to the argument (sorry, I can't remember them right now). Some other functions return a tuple instead of a single value, and that's the case of some static QFileDialog functions such as getOpenFileName[s] which return both the selected file[s] and the selected filter.
While it is simple to search by using help for most methods that have a clear help(module.method) arrangement, for example help(list.extend), I cannot work out how to look up the method .readline() in python's inbuilt help function.
Which module does .readline belong to? How would I search in help for .readline and related methods?
Furthermore is there any way I can use the interpreter to find out which module a method belongs to in future?
Don't try to find the module. Make an instance of the class you want, then call help on the method of that instance, and it will find the correct help info for you. Example:
>>> f = open('pathtosomefile')
>>> help(f.readline)
Help on built-in function readline:
readline(size=-1, /) method of _io.TextIOWrapper instance
Read until newline or EOF.
Returns an empty string if EOF is hit immediately.
In my case (Python 3.7.1), it's defined on the type _io.TextIOWrapper (exposed publicly as io.TextIOWrapper, but help doesn't know that), but memorizing that sort of thing isn't very helpful. Knowing how to figure it out by introspecting the specific thing you care about is much more broadly applicable. In this particular case, it's extra important not to try guessing, because the open function can return a few different classes, each with different methods, depending on the arguments provided, including io.BufferedReader, io.BufferedWriter, io.BufferedRandom, and io.FileIO, each with their own version of the readline method (though they all share a similar interface for consistency's sake).
From the text of help(open):
open() returns a file object whose type depends on the mode, and
through which the standard file operations such as reading and writing
are performed. When open() is used to open a file in a text mode ('w',
'r', 'wt', 'rt', etc.), it returns a TextIOWrapper. When used to open
a file in a binary mode, the returned class varies: in read binary
mode, it returns a BufferedReader; in write binary and append binary
modes, it returns a BufferedWriter, and in read/write mode, it returns
a BufferedRandom.
See also the section of python's io module documentation on the class hierarchy.
So you're looking at TextIOWrapper, BufferedReader, BufferedWriter, or BufferedRandom. These all have their own sets of class hierarchies, but suffice it to say that they share the IOBase superclass at some point - that's where the functions readline() and readlines() are declared. Of course, each subclass implements these functions differently for its particular mode - if you do
help(_io.TextIOWrapper.readline)
you should get the documentation you're looking for.
In particular, you're having trouble accessing the documentation for whichever version of readline you need, because you can't be bothered to figure out which class it is. You can actually call help on an object as well. If you're working with a particular file object, then you can spin up a terminal, instantiate it, and then just pass it to help() and it'll show you whatever interface is closest to the surface. Example:
x = open('some_file.txt', 'r')
help(x.readline)
I use Python 3 and want to wrap argparse.ArgumentParser with a custom class that sets
formatter_class=argparse.RawDescriptionHelpFormatter by default. I can do this successfully, however IntelliJ IDEA 2017.1 with Python Plugin (PyCharm) gives a warning for the following code:
class CustomParser(argparse.ArgumentParser):
def __init__(self, formatter_class=argparse.RawDescriptionHelpFormatter, **kwargs):
# noinspection PyArgumentList
super().__init__(formatter_class=formatter_class, **kwargs) # warning in this line for the last argument if suppression comment above removed
If one removes the comment with the IntelliJ suppression command the warning on kwargs is "Expected a dictionary, got a dict", however it is working. Is this a false positive warning or can this be done better without the warning? Is there a real issue behind this warning that it helps avoiding?
Side question: Is there a difference in using
formatter_class = kwargs.pop('formatter_class', argparse.RawDescriptionHelpFormatter) instead of explicitly defining the named parameter in the signature? According to PEP20 more explicit in the signature is better, right?
Yes, that appears to be a false positive.
You asked about formatter_class=kwargs.pop('formatter_class', argparse.RawDescriptionHelpFormatter). Please don't do that. Mutating kwargs, which appears as the next argument, seems Bad. Additionally, default keyword args should be set to a simple constant rather than some mutable datastructure, as there is a big difference between evaluating at import time and evaluating at run time. See e.g. http://www.effbot.org/zone/default-values.htm . The usual idiom would be formatter_class=None in the signature, and then in the body you test for None and you mutate kwargs to your heart's content.
Marking all the virtual environment directories stored inside the project (e.g. ./venv) as excluded has solved it for me with PyCharm 2020.2.3.
To do so: right-click on the virtual environment directories in the project tree and select: Mark Directory as -> Excluded. Solution from https://youtrack.jetbrains.com/issue/PY-39715.