We Keep Coding

Show whether a Python module is loaded from bytecode

I'm trying to debug Hy's use of bytecode. In particular, each time a module is imported, I want to see the path it was actually imported from, whether source or bytecode. Under the hood, Hy manages modules with importlib. It doesn't explicitly read or write bytecode; that's taken care of by importlib.machinery.SourceFileLoader. So it looks like what I want to do is monkey-patch Python's importing system to print the import path each time an import happens. How can I do that? I should be able to figure out how to do it for Hy once I understand how to do it for Python.

The easiest way that does not involve coding, is to start Python with two(!) verbose flags:
python -vv myscript.py
you'll get a lot of output, including all the import statements and all the files Python tries to access in order to load the module. In this example I have a simple python script that does import json:
lots of output!
[...]
# trying /tmp/json.cpython-310-x86_64-linux-gnu.so
# trying /tmp/json.abi3.so
# trying /tmp/json.so
# trying /tmp/json.py
# trying /tmp/json.pyc
# /usr/lib/python3.10/json/__pycache__/__init__.cpython-310.pyc matches /usr/lib/python3.10/json/__init__.py
# code object from '/usr/lib/python3.10/json/__pycache__/__init__.cpython-310.pyc'
[...]
Alternatively but more complex: you could change the import statement itself. For that, you can overwrite __import__, which is invoked by the import statement itself. This way you could print out all the files import actually opens.

Seems like a good option would be to dynamically patch importlib.machinery.SourceFileLoader(fullname, path) and importlib.machinery.SourcelessFileLoader(fullname, path) to each print or write to a variable (a) the calling method and (b) the argument passed to the function.
If all you need to do is:
I want to see the path it was actually imported from, whether source or bytecode
And you don't need the import to "work properly", perhaps you can do a modified version of something like this. For example, I quickly modified their sample code to get this, I have not tested it so it may not work exactly, but it should get you on the right track:
# custom class to be the mock return value
class MockSourceLoader:
# mock SourceFileLoader method always returns that the module was loaded from source and its path
def SourceFileLoader(fullname, path):
return {"load type": "source", "fullname": fullname, "path": path}
def check_how_imported(monkeypatch):
# Any arguments may be passed and mock_get() will always return our
# mocked object
def mock_get(*args, **kwargs):
return MockSourceLoader
# apply the monkeypatch
monkeypatch.setattr(importlib.machinery, SourceFileLoader, SourceFileLoader)
You would of course provide a similar mock for Sourceless file loading for SourcelessFileLoader
For reference:
https://docs.python.org/3/library/importlib.html#:~:text=importlib.machinery.SourceFileLoader(fullname%2C%20path)%C2%B6
https://docs.python.org/3/library/importlib.html#:~:text=importlib.machinery.SourcelessFileLoader(fullname%2C%20path)

PTVS IntelliSense not working for Built-in Function

class Class:
def __init__(self, path):
self._path = path
string = open(self._path, 'r'). #HERE
When I try to type read() intelliSense says no completions.
However, I know open() function returns file object, which has read() function. I want to see all supported function after typing a dot.
PyCharm shows me recommanded function list, but PTVS does not support.
I want to know this is casual things in PTVS or only happening to me.
My current Python Enviroment is Anaconda 4.3.0 (Python 3.5.3)
How can I fix it?

We've already fixed the specific case of open for our upcoming update (not the one that released today - the next one), but in short the problem is that you don't really know what open is going to return. In our fix, we guess one of two likely types, which should cover most use cases.
To work around it right now, your best option is to assign the result of open to a variable and force it to a certain type using an assert statement. For example:
f = open(self._path, 'r')
import io
assert isinstance(f, io.TextIOWrapper)
f = open(self._path, 'rb')
import io
assert isinstance(f, io.BufferedIOBase)
Note that your code will now fail if the variable is not the expected type, and that the code for Python 2 would be different from this, but until you can get the update where we embed this knowledge into our code it is the best you can do.

How can I modify modules and packages while keeping the original intact?

I have a program written in my python using the PyPDF2 package to scrape a batch of pdf files. These PDF's aren't in the greatest shape so in order for my program to run, I need to modify the pdf.py file located within the package library as recommended by this website:
https://cheonhyangzhang.wordpress.com/2015/03/31/python-pdffilereader-pdfreaderror-eof-marker-not-found/
Is there a way I can implement this change to the file while keeping the original file intact? I've tried creating a child class of PdfFileReader class and modifying the 'read' method as prescribed by my link above, however, I've found that that leads to several import dependency issues that I'd like to avoid.
Is there an easier way to do this?

I would recommend to copy the pdf.py file into our script directory and rename it to mypdf.py. You can then modify the copy as you please without affecting the original. You can import the package using
import mypdf
I have done something similar for shutil.py as the default buffer size is too small in Windows for transferring large files.

You can add (or redefine) a method of class using setattr() like this (where the class has been defined inline rather than being imported only for purposes of illustration):
class Class(object):
pass
def func(self, some_other_argument):
return some_other_argument
setattr(Class, 'func', func)
if __name__ == '__main__':
c = Class()
print(c.func(42)) # -> 42

How to set the filename encoding globally for a python interpreter?

In a project, we need to open many files all over the place in the source code. We know that all filenames on disk are encoded in utf-8 and all filenames are proceed as Unicode internally.
Is there a way to set the filename encoding globally for the running interpreter, so that we change the following command:
open(filename.encode('utf-8'))
to this simpler version:
open(filename)
This would reduces errors and confusion among the developers. We use python 2.7.

Assuming this is really what you meant (and not the encoding of the file's contents), this would do it:
open = lambda fname, *a, **kw: __builtins__.open(fname.encode('utf-8'), *a, **kw)
This will only affect modules that include (or import) the redefinition, so it's reasonably safe. But it might be less confusing, and certainly less trouble-prone over the long run, to provide a different command for opening files in your environment for which this applies:
def fsopen(fname, *args, **kwargs):
"""(explain when and why you want this)"""
return open(fname.encode('utf-8'), *args, **kwargs)

There is no such option, because the open function has documented behaviour not to UTF-8 encode the strings.
There's a simple reason for this: open is more or less just a delegate of the underlying OS function, and must work on any string that the OS function would work with. Hence, it expects raw Python strings and not unicode.
If you need this, implement a wrapper function to do this.
In Python, you can just override the local name -- or the global one, for that matter:
__uni_open = lambda string: __builtins__.open(string.encode("utf-8"))
open = __uni_open
in whatever module all your applications work with.
I don't recommend doing this -- it might break stuff beyond your project's control.

How to get PyCharm to auto-complete code in methods?

When I'm using a 3rd party l
ibrary such as boto, PyCharm seems to be able to auto-complete quite nicely
However, as soon as I define a function of my own, auto-complete breaks down inside that function. I understand why, since I can't give the function any type information about its arguments, so it can't guess how to auto-complete. Is there a way around this issue?
Edit
I tried using the docstring (for Python 2), but still no auto-complete
def delete_oldest_backups(conn, backups_to_keep, backup_description):
"""
delete_oldest_backups(EC2Connection, int, string)
"""
(Also tried boto.ec2.connection.EC2Connection instead of just EC2Connection)

You can use type hints: http://www.jetbrains.com/pycharm/webhelp/type-hinting-in-pycharm.html
def some_method(self, conn):
"""
#type conn: EC2Connection
"""
conn.<autocomplete>

You can specify the type information about the parameters of the function using Python 3 parameter and return value annotations. If you're using Python 2, you can also specify information in the function's docstring. PyCharm understands the format used by docstrings of binary modules in the standard library, for example:
"""
foo(int, string) -> list
Returns the list of something
"""

In order for PyCharm to recognize an instance of an object and retrieve all its methods, we have to use the following statements. But I think that both is a terrible way of wasting programming and run time.
assert isinstance(instanceX, ClassOfInstanceX)
instanceX.{#list of method/properties appears}
Alternatively, you can also use the class name will recall the method or property everytime you want to invoke it and pass in the instance to the self parameter. But this is too verbose, for my liking, esp for nested class
ClassOfInstanceX.{#list of method/properties appears}
# then you will have...
ClassOfInstance.method(instanceX, args...)

You can install the library via pyCharm "package manager".
Go to Settings -> Project Interpreter -> Python Interpreters
And in the Packages list, click on install and search for the library you want to install
Once installed, auto-complete will be available on editor.
Hope this is what you are looking for.