We Keep Coding

Problems to access to other module with the same parent name in Python

In my company we decided to structure own python modules using this convention:
dsc.<package_name>
It works without any problem when two modules is used in other project that doesn't follow this convention. However, when in a develop environment I try to develop a new module "dsc.new_module" that references to other, for example, "dsc.other_module", the import raises a not module found exception. Is there any way to solve this?
If I package the module and install, everything is correct but not when I'm developing the module that is not able to find it. The only way I overcome this problem is doing that:
try:
from dsc.other_module import send_message
except ImportError:
def dummy(a, b):
pass
send_message = dummy
Beacause the function is not essential.

What you can do is install your packages in development mode. pip install -e . (from the parent folder) After this the imports should work as you envision them, so the same as other packages that use them.
Development mode is not required, but it adds the benefit of implementing changes made to your code immediately.

Pycharm venv packages not available to me?

I just created a pycharm virtual enviorment, and would like to use some specific packages inside of it. It's these two cryptography ones:
But when I try to import them in my code, the import doesnt complain, but when I try to acces the library, no suggestions are given.
What should I do in order to actually use a library?

I'm not familiar with crypto, but its top-level __init__.py contains nothing and hence there is nothing to suggest.
You'll receive completion with non-empty modules, e.g.:
import crypto.library.hash
crypto.library.hash.<here>
# or
from crypto.library import hash
hash.<here>

Python: How do I find which pip package a library belongs to?

I got a script transferred from someone else. And there is a module imported into the script. I'm wondering what is the best way to find out which pip package installed this library (other than search online).
I tried to import the package and then just do help() on it but didn't got much information. Is there a reliable and pythonic way to achieve this?
For example:
In the script it has a line
from impala.dbapi import connect
Without searching on internet, how can I find out that following package can install this library? as you can see in this case the package name is is different from the name used in pip.
pip install impyla

Short answer: You can't.
The core of the reason is that the name to import the package and the name to install the package come from different namespaces. When you run the import command, Python is looking for the package in the local environment. When you tell pip to install a package, it's looking for something to install from PyPI (or somewhere else if you told pip to look elsewhere).
It would make sense for these two names to always be the same, but there's no guarantee. Installation is a matter of choosing which set of files to download and install, while importing is a matter of choosing what installed files to run, but the names for those files do not have to stay the same during the installation process. And that's how we get confusion like pip install impyla, import impala.
If you had access to the setup.py file for the package, you could look in there for the name (if you look at the GitHub for impyla/impala, you'll see a name='impyla', line inside the call to setup), but if the package was installed via pip, you won't have the setup.py file locally, so this option is pretty much right out.
It's not a great state of affairs. There's simply no guarantee that you can find the PyPI name for a package from just having local access to the package code and the "real" import name for the package. That said, if you're unfamiliar with the package anyway, you're probably going to want to look up documentation and more info on the internet anyway. Just one more thing to look up, I guess.

If you do not have the package installed, you'll have to use a pip search or a poetry search (or a Google search).
In the case you have the package already installed (i.e. you can import it) , you can get the package name with importlib.metadata:
In Python >=3.8, you can use the standard library importlib.metadata
For Python <3.8, there is importlib_metadata (link to documentation). Replace importlib.metadata with importlib_metadata in the examples below.
Example on usage:
>>> from importlib.metadata import packages_distributions
>>> packages_distributions()
'asttokens': ['asttokens'],
'backcall': ['backcall'],
'bitarray': ['bitarray'],
'colorama': ['colorama'],
'decorator': ['decorator'],
'executing': ['executing'],
'importlib_metadata': ['importlib-metadata'],
'impala': ['impyla'],
'IPython': ['ipython'],
'jedi': ['jedi'],
'matplotlib_inline': ['matplotlib-inline'],
'parso': ['parso'],
'pickleshare': ['pickleshare'],
'pip': ['pip'],
'prompt_toolkit': ['prompt-toolkit'],
'pure_eval': ['pure-eval'],
'puresasl': ['pure-sasl'],
'pygments': ['Pygments'],
'_distutils_hack': ['setuptools'],
'pkg_resources': ['setuptools'],
'setuptools': ['setuptools'],
'six': ['six'],
'stack_data': ['stack-data'],
'thrift': ['thrift'],
'thrift_sasl': ['thrift-sasl'],
'traitlets': ['traitlets'],
'wcwidth': ['wcwidth'],
'zipp': ['zipp']}
In this case, you would look for the "impala" entry in the packages_distributions() dictionary:
>>> packages_distributions()['impala']
['impyla']
You may also check for example the version of the package:
>>> from importlib.metadata import version
>>> # Note: Using the name of the package, not the "import name"
>>> version('impyla')
'0.18.0'
Bonus: Checking the name of the module from a function or class name
>>> connect.__module__.split('.')[0]
'impala'
Note that this does not guarantee that a package was installed from PyPI. You could, if you want, create your own package called "impyla", "matplotlib", whatsoever, install it, and use it as you wish.

What is the cleanest way to add a directory of third-party packages to the beginning of the Python path?

My context is appengine_config.py, but this is really a general Python question.
Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:
dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)
For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):
sys.path.insert(1, dirpath)
However, that won't work if any of the packages in that directory are namespace packages.
To support namespace packages we can instead use:
site.addsitedir(dirpath)
But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.
The solution I have so far is this bit of code which I'd really like to simplify:
sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)
Is there a cleaner or more Pythonic way of accomplishing this?

For this answer I assume you know how to use setuptools and setup.py.
Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:
import os
import sys
if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
# If we are unittesting, fake the non-existence of appengine_config.
# The error message of the import error is handled by gae and must
# exactly match the proper string.
raise ImportError('No module named appengine_config')
# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
for lib in os.listdir(lib_dir):
if lib.endswith('.egg'):
lib = os.path.join(lib_dir, lib)
# Insert to override default libraries such as webob 1.1.1.
sys.path.insert(0, lib)
And this piece of code in setup.cfg:
[develop]
install-dir = libs
always-copy = true
If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.
We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.

You may want to have a look at the answers in the Stack Overflow thread, "How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)," but for your particular predicament with namespace packages, you're running up against a long-standing issue I filed against site.addsitedir's behavior of appending to sys.path instead of inserting after the first element. Please feel free to add to that discussion with a link to this use case.
I do want to address something else that you said that I think is misleading:
My context is appengine_config.py, but this is really a general Python
question.
The question actually arises from the limitations of Google App Engine and the inability to install third-party packages, and hence, seeking a workaround. Rather than manually adjusting sys.path and using site.addsitedir. In general Python development, if your code uses these, you're Doing It Wrong.
The Python Packaging Authority (PyPA) describes the best practices to put third party libraries on your path, which I outline below:
Create a virtualenv
Mark out your dependencies in your setup.py and/or requirements files (see PyPA's "Concepts and Analyses")
Install your dependencies into the virtualenv with pip
Install your project, itself, into the virtualenv with pip and the -e/--editable flag.
Unfortunately, Google App Engine is incompatible with virtualenv and with pip. GAE chose to block this toolset in an attempt sandbox the environment. Hence, one must use hacks to work around the limitations of GAE to use additional or newer third party libraries.
If you dislike this limitation and want to use standard Python tooling for managing third-party package dependencies, other Platform as a Service providers out there eagerly await your business.

How to import 3rd party module to python if its name is taken?

I want to use this class: google-mail-oauth2-tools but if i'll do something like import oauth2 the imported class would be python-oauth2 which is deprected and doesn't support oauth2 )even though the name is oauth2)
How can I use google module ? do I need to install it first?

Looking at the docs you linked, it looks like the Google Mail oauth2 module is meant to be downloaded and used in-place.
You can, of course, install it… but you can't have two (top-level) modules installed with the same name, so you'd have to uninstall python-oauth2 first.
But if you just use it in-place, in Python 2.7, you can have an oauth2.py in one directory and one in the stdlib. Whichever one you import first will "win"; any subsequent attempts to import oauth2 will get the first one.
To force it to import the right one (in a way that will also work for older Python 2.x and for Python 3.x), you may want to use the imp module to give it the path explicitly.
For example, if you plan to put oauth2.py right alongside the script that imports it, instead of just import oauth2, do:
script_path = os.path.abspath(os.path.dirname(__file__))
f, path, desc = imp.find_module('oauth2', [script_path])
oauth2 = imp.load_module('oauth2', f, path, desc)
… although in some situations, you can get away with cheating by assuming the current working directory is the script directory, and/or by permanently munging sys.path, etc., so you can simplify it in various different ways—ultimately, if it's safe, just this:
sys.path = ['.'] + sys.path
import oauth2
Still, I would consider doing one of the following for safety (and readability):
Uninstall python-oauth2.
Rename the downloaded oauth2.py to something else, like google_oauth2.py, and then import google_oauth2.
Put the downloaded oauth2.py into a package, so you can, e.g., import googletools.oauth2.