We Keep Coding

Why does Python's filterwarning module regex not work as expected?

According to the Python warnings documentation the filterwarnings pattern is action:message:category:module:line, and the module part should allow for matching specific modules against a regex. I'm trying to make this work to filter specific warnings for third party library modules, but the feature doesn't seem to work.
Minimal example
A simpler way to reproduce the issue is to create a file
my_module.py
print("\d")
and then run PYTHONWARNINGS="error,ignore:::.*" python -c "import my_module" from the same directory. Clearly the regex .* should match the module name my_module, but somehow it doesn't. The question is why?
Original example
As an example: Importing rosbag package (some package that comes with the Robot Operating System (ROS) distribution) triggers an error when running in "strict" report-warnings-as-errors mode:
$ PYTHONWARNINGS='error' python -c "import rosbag"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/<PATH-TO-ROS-DISTRIBUTION>/lib/python3/dist-packages/rosbag/__init__.py", line 33, in <module>
from .bag import Bag, Compression, ROSBagException, ROSBagFormatException, ROSBagUnindexedException
File "/<PATH-TO-ROS-DISTRIBUTION>/lib/python3/dist-packages/rosbag/bag.py", line 1568
matches = re.match("#ROS(.*) V(\d).(\d)", version_line)
^
SyntaxError: invalid escape sequence \d
That makes sense, it should use a raw string literal.
Now I'm trying to use a module regex to silence the warning from the rosbag module. For the purpose of the example I'm specifying the filterwarnings via PYTHONWARNINGS but other methods like via pytests settings result in the same behavior.
I'm getting the following unexpected behavior:
using PYTHONWARNINGS='error,ignore:invalid escape sequence::' works. Of course that filters all "invalid escape sequence" warnings. So let's add a module regex at the end to be module specific...
using PYTHONWARNINGS='error,ignore:invalid escape sequence::rosbag.*' should do exactly that, but the filter doesn't seem to match. The warning is still reported as error as with plain error.
even using PYTHONWARNINGS='error,ignore:::.*', which should in theory match any module, fails to match. Only removing the regex entirely like PYTHONWARNINGS='error,ignore:::' matches, but of course that is essentially equivalent to just PYTHONWARNINGS='ignore'.
Any ideas why the module regex is not matching at all?! Is this a Python distribution bug? I'm on Python 3.8.10 / Ubuntu 20.04.

As of Python 3.10, this is the intended behavior. Warning filters set through -W and PYTHONWARNINGS currently only match literal strings for the message and module.
This is a known discrepancy in the Python documentation, and is tracked in this CPython issue. There were some discussions and a PR to add regex support, but these ultimately went nowhere. This open PR adds further clarification to the docs.

For ones looking for a workaround to suppress warnings in third-party modules and keep shown for developed packages and/or modules. Say I have developing mypackage. I use a shell wrapper runner script that does following:
FLT="$(cd "./src"; find mypackage/ -name '*.py' | \
sed -e 's/\.py//g' -e 's/\/__init__//g' -e 's/\//./g' | \
{ while read module; do
echo "default:::$module"
done; echo "default:::__main__"; } | paste -sd,)"
export PYTHONWARNINGS="ignore,$FLT"

How can I configure IPython to issue the same "magic" commands at every startup?

I'd like to be able to use %cd "default_dir" and %matplotlib whenever I call ipython from my terminal. I tried writing this in a .py file in .ipython/profile_default/startup/file.py but it results in the following error:
[TerminalIPythonApp] WARNING | Unknown error in handling startup files:
File "/Users/<name>/Dropbox/.ipython/profile_default/startup/startup.py", line 18
%cd "~/Dropbox/"
^
SyntaxError: invalid syntax

You just need to use the magic in your startup scripts:
get_ipython().magic('cd ~/Dropbox')
get_ipython().magic('matplotlib')
Put that in the contents of your startup script and it should do the magic you need ✨🔮✨

I just wanted to elaborate the Wayne's answer, but do not have enough reputation to do a comment. You can have the following in the start up script to run the required magic commands
from IPython.core import getipython
getipython.get_ipython().magic(u"%reload_ext autoreload")
getipython.get_ipython().magic(u"%autoreload 2")
Module reference is here Ipython module
To run the above start up at terminal, do this
ipython -i startup.py

What happened to my Ubuntu Desktop 16 today - cannot execute Python file as usual

In brief
I can't run a simple Python file with +x permission set and shebang line.
In Details
Let's take a simple Python code in myApp.py file at some $CODE_HOME folder
#!/usr/bin/python
print 122
When cd $CODE_HOME and running this file from console
. ./myApp.py
I got error as
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/%{ <-- HERE (.*?)}/ at /usr/bin/print line 528.
Error: no such file "122333"
Though running by python myApp.py will get thing work.
The question
What's wrong is that? How to fix it?

. myApp.py is an instruction to Bash to source the passed file, ie execute it within the current process.
To execute a script or other file, you need to reference it by path:
./myApp.py (or just python myApp.py)
i.e. omitting the starting '.' in your call

To answer your question as is, . is the source command, which just runs each of the commands in the argument script in the context of the calling terminal. In your case, this doesn't do anything for the first line, then tries to call print, as you can see in
at /usr/bin/print line 528
Use ./myApp.py instead.

Why my python script is not printing anything

I have a problem..whenever I am writing any Python script say like this
#!/usr/local/bin/python
print "hello"
Then using
chmod +x a.py
And then write ./a.py then it is not printing anything in the terminal
Moreover whenever I write any comment below the shabang line, it gives me an error saying
#: bad interpreter : No such file or directory
but when I run the script like this python a.py it works as usual..
Can someone tell me what's wrong and how to fix this..

This is almost certainly because your line ending is a carriage-return/line feed combination - which Windows-style editors will create. Unix regards the LF as the end of the line, so it's looking for an executable called "python\r". When you run it with an explicit call to the interpreter the shebang line is just treated as a comment.

Under linux text replacement fix:
sed -i 's/^ M//g' filename
(note that ^ M is written in linux, press ^ M is carriage return and line feed, the input method is to hold down CTRL + v, release v, press m)
This solve bad endline for shell in first line if error message looks like this:
/usr/bin/python3 ^M: bad interpreter: No such file or directory

Ignore IPython magic in python

What is the best way to ignore IPython magic when running scripts using the python interpreter?
I often include IPython magic in my script files because it work with the code interactively. For example, with the autoreload magic, I don't have to keep reload-ing the modules after I make some changes and fix bugs:
%load_ext autoreload
%autoreload 2
However, when I try to run this script using a usual python interpreter, I get an error:
File "<string>", line 1
%load_ext autoreload
^
SyntaxError: invalid syntax
Wrapping IPython magic inside an if statement does not work, because incorrect syntax is detected before the file is actually ran.
So what is the best way to get python to ignore IPython magic?
It's annoying to have to change your scripts whenever you want to run then in python, pdb, sphinx, etc.

For all tools that can read from standard input you could use grep to remove any magic lines and pipe the result into python:
grep -v '^%' magicscript.ipy | python
Works well as a bash alias:
alias pynomagic='( grep -v "^%" | python ) < '
pynomagic magicscript.ipy
Tools like pdb that only accept filenames could be called like this (bash again):
pdb <(grep -v '^%' magicscript.ipy)

In case this helps anyone.
At least for Databricks, when syncing a notebook with a .py file in Github, a magic function can be specified with a specially formatted comment.
Like this:
# MAGIC %run ./my_external_file

You should load such magic in your config file, not in your scripts! It is just not valid Python.
Put the following in your ~/.ipython/profile_default/ipython_config.py:
c = get_config()
c.InteractiveShellApp.extensions = ['autoreload']
c.InteractiveShellApp.exec_lines = ['%autoreload 2']
c.InteractiveShellApp.exec_lines.append('print("Warning: disable autoreload in ipython_config.py to improve performance.")')

Create a template file named simplepython.tpl. Copy the below statements.
{% extends 'python.tpl'%}
{% block codecell %}
{{ super().replace('get_ipython','#get_ipython') if "get_ipython" in super() else super() }}
{% endblock codecell %}
Save simplepython.tpl.
Type in command line:
jupyter nbconvert --to python 'IPY Notebook' --template=simplepython.tpl --stdout

Spyder gives warning (as given in the picture below), when a coder use this type of code and says that it is not a valid Python code.
So, in order to use IPython magics, saving files with the .ipy extension may be a solution.
Spyder screenshot