We Keep Coding

Running an Nsight Systems report python script independently

I've tweaked a copy of one of the Nsight Systems report scripts (gpukernsum), and I now want to run it myself. So, I write:
./gpukernsum.py report.sqlite
This doesn't work; I get:
ERROR: Script 'gpukernsum.py' encountered an internal error.
$ ./gpukernsum.py report.sqlite
File "./gpukernsum.py", line 40
"""
^
SyntaxError: invalid syntax
I know this is because f"""whatever""" is Python-3 syntax, so I change the script's hash-bang line from:
#!/usr/bin/env python
to:
#!/usr/bin/env python3
and now I get:
$ ./gpukernsum.py report.sqlite
Traceback (most recent call last):
File "/path/to/./gpukernsum.py", line 7, in <module>
import nsysstats
ModuleNotFoundError: No module named 'nsysstats'
So I added the relevant directory to the lookup path:
export PYTHONPATH="$PYTHONPATH:/opt/nvidia/nsight-systems/2022.1.1/host-linux-x64/python/lib"
and now I get:
$ ./gpukernsum.py report.sqlite
near "WITH": syntax error
... and I'm stuck. The relevant area of the code is:
and not a percentage of the application wall or CPU execution time.
"""
query_stub = """
WITH
summary AS (
SELECT
coalesce({NAME_COL_NAME}, demangledName) AS nameId,
i.e. the "WITH" is part of a string literal which is an SQL query. So, what's the problem? Is Python complaining? Is sqlite complaining?
Note:
Nsight Systems 2022.1.1
CentOS 7
I'm using the original gpukernsum.py code - I have not made any changes to it (other than as described above).
My system has Python 3.9.1 for python3.

A workaround answer:
Nsight Systems bundles its own version of Python, with lib and bin directories.
If you run your script with this specific version, having set PYTHONPATH as described in your question - then the script will work. It's what Nsight itself does, after all.

Protobuf 3.19.1 and Python 3.10 issue

This is all about and issue when using the latest Python Protobuf (3.19.1) and Python 3.10, in Linux (tested in Fedora 35 and Ubuntu 20.04.
It broke our library but it can easily tested using the addressbook.proto from the Python Protobuf tutorial and tried to get the proto2 message class as follows:
import addressbook_pb2
from google.protobuf import (
descriptor_database,
descriptor_pb2,
descriptor_pool,
message_factory,
)
_DESCRIPTOR_DB = descriptor_database.DescriptorDatabase()
_DESCRIPTOR_POOL = descriptor_pool.DescriptorPool(_DESCRIPTOR_DB)
_DESCRIPTOR_DB.Add(
descriptor_pb2.FileDescriptorProto.FromString(
addressbook_pb2.DESCRIPTOR.serialized_pb
)
)
factory = message_factory.MessageFactory()
cls = factory.GetPrototype(_DESCRIPTOR_POOL.FindMessageTypeByName("tutorial.Person"))
It raises the following error:
[libprotobuf ERROR google/protobuf/pyext/descriptor_database.cc:64] DescriptorDatabase method raised an error
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
Traceback (most recent call last):
File "/dev/protobuf/test/test.py", line 21, in <module>
ls = factory.GetPrototype(_DESCRIPTOR_POOL.FindMessageTypeByName("tutorial.Person"))
`KeyError: "Couldn't find message tutorial.Person"
Now, it works as expected if I use an older Python Protobuf version, such as 3.18.1.
I've opened a bug https://github.com/protocolbuffers/protobuf/issues/9245, but apparently, it was not considered a bug.
Python Protobuf introduced the PY_SSIZE_T_CLEAN macro in 3.19.1 and broke something, probably by using int instead of Py_ssize_t when using # formats.
Have anyone have this issue or can confirm it?

Yes I am also getting same issue.
We changes the version as below:
*protobuf >= 3.19* # Not working
*protobuf >= 3.15.6, <= 3.20.1* # Working

There are actually two errors here:
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
This error is caused by Python 3.10 dropping support for old default conversions when passing data from C to Python side. In this case in the protobuf library, the error only occurs when passing an exception from C code to Python.
The python-protobuf library was fixed to work with Python 3.10 back in October 2021, and the fix should be included in python-protobuf 3.20.0 and later.
Try adding this to your script to check the version:
import google.protobuf
print(google.protobuf.__version__)
For me the error does not occur with the latest versions 3.19.4, 3.20.1 or 4.21.1, but does occur with 3.19.2 and older.

Flake8 reports E999 SyntaxError

I am unable to solve the flake8 SyntaxError and although the code executes just fine.
Code without comments
import math
def answer(str_n):
sume = ((str_n * (str_n + 1)) / 2) * math.sqrt(2)
sume = int(sume)
return sume
def answer1(str_n):
sume = 0
for i in range(str_n + 1):
sume += math.floor(i * math.sqrt(2))
# print i,math.floor(i*math.sqrt(2))
return sume
print "Test answer:", answer(77)
print "Actual answer:", answer1(77)

As #jonrsharpe says, and I agree, this is because the code is being run in Python 2, but linted in Python 3.
From the flake8 documentation on error codes:
We report E999 when we fail to compile a file into an Abstract Syntax Tree for the plugins that require it.
So to prove this is correct, using a file called bad_syntax.py and using the same print syntax as above:
print "test answer", len([])
When I run this with Python 2, everything is happy:
james#codebox:/tmp/lint$ python --version
Python 2.7.12
james#codebox:/tmp/lint$ python bad_syntax.py
test answer 0
Linting with flake8 invoked with a Python 2 environment also passes.
But when I lint with Python 3 (this is running in a virtualenv venv with Python 3 installed), the E999 is returned:
(venv) james#codebox:/tmp/lint$ flake8 --version
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 3.5.2 on Linux
(venv) james#codebox:/tmp/lint$ flake8 bad_syntax.py
bad_syntax.py:1:19: E999 SyntaxError: invalid syntax
I do not think that this is a setting that needs changing inside linter-flake8 because Flake8 will use the version of Python that it is run through. My guess would be that Flake8 is being run on Python 3 because it has been installed inside a Python 3 environment, even though the code is being run on Python 2.

Flake8 launcher has Python3 hardcoded as main python.
How to fix:
1) install flake8 package using pip
$ pip install flake8
pip will tell you that flake8 script hasn't been added to path and print path to it (/Library/Frameworks/Python.framework/Versions/2.7/bin/ in my case)
2) tune your IDE (Atom/PyCharm/etc) to use this script with your default Python 2.7 (my example is from PyCharm # MacOS):
PyCharm -> Preferences -> External tools -> "flake8 - current file"
Program: /usr/local/bin/python
Arguments: /Library/Frameworks/Python.framework/Versions/2.7/bin/flake8 --ignore=E501,E124,E127,E128 $FilePath$
Working directory: $FileDir$
[x] open console for tool output
Output filters: $FILE_PATH$\:$LINE$\:.*
It will work correctly without reporting E999.has

Why does python 2.4 keep ignoring my exit function?

I've written a script that only runs on version 2.6 or 2.7. I've placed in a safety check to see if another version is installed and to also check if there are any compatible versions available. Here is my code snippet;
# !/usr/bin/python
import sys, os, re
from sys import exit
if sys.version[0:3] in ['2.6', '2.7']:
import subprocess, datetime, os.path, json, urllib2, socket
def python_version():
version = sys.version[0:3]
while version not in ['2.6', '2.7']:
print '\n\nYou\'re using an incompatible version of Python (%s)' % version
python_installed_version = []
for files in os.listdir("/usr/bin/"): # check to see version of python already on the box
python_version = re.search("(python2(.+?)[6-7]$)", files)
if python_version:
python_installed_version.append(python_version.group())
if python_installed_version:
print 'Fortunately there are compatible version(s) installed. Try the following command(s): \n\n',
for version in python_installed_version:
print 'curl http://script.py | %s\n' % version
sys.exit()
python_version()
with p.stdout:
print 'hello'
When I run the above in python 2.4 I get this error;
File "<stdin>", line 46
with p.stdout:
^
SyntaxError: invalid syntax
I don't understand why the script doesn't exit as soon as it detects that you're using python 2.4 with sys.exit(), but instead carries on reading the script and gives the error above where the with p.stout: is read.
I have removed the with p.stdout: line, and it will work fine, it won't read the print 'hello'. I don't know why the with p.stdout: is causing the script to break. This works perfectly fine on 2.6 and 2.7.
Any ideas on why python 2.4 will still read the with p.stdout: line? Thanks.

Python 2.4 doesn't have the with syntax support yet, so the script fails at the parsing stage, rather than at runtime. You can work around that by having some wrapper like:
if version_check_ok():
import actual_app
actual_app.run()
else:
...
This way the new files with the new syntax will not be imported/parsed until you know it's safe to do so. You can't move the import above the version check however.

Get the version from distutils setup.py

How can I import or read the VERSION from the setup.py file so that I can log the version at runtime.
This way I can make sure that the results obtained are from this particular version of my package.
The following is the contents of my setup.py file (simplified to have the necessary part)
import distutils.core
VERSION = '0.1.0'
LICENSE = 'GPLv2'
distutils.core.setup(**KWARGS)
When I try to do :
import setup
I get the following error:
distutils.core.setup(**KWARGS)
usr/lib/python2.6/distutils/core.pyc in setup(**attrs)
ok = dist.parse_command_line()
except DistutilsArgError, msg:
raise SystemExit, gen_usage(dist.script_name) + "\nerror: %s" % msg
if DEBUG:
SystemExit:
error: no commands supplied

There is a way to get the version from your setup script:
python setup.py --version
But I’m not sure I understand what you mean with “log the version at runtime”; the setup script is normally not installed with your modules, so people use other ways to put a version number in their code, like a __version__ attribute in their module or __init__.py file.

In yor example, setup is excecuted automatically, you have to replace:
distutils.core.setup(**KWARGS)
with:
if __name__ == '__main__':
distutils.core.setup(**KWARGS)
Like this, setup is only executed if you actually run the setup.py