We Keep Coding

BeamRunPythonPipelineOperator on DataFlowRunner keeps throwing error missing service_account

I am running following DataFlow config
test_dataflow= BeamRunPythonPipelineOperator(
task_id="xxxx",
runner="DataflowRunner",
py_file=xxxxx,
pipeline_options = dataflow_options,
py_requirements=['apache-beam[gcp]==2.39.0'],
py_interpreter='python3',
dataflow_config=DataflowConfiguration(job_name="{{task.task_id}}", location=LOCATION, project_id=PROJECT, wait_until_finished=False,gcp_conn_id="google_cloud_default")
#dataflow_config={"job_name":"{{task.task_id}}", "location":LOCATION, "project_id":PROJECT, "wait_until_finished":True,"gcp_conn_id":"google_cloud_default"}
)
It keeps throwing error . airflow-2.2.5 version.
Error - Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/apache/beam/operators/beam.py", line 287, in execute
) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/apache/beam/operators/beam.py", line 183, in _init_pipeline_options
dataflow_job_name, pipeline_options, process_line_callback = self._set_dataflow(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/apache/beam/operators/beam.py", line 63, in _set_dataflow
pipeline_options = self.__get_dataflow_pipeline_options(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/apache/beam/operators/beam.py", line 92, in __get_dataflow_pipeline_options
if self.dataflow_config.service_account:
AttributeError: 'DataflowConfiguration' object has no attribute 'service_account'
If I give service_account, then it errors saying parameter invalid

I ran into the same issue.
This is because of the inconsistency between the dataflow_configuration in dataflow and the one expected by beam. The DataflowConfiguration doesn't accepting the service_account.
I resolved my issue by upgrading the composer in place, so it gets the latest package related to dataflow where it has been fixed.
The service_account attribute has been added in this commit https://github.com/apache/airflow/commit/de65a5cc5acaa1fc87ae8f65d367e101034294a6
If you can't upgrade composer, try updating the google providers package to the latest version or > version 7.0 ?
You can check the commit in the commit log and identify the minimum version here - https://airflow.apache.org/docs/apache-airflow-providers-google/stable/commits.html#id6
Even though composer uses it's own fork, the oss should work. You can see the list of packages in the composer version list https://cloud.google.com/composer/docs/concepts/versioning/composer-versions it says apache-airflow-providers-google==2022.5.18+composer instead of apache-airflow-providers-google==7.0.

assert isinstance(filter_field, BaseCSVFilter)

I am having some issues with my django app since updating my dependencies. Before updating everything was fine I updated django from 2.2.28 to 3.2. The python version in the application is 3.6. I have problem with graphen.
Error:
File ".../lib/python3.6/site-packages/graphene_django/filter/utils.py", line 84, in replace_csv_filters
assert isinstance(filter_field, BaseCSVFilter)
I chose specific libraries:
asgiref==3.5.2
Django==3.2
django-anymail==6.1.0
django-colorfield==0.1.15
django-cors-headers==2.5.3
django-debug-toolbar==3.2.2
django-extensions==2.2.3
django-filter==2.3.0
django-graphiql-debug-toolbar==0.1.3
django-graphql-extensions==0.0.6
django-graphql-jwt==0.2.1
django-health-check==3.14.3
django-model-utils==4.2.0
django-nested-inline==0.4.4
django-qr-code==2.2.0
django-safedelete==0.5.2
django-silk==4.0.1
django-simple-history==2.8.0
django-solo==1.1.3
django-stdimage==2.4.1
django-storages==1.12.2
django-tinymce==2.8.0
django-utils-six==2.0
envs==1.3
graphene==2.1.8
graphene-django==2.15.0
graphene-file-upload==1.2.2
graphql-core==2.3.2
graphql-relay==2.0.1
grpcio==1.39.0
openpyxl==3.0.5
Pillow==8.3.1
python-docx==0.8.11
python-utils==2.5.6
reportlab==3.6.1
requests==2.24.0
six==1.15.0
urllib3==1.25.11
Any idea in something?

error using pip search (pip search stopped working)

I am getting this error in pip search while studying python.
The picture is an error when I pip search. Can you tell me how to fix it?
$ pip search pdbx
ERROR: Exception:
Traceback (most recent call last):
File "*/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 224, in _main
status = self.run(options, args)
File "*/lib/python3.7/site-packages/pip/_internal/commands/search.py", line 62, in run
pypi_hits = self.search(query, options)
File "*/lib/python3.7/site-packages/pip/_internal/commands/search.py", line 82, in search
hits = pypi.search({'name': query, 'summary': query}, 'or')
File "/usr/lib/python3.7/xmlrpc/client.py", line 1112, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python3.7/xmlrpc/client.py", line 1452, in __request
verbose=self.__verbose
File "*/lib/python3.7/site-packages/pip/_internal/network/xmlrpc.py", line 46, in request
return self.parse_response(response.raw)
File "/usr/lib/python3.7/xmlrpc/client.py", line 1342, in parse_response
return u.close()
File "/usr/lib/python3.7/xmlrpc/client.py", line 656, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault -32500: 'RuntimeError: This API has been temporarily disabled due to unmanageable load and will be deprecated in the near future. Please use the Simple or JSON API instead.'>

The pip search command queries PyPI's servers, and PyPI's maintainers have explained that the API endpoint that the pip search command queries is very resource intensive and too expensive for them to always keep open to the public. Consequently they sometimes throttle access and are actually planning to remove it completely soon.
See this GitHub issues thread ...
The solution I am using for now is to pip install pip-search (a utility created by GitHub user #victorgarric).
So, instead of 'pip search', I use pip_search. Definitely beats searching PyPI via a web browser

Follow the suggestion from JRK at the discussion at github (last comment) the search command is temporarily disabled, use your browser to search for packages meanwhile:
Check the thread on github and give him a thumb up ;)

search on website, https://pypi.org/,
then install the package you wanted

The error says
Please use the Simple or JSON API instead
You can try pypi-simple to query the pip repository
https://pypi.org/project/pypi-simple/
It gives an example too, I tried to use it here:
pypi-simple version 0.8.0 DistributionPackage' object has no attribute 'get_digest':
!/usr/bin/env python3
-*- coding: utf-8 -*-
"""
Created on Thu Nov 11 17:40:03 2020
#author: Pietro
"""
from pypi_simple import PyPISimple
def simple():
package=input('\npackage to be checked ')
try:
with PyPISimple() as client:
requests_page = client.get_project_page(package)
except:
print("\n SOMETHING WENT WRONG !!!!! \n\n",
"CHECK INTERNET CONNECTION OR DON'T KNOW WHAT HAPPENED !!!\n")
pkg = requests_page.packages[0]
print(pkg)
print(type(pkg))
print('\n',pkg,'\n')
print('\n'+pkg.filename+'\n')
print('\n'+pkg.url+'\n')
print('\n'+pkg.project+'\n')
print('\n'+pkg.version+'\n')
print('\n'+pkg.package_type+'\n')
#print('\n'+pkg.get_digest()+'\n','ENDs HERE !!!!') #wasnt working
if __name__ == '__main__':
simple()
got -4 so far for this answer don't know why , figureout I can try to check for a package with:
# package_name = input('insert package name : ')
package_name = 'numpy'
import requests
url = ('https://pypi.org/pypi/'+package_name+'/json')
r = requests.get(url)
try:
data = r.json()
for i in data:
if i == 'info':
print('ok')
for j in data[i]:
if j == 'name':
print((data[i])[j])
print([k for k in (data['releases'])])
except:
print('something went south !!!!!!!!!!')

CS50 Finance - NameError: name 'self' is not defined

I am getting 500 Internal Server Error when running flask with the following error message:
NameError: name 'self' is not defined
Yesterday my code worked fine and I didn't make any changes. The error message lists python files which were already imported in the distribution code. Maybe something changed in the background?
192.168.234.116 - - [10/Jul/2019 11:15:56] "GET / HTTP/1.0" 500 -
Error on request:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/werkzeug/serving.py", line 303, in run_wsgi
execute(self.server.app)
File "/usr/local/lib/python3.7/site-packages/werkzeug/serving.py", line 291, in execute
application_iter = app(environ, start_response)
File "/home/ubuntu/environment/pset8/finance/application.py", line 13, in <module>
app = Flask(__name__)
File "/usr/local/lib/python3.7/site-packages/cs50/flask.py", line 54, in _after
self.wsgi_app = ProxyFix(self.wsgi_app, x_proto=1)
NameError: name 'self' is not defined

It's a CS50 bug (regression).
I've submitted [GitHub]: cs50/python-cs50 - Added the 1st (required) argument (self) to flask.Flask's initializer, which was closed (because it was incomplete - as I was too rushed when submitting it, and missed one spot), and [GitHub]: Fix missing self arguments in Flask __init__ replacement was created and merged. Not sure though when will it be available on the market (PyPI, so you can simply pip install it), but you could download the sources from GitHub and overwrite yours.
As an alternative, you could download the patch, and apply the changes locally. Check [SO]: Run/Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (#CristiFati's answer) (Patching utrunner section) for how to apply patches (basically, every line that starts with one "+" sign goes in, and every line that starts with one "-" sign goes out). Or (given the fact that the change is more than trivial), you could:
Open the flask.py file (in your case: "/usr/local/lib/python3.7/site-packages/cs50/flask.py") in a text editor
Go to line #54 (in your case, in mine it's 52)
Replace the current content (do not alter leading SPACEs):
For the current line (def _after(*args, **kwargs):) by: def _after(self, *args, **kwargs):
For the next one (_before(*args, **kwargs)) by: _before(self, *args, **kwargs)
#EDIT0:
As #kaczifant already experienced, the fix is already available for download: pip3 install cs50 --upgrade

The bug was fixed by the CS50 staff.
Select CS50 IDE > Log Out and then log back in at ide.cs50.io. Alternatively, you can run:
sudo pip3 install cs50 --upgrade

"Failed to import google/appengine/ext/deferred/handler.py" in Google App Engine Flexible Environment

I use App Engine Flexible Environment (previously called Managed VMs), and recently upgraded to the latest gcloud SDK. It included some new errors:
ERROR: (gcloud.preview.app.deploy) Error Response: [400] Invalid
character
in filename: lib/setuptools/script (dev).tmpl
ERROR: The [application] field is specified in file [.../app.yaml]. This field is not used
by gcloud and must be removed. Project name should instead be
specified either by `gcloud config set project MY_PROJECT` or by
setting the `--project` flag on individual command executions.
ERROR: (gcloud.preview.app.deploy) There is a Dockerfile in the
current directory, and the runtime field in
.../app.yaml is currently set to
[runtime: python27]. To use your Dockerfile to build a custom runtime,
set the runtime field in .../app.yaml
to [runtime: custom]. To continue using the [python27] runtime, please
omit the Dockerfile from this directory.
I fixed these errors and was able to publish again, but started seeing errors like this:
Failed to import google/appengine/ext/deferred/handler.py
Traceback (most recent call last):
File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/meta_app.py", line 549, in GetUserAppAndServe
app, mod_file = self.GetUserApp(script)
File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/meta_app.py", line 410, in GetUserApp
app = _AppFrom27StyleScript(script)
File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/meta_app.py", line 270, in _AppFrom27StyleScript
app, filename, err = wsgi.LoadObject(script)
File "/home/vmagent/python_vm_runtime/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
ImportError: Import by filename is not supported.

After a bit of digging, I figured out what is going on. Namely, the code that processes this:
builtins:
- remote_api: on
- appstats: on
- deferred: on
is broken with Managed VMs. The correct fix is to eliminate these and inline the builtin includes instead. You can find the relevant includes inside these subdirectories:
In my case, it was to add this to my handlers: directive:
- url: /_ah/queue/deferred
script: google.appengine.ext.deferred.application
login: admin
- url: /_ah/stats.*
script: google.appengine.ext.appstats.ui.app
- url: /_ah/remote_api(/.*)?
script: google.appengine.ext.remote_api.handler.application
As to why, you can understand more here. In google/appengine/ext/builtins/__init__.py#L92, it attempts to find the relevant include file by using the runtime: field in your app.yaml. This means where it previously looked up deferred/include-python27.yaml, it now attempts to find deferred/include-custom.yaml (due to fixing the errors above) and fails. So now it defaulting to deferred/include.yaml, which lists the include scrip by path-name instead of module-name. This then breaks in a python27-custom-VM setup (since it is expecting/needing module-names)