We Keep Coding

Document AI process document fails with invalid argument when processing docs from GCS

I am getting an error very similar to the below, but I am not in EU:
Document AI: google.api_core.exceptions.InvalidArgument: 400 Request contains an invalid argument
When I use the raw_document and process a local pdf file, it works fine. However, when I specify a pdf file on a GCS location, it fails.
Error message:
the processor name: projects/xxxxxxxxx/locations/us/processors/f7502cad4bccdd97
the form process request: name: "projects/xxxxxxxxx/locations/us/processors/f7502cad4bccdd97"
inline_document {
uri: "gs://xxxx/temp/test1.pdf"
}
Traceback (most recent call last):
File "C:\Python39\lib\site-packages\google\api_core\grpc_helpers.py", line 66, in error_remapped_callable
return callable_(*args, **kwargs)
File "C:\Python39\lib\site-packages\grpc\_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "C:\Python39\lib\site-packages\grpc\_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Request contains an invalid argument."
debug_error_string = "{"created":"#1647296055.582000000","description":"Error received from peer ipv4:142.250.80.74:443","file":"src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Request contains an invalid argument.","grpc_status":3}"
>
Code:
client = documentai.DocumentProcessorServiceClient(client_options=opts)
# The full resource name of the processor, e.g.:
# projects/project-id/locations/location/processor/processor-id
# You must create new processors in the Cloud Console first
name = f"projects/{project_id}/locations/{location}/processors/{processor_id}"
print(f'the processor name: {name}')
# document = {"uri": gcs_path, "mime_type": "application/pdf"}
document = {"uri": gcs_path}
inline_document = documentai.Document()
inline_document.uri = gcs_path
# inline_document.mime_type = "application/pdf"
# Configure the process request
# request = {"name": name, "inline_document": document}
request = documentai.ProcessRequest(
inline_document=inline_document,
name=name
)
print(f'the form process request: {request}')
result = client.process_document(request=request)
I do not believe I have permission issues on the bucket since the same set up works fine for a document classification process on the same bucket.

This is a known issue for Document AI, and is already reported in this issue tracker. Unfortunately the only workaround for now is to either:
Download your file, read the file as bytes and use process_documents(). See Document AI local processing for the sample code.
Use batch_process_documents() since by default is only accepts files from GCS. This is if you don't want to do the extra step on downloading the file.

This is still an issue 5 months later, and something not mentioned in the accepted answer is (and I could be wrong but it seems to me) that batch processes are only able to output their results to GCS, so you'll still incur the extra step of downloading something from a bucket (be it the input document under Option 1 or the result under Option 2). On top of that, you'll end up having to do cleanup in the bucket if you don't want the results there, so under many circumstances, Option 2 won't present much of an advantage other than the fact that the result download will probably be smaller than the input file download.
I'm using the client library in a Python Cloud Function and I'm affected by this issue. I'm implementing Option 1 for the reason that it seems simplest and I'm holding out for the fix. I also considered using the Workflow client library to fire a Workflow that runs a Document AI process, or calling the Document AI REST API, but it's all very suboptimal.

Getting KeyError: 'Endpoint' error in Python when calling Custom Vision API

from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
from msrest.authentication import CognitiveServicesCredentials
from azure.cognitiveservices.vision.customvision import prediction
from PIL import Image
endpoint = "https://southcentralus.api.cognitive.microsoft.com/"
project_id = "projectidhere"
prediction_key = "predictionkeyhere"
predict = CustomVisionPredictionClient(prediction_key, endpoint)
with open("c:/users/paul.barbin/pycharmprojects/hw3/TallowTest1.jpg", mode="rb") as image_data:
tallowresult = predict.detect_image(project_id, "test1", image_data)
Python 3.7, and I'm using Azure Custom Vision 3.1? (>azure.cognitiveservices.vision.customvision) (3.1.0)
Note that I've seen the same question on SO but no real solution. The posted answer on the other question says to use the REST API instead.
I believe the error is in the endpoint (as stated in the error), and I've tried a few variants - with the slash, without, using an environment variable, without, I've tried appending various strings to my endpoint but I keep getting the same message. Any help is appreciated.
Full error here:
Traceback (most recent call last):
File "GetError.py", line 15, in <module>
tallowresult = predict.detect_image(project_id, "test1", image_data)
File "C:\Users\paul.barbin\PycharmProjects\hw3\.venv\lib\site-packages\azure\cognitiveservices\vision\customvision\prediction\operations\_custom_vision_
prediction_client_operations.py", line 354, in detect_image
request = self._client.post(url, query_parameters, header_parameters, form_content=form_data_content)
File "C:\Users\paul.barbin\PycharmProjects\hw3\.venv\lib\site-packages\msrest\service_client.py", line 193, in post
request = self._request('POST', url, params, headers, content, form_content)
File "C:\Users\paul.barbin\PycharmProjects\hw3\.venv\lib\site-packages\msrest\service_client.py", line 108, in _request
request = ClientRequest(method, self.format_url(url))
File "C:\Users\paul.barbin\PycharmProjects\hw3\.venv\lib\site-packages\msrest\service_client.py", line 155, in format_url
base = self.config.base_url.format(**kwargs).rstrip('/')
KeyError: 'Endpoint'

CustomVisionPredictionClient takes two required, positional parameters: endpoint and credentials. Endpoint needs to be passed in before credentials, try swapping the order:
predict = CustomVisionPredictionClient(endpoint, prediction_key)

Error when trying to communicate with scope, VisaIOError: VI_ERROR_TMO (-1073807339): Timeout expired before operation completed

I need to communicate with a scope, Agilent Infiniium DCA-J 86100C, with python 2.7. The company Keysight offers various python code, although I'm trying to run one of them to help me learn but it crashed. I'm using GPIB and pyvisa for the connection.
I've already tried to change to termination characters but it didn't change anything. I'm not sure what band rate I could try.
SCOPE_VISA_ADDRESS = "GPIB0::7::INSTR"
rm = visa.ResourceManager('C:\\Windows\\System32\\visa32.dll')
KsInfiniiVisionX = rm.open_resource(SCOPE_VISA_ADDRESS)
KsInfiniiVisionX.clear()
KsInfiniiVisionX.query(':SYSTEM:DSP "";*OPC?')
KsInfiniiVisionX.write(":HARDcopy:INKSaver OFF")
KsInfiniiVisionX.write(":DISPlay:DATA? PNG,SCReen,COLor")
my_image = KsInfiniiVisionX.read_raw()
Traceback (most recent call last):
File "X:\...\Get_screen_image_VISA_Python_modified\InfiniiVision_Save_ScreenShot_to_PC_Python-2.7_modified.py", line 201, in <module>
my_image = KsInfiniiVisionX.read_raw()
File "C:\Python27\lib\site-packages\pyvisa\resources\messagebased.py", line 306, in read_raw
chunk, status = self.visalib.read(self.session, size)
File "C:\Python27\lib\site-packages\pyvisa\ctwrapper\functions.py", line 1582, in read
ret = library.viRead(session, buffer, count, byref(return_count))
File "C:\Python27\lib\site-packages\pyvisa\ctwrapper\highlevel.py", line 188, in _return_handler
raise errors.VisaIOError(ret_value)
VisaIOError: VI_ERROR_TMO (-1073807339): Timeout expired before operation completed.

I was able to get help. The goal was to take a screenshot of the display on the scope and save this screenshot to a connected PC. The picture had to be modify before being saved. Also, the reason why the function ".read_raw()" wouldn't work is that I had to do an *OPC before but only in a .write() command not in a .query().
KsInfiniiVisionX.write('DISK:SIMAGE "D:\User Files\screen images\TEST.jpg",SCR,INV')
KsInfiniiVisionX.write('*OPC?')
complete = KsInfiniiVisionX.read()
KsInfiniiVisionX.write('DISK:BFILE? "D:\User Files\screen images\TEST.jpg"')
my_image = KsInfiniiVisionX.read_raw()
dum = (my_image[0:1].decode())
length = int(my_image[1:2].decode())
size = int(my_image[2:2+length].decode())
search = dum+str(length)+str(size)
my_file=my_image.partition(search.encode())
base_directory = "X:\\..."
target = open(base_directory + '{}.jpg'.format(file_name), 'wb')
target.write(my_file[2])
target.close()
Unfortunately, I am not an expert so I can't explain WHY it works.

google.gax.errors.RetryError StatusCode.DEADLINE_EXCEEDED

from this github:
https://github.com/GoogleCloudPlatform/python-docs-samples
i am trying to test Video intelligence API and do label analysis.
import argparse
import sys
import time
import io
import base64
from google.cloud.gapic.videointelligence.v1beta1 import enums
from google.cloud.gapic.videointelligence.v1beta1 import (
video_intelligence_service_client)
# [END imports]
#python labels.py /Users/rockbaek/tildawatch-contents/EpicSkillShot/M7-_VukSueY/SKT\ vs\ KT\ Game\ 3\ _\ Grand\ Finals\ S7\ LCK\ Spring\ 2017\ _\ KT\ Rolster\ vs\ SK\ Telecom\ T1\ G3\ 1080p-M7-_VukSueY.mp4
def analyze_labels_file(path):
""" Detects labels given a file path. """
video_client = (video_intelligence_service_client.
VideoIntelligenceServiceClient())
features = [enums.Feature.LABEL_DETECTION]
with io.open(path, "rb") as movie:
content_base64 = base64.b64encode(movie.read())
operation = video_client.annotate_video(
'', features, input_content=content_base64)
print('\nProcessing video for label annotations:')
while not operation.done():
sys.stdout.write('.')
sys.stdout.flush()
time.sleep(15)
print('\nFinished processing.')
# first result is retrieved because a single video was processed
results = operation.result().annotation_results[0]
for i, label in enumerate(results.label_annotations):
print('Label description: {}'.format(label.description))
print('Locations:')
for l, location in enumerate(label.locations):
positions = 'Entire video'
if (location.segment.start_time_offset != -1 or
location.segment.end_time_offset != -1):
positions = '{} to {}'.format(
location.segment.start_time_offset / 1000000.0,
location.segment.end_time_offset / 1000000.0)
print('\t{}: {}'.format(l, positions))
print('\n')
if __name__ == '__main__':
# [START running_app]
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('path', help='GCS file path for label detection.')
args = parser.parse_args()
analyze_labels_file(args.path)
# [END running_app]
# [END full_tutorial]
and then i run it from terminal
python labels.py MP4_FILE_PATH
After quite a while, it fails with this error code:
Traceback (most recent call last):
File "labels.py", line 123, in <module>
analyze_labels_file(args.path)
File "labels.py", line 52, in analyze_labels_file
'', features, input_content=content_base64)
File "/Library/Python/2.7/site-packages/google/cloud/gapic/videointelligence/v1beta1/video_intelligence_service_client.py", line 237, in annotate_video
self._annotate_video(request, options), self.operations_client,
File "/Library/Python/2.7/site-packages/google/gax/api_callable.py", line 428, in inner
return api_caller(api_call, this_settings, request)
File "/Library/Python/2.7/site-packages/google/gax/api_callable.py", line 416, in base_caller
return api_call(*args)
File "/Library/Python/2.7/site-packages/google/gax/api_callable.py", line 376, in inner
return a_func(*args, **kwargs)
File "/Library/Python/2.7/site-packages/google/gax/retry.py", line 144, in inner
raise exc
google.gax.errors.RetryError: GaxError(Retry total timeout exceeded with exception, caused by <_Rendezvous of RPC that terminated with (StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded)>)
Please help why it is not working! :(

For the visitors from future, I faced same error with Cloud Speech API.
I just increased the timeout value while calling operation.result. That solved it. Though this fragment isn't in OP's code, it should be in Google's example code OP mentioned.
operation.result(timeout=90) # increase this numeric value

I tried your code with a small video and it seems to work just fine for me. Maybe you are hitting some form of quota or limits (refer: https://cloud-dot-devsite.googleplex.com/video-intelligence/limits)? I have also run large videos loaded in Google Storage and using Python client library without issue.
Other step to try: send the request to the service via a curl command.

How to extract variable from within a function in Python?

I'm making a program which scrapes bus information from a server and sends it to a user via Facebook messenger. It works fine, but I'm trying to add functionality which splits really long timetables into separate messages. To do this, I had to make an if statement that detects really long timetables, splits them and calls the send_message function from my main file, app.py
Here is the part of the function in app with the variable I need to extract:
for messaging_event in entry["messaging"]:
if messaging_event.get("message"): # someone sent us a message
sender_id = messaging_event["sender"]["id"] # the facebook ID of the person sending you the message
recipient_id = messaging_event["recipient"]["id"] # the recipient's ID, which should be your page's facebook ID
message_text = messaging_event["message"]["text"] # the message's text
tobesent = messaging_event["message"]["text"]
send_message(sender_id, fetch.fetchtime(tobesent))
and here is the if statement in fetch which detects long messages, splits them and calls the send_message function from the other file, app.py:
if len(info["results"]) > 5:
for i, chunk in enumerate(chunks(info, 5), 1):
app.send_message((USER ID SHOULD BE HERE, 'Listing part: {}\n \n{}'.format(i, chunk)))
I'm trying to call the send_message function from app.py, but it requires two arguments, sender_id and the message text. How can I go about getting the sender_id variable from this function and using it in fetch? I've tried returning it and calling the function, but it doesn't work for me.
EDIT:error
Traceback (most recent call last):
File "bus.py", line 7, in <module>
print fetch.fetchtime(stopnum)
File "/home/ryan/fb-messenger-bot-master/fetch.py", line 17, in fetchtime
send_message((webhook(),'Listing part: {}\n \n{}'.format(i, chunk)))
File "/home/ryan/fb-messenger-bot-master/app.py", line 30, in webhook
data = request.get_json()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/local.py", line 343, in __getattr__
return getattr(self._get_current_object(), name)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/local.py", line 302, in _get_current_object
return self.__local()
File "/usr/local/lib/python2.7/dist-packages/flask/globals.py", line 37, in _lookup_req_object
raise RuntimeError(_request_ctx_err_msg)
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.