We Keep Coding

Error with Entity matching package deepmatcher

Getting below Error:
ValueError Traceback (most recent call last)
<ipython-input-6-2d323ffe212f> in <module>()
----> 1 train, validation, test = dm.data.process(path='/content/', train='train.csv', validation='validation.csv', test='test.csv')
1 frames
/usr/local/lib/python3.7/dist-packages/deepmatcher/data/process.py in _check_header(header, id_attr, left_prefix, right_prefix, label_attr, ignore_columns)
32 if attr not in (id_attr, label_attr) and attr not in ignore_columns:
33 if not attr.startswith(left_prefix) and not attr.startswith(right_prefix):
---> 34 raise ValueError('Attribute ' + attr + ' is not a left or a right table '
35 'column, not a label or id and is not ignored. Not sure '
36 'what it is...')
ValueError: Attribute ltable_id is not a left or a right table column, not a label or id and is not ignored. Not sure what it is...
I am using the
http://pages.cs.wisc.edu/~anhai/data1/deepmatcher_data/Textual/Company/company_exp_data.zip
dataset for this learning, because previous test with my own dataset gave the same error.
Code:
import deepmatcher as dm
train, validation, test = dm.data.process(path='/content/', train='train.csv', validation='validation.csv', test='test.csv')
That's it. I am following below repo github.com/anhaidgroup/deepmatcher
Looking for better understanding and possible solve. Thanks in advance.

I didn't test it but error message may suggest that it needs special columns to work.
And first look at page with repo DeepMatch can confirm it.
There is example table which has columns with name Left ... and Right ....
There was also link to Get Started in which you can see
Step 1. Process labeled data¶
Left" attributes (required): ...
These column names are expected to be prefixed with "left_" by default.
"Right" attributes (required): "...
These column names are expected to be prefixed with "right_" by default.
Which shows that columns need prefix left_, right_ but your data has columns ltable_id, rtable_id. So you have to change column names after you load it and before you use it with DeepMatch
EDIT:
In repo is also link Data Processing
And there you can see
"Left" attributes (required): ...
This can be customized by setting the left_prefix parameter (e.g., use "ltable_" as the prefix).
"Right" attributes (required): ...
This can be customized by setting the right_prefix parameter (e.g., use "rtable_" as the prefix).
and it shows example code
dm.data.process(... left_prefix='left_', right_prefix='right_', ...)
which can means you can do
dm.data.process(... left_prefix='ltable_', right_prefix='rtable_', ...)
EDIT:
I tested it and it resolves this problem with company_exp_data.zip.
import deepmatcher as dm
train, validation, test = dm.data.process(
path='/content/',
#path='exp_data',
train='train.csv',
validation='valid.csv',
test='test.csv',
left_prefix='ltable_',
right_prefix='rtable_',
)
But next it gives other problem.
RuntimeError: Google drive link https://drive.google.com/uc?export=download&id=1Vih8gAmgBnuYDxfblbT94P6WjB7s1ZSh is currently unavailable, because the quota was exceeded.
It tries to read some data from Google Drive but the quota was exceeded.
Maybe it will need to download it manually and change source code to load it from local computer. But it is problem for new question. Or this problem should be send to author of this module. And he should put data on other server and change source code.
Summarizing: all your problem is you didn't read documentation.

DocuSign python (eg-03-python-auth-code-grant) authentication server problem

I am able to get on Docusign example page on my localhost:5000 fine but when I click on any of the 23 links it redirects me to localhost/ds/must_authenticate.
- I click on “authenticate with DocuSign” link and it takes me to account-d.docusign.com/#/password
-I enter my credentials and then it takes me appdemo.docusign.com/home
I am trying to do one of the many examples from the DocuSign GitHub repo on https://github.com/docusign/eg-03-python-auth-code-grant . The only thing I have changed is In the ds_config to add my client_id, client_secret, signer_email, signer_name, session_secret, and I had to add /oauth on line 14 in the ds_config. Changing it from authorization_server": "https://account-d.docusign.com to authorization_server": "https://account-d.docusign.com/oauth . The reason I added /oauth is because I get a client id error. The redirect URI I used in my integration key is http://localhost:5000/ds/callback. I guess I am having an issue with with the authorization_server. If I omit or keep brackets I get same results and can get past login. The url paths are : 127.0.0.1:5000 for homepage, then I click on any of the example links and get redirected to sign in page 127.0.0.1:5000/ds/must_authenticate, Then I click on the "authenticate with DocuSign" and get redirected to account.d.docusign.com/username#/password, then ""/password, then appdemo.docusign/authentication, then appdemo.docusign/redirect?to, then appdemo.docusign.com/home. I keep getting redirected to Docusign Homepage. If I take /oauth out of authorization server then I cant get past sign in page.
DocuSignIntegrationKeyandRedirectURI
OAuthSignInError DS_Config w/o brackets and w/o /oauth
DS_Config.py w/o brackets and w/o oauth
DS_CONFIG.PY

Thank you for submitting your question. On first glance, I see that your DS_Config.Py needs a couple of quick alterations. Can you try the following and let me know if you're still having issues with the OAuth flow?
1) Remove the { }'s from the config file. if you're listing your ds_client_id as {client_Id} it reads the {} as part of the ID, which will return an error indicating the key isn't properly registered with DocuSign. You'll want to remove the brackets from the first 5 variables in your config file.
2) Move the authorization_server back to the original URL (excluding '/oauth')
Let me know if that works for you, if you see another error on your side please reply with the exact error message with a screenshot if possible, then we can pick it up from there.

A current workaround I have found is editing the views.py ds_token_ok function. On line 177, the ok boolean variables are not returning true. Not sure if it is a session problem. I edited the return statement to return True instead of ok, this has let me open all the links in the python application and run all the examples.
def ds_token_ok(buffer_min=60):
# :param buffer_min: buffer time needed in minutes
#:return: true iff the user has an access token that will be good for #another buffer min
ok = "ds_access_token" in session and "ds_expiration" in session
ok = ok and (session["ds_expiration"] - timedelta(minutes=buffer_min)) > datetime.utcnow()
#comment: was return ok but I am not able to run program. Problem in line 177
return True
This is just a temporary solution to work in the sandbox.

Getting SalesforceMalformedRequest: Malformed request error

I am trying to execute this following code to push data to Salesforce using the simple_salesforce python library :
from simple_salesforce import Salesforce
staging_df = hive.execute("select * from hdmni")
staging_df = staging_df.toPandas()
# # staging_df['birth_date']= staging_df['birth_date'].dt.date
staging_df['birth_date'] = staging_df['birth_date'].astype(str)
staging_df['encounter_start_date'] = staging_df['encounter_start_date'].astype(str)
staging_df['encounter_end_date'] = staging_df['encounter_end_date'].astype(str)
bulk_data = []
for row in staging_df.itertuples():
d= row._asdict()
del d['Index']
bulk_data.append(d)
sf = Salesforce(password='', username='', security_token='')
sf.bulk.Delivery_Detail__c.insert(bulk_data)
I am getting this error while trying to send dictionary to salesforce :
SalesforceMalformedRequest: Malformed request
https://subhotutorial-dev-ed.my.salesforce.com/services/async/38.0/job/7500o00000HtWP6AAN/batch/7510o00000Q15TnAAJ/result.
Response content: {'exceptionCode': 'InvalidBatch',
'exceptionMessage': 'Records not processed'}

There's something about your query that is not correct. While I don't know your use case, by reading this line, you can tell that you are attempting to insert into a custom object/entity in Salesforce:
sf.bulk.Delivery_Detail__c.insert(bulk_data)
The reason you can tell is because of the __c suffix, which gets appended onto custom objects and fields (that's two underscores, by the way).
Since you're inserting into a custom object, your fields would have to be custom, too. And note, you've not appended that suffix onto them.
Note: Every custom object/entity in Salesforce does come with a few standard fields to support system features like record key (Id), record name (Name), audit fields (CreatedById, CreatedDate, etc.). These wouldn't have a suffix. But none of the fields you reference are any of these standard system fields...so the __c suffix would be expected.
I suspect that what Salesforce is expecting in your insert operation are field names like this:
Birth_Date__c
Encounter_Start_Date__c
Encounter_End_Date__c
These are referred to as the API name for both objects and fields, and anytime code interacts with them (whether via integration, or on code that executes directly on the Salesforce platform) you need to make certain you're using this API name.
Incidentally, you can retrieve this API name through a number of ways. Probably easiest is to log into your Salesforce org, and in Setup > Object Manager > [some object] > Fields and Relationships you can view details of each field, including the API name. Here's a screen shot.
You can also use SObject describe APIs, either in native Apex code, or via integration and either the REST or SOAP APIs. Here's part of the response from the describe API request to the describe REST endpoint for the same object as my UI example above, found here at https://[domain]/services/data/v47.0/sobjects/Expense__c/describe:
Looking at the docs for the simple-salesforce python library you're using, they've surfaced the describe API. You can find some info under Other Options. You would invoke it as sf.SObject.describe where "SObject" is the actual object you want to find the information about. For instance, in your case you would use:
sf.Delivery_Detail__c.describe()
As a good first troubleshooting step when interacting with a Salesforce object, I'd always recommend double-checking correctly referencing an API name. I can't tell you how many times I've bumped into little things like adding or missing an underscore. Especially with the __c suffix.

Fetching place details (specifically reviews) with GooglePlaces in Python 3

I am completely new to this module and Python in general, yet wanted to start some sort of a fun project in my spare time.
I have a specific question concerning the GooglePlaces module for Python - how do I retrieve the reviews of a place by only knowing its Place ID.
So far I have done...
from googleplaces import GooglePlaces, types, lang
google_places = GooglePlaces('API KEY')
query_result = google_places.get_place(place_id="ChIJB8wSOI11nkcRI3C2IODoBU0")
print(query_result) #<Place name="Starbucks", lat=48.14308250000001, lng=11.5782337>
print(query_result.get_details()) # Prints None
print(query_result.rating) # Prints the rating of 4.3
I am completely lost here, because I cannot get access to the object's details. Maybe I am missing something, yet would be very thankful for any guidance through my issue.

If you are completly lost just read the docs :)
Example from https://github.com/slimkrazy/python-google-places:
for place in query_result.places:
# Returned places from a query are place summaries.
# The following method has to make a further API call.
place.get_details()
# Referencing any of the attributes below, prior to making a call to
# get_details() will raise a googleplaces.GooglePlacesAttributeError.
print place.details # A dict matching the JSON response from Google.
See the Problem with your code now?
print(query_result.get_details()) # Prints None
should be
query_result.get_details() # Fetch details
print(query_result.details) # Prints details dict
Regarding the results, the Google Docs states:
reviews[] a JSON array of up to five reviews. If a language parameter
was specified in the Place Details request, the Places Service will
bias the results to prefer reviews written in that language. Each
review consists of several components:

gitpython to check for if there are any changes in files using PYTHON

I am experimenting with gitpython and I am new to it. I am trying to detect if there are any changes staged for commit.
Currently, I have a function that looks like this:
def commit(dir):
r = Repo(dir)
r.git.add(A=True)
r.git.commit(m='commit all')
But this is just the code to commit the directory. I wanna do something like, if there are changes, then display some message, else, display another message.
Anybody have any idea how do I do it in python ?

You can check for all unstaged changes like this:
for x in r.index.diff("HEAD"):
# Just print
print(x)
# Or for each entry you can find out information about it, e.g.
print(x.new_file)
print(x.b_path)
Basically you are comparing the staging area (i.e. index) to the active branch.

To get a definitive list of what's changed (but not yet staged):
# Gives a list of the differing objects
diff_list = repo.head.commit.diff()
for diff in diff_list:
print(diff.change_type) # Gives the change type. eg. 'A': added, 'M': modified etc.
# Returns true if it is a new file
print(diff.new_file)
# Print the old file path
print(diff.a_path)
# Print the new file path. If the filename (or path) was changed it will differ
print(diff.b_path)
# Too many options to show. This gives a comprehensive description of what is available
help(diff_list[0])
I found the diff object to be very useful and should give any info you require.
For staged items, use repo.index
From my testing, I found that the previous answer gave a diff output the wrong way round (ie. added files would show up as deleted).
The other option is repo.git.diff(...) which I found less useful as it gives long text strings for output rather than objects that can be easily parsed.