Was trying to figure out a way to get simple salesforce to just give me all the field names in a list. I want to create soql query that pretty much does the same thing as a Select * does in sql.
for obj in objects:
fields = [x["name"] for x in sf[obj].describe()["fields"]]
thanks
A list of field names in an object can be achieved as follow:
def getObjectFields(obj):
fields = getattr(sf,obj).describe()['fields']
flist = [i['name'] for i in fields]
return flist
getObjectFields('Contact')
Your query to get the effect of SELECT * would then look something like this:
sf.query_all('SELECT {} FROM Contact LIMIT 10'.format(','.join(getObjectFields('Contact'))))
On a related note:
In case it is helpful, a dictionary of label/name pairs can be achieved as follows:
def getObjectFieldsDict(obj):
fields = getattr(sf,obj).describe()['fields']
fdict = {}
for i in fields:
fdict[i['label']] = i['name']
return fdict
getObjectFieldsDict('Contact')
I find this can be useful for figuring out the names of fields with labels that do not follow the standard format (i.e. "My Favorite Website" field label for "Favorite_Website__c" field name)
This method will return a query string with all fields for the object passed in. Well all the fields the user has access to.
public static string getFullObjectQuery(String sObjectName){
Schema.SObjectType convertType = Schema.getGlobalDescribe().get(sObjectName);
Map<String,Schema.sObjectField> fieldMap = convertType.getDescribe().Fields.getMap();
Set<String> fields = fieldMap.keySet();
String Query = 'SELECT ';
for(String field: fields){
Schema.DescribeFieldResult dfr = fieldMap.get(field).getDescribe();
if(dfr.isAccessible()){
Query += field + ',';
}
}
Query = query.SubString(0,Query.length() - 1);
Query += ' FROM ' + sObjectName;
return Query;
}
#!/usr/bin/env python3
import argparse
import os
import simple_salesforce
parser = argparse.ArgumentParser()
parser.add_argument('--sandbox', action='store_true',
help='Use a sandbox')
parser.add_argument('sfobject', nargs='+', action='store',
help=('Salesforce object to query (e.g. Contact)'))
args = parser.parse_args()
sf = simple_salesforce.Salesforce(
username = os.getenv('USERNAME'),
password = os.getenv('PASSWORD'),
security_token = os.getenv('SECURITY_TOKEN'),
sandbox = args.sandbox)
for sfobject in args.sfobject:
print(sfobject)
fields = [x['name'] for x in getattr(sf, sfobject).describe()['fields']]
print(fields)
Related
What i get from api:
"name":"reports"
"col_type":"array<struct<imageUrl:string,reportedBy:string>>"
So in hive schema I got:
reports array<struct<imageUrl:string,reportedBy:string>>
Note: I got hive array schema as string from api
My target:
bigquery.SchemaField("reports", "RECORD", mode="NULLABLE",
fields=(
bigquery.SchemaField('imageUrl', 'STRING'),
bigquery.SchemaField('reportedBy', 'STRING')
)
)
Note: I would like to create universal code that can handle when i receive any number of struct inside of the array.
Any tips are welcome.
I tried creating a script that parses your input which is reports array<struct<imageUrl:string,reportedBy:string>>. This converts your input to a dictionary that could be used as schema when creating a table. The main idea of the apporach is instead of using SchemaField(), you can create a dictionary which is much easier than creating SchemaField() objects with parameters using your example input.
NOTE: The script is only tested based on your input and it can parse more fields if added in struct<.
import re
from google.cloud import bigquery
def is_even(number):
if (number % 2) == 0:
return True
else:
return False
def clean_string(str_value):
return re.sub(r'[\W_]+', '', str_value)
def convert_to_bqdict(api_string):
"""
This only works for a struct with multiple fields
This could give you an idea on constructing a schema dict for BigQuery
"""
num_even = True
main_dict = {}
struct_dict = {}
field_arr = []
schema_arr = []
# Hard coded this since not sure what the string will look like if there are more inputs
init_struct = sample.split(' ')
main_dict["name"] = init_struct[0]
main_dict["type"] = "RECORD"
main_dict["mode"] = "NULLABLE"
cont_struct = init_struct[1].split('<')
num_elem = len(cont_struct)
# parse fields inside of struct<
for i in range(0,num_elem):
num_even = is_even(i)
# fields are seen on even indices
if num_even and i != 0:
temp = list(filter(None,cont_struct[i].split(','))) # remove blank elements
for elem in temp:
fields = list(filter(None,elem.split(':')))
struct_dict["name"] = clean_string(fields[0])
# "type" works for STRING as of the moment refer to
# https://cloud.google.com/bigquery/docs/schemas#standard_sql_data_types
# for the accepted data types
struct_dict["type"] = clean_string(fields[1]).upper()
struct_dict["mode"] = "NULLABLE"
field_arr.append(struct_dict)
struct_dict = {}
main_dict["fields"] = field_arr # assign dict to array of fields
schema_arr.append(main_dict)
return schema_arr
sample = "reports array<struct<imageUrl:string,reportedBy:string,newfield:bool>>"
bq_dict = convert_to_bqdict(sample)
client = bigquery.Client()
project = client.project
dataset_ref = bigquery.DatasetReference(project, '20211228')
table_ref = dataset_ref.table("20220203")
table = bigquery.Table(table_ref, schema=bq_dict)
table = client.create_table(table)
Output:
My backend in Python :
def resview(request, *args, **kwargs):
if 'uid' not in kwargs:
kwargs = kwargs.copy()
kwargs['filter'] = {}
username = request.GET.get('username')
if username:
kwargs['filter']['user__username__contains'] = username
return resource_view(request, UserProfile, get_userprof, put_userprof,
deleter=del_userprof, api=request.api, order_by='user__username', **kwargs)
I can successfully search for the username but I want to be able to search on multiple fields with a single input and I don't know how to do it.
I have once in similar situation used Regular Expression to search based on keywords & then pulling out entire line until newline. Here I'm mentioning my small code.
Step1: Save the content in Text file
str = open('file.txt', 'r').read()
import re
#to print id
m = re.search('(?<=id: )(.*)', str)
print ("id= " , (m.groups()))
#to print username
m = re.search('(?<=username: )(.*)', str)
print (username= " , (m.groups()))
############
#and so on----Replace keywords whatever you need...
#############
Considering you're using django, I would suggest to take look at SearchFilter which supports searching multiple fields at once.
Beside of that, you can filter queryset by hands with custom implementation of Filter:
class F(django_filters.FilterSet):
username = CharFilter(method='my_custom_filter')
class Meta:
model = User
fields = ['username', 'firstname', 'lastname', 'email']
def my_custom_filter(self, queryset, name, value):
return queryset.filter(
Q(username__ilike=value)
| Q(firstname__ilike=value)
| Q(lastname__ilike=value)
| Q(email__ilike=value)
)
You can store the JSON list in a list variable and filter the results.
filterData(searchString){
searchString = searchString.trim();
return this.data.filter(user=> user.username.toLowerCase().indexOf(searchString.toLowerCase()) !== -1 || user.firstname.toLowerCase().indexOf(searchString.toLowerCase()) !== -1 || user.lastname.toLowerCase().indexOf(searchString.toLowerCase()) !== -1);
}
I suggest that you use two different lists to save data. One filtered and the other un filtered. This way you won't have to call the service again to get original data. Filter is always applied on unfiltered data.
In Django, can I re-use an existing Q object on multiple models, without writing the same filters twice?
I was thinking about something along the lines of the pseudo-Django code below, but did not find anything relevant in the documentation :
class Author(Model):
name = TextField()
company_name = TextField()
class Book(Model):
author = ForeignKey(Author)
# Create a Q object for the Author model
q_author = Q(company_name="Books & co.")
# Use it to retrieve Book objects
qs = Book.objects.filter(author__matches=q_author)
If that is not possible, can I extend an existing Q object to work on a related field? Pseudo-example :
# q_book == Q(author__company_name="Books & co.")
q_book = q_author.extend("author")
# Use it to retrieve Book objects
qs = Book.objects.filter(q_book)
The only thing I've found that comes close is using a subquery, which is a bit unwieldy :
qs = Book.objects.filter(author__in=Author.objects.filter(q_author))
From what I can tell by your comment, it just looks like you're trying to pass a set of common arguments to multiple filters, to do that you can just unpack a dictionary
The values in the dictionary can still be q objects if required as if it were a value you would pass in to the filter argument normally
args = { 'author__company_name': "Books & co" }
qs = Book.objects.filter(**args)
args['author_name'] = 'Foo'
qs = Book.objects.filter(**args)
To share this between different models, you'd have to do some dictionary mangling
author_args = { k.lstrip('author__'): v for k, v in args.items }
You can do this
books = Book.objects.filter(author__company_name="Books & co")
I have a simple one-to-many structure like this:
class User(db.Model):
userEmail = db.StringProperty()
class Comment(db.Model):
user = db.ReferenceProperty(User, collection_name="comments")
comment = db.StringProperty()
date = db.DateTimeProperty()
I fetch a user from by his email:
q = User.all() # prepare User table for querying
q.filter("userEmail =", "az#example.com") # apply filter, email lookup
results = q.fetch(1) # execute the query, apply limit 1
the_user = results[0] # the results is a list of objects, grab the first one
this_users_comments = the_user.comments # get the user's comments
How can I order the user's comments by date, and limit it to 10 comments?
You will want to use the key keyword argument of the built-in sorted function, and use the "date" property as the key:
import operator
sorted_comments = sorted(this_users_comments, key=operator.attrgetter("date"))
# The comments will probably be sorted with earlier comments at the front of the list
# If you want ten most recent, also add the following line:
# sorted_comments.reverse()
ten_comments = sorted_comments[:10]
That query fetches the user. You need to do another query for the comments:
this_users_comments.order('date').limit(10)
for comment in this_users_comments:
...
class Tag(models.Model):
name = models.CharField(maxlength=100)
class Blog(models.Model):
name = models.CharField(maxlength=100)
tags = models.ManyToManyField(Tag)
Simple models just to ask my question.
I wonder how can i query blogs using tags in two different ways.
Blog entries that are tagged with "tag1" or "tag2":
Blog.objects.filter(tags_in=[1,2]).distinct()
Blog objects that are tagged with "tag1" and "tag2" : ?
Blog objects that are tagged with exactly "tag1" and "tag2" and nothing else : ??
Tag and Blog is just used for an example.
You could use Q objects for #1:
# Blogs who have either hockey or django tags.
from django.db.models import Q
Blog.objects.filter(
Q(tags__name__iexact='hockey') | Q(tags__name__iexact='django')
)
Unions and intersections, I believe, are a bit outside the scope of the Django ORM, but its possible to to these. The following examples are from a Django application called called django-tagging that provides the functionality. Line 346 of models.py:
For part two, you're looking for a union of two queries, basically
def get_union_by_model(self, queryset_or_model, tags):
"""
Create a ``QuerySet`` containing instances of the specified
model associated with *any* of the given list of tags.
"""
tags = get_tag_list(tags)
tag_count = len(tags)
queryset, model = get_queryset_and_model(queryset_or_model)
if not tag_count:
return model._default_manager.none()
model_table = qn(model._meta.db_table)
# This query selects the ids of all objects which have any of
# the given tags.
query = """
SELECT %(model_pk)s
FROM %(model)s, %(tagged_item)s
WHERE %(tagged_item)s.content_type_id = %(content_type_id)s
AND %(tagged_item)s.tag_id IN (%(tag_id_placeholders)s)
AND %(model_pk)s = %(tagged_item)s.object_id
GROUP BY %(model_pk)s""" % {
'model_pk': '%s.%s' % (model_table, qn(model._meta.pk.column)),
'model': model_table,
'tagged_item': qn(self.model._meta.db_table),
'content_type_id': ContentType.objects.get_for_model(model).pk,
'tag_id_placeholders': ','.join(['%s'] * tag_count),
}
cursor = connection.cursor()
cursor.execute(query, [tag.pk for tag in tags])
object_ids = [row[0] for row in cursor.fetchall()]
if len(object_ids) > 0:
return queryset.filter(pk__in=object_ids)
else:
return model._default_manager.none()
For part #3 I believe you're looking for an intersection. See line 307 of models.py
def get_intersection_by_model(self, queryset_or_model, tags):
"""
Create a ``QuerySet`` containing instances of the specified
model associated with *all* of the given list of tags.
"""
tags = get_tag_list(tags)
tag_count = len(tags)
queryset, model = get_queryset_and_model(queryset_or_model)
if not tag_count:
return model._default_manager.none()
model_table = qn(model._meta.db_table)
# This query selects the ids of all objects which have all the
# given tags.
query = """
SELECT %(model_pk)s
FROM %(model)s, %(tagged_item)s
WHERE %(tagged_item)s.content_type_id = %(content_type_id)s
AND %(tagged_item)s.tag_id IN (%(tag_id_placeholders)s)
AND %(model_pk)s = %(tagged_item)s.object_id
GROUP BY %(model_pk)s
HAVING COUNT(%(model_pk)s) = %(tag_count)s""" % {
'model_pk': '%s.%s' % (model_table, qn(model._meta.pk.column)),
'model': model_table,
'tagged_item': qn(self.model._meta.db_table),
'content_type_id': ContentType.objects.get_for_model(model).pk,
'tag_id_placeholders': ','.join(['%s'] * tag_count),
'tag_count': tag_count,
}
cursor = connection.cursor()
cursor.execute(query, [tag.pk for tag in tags])
object_ids = [row[0] for row in cursor.fetchall()]
if len(object_ids) > 0:
return queryset.filter(pk__in=object_ids)
else:
return model._default_manager.none()
I've tested these out with Django 1.0:
The "or" queries:
Blog.objects.filter(tags__name__in=['tag1', 'tag2']).distinct()
or you could use the Q class:
Blog.objects.filter(Q(tags__name='tag1') | Q(tags__name='tag2')).distinct()
The "and" query:
Blog.objects.filter(tags__name='tag1').filter(tags__name='tag2')
I'm not sure about the third one, you'll probably need to drop to SQL to do it.
Please don't reinvent the wheel and use django-tagging application which was made exactly for your use case. It can do all queries you describe, and much more.
If you need to add custom fields to your Tag model, you can also take a look at my branch of django-tagging.
This will do the trick for you
Blog.objects.filter(tags__name__in=['tag1', 'tag2']).annotate(tag_matches=models.Count(tags)).filter(tag_matches=2)