I have a working model for a chat application. The requirement is such that upon service restart, we design an in-mem mapper and fetch the first page details for each DM / group from that mapper based on the ID.
The working model is as follows:
'''
RECEIVER_SENDER_MAPPER = {"61e7dbcf9edba13755a4eb07" : {"61e7a5559edba13755a4ea65":[{},{},{},{},first page entries(25)],
"61de751742fc165ec8b729c9":[{},{},{},{},first page entries(25)]},
"61e7a5559edba13755a4ea65" : {"61e7dbcf9edba13755a4eb07":[{},{},{},{},first page entries(25)],
"61de751742fc165ec8b729c9":[{},{},{},{},first page entries(25)]}
}
'''
RECEIVER_SENDER_MAPPER = {}
def sync_to_inmem_from_db():
global RECEIVER_SENDER_MAPPER
message = db.messages.find_one()
if message:
#Fetch all users from db
users = list(db.users.find({},{"_id":1, "username":1}))
prepared_data = {}
counter = 0
for user in users:
counter += 1
#find all message groups which user is a part of
user_channel_ids = list(db.message_groups.find({"channel_members":{"$in":[user["_id"]]}},{"_id":1, "channel_name":1}))
combined_list = user_channel_ids + users
users_mapped_data = {}
for x in combined_list:
counter += 1
if x["_id"] == user["_id"]:
continue
find_query = {"receiver_type":"group", "receiver_id":x["_id"], "pid":"0"}
if x.get("username"):
find_query = {"pid":"0", "receiver_type":"user"}
messages = list(db.messages.find(find_query).sort("created_datetime",
-1).limit(50))
if messages:
users_mapped_data[x["_id"]] = messages
prepared_data[user["_id"]] = users_mapped_data
RECEIVER_SENDER_MAPPER = prepared_data
if not RECEIVER_SENDER_MAPPER:
sync_to_inmem_from_db()
The value of the counter for 70 users and 48 message groups is : 5484, It takes close to 9 mins to create the RECEIVER_SENDER_MAPPER.
I have to reduce this atleast to 1/4th of the value
One optimization i found was, since group messages will be same for all the users of the particular group, i can just create a dictionary this way:
all_channels = list(db.message_groups.find())
channels_data = {channel["_id"] : list(db.messages.find({"receiver_id":channel["_id"]}).limit(50)) for channel in all_channels}
But here again, while looping the users, i have to again loop the groups to find if the "user" is a part of that group or not.!
Any idea to reduce the complexity of this ? Thanks in advance.
Related
I'm trying to set inventory quantity of a product in shopify using the Shopify Python Api.
As i understand it, i need to set the 'inventory_level' of the 'inventory_item' that belongs to the product, but after a few days of searching and testing i still have no luck.
Where i'm at
I have my products showing up in my store with all the data but the inventory quantity.
I'm not sure how to proceed as there's not a whole lot of documentation.
Here's my code for creating a product
def CreateShopifyProduct(data):
# CREATE PRODUCT
product = shopify.Product()
# Add stuff to product, variant and inventoryItem here
product.title = data['title']
#product.status = ""
#product.tags = data['tags']
product.body_html = data['description']
if 'catagory' in data:
product.product_type = data['category']
if 'vendor' in data:
product.vendor = data['vendor']
if 'image_url' in data:
image_path = data['image_url']
image = shopify.Image()
image.src = image_path
product.images = [image]
else:
try:
image = GetLocalImageFiles(data['id'])
product.images = [image]
except:
print("No local images found")
success = product.save() #returns false if the record is invalid
# CREATE VARIANT
variant = shopify.Variant()
if 'ean' in data:
variant.barcode = data['ean']
variant.price = data['gross_price']
variant.weight = data['weight']
#variant.count = data['inventory']
variant.inventory_management = 'shopify'
product.variants = [variant]
variant.product_id = product.id
s = variant.save()
success = product.save() #returns false if the record is invalid
# CREATE INVENTORYITEM
inventoryItem = shopify.InventoryItem()
#inventoryItem = variant.inventory_item
inventoryItem.tracked = True
inventoryItem.id = product.id
variant.inventory_quantity = data['inventory']
inventoryItem.inventory_quantity = data['inventory']
variant.inventory_item = inventoryItem
s = variant.save()
success = product.save()
#ii = inventoryItem.save() # this returns 406
#inv_level = shopify.InventoryLevel.find(inventory_item_ids=6792364982390, location_ids=61763518582)
#quantity = inv_level[0].__dict__['attributes']['available']
#shopify.InventoryLevel.set(location_id=61763518582, inventory_item_id=variant.inventory_item.id, available=data['inventory'])
#shopify.InventoryLevel.connect(61763518582, variant.inventory_item.id)
if product.errors:
#something went wrong, see new_product.errors.full_messages() for example
print("error")
print(product.errors.full_messages())
If i try to set the InventoryLevel with
shopify.InventoryLevel.set(61527654518, inventoryItem.id, 42)
# or
shopify.InventoryLevel.set(location_id=61527654518, inventory_item_id=inventoryItem.id, available=17)
I recieve a
pyactiveresource.connection.ResourceNotFound: Not Found: https://domain.myshopify.com/admin/api/2022-07/inventory_levels/set.json
You need three things to update an inventory level. One, you need a valid location ID. Two, you need the inventory item ID. Finally, you need the amount to adjust inventory to, that will adjust the inventory there to match your needs.
You should really play at the command-line and ensure you can quickly get the information you need, then try your updates. In other words, ensure you are getting a good location ID, inventory item ID and finally, that you know the amount of inventory already in Shopify. Since you have to calculate a delta change, these are the minimum steps most people take.
Note that once you get good at doing one item, you'll realize Shopify also accepts up to 100 at a time, making your updates a lot faster.
I have a list of IDs which corresponds to a set of records (opportunities) in a database. I then pass this list as a parameter in a RESTful API request where I am filtering the results (tickets) by ID. For each match, the query returns JSON data pertaining to the individual record. However, I want to handle when the query does not find a match. I would like to assign some value for this case such as the string "None", because not every opportunity has a ticket. How can I make sure there exists some value in presales_tickets for every ID in opportunity_list? Could I provide a default value in the request for this case?
views.py
opportunities = cwObj.get_opportunities()
temp = []
opportunity_list = []
cw_presales_engineers = []
for opportunity in opportunities:
temp.append(str(opportunity['id']))
opportunity_list = ','.join(temp)
presales_tickets = cwObj.get_tickets_by_opportunity(opportunity_list)
for opportunity in opportunities:
try:
if opportunity['id'] == presales_tickets[0]['opportunity']['id']:
try:
for presales_ticket in presales_tickets:
cw_engineer = presales_ticket['owner']['name']
cw_presales_engineers.append(cw_engineer)
except:
pass
else:
cw_engineer = 'None'
cw_presales_engineers.append(cw_engineer)
except AttributeError:
cw_engineer = ''
cw_presales_engineers.append(cw_engineer)
So, lets say you have a Ticket model and Opportunity model. Connected via a foreign key.
class Opportunity(models.Model):
... some fields here ...
class Ticket(models.Model):
opportunity = models.ForeignKey(Opportunity)
and in your view, you get a list of opportunity ids
def some_view(request):
ids = request.GET['ids']
It sounds, like what you want is to fetch all the tickets for the supplied opportunities and add some default processing for the opportunities that do not have tickets. If that is the case, why not do something like
def some_view(request):
ids = request.GET['ids']
tickets = Ticket.objects.filter(opportunity__id__in=ids)
results = []
for ticket in tickets:
result = ... do your thing here ...
results.append(result)
# now handle missing opportunities
good_ids = tickets.values_list('opportunity__id', flat=True).distinct()
for id in ids:
if id not in good_ids:
result = ... do your default processing ...
results.append(result)
Is that what you are trying to do?
Write a code to print all the unique customers visited in last hour
My try:
import datetime
def find_repeated_customer():
file_obj = open(" my file path","r")
customer_last_visit = {}
repeat_customer = set()
while line in file_obj:
timestamp,customer_id,page_id = line.split(" : ")
last_visit = customer_last_vist.get(customer_id,None)
if not last_visit:
customer_last_visit[customer_id] = last_visit
else:
# assuming time stamp looks like 2016-10-29 01:03:26.947000
year,month,date = timestamp.split(" ")[0].split("-")
current_visit = datetime.date(year,month,date)
day_diff = current_visit - last_visit
if day_diff >=1:
repeat_customer.add(customer_id)
customer_last_visit[customer_id] = current_visit
I am completely failing over in order to get my desired output. By doing this I am able to get repeated customers in last one day but how to get unique users?
You can't do this kind of manipulation in one pass. You have to pass once through lines to get customers, and only then You can check who came once. In another pass, You check if current customer is in list on once-customers and do something with him.
I have the following class to keep my records:
class List(ndb.Model):
'''
Index
Key: sender
'''
sender = ndb.StringProperty()
...
counter = ndb.IntegerProperty(default=0)
ignore = ndb.BooleanProperty(default=False)
added = ndb.DateTimeProperty(auto_now_add=True, indexed=False)
updated = ndb.DateTimeProperty(auto_now=True, indexed=False)
The following code is used to return all entities I need:
entries = List.query()
entries = entries.filter(List.counter > 5)
entries = entries.filter(List.ignore == False)
entries = entries.fetch()
How should I modify the code to get 10 random records from entries? I am planning to have a daily cron task to extract random records, so they should be really random. What is the best way to get these records (to minimize number of read operations)?
I don't think that the following code is the best:
entries = random.sample(entries, 10)
Well after reading the comments the only improvement you can make as far I can see is to fetch the keys only and limit if possible.
Haven't tested but like so
list_query = List.query()
list_query = list_query.filter(List.counter > 5)
list_query = list_query.filter(List.ignore == False)
list_keys = list_query.fetch(keys_only=True) # maybe put a limit here.
list_keys = random.sample(list_keys, 10)
lists = [list_key.get() for list_key in list_keys]
I try to use extjs with django, i started extjs with php. for create a paginate grid i used to get total count of the data and get the start and limit value. In django, the pagination does not work. what am i forgot? is it my query? i use postgresql. this is my code. i
if request.POST['task'] == 'OK':
pers = Plante.objects.all().values('id','name','year')
nbrows = len(pers)
if request.POST['start']:
start = request.POST['start']
else:
start = request.GET['start']
if request.POST['limit']:
end = request.POST['limit']
else:
end = request.GET['limit']
pers = Plante.objects.all().values('id','name','year')[start:end]
start = int(request.POST.get('start') or request.GET.get('start'))
limit = int(request.POST.get('limit') or request.GET.get('limit'))
pers = Plante.objects.all().values('id','name','year')[start:start+limit]
I know it's quite late but here's a way how you can achieve it using the "start" & "limit" pagination params sent by EXTJS.
def fetchRecords(self, params):
totalCount = 0
pageNumber = 1
records = []
ids = []
#Instanciate your query object
query = Q()
#Not really relevant for this case but in case you have any filter criteria params then put them here
if(params.get("searchStartDate")):
startDate = datetime.strptime(params.get("searchStartDate"), '%Y-%m-%d').date()
query &= Q(date_created__gte=startDate)
if(params.get("searchEndDate")):
endDate = datetime.strptime(params.get("searchEndDate"), '%Y-%m-%d').date()
query &= Q(date_created__lte=endDate)
# Get the total count, EXT JS Grids need the total count value to be able to paginate
totalCount = YourModel.objects.filter(query).count()
#Get the primary keys, we do this because we don't want to get all the objects onto memory. The paginator doesn't
#Optimize the fetched data. If your table has millions of records and you load all the record objects to mem, the
#execution might be quite slow
your_model_ids_list = YourModel.objects.filter(query).order_by("-id").only('id')
#Compute the page number based on the pagination "start" & "limit" params sent by EXT grid
if(int(params.get("start")) != 0 ):
pageNumber = (int(params.get("start")) / int(params.get("limit"))) + 1
#Instanciate the paginator object with the unique id's list matching your filter criteria & limit
paginator = Paginator(your_model_ids_list, int(params.get("limit")))
#Get the records that fall on the particular page number that we computed above
recordIds = paginator.page(pageNumber)
#Iterate through the record IDs and place them in an array list
for recordId in recordIds.object_list:
ids.append(recordId.id)
#Now fetch the records from your model based on the unique ids that fall on the particular page fetched
#above
result = YourModel.objects.filter(Q(pk__in=ids)).order_by("-id")
#Formulate your response object and return the data
return {'totalCount': totalCount, 'records': result}