Python Web Scrapping - python

I'm trying to write a code in Python that gets specific data from this site:
exchange data
The information i'm interested in, is in the table at the middle of the page and I want to get the Price for each Q in the corresponding row. For example Q=0 price=18.9, Q=1 price = 19, etc.
Any suggestions how that can be done?

You can get the desired data from API. Because data is loaded dynamically by JS via API.
import requests
api_url='https://simcotools.app/api/resource/18'
req=requests.get(api_url).json()
data=[]
for item in req['qualities']:
Quantity = item['quality']
Price = item['price']
data.append({
'Quantity':Quantity,
'Price': Price
})
print(data)
Output:
[{'Quantity': 0, 'Price': 18.8}, {'Quantity': 1, 'Price': 19.0}, {'Quantity': 2, 'Price': 19.1}, {'Quantity': 3, 'Price': 19.2}, {'Quantity': 4, 'Price': 19.3}, {'Quantity': 5, 'Price': 20.75}, {'Quantity': 6, 'Price': 23.5}, {'Quantity': 7, 'Price': 23.5}, {'Quantity': 8, 'Price': 39.0}, {'Quantity': 9, 'Price': 50.0}, {'Quantity': 10, 'Price': 100.0}]

Related

Get unique elements in an array and add the value if same element is found - django

<QuerySet
[{'product': 6, 'quantity': 1},
{'product': 6, 'quantity': 10},
{'product': 7, 'quantity': 2},
{'product': 9, 'quantity': 3}]
>
how to get unique elements only in the array and add the quantity if same product is found. In the above eg: there are two "product: 6", so the new QuerySet should be
<QuerySet
[{'product': 6, 'quantity': 11},
{'product': 7, 'quantity': 2},
{'product': 9, 'quantity': 3}]
>

Repeating values when we use append function in list or update in dict

when I try to append a Dictonary values to a list I'm getting the value appended multiple time but when I try to see what is really happening in there by using print statement, the values are iterating properly but while appending it is appending same value multiple times
Here is my code
from random import randint
x = {'1':
{
'item_id': 6,
'item_name': 'burger',
'item_price': 10,
'item_quantity': 2
},
'2':
{
'item_id': 7,
'item_name': 'pizza',
'item_price': 15,
'item_quantity': 4
},
'3':
{
'item_id': 8,
'item_name': 'Biryani',
'item_price': 20,
'item_quantity': 6
}
}
cart=[]
items = {}
for y in x.values():
items['name'] = y['item_name']
items['price'] = y['item_price']
items['quantity'] = y['item_quantity']
print(items)
cart.append(items)
print(cart)
And This is the Output:
{'name': 'burger', 'price': 10, 'quantity': 2}
{'name': 'pizza', 'price': 15, 'quantity': 4}
{'name': 'Biryani', 'price': 20, 'quantity': 6}
[{'name': 'Biryani', 'price': 20, 'quantity': 6}, {'name': 'Biryani', 'price': 20, 'quantity': 6}, {'name': 'Biryani', 'price': 20, 'quantity': 6}]
>
You are mutating and appending the same dict items to cart again and again. Instantiate a new dict items = {} in each iteration of the for loop.
try initializing your dictionary every time inside the loop
cart=[]
for y in x.values():
items = {}
items['name'] = y['item_name']
items['price'] = y['item_price']
items['quantity'] = y['item_quantity']
print(items)
cart.append(items)
print(cart)

Table of dictionaries

I have a table of dicts looking like:
[{
'variant_id': 4126274,
'stock': [
{'stock_id': 6, 'quantity': 86},
{'stock_id': 4, 'quantity': 23},
{'stock_id': 3, 'quantity': 9}
]
}, ...]
My goal is to unzip every piece of stock to look like this:
[{'variant_id': 4126274, 'stock_id': 6, 'quantity':86}
{'variant_id': 4126274, 'stock_id': 4, 'quantity':23}
{'variant_id': 4126274, 'stock_id': 3, 'quantity':9}...]
Is there any fast and optimal way to do this?
You could do something like:
result = [{'variant_id': entry['variant_id'],
'stock_id': stock_entry['stock_id'],
'quantity': stock_entry['quantity']} for entry in table for stock_entry in entry['stock']]
This gives
[{'quantity': 86, 'stock_id': 6, 'variant_id': 4126274},
{'quantity': 23, 'stock_id': 4, 'variant_id': 4126274},
{'quantity': 9, 'stock_id': 3, 'variant_id': 4126274}]
Here are two approaches: one with nested for loops, and one with a list comprehension.
data = [{'variant_id': 4126274, 'stock': [{'stock_id': 6, 'quantity': 86}, {'stock_id': 4, 'quantity': 23}, {'stock_id': 3, 'quantity': 9}]}]
result = []
for entry in data:
for stock in entry['stock']:
result.append({'variant_id': entry['variant_id'], 'stock_id': stock['stock_id'], 'quantity': stock['quantity']})
print(result)
result_list_comprehension = [{'variant_id': entry['variant_id'], 'stock_id': stock['stock_id'], 'quantity': stock['quantity']} for entry in data for stock in entry['stock']]
print(result_list_comprehension)

How to add new keys and values to dictionaries inside of a list in python?

I'm sorry if this is a basic question, but how can i add a new key-value pair to my dictionaries inside my list?
I want to loop over the first list so the first list-value goes to the first list dictionary with a key of 'quantity' and so on, see the output below.
Output:
[1, 2]
[{'id': 1, 'title': 'Tights 1', 'price': Decimal('200.00'), 'category_id': 1}, {'id': 2, 'title': 'Tights 2', 'price': Decimal('400.00'), 'category_id': 1}]
Desired output:
[{'id': 1, 'title': 'Tights 1', 'price': Decimal('200.00'), 'quantity': 1, 'category_id': 1}, {'id': 2, 'title': 'Tights 2', 'price': Decimal('400.00'), 'quantity': 2, 'category_id': 1}]
To add a new key pair value into your dictionary you do:
dict['new_key'] = new_value
So adapting your code to create a new entry:
from decimal import Decimal
order_item_values = [{'id': 1, 'title': 'Tights 1', 'price': Decimal('200.00'), 'category_id': 1}, {'id': 2, 'title': 'Tights 2', 'price': Decimal('400.00'), 'category_id': 1}]
for item in order_item_values:
item['quantity'] = item['id']
print(order_item_values)
Loop through the list and add the elements to each dictionary.
lst = list(item.values())
for i in range(len(lst)):
lst[i]["quantity"] = i + 1

List of dictionaries - stack one value of dictionary

I have trouble in adding one value of dictionary when conditions met, For example I have this list of dictionaries:
[{'plu': 1, 'price': 150, 'quantity': 2, 'stock': 5},
{'plu': 2, 'price': 150, 'quantity': 7, 'stock': 10},
{'plu': 1, 'price': 150, 'quantity': 6, 'stock': 5},
{'plu': 1, 'price': 200, 'quantity': 4, 'stock': 5},
{'plu': 2, 'price': 150, 'quantity': 3, 'stock': 10}
]
Then output should look like this:
[{'plu': 1, 'price': 150, 'quantity': 8, 'stock': 5},
{'plu': 1, 'price': 200, 'quantity': 4, 'stock': 5},
{'plu': 2, 'price': 150, 'quantity': 10, 'stock': 10}
]
Quantity should be added only if plu and price are the same, it should ignore key:values other than that (ex. stock). What is the most efficient way to do that?
#edit
I tried:
import itertools as it
keyfunc = lambda x: x['plu']
groups = it.groupby(sorted(new_data, key=keyfunc), keyfunc)
x = [{'plu': k, 'quantity': sum(x['quantity'] for x in g)} for k, g in groups]
But it works only on plu and then I get only quantity value when making html table in django, other are empty
You need to sort/groupby the combined key, not just one key. Easiest/most efficient way to do this is with operator.itemgetter. To preserve an arbitrary stock value, you'll need to use the group twice, so you'll need to convert it to a sequence:
from operator import itemgetter
keyfunc = itemgetter('plu', 'price')
# Unpack key and listify g so it can be reused
groups = ((plu, price, list(g))
for (plu, price), g in it.groupby(sorted(new_data, key=keyfunc), keyfunc))
x = [{'plu': plu, 'price': price, 'stock': g[0]['stock'],
'quantity': sum(x['quantity'] for x in g)}
for plu, price, g in groups]
Alternatively, if stock is guaranteed to be the same for each unique plu/price pair, you can include it in the key to simplify matters, so you don't need to listify the groups:
keyfunc = itemgetter('plu', 'price', 'stock')
groups = it.groupby(sorted(new_data, key=keyfunc), keyfunc)
x = [{'plu': plu, 'price': price, 'stock': stock,
'quantity': sum(x['quantity'] for x in g)
for (plu, price, stock), g in groups]
Optionally, you could create getquantity = itemgetter('quantity') at top level (like the keyfunc) and change sum(x['quantity'] for x in g) to sum(map(getquantity, g)) which pushes work to the C layer in CPython, and can be faster if your groups are large.
The other approach is to avoid sorting entirely using collections.Counter (or collections.defaultdict(int), though Counter makes the intent more clear here):
from collections import Counter
grouped = Counter()
for plu, price, stock, quantity in map(itemgetter('plu', 'price', 'stock', 'quantity'), new_data):
grouped[plu, price, stock] += quantity
then convert back to your preferred form with:
x = [{'plu': plu, 'price': price, 'stock': stock, 'quantity': quantity}
for (plu, price, stock), quantity in grouped.items()]
This should be faster for large inputs, since it replaces O(n log n) sorting work with O(n) dict operations (which are roughly O(1) cost).
Using pandas will make this a trivial problem:
import pandas as pd
data = [{'plu': 1, 'price': 150, 'quantity': 2, 'stock': 5},
{'plu': 2, 'price': 150, 'quantity': 7, 'stock': 10},
{'plu': 1, 'price': 150, 'quantity': 6, 'stock': 5},
{'plu': 1, 'price': 200, 'quantity': 4, 'stock': 5},
{'plu': 2, 'price': 150, 'quantity': 3, 'stock': 10}]
df = pd.DataFrame.from_records(data)
# df
#
# plu price quantity stock
# 0 1 150 2 5
# 1 2 150 7 10
# 2 1 150 6 5
# 3 1 200 4 5
# 4 2 150 3 10
new_df = df.groupby(['plu','price','stock'], as_index=False).sum()
new_df = new_df[['plu','price','quantity','stock']] # Optional: reorder the columns
# new_df
#
# plu price quantity stock
# 0 1 150 8 5
# 1 1 200 4 5
# 2 2 150 10 10
And finally, if you want to, port it back to dict (though I would argue pandas give you a lot more functionality to handle the data elements):
new_data = df2.to_dict(orient='records')
# new_data
#
# [{'plu': 1, 'price': 150, 'quantity': 8, 'stock': 5},
# {'plu': 1, 'price': 200, 'quantity': 4, 'stock': 5},
# {'plu': 2, 'price': 150, 'quantity': 10, 'stock': 10}]

Categories