Related
I'm trying to use this create_df() function in Streamlit to gather a list of user-provided URLs called "recipes" and loop through each URL to return a df I've labeled "res" towards the end of the function. I've tried several approaches with the Streamlit syntax but I just cannot get this to work as I'm getting this error message:
recipe_scrapers._exceptions.WebsiteNotImplementedError: recipe-scrapers exception: Website (h) not supported.
Have a look at my entire repo here. The main.py script works just fine once you've installed all requirements locally, but when I try running the same script with Streamlit syntax in the streamlit.py script I get the above error. Once you run streamlit run streamlit.py in your terminal and have a look at the UI I've create it should be quite clear what I'm aiming at, which is providing the user with a csv of all ingredients in the recipe URLs they provided for a convenient grocery shopping list.
Any help would be greatly appreciated!
def create_df(recipes):
"""
Description:
Creates one df with all recipes and their ingredients
Arguments:
* recipes: list of recipe URLs provided by user
Comments:
Note that ingredients with qualitative amounts e.g., "scheutje melk", "snufje zout" have been ommitted from the ingredient list
"""
df_list = []
for recipe in recipes:
scraper = scrape_me(recipe)
recipe_details = replace_measurement_symbols(scraper.ingredients())
recipe_name = recipe.split("https://www.hellofresh.nl/recipes/", 1)[1]
recipe_name = recipe_name.rsplit('-', 1)[0]
print("Processing data for "+ recipe_name +" recipe.")
for ingredient in recipe_details:
try:
df_temp = pd.DataFrame(columns=['Ingredients', 'Measurement'])
df_temp[str(recipe_name)] = recipe_name
ing_1 = ingredient.split("2 * ", 1)[1]
ing_1 = ing_1.split(" ", 2)
item = ing_1[2]
measurement = ing_1[1]
quantity = float(ing_1[0]) * 2
df_temp.loc[len(df_temp)] = [item, measurement, quantity]
df_list.append(df_temp)
except (ValueError, IndexError) as e:
pass
df = pd.concat(df_list)
print("Renaming duplicate ingredients e.g., Kruimige aardappelen, Voorgekookte halve kriel met schil -> Aardappelen")
ingredient_dict = {
'Aardappelen': ('Dunne frieten', 'Half kruimige aardappelen', 'Voorgekookte halve kriel met schil',
'Kruimige aardappelen', 'Roodschillige aardappelen', 'Opperdoezer Ronde aardappelen'),
'Ui': ('Rode ui'),
'Kipfilet': ('Kipfilet met tuinkruiden en knoflook'),
'Kipworst': ('Gekruide kipworst'),
'Kipgehakt': ('Gemengd gekruid gehakt', 'Kipgehakt met Mexicaanse kruiden', 'Half-om-halfgehakt met Italiaanse kruiden',
'Kipgehakt met tuinkruiden'),
'Kipshoarma': ('Kalkoenshoarma')
}
reverse_label_ing = {x:k for k,v in ingredient_dict.items() for x in v}
df["Ingredients"].replace(reverse_label_ing, inplace=True)
print("Assigning ingredient categories")
category_dict = {
'brood': ('Biologisch wit rozenbroodje', 'Bladerdeeg', 'Briochebroodje', 'Wit platbrood'),
'granen': ('Basmatirijst', 'Bulgur', 'Casarecce', 'Cashewstukjes',
'Gesneden snijbonen', 'Jasmijnrijst', 'Linzen', 'Maïs in blik',
'Parelcouscous', 'Penne', 'Rigatoni', 'Rode kidneybonen',
'Spaghetti', 'Witte tortilla'),
'groenten': ('Aardappelen', 'Aubergine', 'Bosui', 'Broccoli',
'Champignons', 'Citroen', 'Gele wortel', 'Gesneden rodekool',
'Groene paprika', 'Groentemix van paprika, prei, gele wortel en courgette',
'IJsbergsla', 'Kumato tomaat', 'Limoen', 'Little gem',
'Paprika', 'Portobello', 'Prei', 'Pruimtomaat',
'Radicchio en ijsbergsla', 'Rode cherrytomaten', 'Rode paprika', 'Rode peper',
'Rode puntpaprika', 'Rode ui', 'Rucola', 'Rucola en veldsla', 'Rucolamelange',
'Semi-gedroogde tomatenmix', 'Sjalot', 'Sperziebonen', 'Spinazie', 'Tomaat',
'Turkse groene peper', 'Veldsla', 'Vers basilicum', 'Verse bieslook',
'Verse bladpeterselie', 'Verse koriander', 'Verse krulpeterselie', 'Wortel', 'Zoete aardappel'),
'kruiden': ('Aïoli', 'Bloem', 'Bruine suiker', 'Cranberrychutney', 'Extra vierge olijfolie',
'Extra vierge olijfolie met truffelaroma', 'Fles olijfolie', 'Gedroogde laos',
'Gedroogde oregano', 'Gemalen kaneel', 'Gemalen komijnzaad', 'Gemalen korianderzaad',
'Gemalen kurkuma', 'Gerookt paprikapoeder', 'Groene currykruiden', 'Groentebouillon',
'Groentebouillonblokje', 'Honing', 'Italiaanse kruiden', 'Kippenbouillonblokje', 'Knoflookteen',
'Kokosmelk', 'Koreaanse kruidenmix', 'Mayonaise', 'Mexicaanse kruiden', 'Midden-Oosterse kruidenmix',
'Mosterd', 'Nootmuskaat', 'Olijfolie', 'Panko paneermeel', 'Paprikapoeder', 'Passata',
'Pikante uienchutney', 'Runderbouillonblokje', 'Sambal', 'Sesamzaad', 'Siciliaanse kruidenmix',
'Sojasaus', 'Suiker', 'Sumak', 'Surinaamse kruiden', 'Tomatenblokjes', 'Tomatenblokjes met ui',
'Truffeltapenade', 'Ui', 'Verse gember', 'Visbouillon', 'Witte balsamicoazijn', 'Wittewijnazijn',
'Zonnebloemolie', 'Zwarte balsamicoazijn'),
'vlees': ('Gekruide runderburger', 'Half-om-half gehaktballetjes met Spaanse kruiden', 'Kipfilethaasjes', 'Kipfiletstukjes',
'Kipgehaktballetjes met Italiaanse kruiden', 'Kippendijreepjes', 'Kipshoarma', 'Kipworst', 'Spekblokjes',
'Vegetarische döner kebab', 'Vegetarische kaasschnitzel', 'Vegetarische schnitzel'),
'zuivel': ('Ei', 'Geraspte belegen kaas', 'Geraspte cheddar', 'Geraspte grana padano', 'Geraspte oude kaas',
'Geraspte pecorino', 'Karnemelk', 'Kruidenroomkaas', 'Labne', 'Melk', 'Mozzarella',
'Parmigiano reggiano', 'Roomboter', 'Slagroom', 'Volle yoghurt')
}
reverse_label_cat = {x:k for k,v in category_dict.items() for x in v}
df["Category"] = df["Ingredients"].map(reverse_label_cat)
col = "Category"
first_col = df.pop(col)
df.insert(0, col, first_col)
df = df.sort_values(['Category', 'Ingredients'], ascending = [True, True])
print("Merging ingredients by row across all recipe columns using justify()")
gp_cols = ['Ingredients', 'Measurement']
oth_cols = df.columns.difference(gp_cols)
arr = np.vstack(df.groupby(gp_cols, sort=False, dropna=False).apply(lambda gp: justify(gp.to_numpy(), invalid_val=np.NaN, axis=0, side='up')))
# Reconstruct DataFrame
# Remove entirely NaN rows based on the non-grouping columns
res = (pd.DataFrame(arr, columns=df.columns)
.dropna(how='all', subset=oth_cols, axis=0))
res = res.fillna(0)
res['Total'] = res.drop(['Ingredients', 'Measurement'], axis=1).sum(axis=1)
res=res[res['Total'] !=0] #To drop rows that are being duplicated with 0 for some reason; will check later
print("Processing complete!")
return res
Your function create_df needs a list as an argument but st.text_input returs always a string.
In your streamlit.py, replace this df_download = create_df(recs) by this df_download = create_df([recs]). But if you need to handle multiple urls, you should use str.split like this :
def create_df(recipes):
recipes = recipes.split(",") # <--- add this line to make a list from the user-input
### rest of the code ###
if download:
df_download = create_df(recs)
# Output :
I receive every day a report with some values and I have to match postal codes from countries all over the world to get the right region. Then I upload the result in my Django app.
Here's a look at my report:
Order Number
Date
City
Postal code
930276
27/09/2022
Madrid
cp: 28033
929670
27/09/2022
Lisboa
cp: 1600-812
I have thousands of rows like this. The objective is to retrieve the region in ISO 3166-2 format. To help me, I accessed the following page Geonames and downloaded all the countries' information (example: "FR.txt", "ES.txt"...)
Because this is a huge txt file, I chose to store it on a S3 Server.
Here is what I tried:
def access_scaleway(region_name, endpoint_url, access_key, secret_key):
""" Accessing Scaleway Bucket """
scaleway = boto3.client('s3', region_name=region_name, endpoint_url=endpoint_url, aws_access_key_id=access_key,
aws_secret_access_key=secret_key)
return scaleway
def get_region_code_accessing_scaleway(countries, regions):
''' Retrieves the region code from the region name. '''
list_countries = countries
list_regions = regions
list_regions_codes = []
scaleway_session = access_scaleway(region_name=settings.SCALEWAY_S3_REGION_NAME,
endpoint_url=settings.SCALEWAY_S3_ENDPOINT_URL,
access_key=settings.SCALEWAY_ACCESS_KEY_ID,
secret_key=settings.SCALEWAY_SECRET_ACCESS_KEY)
for country, region in zip(list_countries, list_regions):
try:
obj = scaleway_session.get_object(Bucket=settings.SCALEWAY_STORAGE_BUCKET_NAME, Key=f'countries/{country}.txt')
df = pd.read_csv(io.BytesIO(obj['Body'].read()), sep='\t', header=None)
df.columns = ['country code', 'postal code', 'place name', 'admin name1', 'admin code1', 'admin name2', 'admin code2', 'admin name3', 'admin code3', 'latitude', 'longitude', 'accuracy']
df['postal code'] = df['postal code'].astype(str)
df['postal code'] = df['postal code'].str.zfill(5)
# Removing all spaces and special characters
postal_code = re.sub("[^0-9^-]", '', region).strip()
region_code = country + "-" + df[df['postal code'] == postal_code]['admin code1'].values[0]
list_regions_codes.append(region_code)
except AttributeError:
list_regions_codes.append(None)
except ValueError:
list_regions_codes.append(None)
return list_regions_codes
But it is way too long. For a simple report of 1000 rows, it takes like 30 min.
My second try was to go with the OpenDataSoft public API. Here is what I tried:
def fetch_data(url, params, headers=None):
response = requests.get(url=url, params=params, headers=headers)
return response
def get_region_code_accessing_scaleway(countries, regions):
''' Retrieves the region code from the region name. '''
list_countries = countries
list_regions = regions
list_regions_codes = []
for country, region in zip(list_countries, list_regions):
try:
#Get response from API
postal_code = re.sub("[^0-9^-]", '', region).strip()
response = fetch_data(
url="https://data.opendatasoft.com/api/v2/catalog/datasets/geonames-postal-code%40public/records?",
params="select=country_code%2C%20postal_code%2C%20admin_code1&where=country_code%3D%22" + country + "%22%20and%20postal_code%3D%22" + postal_code + "%22")
if response.status_code == 200:
data = response.json()
if len(data['records']) > 0:
list_regions_codes.append(country + "-" + data['records'][0]['record']['fields']['admin_code1'])
else:
list_regions_codes.append(None)
else:
print('Error:" + response.status_code')
list_regions_codes.append(None)
But once again, it takes like forever to get matching values.
The last thing I tried was to go with pgeocode but it was also too long.
I don't understand why it is so long because the desired output is this one:
Order Number
Date
City
Postal code
Region code
930276
27/09/2022
Madrid
cp: 28033
ES-MD
929670
27/09/2022
Lisboa
cp: 1600-812
PT-08
Do you have any idea to speed up the process?
I have the following code:
import pandas as pd
y = pd.ExcelFile('C:\\Users\\vibhu\\Desktop\\Training docs\\excel training\\super store data transformation\\Sample - Superstore data transformation by Vaibhav.xlsx')
superstore_orders = y.parse(sheet_name='Orders Input data')
superstore_orders.dtypes
factual_table= superstore_orders[['Order ID','Customer ID','Postal Code','Product ID','Product Name','Sales','Quantity','Discount','Profit' ]]
Order_table= superstore_orders[['Order ID','Order Date','Ship Date','Ship Mode']]
Order_table1= Order_table.drop_duplicates(subset='Order ID', keep='first', inplace=False)
Customer_table= superstore_orders[['Customer ID','Customer Name','Segment']]
Customer_table1= Customer_table.drop_duplicates(subset='Customer ID', keep='first', inplace=False)
Geographical_table= superstore_orders[['Postal Code','Country','City','State','Region']]
Geographical_table1= Geographical_table.drop_duplicates(subset='Postal Code', keep='first', inplace=False)
Product_table= superstore_orders[['Product ID','Category','Sub-Category','Product Name']]
Product_table1= Product_table.drop_duplicates(subset=['Product ID','Product Name'], keep='first', inplace=False)
Final_factual_data = pd.merge(Order_table1, factual_table, how='left', on='Order ID')
Final_factual_data = pd.merge(Customer_table1, Final_factual_data, how='left', on='Customer ID')
Final_factual_data = pd.merge(Geographical_table1,Final_factual_data,how='left', on='Postal Code')
Final_factual_data = pd.merge(Product_table1,Final_factual_data,how='left', on=['Product ID','Product Name'] )
Output is this format:- Product ID Category Sub-Category Product Name Postal Code Country City State Region Customer ID Customer Name Segment Order ID Order Date Ship Date Ship Mode Sales Quantity Discount Profit
I require reformatting in this order :
Order ID order date ship date ship mode Customer ID cutomer name segment Postal Code country city state reion Product ID Product Name product key cateory subcategory Sales Quantity Discount Profit
Final_factual_data1 = Final_factual_data [['Order ID','Order Date','Ship Date','Ship Mode','Customer ID','Customer Name','Segment','Country','City','State','Postal Code','Region','Product ID','Category','Sub-Category','Product Name','Sales','Quantity','Discount','Profit']]
this code help me to get the desired answer
Just assign the intended ordered sequence to columns attribute:
Final_factual_data.columns = ['Order ID', 'order date', 'ship date', 'ship mode', 'Customer ID', 'cutomer name', 'segment', 'Postal Code', 'country', 'city', 'state reion', 'Product ID', 'Product Name', 'product key', 'cateory', 'subcategory', 'Sales', 'Quantity', 'Discount', 'Profit']
I am developping a module to use a custom formula for the amount calculation method in invoices and sales orders.
The line amount formula should be : nbJours * price_unit * quantity instead of the default formula : price_unit * quantity
I added a custom field to AccountInvoiceLine class by inheriting it as following :
class AccountInvoiceLine(models.Model):
_inherit = "account.invoice.line"
# Nombre de jours de location
nombreJours = fields.Integer("Nombre de jours",default=1,required=True)
#api.multi
#api.depends('nombreJours','price_unit', 'discount', 'invoice_line_tax_ids', 'quantity', 'product_id', 'invoice_id.partner_id', 'invoice_id.currency_id', 'invoice_id.company_id', 'invoice_id.date_invoice', 'invoice_id.date')
def _compute_price(self):
...
...
if self.invoice_line_tax_ids:
taxes = self.invoice_line_tax_ids.compute_all(self.nombreJours, price, currency, self.quantity, product=self.product_id, partner=self.invoice_id.partner_id)
# Calcul du sous-total de la ligne
self.price_subtotal = price_subtotal_signed = taxes['total_excluded'] if taxes else self.quantity * price * self.nombreJours
self.price_total = taxes['total_included'] if taxes else self.price_subtotal
...
...
And I also need to set the custom formula in the AccountTax class. I tried to override the compute_all() method :
# Modification du modèle de Taxes
class AccountTax(models.Model):
_inherit = 'account.tax'
#api.multi
def compute_all(self, nbJrs=1, price_unit=1, currency=None, quantity=1.0, product=None, partner=None):
...
...
if not base_values:
odooAmount = price_unit * quantity
customAmount = nbJrs * odooAmount
total_excluded = total_included = base = round( customAmount , prec)
else:
total_excluded, total_included, base = base_values
...
...
return {
'taxes': sorted(taxes, key=lambda k: k['sequence']),
'total_excluded': currency.round(total_excluded) if round_total else total_excluded,
'total_included': currency.round(total_included) if round_total else total_included,
'base': base,
}
I've successfully updated my module, but when I try to add a product in a new invoice I get this error :
... ... ...
File "/OdooERP/Odoo 11.0/addons/account/models/account_invoice.py", line 618, in _onchange_invoice_line_ids taxes_grouped = self.get_taxes_values()
File "/OdooERP/Odoo 11.0/addons/account/models/account_invoice.py", line 889, in get_taxes_values taxes = line.invoice_line_tax_ids.compute_all(price_unit, self.currency_id, line.quantity, line.product_id, self.partner_id)['taxes']
File "/OdooERP/Instances/xaymalab/addons/sunu_location_event/models/accountinvoice.py",line 82, in compute_all odooAmount = price_unit * quantity
TypeError: unsupported operand type(s) for *: 'res.currency' and 'product.product'
When I remove "nbJrs" parameter in the definition, it works!
The values shift one parameter to the right.
Can anybody help me out regarding it?
You've made a mistake on overriding compute_all() of account.tax. Seems you've added a new parameter nbJrs. The original Method:
def compute_all(self, price_unit, currency=None,
quantity=1.0, product=None, partner=None)
and yours:
def compute_all(self, nbJrs=1, price_unit=1, currency=None,
quantity=1.0, product=None, partner=None):
That just won't work this way. The error is: You're multiplying the value of currency with the value of product, because the values shift one parameter to the right.
How I solved this :
I was trying to override the compute_all() by adding a new parameter. It was the wrong way.
I updated the _compute_price() by adding a new variable "rentalprice" and passing it to compute_all():
def _compute_price(self):
currency = self.invoice_id and self.invoice_id.currency_id or None
price = self.price_unit * (1 - (self.discount or 0.0) / 100.0)
# Prix unitaire d'un article pendant la durée totale de location:
rentalprice = self.nombrejours * price
taxes = False
if self.invoice_line_tax_ids:
taxes = self.invoice_line_tax_ids.compute_all(rentalprice, currency, self.quantity, product=self.product_id, partner=self.invoice_id.partner_id)
I removed the AccountTax inheritance code because I no longer need it, updated my module and now it's working.
Thanks to #CZoellner for helping me to better understand what I was missing.
I'm creating a custom module for Odoo.
I have Fabrication Orders with different phases : Order , Preparation , Fabrication , Shipping and Final.
And I have the product_id for the product that I want to Fabricate.
When the "Final" phase comes , the stock for that product need to be increased with the quantity that I choose from a field called "quantity".
Here's the code :
class fabrication_orders(models.Model):
_name = 'proyecto.fabrication_orders'
order_id = fields.Many2one('proyecto.products',required="true",string="Product Id")
name = fields.Char(related='order_id.name',required="true",string="Product Name")
order_number = fields.Char(compute='_get_order_number',string="Order Nº",store="true")
image = fields.Binary(related='order_id.image_medium',string="Image")
quantity = fields.Float(required="true")
date = fields.Datetime(required="true",string="Order Date") ### Order Date
end_date = fields.Datetime(compute='_get_end_date',string="Finish Date",store="true") ### Finish Date
initial_cost = fields.Float(related='order_id.standard_price',string="Initial Cost")
final_cost = fields.Float(compute='_get_order_cost',string="Fabrication Cost")
#venue = fields.Many2one('proyecto.venues',required="true",string="Ship to")
order_state = fields.Selection([
('orden', "Order"),
('preparacion', "Preparation"),
('fabricacion', "Fabrication"),
('envio', "Shipping"),
('final', "Final"),
], default='orden')
#Este metodo pone la fecha final segun la cantidad
#api.depends('date')
def _get_end_date(self):
for d in self:
if d.date:
d.end_date = datetime.now() + timedelta(hours=d.quantity)
#api.depends('order_id')
def _get_order_number(self):
for r in self:
if r.order_id:
r.order_number=str(random.randint(1, 1e6))
#api.multi
def change_order_state(self):
for r in self:
if r.order_state == 'orden':
r.write({'order_state':'preparacion'})
elif r.order_state == 'preparacion':
r.write({'order_state':'fabricacion'})
elif r.order_state == 'fabricacion':
r.write({'order_state':'envio'})
elif r.order_state == 'envio':
r.write({'order_state':'final'})
r.end_date = datetime.now()
elif r.order_state == 'final':
raise ValidationError("La fabricación ha finalizado !")
Can you , please , help to increase the stock quantity of the product ?
Thanks a lot.
I am using Odoo 11. In my use case I wanted to reduce the amount of a product when the stage of my custom model is set to 'done'. Inventory amount is changed by stock.move and stock.move.line. So what I did is creating a stock.move and linked a stock.move.line to it when the state changes to 'done'.
Examples of stock.move creation can be found in addons/stock/tests/test_move.py
Here is a recipe:
(1.) If you do not have one yet, create a location
<record id="location_mylocation" model="stock.location">
<field name="name">MyLocation</field>
<field name="location_id" ref="stock.stock_location_locations_virtual"/>
<field name="usage">inventory</field>
<field name="company_id"></field>
</record>
The usage is set to 'inventory' to reduce the amount of the product. Scrap Orders use the same mechanism.
(2.) Create a stock.move
stock_location = self.env.ref('stock.stock_location_stock')
move = self.env['stock.move'].create({
'name': 'Use on MyLocation',
'location_id': stock_location.id,
'location_dest_id': mylocation.id,
'product_id': product.id,
'product_uom': product.uom_id.id,
'product_uom_qty': qty,
})
move._action_confirm()
move._action_assign()
# This creates a stock.move.line record.
# You could also do it manually using self.env['stock.move.line'].create({...})
move.move_line_ids.write({'qty_done': qty})
move._action_done()