I am trying to write to a csv in scrapy. I have a dictionary named features. I add keys and then values. There are some particular keys that are being added in a loop like this:
for i in paint_variable:
print("inside")
print(str(i[0]).replace(":",""))
features[str(i[0]).replace(":","")] = ""
features[str(i[0]).replace(":", "")]=(i[1])
pain_variables is a list. It is adding keys correctly to it.
title Sahibinden Peugeot 407 2.0 HDi Executive Premium 2006 Model
price 43.000 TL
İlan No: 14215895
İlan Tarihi: 15 Nisan 2020
Marka: Peugeot
Seri: 407
Model: 2.0 HDi Executive Premium
Yıl: 2006
Yakıt Tipi: Dizel
Vites Tipi: Otomatik
Motor Hacmi: 1997 cc
Motor Gücü: 138 hp
Kilometre: 295.000 km
Boya-değişen: Takasa Uygun
Takasa Uygun: Sahibinden
Kimden:
user_name [' Ayhan BOYSAN ']
explanation ['-', '- Tuna Caddesi, Hayır Sokak, Lena Reklam. Hemen cadde üzerinde, köşede araç görülebilir. Arkadaşımın ofisinin önünde durmakta.', '* Muayenesi yok, biteli iki ay kadar oldu, satışta anlaşılırsa yaptırıp vereceğim.', '* Sol arka ABS sensörü arıza veriyor, 150 lira parça fiyatı var, parçacılar açılır açılmaz sipariş verip yaptıracağım.', '* 295000 bakımı yapılmadı. Satışta anlaşılırsa yaptırıp vereceğim.', '* Aküsü 2 yıllık, sorunu yok ama araç kullanılmadığı için bir iki gün ya şarj edilmeli ya da bir iki saat dolaşılmalı.\xa0', '* Düz yolda iyi ama yokuşta çekişi düşük.\xa0', '* Cam tavan perdesinin ikisi eksik. Biz perdesiz kullanıyorduk çünkü 3 numara Ziebart noktasında film yaptırmıştım, çok rahat.', '* Isıtmalı koltukların rolesi değişmesi gerek, koltuk ısıtmaları çalışmıyor, geçen kış arızalandı, aracı kullanmadığımız için\xa0 \xa0 \xa0 \xa0yaptırmaya ihtiyaç duymadık.', '* Ezik yok, çizikler var bazı yerlerde, tampon köşelerinde de resimlerde gözüken yıpranmalar var.\xa0', 'Aracın ARTILARI', '* Orjinal, değişeni yok, kazası yok, macun yok.', '* Kaskolu hala. Boyattığım için sadece 2500 lira trameri var. Tamamen kozmetik amaçlı boyattım, gören usta anlar.', '* Ön cam 1 numara, yanlar 2 numara, cam tavan 3 numara Ziebarth filmi mevcut.', '* Derileri ve döşemeleri çok iyi.', '* Kesinlikle ezik bir araç değil.\xa0', '* Cd Changer mevcut', '* Yedek anahtarı mevcut', '* 17" Jant ve Lastikleri iyi durumda.', '* Xenon - Far Yıkama - Cam Tavan vb.', '* Her yıl bakımını orjinal parça kullanarak yaptım.', 'Evde iki kişiyiz ve bunu babamın kullanımı için almıştım ama yaştan ötürü artık kullanamaz hale geldi, evdeki 3. araç. İlk kez satışa koyuyorum ve masrafları olduğu için bu rakamı yazıyorum. Piyasasının ne olduğunu hepimiz biliyoruz. Ucuz diye kazalı sanmayın.', 'Bedavacı kişiler ararsa eğer karşılıklı şekilde üzülürüz uyarıyorum :)', 'Göz boyamak, kandırmak niyetindeki insanlardan olmadığım için ne biliyorsam yazdım. Yıkamadan resimleri ekledim. Buna göre yakışıksız teklifler mesajlar atmayın.', 'Değerinde toprak ile takasa girerim dağın başında olmaması kaydı ile.', 'Aracı görmek isteyenler Bayrampaşa eski cezaevi tarafına USTASI ile gelebilir. Oraya buraya gitmez aracım. Olumlu olumsuz yazdıklarım geçerlidir.', 'MESAJLA SON RAKAM YAZANLARA CEVAP VERMEM. Pazarlık gönül kırmamak adına hak ettiği kadar yapılır.', 'Saat 20.00 den sonra telefonlara bakamayabilirim, mesaj atarsanız dönerim.', 'Teşekkürler']
Sağ Arka Çamurluk Boyanmış
Arka Kaput Boyanmış
Sol Arka Çamurluk Boyanmış
Sağ Arka Kapı Boyanmış
Sağ Ön Kapı Boyanmış
Tavan Orijinal
Sol Arka Kapı Boyanmış
Sol Ön Kapı Boyanmış
Sağ Ön Çamurluk Boyanmış
Motor Kaputu Boyanmış
Sol Ön Çamurluk Boyanmış
Ön Tampon Boyanmış
Arka Tampon Boyanmış
year 2006
feul_type Dizel
gear_type Otomatik
case_type Station wagon
kilometer 295.000 km
vehicle_type Bireysel Araç
color Bej
nationality (TR) Türkiye
warranty Garantisi Yok
first_owner İlk Sahibi Değilim
suitable_for_clearing Takasa Uygun
from_who Sahibinden
annual_mtv 1.124 TL
traction Motor ve Performans
number_of_cylinders Önden Çekiş
maxiumum_power 138 hp
minimum_power 11,2 sn
acceleration 201 km/s
maximumspeed Ortalama Yakıt Tüketimi
Keys in loop start after explanation key. They are present here. But they are not written to in csv.
enter image description here
As you can see these keys are not present.
Here is a fully functional code if anyone wants to recreate:
class Myspider(SitemapSpider):
name = 'spidername'
sitemap_urls = ['https://www.arabam.com/sitemap/otomobil_1.xml']
sitemap_rules = [
('/otomobil/', 'parse'),
# ('/category/', 'parse_category'),
]
custom_settings = {'FEED_FORMAT':'csv','FEED_URI': str(datetime.today().strftime('%d%m%y'))+'.csv'}
def parse(self,response):
for td in response.xpath("/html/body/div[3]/div[6]/div[4]/div/div[2]/table/tbody/tr/td[4]/div/a/#href").extract():
checks = str(td.split("/")[3]).split("-")
for items in checks:
if items.isdigit():
if int(items) > 2001:
url = "https://www.arabam.com/"+ td
yield scrapy.Request(url, callback=self.parse_dir_contents)
def parse_dir_contents(self,response):
# print("hi here")
features = {}
features["ad_url"] = response.request.url
features["ad_path"] =""
for paths in response.xpath(" /html/body/div[3]/div[6]/div[1]/div/div/div/div[1]/div/span/a/text()").extract():
features["ad_path"]+=paths
features["ad_path"]+="/"
features["title"] = response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[1]/h1/text()").extract()[0]
features["price"] = response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[1]/div[1]/div[2]/div[1]/div/span/text()").extract()[0]
for items in response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[1]/div[1]/div[2]/ul/li/span[1]/text()").extract():
features[(str(items) )] = ""
counter = 1
for items in response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[1]/div[1]/div[2]/ul/li/span[2]/text()").extract():
query = (response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[1]/div[1]/div[2]/ul/li["+str(counter)+"]/span[1]/text()").extract())
features[str(query[0])] = items
counter+=1
features["user_name"] = response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[3]/div/div/div[1]/p/text()").extract()
if not features["user_name"]:
features["user_name"] = response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[3]/div/div/div[1]/a/span/text()").extract()
features["explanation"] = response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[1]/div[3]/div/div[1]/div/div/p/text()").extract()
#change this
paint_variable = []
for li in response.xpath("/html/body/div[3]/div[6]/div[3]/div/div[1]/div[3]/div/div[2]/div/div[2]/ul[1]/li"):
paint_variable.append(li.xpath('span/text()').extract())
for i in paint_variable:
print("inside")
print(str(i[0]).replace(":",""))
features[str(i[0]).replace(":","")] = ""
features[str(i[0]).replace(":", "")]=(i[1])
options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_argument('window-size=1200x600')
d = webdriver.Chrome(chrome_options=options,
executable_path='/Users/fatima.arshad/Downloads/chromedriver')
d.get(features["ad_url"])
# Use send_keys(Keys.HOME) to scroll up to the top of page
d.find_element_by_tag_name('body').send_keys(
Keys.END)
while True:
d.find_element_by_tag_name('body').send_keys(
Keys.UP)
e = d.find_element_by_xpath("/html/body/div[3]/div[6]/div[3]/div/div[1]/div[3]/div/div[3]/div")
if e.text:
break
overview1 = e.text.split("\n")
features["year"] = overview1[2]
features["feul_type"] = overview1[4]
features["gear_type"] = overview1[6]
features["case_type"] = overview1[8]
features["kilometer"] = overview1[10]
features["vehicle_type"] = overview1[12]
features["color"] = overview1[14]
features["nationality"] = overview1[16]
features["warranty"] = overview1[18]
features["first_owner"] = overview1[20]
features["suitable_for_clearing"] = overview1[22]
features["from_who"] = overview1[24]
features["annual_mtv"] = overview1[26]
features["traction"] = overview1[29]
features["number_of_cylinders"] = overview1[31]
features["maxiumum_power"] = overview1[39]
features["minimum_power"] = overview1[41]
features["acceleration"] = overview1[43]
features["maximumspeed"] = overview1[45]
print("feature starting")
for key, value in features.items():
print(key, value)
# print("s"+ str(overview1))
yield features
process = CrawlerProcess({
})
process.crawl(Myspider)
process.start() # the script wi
[1]: https://i.stack.imgur.com/qfS2e.png
You must do one of the following:
Define all output fields in the FEED_EXPORT_FIELDS setting
Use an Item subclass instead of a dict
Related
My goal is to write a python 3 function that takes lastnames as row from a csv and splits them correctly into lastname_1 and lastname_2.
Spanish names have the following structure: firstname + lastname_1 + lastname_2
Forgettin about the firstname, I would like a code that splits a lastname in these 2 categories (lastname_1, lastname_2) Which is the challenge?
Sometimes the lastname(s) have prepositions.
DE BLAS ZAPATA "de blas" ist the first lastname and "Zapata" the second lastname
MATIAS DE LA MANO "Matias" is lastname_1, "de la mano" lastname_2
LOPEZ FERNANDEZ DE VILLAVERDE Lopez Fernandez is lastname_1, de villaverda lastname_2
DE MIGUEL DEL CORRAL De Miguel is lastname_1, del corral lastname_2 More: VIDAL DE LA PEÑA SOLIS Vidal is surtname_1, de la pena solis surname_2
MONTAVA DEL ARCO Montava is surname_1 Del Arco surname_2
and the list could go on and on.
I am currently stuck and I found this code in perl but I struggle to understand the main idea behind it to translate it to python 3.
import re
preposition_lst = ['DE LO ', 'DE LA ', 'DE LAS ', 'DEL ', 'DELS ', 'DE LES ', 'DO ', 'DA ', 'DOS ', 'DAS', 'DE ']
cases = ["DE BLAS ZAPATA", "MATIAS DE LA MANO", "LOPEZ FERNANDEZ DE VILLAVERDE", "DE MIGUEL DEL CORRAL", "VIDAL DE LA PEÑA", "MONTAVA DEL ARCO", "DOS CASAS VALLE"]
for case in cases:
for prep in preposition_lst:
m = re.search(f"(.*)({prep}[A-ZÀ-ÚÄ-Ü]+)", case, re.I) # re.I makes it case insensitive
try:
print(m.groups())
print(prep)
except:
pass
Try this one:
import re
preposition_lst = ['DE','LO','LA','LAS','DEL','DELS','LES','DO','DA','DOS','DAS']
cases = ["DE BLAS ZAPATA", "MATIAS DE LA MANO", "LOPEZ FERNANDEZ DE VILLAVERDE", "DE MIGUEL DEL CORRAL", "VIDAL DE LA PEÑA", "MONTAVA DEL ARCO", "DOS CASAS VALLE"]
def last_name(name):
case=re.findall(r'\w+',name)
res=list(filter(lambda x: x not in preposition_lst,case))
return res
list_final =[]
for case in cases:
list_final.append(last_name(case))
for i in range(len(list_final)):
if len(list_final[i])>2:
name1=' '.join(list_final[i][:2])
name2=' '.join(list_final[i][2:])
list_final[i]=[name1,name2]
print(list_final)
#[['BLAS', 'ZAPATA'], ['MATIAS', 'MANO'], ['LOPEZ FERNANDEZ', 'VILLAVERDE'], ['MIGUEL', 'CORRAL'], ['VIDAL', 'PE�A'], ['MONTAVA', 'ARCO'], ['CASAS', 'VALLE']]
Does this suit your requirements?
import re
preposition_lst = ['DE LO', 'DE LA', 'DE LAS', 'DE LES', 'DEL', 'DELS', 'DO', 'DA', 'DOS', 'DAS', 'DE']
cases = ["DE BLAS ZAPATA", "MATIAS DE LA MANO", "LOPEZ FERNANDEZ DE VILLAVERDE", "DE MIGUEL DEL CORRAL", "VIDAL DE LA PENA SOLIS", "MONTAVA DEL ARCO", "DOS CASAS VALLE"]
def split_name(name):
f1 = re.compile("(.*)({preps}(.+))".format(preps = "(" + " |".join(preposition_lst) + ")"))
m1 = f1.match(case)
if m1:
if len(m1.group(1)) != 0:
return m1.group(1).strip(), m1.group(2).strip()
else:
return " ".join(name.split()[:-1]), name.split()[-1]
else:
return " ".join(name.split()[:-1]), name.split()[-1]
for case in cases:
first, second = split_name(case)
print("{} --> name 1 = {}, name 2 = {}".format(case, first, second))
# DE BLAS ZAPATA --> name 1 = DE BLAS, name 2 = ZAPATA
# MATIAS DE LA MANO --> name 1 = MATIAS, name 2 = DE LA MANO
# LOPEZ FERNANDEZ DE VILLAVERDE --> name 1 = LOPEZ FERNANDEZ, name 2 = DE VILLAVERDE
# DE MIGUEL DEL CORRAL --> name 1 = DE MIGUEL, name 2 = DEL CORRAL
# VIDAL DE LA PENA SOLIS --> name 1 = VIDAL, name 2 = DE LA PENA SOLIS
# MONTAVA DEL ARCO --> name 1 = MONTAVA, name 2 = DEL ARCO
# DOS CASAS VALLE --> name 1 = DOS CASAS, name 2 = VALLE
This question already has answers here:
How do I split a list into equally-sized chunks?
(66 answers)
Closed 2 years ago.
I have string and I will do split for this string and then I will get 320 elements. I need to create 4 groups. Every group will be with 100 elements and the last group must be with 20 last elements. And the last step is that all groups must be string and not list. how can I do that?
I can do that if I know how many elements I have:
s_l = 'AA AAL AAOI ACLS ADT ADTX ADVM AEL AIG ALEC ALLY ALT AMCX ANGI APA APDN APLS APPS APRN AQUA ARMK ARNC ARVN ATNM ATOM ATRA ATSG AVCT AVT AX AXDX BCLI BE BEAM BJRI BKE BKU BLDR BLNK BMRA BOOT BXS BYD CAKE CALX CAPR CARG CARR CARV CATM CC CCL CELH CEQP CFG CHEF CHRS CIT CLDX CLR CLSK CNK CNST CODX COLB COOP CPE CRS CTVA CUK CVET CVI CVM CYTK DAL DBX DCP DDS DEI DISCA DISCK DK DNB DRNA DVAX DXC ECOM EIGR ELAN ELF ELY ENVA EQ EQT EXEL FE FHI FIXX FL FLWS FMCI FORM FOX FOXA FRTA FUN GBX GIII GM GNMK GOCO GPRE GRAF GRPN GRWG GTHX GWB HALO HCC HCSG HEAR HFC HGV HIBB HMSY HOG HOME HP HSC HTH HWC IMUX IMVT INO INOV INSG INSM INT IOVA IRDM ITCI JELD JWN KMT KODK KPTI KSS KTB KTOS KURA LAKE LB LCA LL LPI LPRO LSCC LYFT MAXR MBOT MCRB MCS MD MDP MGM MGNX MIC MLHR MOS MRSN MTOR MXL MYGN NCLH NCR NK NKTR NLS NMIH NOVA NTLA NTNX NUAN NVST NXTC ODP OFC OKE OMER OMF OMI ONEM OSPN OSUR OXY OZK PACW PD PDCE PDCO PEAK PGNY PLAY PLCE PLT PLUG PPBI PRPL PRTS PRVB PS PSNL PSTX PSXP PTGX PVAC RCUS REAL REZI RKT RMBL RPAY RRGB RRR RVLV RVP RXN SANM SAVE SBGI SC SCPL SEAS SEM SFIX SFM SGMS SGRY SHLL SHOO SHYF SIX SKX SLQT SMCI SNAP SNDX SNV SONO SPAQ SPCE SPR SPWH SPWR SRG SRNE SSNT SSSS STOR SUM SUN SUPN SVMK SWBI SYF SYRS TBIO TCDA TCF TCRR TDC TEX TFFP TGTX THC TMHC TRGP TRIP TSE TUP TVTY UBX UCBI UCTT UFS UNFI UONE UPWK URBN USFD VCRA VERI VIAC VIRT VIVO VREX VSLR VSTO VXRT WAFD WBS WFC WHD WIFI WKHS WORK WORX WRK WRTC WW WWW WYND XEC XENT XPER XRX YELP ZGNX ZUMZ ZYXI'
split_s_l = s_l.split(" ")
part_1 = ' '.join(split_s_l[:100])
part_2 = ' '.join(split_s_l[100:200])
part_3 = ' '.join(split_s_l[200:300])
part_4 = ' '.join(split_s_l[300:])
for part in (part_1, part_2, part_3, part_4):
print(part)
but I don't know how to do that If I have many elements in list.
For a variable number of items, you can loop using:
sep = ' '
num = 100
split_s_l = s_l.split(sep)
for i in range(0, len(split_s_l), num):
part = sep.join(split_s_l[i : i+num])
print(part)
Bear in mind that for the last slice in the example case ([300:400]) it does not matter that there are only 320 elements -- just the last 20 items will be included (no error).
Something like this?
def break_up(s, nwords, separator):
words = s.split()
return [separator.join(words[n:n+nwords]) for n in range(0, len(words), nwords)]
print(break_up('a b c d e f g h', 3, ' '))
Result:
['a b c', 'd e f', 'g h']
Of course, you might call as print(break_up(s_l, 100, ' '))
Below
s_l = 'AA AAL AAOI ACLS ADT ADTX ADVM AEL AIG ALEC ALLY ALT AMCX ANGI APA APDN APLS APPS APRN AQUA ARMK ARNC ARVN ATNM ATOM ATRA ATSG AVCT AVT AX AXDX BCLI BE BEAM BJRI BKE BKU BLDR BLNK BMRA BOOT BXS BYD CAKE CALX CAPR CARG CARR CARV CATM CC CCL CELH CEQP CFG CHEF CHRS CIT CLDX CLR CLSK CNK CNST CODX COLB COOP CPE CRS CTVA CUK CVET CVI CVM CYTK DAL DBX DCP DDS DEI DISCA DISCK DK DNB DRNA DVAX DXC ECOM EIGR ELAN ELF ELY ENVA EQ EQT EXEL FE FHI FIXX FL FLWS FMCI FORM FOX FOXA FRTA FUN GBX GIII GM GNMK GOCO GPRE GRAF GRPN GRWG GTHX GWB HALO HCC HCSG HEAR HFC HGV HIBB HMSY HOG HOME HP HSC HTH HWC IMUX IMVT INO INOV INSG INSM INT IOVA IRDM ITCI JELD JWN KMT KODK KPTI KSS KTB KTOS KURA LAKE LB LCA LL LPI LPRO LSCC LYFT MAXR MBOT MCRB MCS MD MDP MGM MGNX MIC MLHR MOS MRSN MTOR MXL MYGN NCLH NCR NK NKTR NLS NMIH NOVA NTLA NTNX NUAN NVST NXTC ODP OFC OKE OMER OMF OMI ONEM OSPN OSUR OXY OZK PACW PD PDCE PDCO PEAK PGNY PLAY PLCE PLT PLUG PPBI PRPL PRTS PRVB PS PSNL PSTX PSXP PTGX PVAC RCUS REAL REZI RKT RMBL RPAY RRGB RRR RVLV RVP RXN SANM SAVE SBGI SC SCPL SEAS SEM SFIX SFM SGMS SGRY SHLL SHOO SHYF SIX SKX SLQT SMCI SNAP SNDX SNV SONO SPAQ SPCE SPR SPWH SPWR SRG SRNE SSNT SSSS STOR SUM SUN SUPN SVMK SWBI SYF SYRS TBIO TCDA TCF TCRR TDC TEX TFFP TGTX THC TMHC TRGP TRIP TSE TUP TVTY UBX UCBI UCTT UFS UNFI UONE UPWK URBN USFD VCRA VERI VIAC VIRT VIVO VREX VSLR VSTO VXRT WAFD WBS WFC WHD WIFI WKHS WORK WORX WRK WRTC WW WWW WYND XEC XENT XPER XRX YELP ZGNX ZUMZ ZYXI'
words= s_l.split(" ")
num_of_100_groups = int(len(words) / 100)
groups = []
for i in range(0,num_of_100_groups):
groups.append(words[i * 100 : (i+1) * 100])
groups.append(words[num_of_100_groups * 100:])
for sub_group in groups:
print(' '.join(sub_group))
import requests
from datetime import datetime
import time
class Bot:
def __init__(self):
self.url = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest'
self.params = {
'start': '1',
'limit': '100',
'convert': 'USD'
}
self.headers = {
'Accepts': 'application/json',
'X-CMC-PRO-API-KEY': 'my key',
}
self.orders = []
def fetchCurrenciesData(self):
r = requests.get(url=self.url, headers=self.headers, params=self.params).json()
return r['data']
def canBuy(self):
#controlla se le operazioni di acquisto non sono chiuse
for order in self.orders:
if order['status'] == 'open':
return False
return True
impactBot = Bot()
while(1):
now = datetime.now()
currencies = impactBot.fetchCurrenciesData()
print(currencies)
i = 1 #incremento in percentuale
r = 4 #valore sopra il quale fare partire l'operazione
n = 0 #numero di valute cui prezzo ha subito un incremento migliore di i dall'ultima volta
z = -1 #percentuale sotto la quale vendi la valuta
bestCurrency = None #valuta con rivalutazione del prezzo maggiore
#logica
if impactBot.canBuy():
print('Non ci sono posizioni aperte - Controllo se trovo valute che hanno guadagnato piu di (i)% nell ultima ora ')
for currency in currencies:
if not bestCurrency or currency['quote'] ['USD'] ['percent_change_1h'] > bestCurrency['quote'] ['USD'] ['percent_change_1h']:
bestCurrency = currency
if currency['quote'] ['USD'] ['percent_change_1h'] > i:
n = n +1
if n > 4:
print ('Ho trovato piu di quattro valute - Creo un nuovo ordine')
newOrder = {
'datetime': now,
'symbol': currency['symbol'],
'enterPrice': currency['quote'] ['USD'] ['price'], #prezzo con cui abbiamo acquistato
'exitPrice': None,
'status': 'open'
}
impactBot.orders.append(newOrder)
else:
print('Controllo gli ordini ancora aperti - Se si verifica la condizione di svalutazioneallora vendo')
for currency in currencies:
if currency['quote'] ['USD'] ['percent_change_1h']:
for order in impactBot.orders:
if order['status'] == 'open' and order['symbol'] == currency['symbol']:
#vendi
order['status'] = 'close'
order['exitPrice'] = currency['quote'] ['USD'] ['price']
#overview
initialAmount = 10000
profit = 0
for order in impactBot.orders:
if order['status'] == 'close':
profit += initialAmount * order['exitPrice'] / order['enterPrice']
finalAmount = initialAmount + profit
print('Ho realizzato {len(impactBot.orders)} compravendite aono partito con {initialAmount}$ e adesso ne ho {finalAmount}$')
#routine
minutes = 10
seconds = minutes * 60
time.sleep(seconds)
This are the errors:
Traceback (most recent call last):
File "/Users/andyduma/PycharmProjects/esercitazioni/Bot.py", line 38, in <module>
currencies = impactBot.fetchCurrenciesData()
File "/Users/andyduma/PycharmProjects/esercitazioni/Bot.py", line 24, in fetchCurrenciesData
return r['data']
KeyError: 'data'
When you make requests like these you should make a condition to test the results so you can see what is going on. I can't be positive that my way of testing the error code is exactly how you should do it, but it is definitely in the ballpark.
def fetchCurrenciesData(self):
r = requests.get(url=self.url, headers=self.headers, params=self.params).json()
status = r['status']
if 'error_code' not in [*status] or status['error_code'] == 0:
#print(r) #uncomment if the next line throws an error
return r['data']
raise ValueError(f"Error {status['error_code']}: {status['error_message']}")
and
while(1):
now = datetime.now()
try:
currencies = impactBot.fetchCurrenciesData()
print(currencies)
except ValueError as err:
print(err)
break
If you look at the python example for the API you are not doing it the way they tell you to. Your way may work regardless, but why invent your own solutions when the company that is providing you with the API has given you a copy/paste solution to use? They want you to create a Session and you have completely failed to do that.
I am trying to make a colormap plotting a function that calculates the pH of the ocean from a set of values for CO2 and DIC (Carbon Dioxide and Dissolved Inorganic Carbon), for CO2 (x axis) and DIC (y axis) I make a set of values between 0.000090 and 0.001100, 0.001526 and 0.002100 respectively. I used linspace to make 100 points.
When I make the plot, in the x and y axis the ticks show values from 0 to 100 in both axis. The function works OK and the expected output plot makes sense but I do not know why the ticks labels show those values. Any help will be appreciated.
Below you have the code tha makes the plot.
import numpy as np
def K0_Weiss(S, TC):
TK=TC+273.15 #Temperatura de Celsius a Kelvin
lnK0 = 9345.17/TK - 60.2409 + 23.3585 * np.log(TK/100) + S * (0.023517 - 0.00023656 * TK + 4.7036e-07 *
TK * TK)
K0=np.exp(lnK0)
return K0
"""
Created on Mon Sep 3 11:51:00 2018
#author: fergomez
Esta funcion calcula K1 y K2 del sistema CO2-H20 segun
Roy et al., (1993), The dissociation constant of carbonic acid in sewater
at salinities of 5 to 45 and temperatures of 0 to 45 desgrees Celsius.
Marine Chemistry 44(2-4), 249-267.
Unidades
K1, K2=mol*kg-soln-1
Ejemplo de uso:
Input
K1, K2=K1_K2_Roy(35, 25)
print(K1, K2)
Output:
1.3921075396202872e-06 1.1887254858040348e-09
"""
def K1_K2_Roy(S, TC):
TK=TC+273.15 #Temperatura de Celsius a Kelvin
#K1 usando Roy et al., 1993
tmp1 = 2.83655 - 2307.1266/TK - 1.5529413 * np.log(TK)
tmp2 = -(0.207608410 + 4.0484/TK) * np.sqrt(S)
tmp3 = 0.08468345 * S - 0.00654208 * S**1.5
tmp4 = np.log(1 - 0.001005 * S)
lnK1roy = tmp1 + tmp2 + tmp3 + tmp4
K1 = np.exp(lnK1roy)
#K2 usando Roy et al., 1993
tmp1 = -9.226508 - 3351.6106/TK - 0.2005743 * np.log(TK)
tmp2 = (-0.106901773 - 23.9722/TK) * np.sqrt(S)
tmp3 = 0.1130822 * S - 0.00846934 * S**1.5 + np.log(1 -
0.001005 * S)
lnK2roy = tmp1 + tmp2 + tmp3
K2 = np.exp(lnK2roy)
return K1, K2
def K1_K2_SBY(S):
pK1=6.1568-0.00352*S
pK2=8.5503-0.0080*S
K1=10**(-pK1)
K2=10**(-pK2)
return K1, K2
"""
Created on Mon Sep 3 11:53:53 2018
#author: fergomez
Esta funcion calcula K0 del sistema CO2-H20 segun recomienda Dickson et al 2007
Guide for best practices for ocean CO2 measurements. PICES SP 3, 191 pp.
Conforme a Mucci, A., 1983. The solubility of calcite and aragonite in seawater
at various salinities and temperatures and one atmosphere of total pressure.
American journal of Science, 283, 780-799.
Ejemplo de uso:
Input
Ksp_a=Kspa_Mucci(35, 25)
print(Ksp_a)
Output:
6.48175906801198e-07
"""
def Kspa_Mucci(S, TC):
TK=TC+273.15 #Temperatura de Celsius a Kelvin
tmp1 = -171.945 - 0.077993 * TK + 2903.293/TK + 71.595 * np.log10(TK)
tmp2 = +(-0.068393 + 0.0017276 * TK + 88.135/TK) * np.sqrt(S)
tmp3 = -0.10018 * S + 0.0059415 * S**1.5
log10Kspa = tmp1 + tmp2 + tmp3
Ksp_a=10**(log10Kspa)
return Ksp_a
"""
Created on Mon Sep 3 11:54:42 2018
#author: fergomez
Esta funcion calcula K0 del sistema CO2-H20 segun recomienda Dickson et al 2007
Guide for best practices for ocean CO2 measurements. PICES SP 3, 191 pp.
Conforme a Mucci, A., 1983. The solubility of calcite and aragonite in seawater
at various salinities and temperatures and one atmosphere of total pressure.
American journal of Science, 283, 780-799.
Ejemplo de uso:
Input
Ksp_c=Kspc_Mucci(35, 25)
print(Ksp_c)
Output:
4.2723509278626e-07
"""
def Kspc_Mucci(S, TC):
TK=TC+273.15 #Temperatura de Celsius a Kelvin
tmp1 = -171.9065 - 0.077993 * TK + 2839.319/TK + 71.595 * np.log10(TK)
tmp2 = +(-0.77712 + 0.0028426 * TK + 178.34/TK) * np.sqrt(S)
tmp3 = -0.07711 * S + 0.0041249 * S**1.5
log10Kspc = tmp1 + tmp2 + tmp3
Ksp_c=10**(log10Kspc)
return Ksp_c
# -*- coding: utf-8 -*-
"""
Created on Mon Sep 3 11:55:56 2018
Calcula el factor de correccion de la presion para las constantes de
equilibrio Ki usando Millero, F.J., (1995). Thermodynamics of the carbon
dioxide system in the oceans. Geochimica et Cosmochimica Acta 59:661-677.
#author: fergomez
Ejemplo de uso:
A presion atmosferica Kspa=4.27 e-07
Input (para aragonita):
fcorr=P_corr_Millero(-45.96, 0.5304, 0.0, -11.76e-03, 0.3692e-03, 0.0, 25, 300)
print(fcorr)
Output:
1.4787577790538065
entonces Kspa(presion atmosferica)*fcorr=Kspa_corregido
4.27 e-07*1.4787577790538065= 9.58 e-07
"""
#pagina 186 Andreas Hoffman PhD Thesis
def P_corr_Millero(a0, a1, a2, b0, b1, b2, TC, P):
TK=TC+273.15 #Temperatura de Celsius a Kelvin
R=83.131
delta_V= a0+a1*TC+a2*TC*TC
delta_k= (b0+b1*TC+b2*TC*TC)
lnKp=-(delta_V/(R * TK))*P+0.5*(delta_k/(R*TK))*P*P
fcorr=np.exp(lnKp)
return fcorr
#Llamamos las librerias que utilizaremos
import numpy as np
#import matplotlib.pyplot as plt
import math
from numpy import exp,arange
from pylab import meshgrid,cm,imshow,contour, clabel,colorbar,axis,title,show, contourf
#from constantes import K0_Weiss, K1_K2_Roy, Kspc_Mucci, Kspa_Mucci, P_corr_Millero
import matplotlib.pyplot as plt
#============================================================================
#Inicio de seccion de INPUT==================================================
#============================================================================
S=6.88 #ppt
TC=6.42 #Celsius
P=1.0 #atmosferas
K0=K0_Weiss(S, TC)
K1, K2=K1_K2_Roy(S, TC)
Ksp_c=Kspc_Mucci(S, TC)
Ksp_a=Kspa_Mucci(S, TC)
#============================================================================
#Fin de seccion de INPUT======================================================
#============================================================================
#=============================================================================
#Estimacion de Calcio de Tyrrell et al., 2008 y valores de ejemplo agua de mar
#============================================================================
#Ca=0.0103 #valores agua de mar
Ca=(0.331*S+0.392)*1e-03 #Mar Baltico
#Ca=(0.375*S+0.0368)*1e-03 # Bothnian Bay (parte norte del mar Baltico)
#=============================================================================
#def co2_t(pco2, K0):
# co2=pco2*K0
# return co2
#=============================================================================
#vco2_t = np.vectorize(co2_t)
#Pasamos de pco2 a co2
#co2=co2_t(pco2, K0)
#=============================================================================
#=============================================================================
#Calculo del factor de correccion por presion
#=============================================================================
fcorr_K1=P_corr_Millero(-25.50, 0.1271, 0.0, -3.08e-03, 0.0877e-03, 0.0, TC, P)
fcorr_K2=P_corr_Millero(-15.82, -0.219, 0.0, 1.136e-03, -0.1475e-03, 0.0, TC, P)
fcorr_Kspc=P_corr_Millero(-48.76, 0.5304, 0.0, -11.76e-03, 0.3692e-03, 0.0, TC, P)
fcorr_Kspa=P_corr_Millero(-45.96, 0.5304, 0.0, -11.76e-03, 0.3692e-03, 0.0, TC, P)
#=============================================================================
#=============================================================================
#Correccion de ctes de equilibrio por presion
#=============================================================================
K1=K1*fcorr_K1
K2=K2*fcorr_K2
Ksp_c=Ksp_c*fcorr_Kspc
Ksp_a=Ksp_a*fcorr_Kspa
#=============================================================================
#Funcion que realiza calculos del sistema carbonato y omega
#=============================================================================
def carbEq(co2, dic, Ca):
#-----------------------------------------------
# Resolvemos para obtener H+ (cf. a Zeebe and Wolf-Gladrow, 2000)
a1=co2-dic
a2=K1*co2
a3=K1*K2*co2
p= [a1, a2, a3]
r = np.roots(p)
h= max(np.real(r)) # Esto es para seleccionar la raiz real mas grande
#
# Calculamos HCO3, CO3 and CO2aq, usando DIC, AlK y H+
hco3=dic/(1+h/K1+K2/h)
co3=dic/(1+h/K2+h*h/(K1*K2))
#co2=dic/(1+K1/h+K1*K2/(h*h))
fco2=co2 / K0
pH=-math.log10(h)
alk=2*co3+hco3
#Saturacion de Calcita y Aragonita
omega_ar=Ca*co3/Ksp_a
omega_cal=Ca*co3/Ksp_c
return fco2, pH, co2, hco3, co3, h, dic, alk, omega_ar, omega_cal
vcarbEq = np.vectorize(carbEq)
#=============================================================================
#Uso de la funcion - ejemplo
#=============================================================================
#co2=co2*1000000
#dic=dic*1000000
pco2=np.linspace(0.000090, 0.001100, 100)
co2=pco2*K0
dic=np.linspace(0.001526, 0.002100, 100) #moles (para pasar de micromoles a moles, 1587*e-06)
#pco2=pco2*1000
#dic=dic*1000
co2, dic=meshgrid(co2*1000000, dic*1000000)
fco2, pH, co2, hco3, co3, h, dic, alk, omega_ar, omega_cal=vcarbEq(co2=co2, dic=dic, Ca=Ca)
#Ejemplo con valores de referencia de uso para agua de mar normal
#fco2, pH, co2, hco3, co3, h, dic, alk, omega_ar, omega_cal=carbEq(co2=0.00001032997, dic=0.002108, Ca=Ca)
#para este usar calcio=0.0103
#=============================================================================
#Grafico de los resultados
#=============================================================================
print(co2)
print('====================================')
print(dic)
print('====================================')
print(pH)
#print(omega_cal)
#fig, ax = plt.subplots()
im = imshow(pH, cmap=cm.RdBu) # dibujo la funcion
plt.xlabel('CO2')
plt.ylabel('DIC')
plt.ylim(0, 100)
plt.scatter(40, 40, color='k')
#ax.set_xlim(0.00000525, 0.000065)
#ax.set_ylim(0.001, 0.0021)
#plt.xlim(min(co2), max(co2))
# agrego lineas de contorno y rotulos
cset = contour(pH, arange(6.0,9.0,0.2),linewidths=2, cmap=cm.Set2)
clabel(cset,inline=True,fmt='%1.1f',fontsize=10)
colorbar(im) # agrego la barra de colores al costado
I have a problem with my code (trying to make the NIM game with a Tkinter GUI). Anyhow i cant get my functions (Munt1, Munt2) to subtract to my global (aantal_munten). And i dont know how to fix it. Can someone point me in the right direction? Also the TKinter gui is still a bit new for me so if you have any tips or tricks for me they are well appreciated!
import tkinter
import random
def toonStartscherm():
keuzescherm.pack_forget()
spelerscherm.pack_forget()
startscherm.pack()
def toonKeuzescherm():
keuzescherm.pack()
startscherm.pack_forget()
spelerscherm.pack_forget()
def spelOptie1():
keuzescherm.pack_forget()
startscherm.pack_forget()
spelerscherm.pack()
def randomSpeler():
beginnende_speler = random.choice(['1', '2'])
speler = beginnende_speler
if speler == '1':
speler = '2'
else:
speler = '1'
return str(speler)
def Munt1():
eenMunt = 1
aantal_munten -= eenMunt
def Munt2():
tweeMunt = 2
aantal_munten -= tweeMunt
nim_spel = tkinter.Tk()
global aantal_munten
aantal_munten = 7
startscherm = tkinter.Frame(master=nim_spel)
startscherm.pack()
start_welkom = tkinter.Label(startscherm, text= 'Welkom bij het NIM spel! \nHieronder volgende de spelregels.')
start_welkom.pack()
start_uitleg = tkinter.Label(startscherm, text= 'Het spel NIM start met 7 munten, elke speler mag één of twee munten pakken. \n De speler die de laatste munt pakt verliest!')
start_uitleg.pack() # zet uitleg automatisch in venster
doorgaan_knop = tkinter.Button(startscherm, text = 'Ik snap de regels!', fg = 'green', command = toonKeuzescherm) # maakt knop en laat stoppen
doorgaan_knop.pack(side = 'bottom')
keuzescherm = tkinter.Frame(master=nim_spel)
keuzescherm.pack()
keuze_opties = tkinter.Label(keuzescherm, text='Het NIM spel kan op twee manieren gespeeld worden. \n Optie 1: Tegen elkaar \n Optie 2: Tegen de computer')
keuze_opties.pack() # zet opties automatisch in venster
keuze_vraag = tkinter.Label(keuzescherm, text='Voor welke optie kies je?')
keuze_vraag.pack()
optie_1 = tkinter.Button(keuzescherm, text = 'Optie 1', fg = 'green', command = spelOptie1) # maakt knop en laat stoppen
optie_1.pack(side = 'left')
optie_2 = tkinter.Button(keuzescherm, text = 'Optie 2', fg = 'red', command = keuzescherm.quit) # maakt knop en laat stoppen
optie_2.pack(side = 'right')
spelerscherm = tkinter.Frame(master=nim_spel)
spelerscherm.pack()
beurt_speler = tkinter.Label(spelerscherm, text='Speler ' + (randomSpeler()) + ' is aan de beurt!')
beurt_speler.pack()
munten_over = tkinter.Label(spelerscherm, text='Er zijn nog ' + (str(aantal_munten)) + ' aantal munten over, hoeveel pak je er?')
munten_over.pack()
pak_1_munt = tkinter.Button(spelerscherm, text = '1 munt', fg = 'blue', command = Munt1)
pak_1_munt.pack(side = 'left')
pak_2_munt = tkinter.Button(spelerscherm, text = '2 munten', fg = 'blue', command = Munt2)
pak_2_munt.pack(side = 'right')
toonStartscherm()
nim_spel.mainloop()
def Munt1():
global aantal_munten
eenMunt = 1
aantal_munten -= eenMunt
def Munt2():
global aantal_munten
tweeMunt = 2
aantal_munten -= tweeMunt
I just added the global aantal_munten line and checked it, and it's working perfectly now.