So this is pretty much a web scraping program. I feel as if it is nearly finished, but I have no clue how to fix this!
Traceback (most recent call last):
File "AgriMet4.py", line 424, in <module>
orig_column = convert(station, webpage, data_type)
File "AgriMet4.py", line 244, in convert
ag_date, ag_time, ag_data_str = line.split()
ValueError: too many values to unpack
Here are the parts that are flagged:
# Break line into components
ag_date, ag_time, ag_data_str = line.split()
ag_datetime = ag_date + " " + ag_time
ag_datetime = datetime.datetime.strptime(ag_datetime, "%m/%d/%Y %H:%M")
ag_data = float(ag_data_str)
and
columns = []
for data_type in data_types:
webpage = download(station, data_type, effective_begin_date, effective_end_date)
orig_column = convert(station, webpage, data_type)
std_column = fill_gaps(orig_column, interval, data_type,
effective_begin_date, effective_end_date)
adjusted_column = adjust_datetimes(station, time_zone_offset, std_column)
columns.append(adjusted_column)
EDIT: I've made the code shorter. Sorry!
As the error indicates, there are too many values on the right hand side of your equation (line.split()).
You can find the documentation for string splitting. You seem to have more than 3 values to assign. If you don't know how many values you will get, you can use the * syntax to hold any remaining values.
line = "Hello there world"
word1, *remaining_words = line.split()
I've figured out my problem!
My download definition had the incorrect data_types, which caused it to gather all six-seven data types at once instead of one at a time.
def download(station, data_types, begin_date, end_date):
query_string = ('station=' + station
+ '&year=' + str(begin_date.year)
+ '&month=' + str(begin_date.month)
+ '&day=' + str(begin_date.day)
+ '&year=' + str(end_date.year)
+ '&month=' + str(end_date.month)
+ '&day=' + str(end_date.day)
+ '&pcode=' + data_types)
url = AGRIMET_WEBSITE + '?' + query_string
Related
Writing a python script to parse an incoming string from a text file and output word by word to serial (serial parts commented out)
Getting the following error:
Traceback (most recent call last):
File "/Users/di/Desktop/RBDS/rbdsScroll.py", line 23, in <module>
wordLength = len(wordList[stringInc])
IndexError: list index out of range
I know it has to do with the new list (when the contents of the text file have far less words than the previous list) not having a high enough index number to work. I'm not quite sure how to go about remedying this. Any help would be appreciated. Full code below:
import time
#import serial
from config import dataPath
from config import scrollTime
from config import preText
from config import postText
from config import port
stringInc=wordFirst=wordLast=wordList=wordLength=word2Length=0
while True:
f = open(dataPath, 'r')
file_contents = f.read()
f.close()
wordList = file_contents.split()
maxLength = len(wordList)
wordLength = len(wordList[stringInc])
if stringInc < maxLength - 1:
word2Length = len(wordList[stringInc + 1])
wordFirst = 0
wordLast = 8
if wordLength > 8:
longString = (wordList[stringInc])
while wordLength + 1 > wordLast:
# ser = serial.Serial(port)
# ser.write(preText + longString[wordFirst:wordLast] + postText)
# ser.close()
print(preText + longString[wordFirst:wordLast] + postText)
wordFirst = wordFirst + 1
wordLast = wordLast + 1
time.sleep(scrollTime / 1000)
elif (wordLength + word2Length < 8) and (stringInc + 1 < maxLength):
# ser = serial.Serial(port)
# ser.write(preText + wordList[stringInc] + " " + wordList[stringInc + 1] + postText)
# ser.close()
print(preText + wordList[stringInc] + " " + wordList[stringInc + 1] + postText)
stringInc = stringInc + 1
time.sleep(scrollTime / 1000)
else:
# ser = serial.Serial(port)
# ser.write(preText + wordList[stringInc] + postText)
# ser.close()
print(preText + wordList[stringInc] + postText)
time.sleep(scrollTime / 1000)
stringInc = stringInc + 1
if stringInc == maxLength:
stringInc = 0
If the file content is fixed, you should read it only once, and move the while just before setting wordLength. That sends all the read words one after the other.
In case the file content changes, which would explain the error, if the new content is shorter than the current value of stringInc, you lower maxLength (just after reading the file) but never force stringInc to be lower than this new value, hence the error message...
Adding a check like if stringInc >= maxLength: after changing maxLength should help correcting the code (in that case set stringInc to 0, I guess).
But the behavior is still strange, and I would rather send all the words of the file after reading it once, not reading it in the while, and then later would read the file again, to send if only if modified...
I have an interesting behavior happening with my program.
i have the following methods:
def getMarket(self, Currency):
return self.public_api('GetMarket/' + Currency + '_BTC')
def getBalance(self, Currency):
self.api_params.clear()
self.api_params['Currency'] = Currency
return self.private_api('GetBalance')
my_api = buyBot(API_KEY, API_SECRET)
pumpCoin = my_api.getMarket('OSC')
pumpRawRate = pumpCoin['Data']['High']
pumpRawQty = .02
pumpBuyRate = my_api.calculateBuy(pumpRawRate)
pumpQty = float(pumpRawQty)/float(pumpBuyRate)
pumpSellRate = pumpCoin['Data']['Low']
pumpSellCoin = my_api.getBalance('OSC')
pumpSellAmount = pumpSellCoin["Data"]["Total"]
print str(pumpRawRate) + '\n' + str(pumpBuyRate) + '\n' + str(pumpSellRate) + '\n' + str(pumpQty) + '\n' + str(pumpSellAmount)`
From section: pumpCoin = my_api.getMarket('OSC') to pumpSellRate = pumpCoin['Data']['Low'], i have no problems getting the information and working with it.
Problem seems to be starting with line: pumpSellCoin = my_api.getBalance('OSC')
I get the following Error message:
Traceback (most recent call last):
File "C:\XXXXXX.py", line 92, in <module>
pumpSellAmount = pumpSellCoin["Data"]["Total"]
TypeError: string indices must be integers, not str
if i run: print (my_api.getBalance('OSC'), i am able to see all the private API information that is retrieved by that call, however i am not sure why it is giving me a problem when i try to call 1 specific item in the stack.
Let me know if you need any more information on this.
Any help will be greatly appreciated.
I have looked at the other posts and so far i can't seem to figure out the exact cause.
This is the private_api code
def private_api(self, meth):
time.sleep(1)
params = self.api_params
url = self.apisite + meth
nonce = str(int(time.time()))
post_data = json.dumps(params)
hash = hashlib.md5()
hash.update(post_data)
base64hash = base64.b64encode(hash.digest())
sig = self.apikey + "POST" + urllib.quote_plus(url).lower() + nonce + base64hash
hmacsig = base64.b64encode(hmac.new(base64.b64decode(self.apisecret), sig, hashlib.sha256).digest())
hdr = "amx " + self.apikey + ":" + hmacsig + ":" + nonce
headers = { 'Authorization': hdr, 'Content-Type':'application/json; charset=utf-8' }
request = urllib2.Request(url, data=post_data, headers=headers)
return urllib2.urlopen(request).read()
Please add this to your code:
print('pumpSellCoin', type(pumpSellCoin["Data"]), type(pumpSellCoin["Data"]["Total"]))
pumpSellAmount = pumpSellCoin["Data"]["Total"]
This will show you that one of your variables is a list or a string and not a dictionary and you need to access is using a number and not a name like "Data" or "Total"
Try this example:
test = 'abcde'
print(type(test))
print(test[0])
print(test[2:4])
print(test['whatever']) # this results in TypeError: string indices must be integers
if i run the program as follows:
my_api = buyBot(API_KEY, API_SECRET)
pumpCoin = my_api.getMarket('OSC')
pumpRawRate = pumpCoin['Data']['High']
pumpRawQty = .02
pumpBuyRate = my_api.calculateBuy(pumpRawRate)
pumpQty = float(pumpRawQty)/float(pumpBuyRate)
pumpSellRate = pumpCoin['Data']['Low']
pumpSellBal = my_api.getBalance('OSC')
print pumpSellBal
#print('pumpSellBal', type(pumpSellBal["Data"]), type(pumpSellBal["Data"]["Total"]))
#pumpSellAmount = pumpSellBal['Data']['Total']
print str(pumpRawRate) + '\n' + str(pumpBuyRate) + '\n' + str(pumpSellRate) + '\n' + str(pumpQty) #+ '\n' + str(pumpSellAmount)
i get the following results:
{"Success":true,"Error":null,"Data":[{"CurrencyId":235,"Symbol":"OSC","Total":8561.03652012,"Available":0.00000000,"Unconfirmed":0.00000000,"HeldForTrades":8561.03652012,"PendingWithdraw":0.00000000,"Address":null,"Status":"OK","StatusMessage":null,"BaseAddress":null}]}
1.61e-06
2.415e-06
1.25e-06
8281.57349896
So i am definitely able to communicate back and forward, however the issue only seems to be when i try to work with a single piece of information from pumpSellBal = my_api.getBalance('OSC')
I am making a program that searches the computer for .fts and .fits files in which it opens the file and retrieves info that corresponds to a specific keyword in the header and renames the file to that keyword.
I am having a problem where i keep receiving a KeyError becasue a header keyword I am searching for is not found in the file. Is there a way around this? I want to be able to search various keywords and do something even if that key word does not exist in the file.
Here is code:
from astropy.io import fits
import os
for i in os.listdir(os.getcwd()):
if i.endswith(".fits") or i.endswith(".fts"):
hdulist = fits.open(i)
DATEOBS_header = hdulist[0].header['DATE-OBS']
EXPTIME_header = int(round(hdulist[0].header['EXPTIME']))
CCDTEMP_header = int(round(hdulist[0].header['CCD-TEMP']))
XBINNING_header = hdulist[0].header['XBINNING']
FILTER_header = hdulist[0].header['FILTER']
IMAGETYP_header = hdulist[0].header['IMAGETYP']
OBJECT_header = hdulist[0].header['OBJECT']
DATEandTIME = DATEOBS_header[0:]
YEAR = DATEandTIME[0:4]
MONTH = DATEandTIME[5:7]
DAY = DATEandTIME[8:10]
#TIME = DATEOBS_header[11:]
HOUR = DATEandTIME[11:13]
MINUTE = DATEandTIME[14:16]
SECONDS = DATEandTIME[17:]
DATE = str(YEAR) + str(MONTH) + str(DAY) + 'at' + str(HOUR) + str(MINUTE) + str(SECONDS)
if IMAGETYP_header == 'Light Frame':
newname = str(OBJECT_header) + '_' + str(DATE) + '_' + str(CCDTEMP_header) + 'temp_' + str(XBINNING_header) + 'bin_' + str(EXPTIME_header) + 'exptime_' + str(FILTER_header) + '.fits'
if IMAGETYP_header == 'Dark Frame':
newname = 'Dark_' + str(DATE) + 'at' + str(TIME) + '_' + str(CCDTEMP_header) + 'temp_' + str(XBINNING_header) + 'bin_' + str(EXPTIME_header) + 'exptime' + '.fits'
if IMAGETYP_header == 'Flat Field':
newname = 'Flat_' + str(DATE) + 'at' + str(TIME) + '_' + str(CCDTEMP_header) + 'temp_' + str(XBINNING_header) + 'bin_' + str(EXPTIME_header) + 'exptime_' + str(FILTER_header) + '.fits'
prevname = i
os.rename(prevname, newname)
hdulist.close()
continue
else:
continue
This is the Error I get:
Traceback (most recent call last):
File "glo1.py", line 9, in <module>
DATEOBS_header = hdulist[0].header['DATE-OBS']
File "/home/luisgeesb/.local/lib/python2.7/site-packages/astropy/io/fits/header.py", line 151, in __getitem__
card = self._cards[self._cardindex(key)]
File "/home/luisgeesb/.local/lib/python2.7/site-packages/astropy/io/fits/header.py", line 1723, in _cardindex
raise KeyError("Keyword %r not found." % keyword)
KeyError: "Keyword 'DATE-OBS' not found."
To prevent these kinds of exceptions from stopping your program, you can either catch them, like this:
try:
DATEOBS_header = hdulist[0].header['DATE-OBS']
except KeyError:
DATEOBS_header = None
Or, use the .get() method of dictionaries, which checks if a key exists and if it doesn't returns a default value, instead of raising an exception. The default value returned is None.
If you do this, you will also need to set some sensible defaults, or catch those cases where you are casting the values (since you cannot cast None).
Finally, whenever you are reading from files - you should always assume the data is malformed/junk and do a bit of defensive programming. In your code, you are assuming that the values returned for CCDTEMP is a number, but what if the file is corrupted or has a blank? Your application doesn't handle this case.
Here is some code that attempts to catch as many errors as possible:
DATEOBS_header = hdulist[0].header.get('DATE-OBS')
XBINNING_header = hdulist[0].header.get('XBINNING')
FILTER_header = hdulist[0].header.get('FILTER')
IMAGETYP_header = hdulist[0].header.get('IMAGETYP')
OBJECT_header = hdulist[0].header.get('OBJECT')
# For these two, you need to either set a default
# Here I am setting the default to 0, ------------v
EXPTIME_header = int(round(hdulist[0].header.get('EXPTIME', 0)))
# Or you need to check for a value :
ccdtemp_value = hdulist[0].header.get('CCD-TEMP')
try:
ccdtemp_value = int(round(ccdtemp_value))
except ValueError:
# This means, the value was either None (header does not exist)
# or it was something that can't be converted to a number
# since it cannot be converted to a number, we do not know
# if the value is None or something like an empty string,
# so we explicitly set the value to None
ccdtemp_value = None
CCDTEMP_header = ccdtemp_value
Assuming hdulist[0].header gives you a dict instance, you can do something like
DATEOBS_header = hdulist[0].header.get('DATE-OBS')
Which would return a None if the key 'DATE-OBS' does not exist.
See https://docs.python.org/2/library/stdtypes.html#dict.get for more details.
import MySQLdb
import pyodbc
typesFile = open('servinfo.txt', 'r').readlines()
dataTypes = dict((row.split(',')[0].strip(),row.split(',')[1].strip()) for row in typesFile)
conn = pyodbc.connect('DRIVER={FreeTDS}; SERVER=prsoft; DATABASE=blueseas; UID=sa; PWD=tiger')
msCursor = conn.cursor()
db = MySQLdb.connect(passwd="pr", db="tenable")
myCursor = db.cursor()
msCursor.execute("SELECT * FROM airlinemaster WHERE type='U'")
dbTables = msCursor.fetchall()
noLength = [56, 58, 61]
for tbl in dbTables:
msCursor.execute("SELECT * FROM airlinemaster WHERE airline = air india('%s')" % tbl[0]) #syscolumns: see sysobjects above.
columns = msCursor.fetchall()
attr = ""
for col in columns:
colType = dataTypes[str(col.xtype)]
if col.xtype == 60:
colType = "float"
attr += col.name +" "+ colType + "(" + str(col.length) + "),"
elif col.xtype in noLength:
attr += col.name +" "+ colType + ","
else:
attr += col.name +" "+ colType + "(" + str(col.length) + "),"
attr = attr[:-1]
myCursor.execute("CREATE TABLE " + tbl[0] + " (" + attr + ");") #create the new table and all columns
msCursor.execute("select * from %s" % tbl[0])
tblData = msCursor.fetchall()
#populate the new MySQL table with the data from MSSQL
for row in tblData:
fieldList = ""
for field in row:
if field == None:
fieldList += "NULL,"
else:
field = MySQLdb.escape_string(str(field))
fieldList += "'"+ field + "',"
fieldList = fieldList[:-1]
myCursor.execute("INSERT INTO " + tbl[0] + " VALUES (" + fieldList + ")" )
I'v tried the above code in many ways to import data.. but still keep on getting error as list index out of range.. Dont know where I'm going wrong.. how do I solve this?
Traceback (most recent call last):
File "C:\Python27\programs new\codes.py", line 10, in <module>
dataTypes = dict((row.split(',')[0].strip(),row.split(',')[1].strip()) for row in typesFile)
File "C:\Python27\programs new\codes.py", line 10, in <genexpr>
dataTypes = dict((row.split(',')[0].strip(),row.split(',')[1].strip()) for row in typesFile)
IndexError: list index out of range
pls help.... thnx in advance..
You have at least one row in typesFile that does not have a , comma in it:
>>> 'line without a comma'.split(',')
['line without a comma']
>>> 'line without a comma'.split(',')[1]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range
If servinfo.txt is supposed to be a comma-separated file with two columns, why not use the csv module to read it instead?
import csv
csvreader = csv.reader(open('servinfo.txt', 'rb'))
dataTypes = dict(csvreader)
That won't solve your issue with there being a line with no comma in it in that file though, you'll have to fix that first.
I am using Python 2.7.9. I'm working on a program that is supposed to produce the following output in a .csv file per loop:
URL,number
Here's the main loop of the code I'm using:
csvlist = open(listfile,'w')
f = open(list, "r")
def hasQuality(item):
for quality in qualities:
if quality in item:
return True
return False
for line in f:
line = line.split('\n')
line = line[0]
# print line
itemname = urllib.unquote(line).decode('utf8')
# print itemhash
if hasQuality(itemname):
try:
looptime = time.time()
url = baseUrl + line
results = json.loads(urlopen(url).read())
# status = results.status_code
content = results
if 'median_price' in content:
medianstr = str(content['median_price']).replace('$','')
medianstr = medianstr.replace('.','')
median = float(medianstr)
volume = content['volume']
print url+'\n'+itemname
print 'Median: $'+medianstr
print 'Volume: '+str(volume)
if (median > minprice) and (volume > minvol):
csvlist.write(line + ',' + medianstr + '\n')
print '+ADDED TO LIST'
else:
print 'No median price given for '+itemname+'.\nGiving up on item.'
print "Finished loop in " + str(round(time.time() - looptime,3)) + " seconds."
except ValueError:
print "we blacklisted fool?? cause we skippin beats"
else:
print itemname+'is a commodity.\nGiving up on item.'
csvlist.close()
f.close()
print "Finished script in " + str(round(time.time() - runtime, 3)) + " seconds."
It should be generating a list that looks like this:
AWP%20%7C%20Asiimov%20%28Field-Tested%29,3911
M4A1-S%20%7C%20Hyper%20Beast%20%28Field-Tested%29,4202
But it's actually generating a list that looks like this:
AWP%20%7C%20Asiimov%20%28Field-Tested%29
,3911
M4A1-S%20%7C%20Hyper%20Beast%20%28Field-Tested%29
,4202
Whenever it is ran on a Windows machine, I have no issue. Whenever I run it on my EC2 instance, however, it adds that extra newline. Any ideas why? Running commands on the file like
awk 'NR%2{printf $0" ";next;}1' output.csv
do not do anything. I have transferred it to my Windows machine and it still reads the same. However, when I paste the output into Steam's chat client it concatenates it in the way that I want.
Thanks in advance!
This is where the problem occurs
code:
csvlist.write(line + ',' + medianstr + '\n')
This can be cleared is you strip the space
modified code:
csvlist.write(line.strip() + ',' + medianstr + '\n')
Problem:
The problem is due to the fact you are reading raw lines from the input file
Raw_lines contain \n to indicate there is a new line for every line which is not the last and for the last line it just ends with the given character .
for more details:
Just type print(repr(line)) before writing and see the output