We Keep Coding

TypeError: 'NoneType' object is not subscriptable in a int object

Here is a function to check the data and update it
div , update are my mongodb collection object
def data_updater(user1_id,code):
device_id = dvi.find_one({"user_id":user1_id},{"_id":0,"user_id":0})["device_id"]
prv_data = update.find_one({"device_id":device_id},{"_id":0,"device_id":0})
prv_date = prv_data["date"]
msg = prv_data["message"]
if prv_date < current_date and msg != code:
x = update.find_one_and_update({"device_id":id,},{"$set":message":code,"date":current_date}})
print(x.acknowledged)
and when I am calling the function it is giving TypeError data_updater(95626,972681)
the error
Traceback (most recent call last):
File line 170, in <module>
data_updater(95626,972681)
File line 71, in data_updater
device_id = dvi.find_one({"user_id":int(user1_id)},{"_id":0,"user_id":0})["device_id"]
TypeError: 'NoneType' object is not subscriptable
I am not able to find any mistake please help

Your context isn't very clear, however, from the error trace as generated it seems that your find_one() function returns None with the arguments as passed and you are trying to access the value for the key device_id. I recommend you refactor your find_one() function or make use of the following code to resolve the issue at hand:
def data_updater(user1_id,code):
try:
device_id = dvi.find_one({"user_id":user1_id},{"_id":0,"user_id":0})["device_id"]
prv_data = update.find_one({"device_id":device_id},{"_id":0,"device_id":0})
prv_date = prv_data["date"]
msg = prv_data["message"]
if prv_date < current_date and msg != code:
x = update.find_one_and_update({"device_id":id,},{"message":code,"date":current_date}})
print(x.acknowledged)
except TypeError:
print('No value found for specified parameters :/')
PS: I also didn't understand the use of $ in your code, so I removed it thinking of it as a mistake. Hope this helps! 😊

Extracting SQL Server table data to parquet file

I'm trying to extract one of the SQL Server table data to parquet file format using sqlalchemy, pandas and fastparquet modules, but end up with an exception. Appreciate some help on this, I'm trying this one on a simple table with one column of non null integer type.
Code:
import sqlalchemy as sa
import pandas as pd
import urllib as ul
import fastparquet as fp
def main():
sqlInstance = 'sqlInstance'
database = 'database'
tableName = 'Numbers'
props = ul.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=" + sqlInstance + ";"
"DATABASE=" + database + ";"
"Trusted_Connection=yes;")
con = sa.create_engine("mssql+pyodbc:///?odbc_connect={}".format(props))
fetch_batch_size = 1000
metadata = sa.schema.MetaData(bind=con)
table = sa.Table(tableName, metadata, autoload=True)
# Generate pandas/python compatible datatype mapping
map = {}
data_type_map_lookup = {
'int64': ['smallint', 'tinyint', 'integer'],
'float': ['bigint', 'float', 'real'],
'str': ['char', 'nchar', 'nvarchar', 'nvarchar(max)', 'uniqueidentifier', 'varchar(n)', 'varchar(max)'],
'datetime64[ns]': ['date', 'datetime', 'smalldatetime'],
'bytes': ['binary', 'varbinary', 'varbinary(max)'],
'bool': ['bit']
}
for col in table.columns:
for key, value in data_type_map_lookup.items():
types = data_type_map_lookup[key]
if list(filter(str(col.type).lower().startswith, types)):
if col.nullable and key == 'int64':
map[col.name] = 'float'
else:
map[col.name] = key
#Fetch data
output = table.select().execution_options(stream_results=True).execute()
append = False
while True:
batch = output.fetchmany(fetch_batch_size)
if not len(batch) > 0:
break
else:
df = (pd.DataFrame(data=batch, columns=map)).astype(dtype=map)
print(df.to_string()) # Prints good
fp.write("C:\\temp\\test.parquet", df, write_index=False, compression=False, append=append)
append = True
if __name__ == "__main__":
main()
Exception:
TypeError: Expected unicode, got quoted_name
Exception ignored in: 'fastparquet.cencoding.write_list'
Traceback (most recent call last):
File "C:\...lib\site-packages\fastparquet\writer.py", line 1488, in write_thrift
return f.write(obj.to_bytes())
TypeError: Expected unicode, got quoted_name
TypeError: Expected unicode, got quoted_name
Exception ignored in: 'fastparquet.cencoding.write_list'
Traceback (most recent call last):
File "C:\...lib\site-packages\fastparquet\writer.py", line 1488, in write_thrift
return f.write(obj.to_bytes())
TypeError: Expected unicode, got quoted_name

type object not subscriptable - python

from newsapi.sources import Sources
import json
api_key ='*******************'
s = Sources(API_KEY=api_key)
they input the category of news they want
wanted = input('> ')
source_list = s.get(category=wanted, language='en')
index = 0
sources = []
getting the sources
for source in source_list["sources"]:
data = json.dumps(source_list)
data = json.loads(data)
source = (data["sources"][index]["url"])
sources.append(source)
index += 1
from newspaper import Article
i = len(sources) - 1
looping through the source list and printing the articles
for source in sources:
url_ = sources[i]
a = Article[url_]
print(a)
i -= 1
getting error 'type' object is not subscriptable on the line a = Article[url_] have researched but still do not understand why in my case.

The simple solution to your problem is that the line:
a = Article[url_]
Should be:
a = Article(url_)
Now to get to why you're getting the TypeError: 'type' object is not subscriptable error.
This TypeError is the one thrown by python when you use the square bracket notation object[key] where an object doesn't define the __getitem__ method. So for instance, using [] on an object throws:
>>> object()["foo"]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'object' object is not subscriptable
In this case []s were used accidentally instead of ()s when trying to instantiate a class. Most classes (including this Article class) are instances of the type class, so trying object["foo"] causes the same error you are experiencing:
>>> object["foo"]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'type' object is not subscriptable

JSON sub for loop produces KeyError, but key exists

I'm trying to add the JSON output below into a dictionary, to be saved into a SQL database.
{'Parkirisca': [
{
'ID_Parkirisca': 2,
'zasedenost': {
'Cas': '2016-10-08 13:17:00',
'Cas_timestamp': 1475925420,
'ID_ParkiriscaNC': 9,
'P_kratkotrajniki': 350
}
}
]}
I am currently using the following code to add the value to a dictionary:
import scraperwiki
import json
import requests
import datetime
import time
from pprint import pprint
html = requests.get("http://opendata.si/promet/parkirisca/lpt/")
data = json.loads(html.text)
for carpark in data['Parkirisca']:
zas = carpark['zasedenost']
free_spaces = zas.get('P_kratkotrajniki')
last_updated = zas.get('Cas_timestamp')
parking_type = carpark.get('ID_Parkirisca')
if parking_type == "Avtomatizirano":
is_automatic = "Yes"
else:
is_automatic = "No"
scraped = datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')
savetodb = {
'scraped': scraped,
'id': carpark.get("ID_Parkirisca"),
'total_spaces': carpark.get("St_mest"),
'free_spaces': free_spaces,
'last_updated': last_updated,
'is_automatic': is_automatic,
'lon': carpark.get("KoordinataX_wgs"),
'lat': carpark.get("KoordinataY_wgs")
}
unique_keys = ['id']
pprint savetodb
However when I run this, it gets stuck at for zas in carpark["zasedenost"] and outputs the following error:
Traceback (most recent call last):
File "./code/scraper", line 17, in <module>
for zas in carpark["zasedenost"]:
KeyError: 'zasedenost'
I've been led to believe that zas is in fact now a string, rather than a dictionary, but I'm new to Python and JSON, so don't know what to search for to get a solution. I've also searched here on Stack Overflow for KeyErrror when key exist questions, but they didn't help, and I believe that this might be due to the fact that's a sub for loop.
Update: Now, when I swapped the double quotes for single quotes, I get the following error:
Traceback (most recent call last):
File "./code/scraper", line 17, in <module>
free_spaces = zas.get('P_kratkotrajniki')
AttributeError: 'unicode' object has no attribute 'get'

I fixed up your code:
Added required imports.
Fixed the pprint savetodb line which isn't valid Python.
Didn't try to iterate over carpark['zasedenost'].
I then added another pprint statement in the for loop to see what's in carpark when the KeyError occurs. From there, the error is clear. (Not all the elements in the array in your JSON contain the 'zasedenost' key.)
Here's the code I used:
import datetime
import json
from pprint import pprint
import time
import requests
html = requests.get("http://opendata.si/promet/parkirisca/lpt/")
data = json.loads(html.text)
for carpark in data['Parkirisca']:
pprint(carpark)
zas = carpark['zasedenost']
free_spaces = zas.get('P_kratkotrajniki')
last_updated = zas.get('Cas_timestamp')
parking_type = carpark.get('ID_Parkirisca')
if parking_type == "Avtomatizirano":
is_automatic = "Yes"
else:
is_automatic = "No"
scraped = datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')
savetodb = {
'scraped': scraped,
'id': carpark.get("ID_Parkirisca"),
'total_spaces': carpark.get("St_mest"),
'free_spaces': free_spaces,
'last_updated': last_updated,
'is_automatic': is_automatic,
'lon': carpark.get("KoordinataX_wgs"),
'lat': carpark.get("KoordinataY_wgs")
}
unique_keys = ['id']
pprint(savetodb)
And here's the output on the iteration where the KeyError occurs:
{u'A_St_Mest': None,
u'Cena_dan_Eur': None,
u'Cena_mesecna_Eur': None,
u'Cena_splosno': None,
u'Cena_ura_Eur': None,
u'ID_Parkirisca': 7,
u'ID_ParkiriscaNC': 72,
u'Ime': u'P+R Studenec',
u'Invalidi_St_mest': 9,
u'KoordinataX': 466947,
u'KoordinataX_wgs': 14.567929171694901,
u'KoordinataY': 101247,
u'KoordinataY_wgs': 46.05457609543313,
u'Opis': u'2,40 \u20ac /dan',
u'St_mest': 187,
u'Tip_parkirisca': None,
u'U_delovnik': u'24 ur (ponedeljek - petek)',
u'U_sobota': None,
u'U_splosno': None,
u'Upravljalec': u'JP LPT d.o.o.'}
Traceback (most recent call last):
File "test.py", line 14, in <module>
zas = carpark['zasedenost']
KeyError: 'zasedenost'
As you can see, the error is quite accurate. There's no key 'zasedenost' in the dictionary. If you look through your JSON, you'll see that's true for a number of the elements in that array.
I'd suggest a fix, but I don't know what you want to do in the case where this dictionary key is absent. Perhaps you want something like this:
zas = carpark.get('zasedenost')
if zas is not None:
free_spaces = zas.get('P_kratkotrajniki')
last_updated = zas.get('Cas_timestamp')
else:
free_spaces = None
last_updated = None

Key Error in accessing collections from MongoDB (PyMongo)

Why does this work:
import pymongo
from selenium import webdriver
import smtplib
import sys
import json
from pymongo import MongoClient
client = MongoClient('localhost', 27017)
db = client.properties
collection = db['capitalpacific']
fromDB = []
if collection.count() != 0:
for post in collection.find():
fromDB.append(post)
print(fromDB[0]['url'])
correctly prints url only from document 1 of collection (xxx.com)
but I get a keyError when I do this:
for i in range(0, 2):
print(fromDB[i]['url'}
KeyError: 'url'
The documents stored in the DB look like so :
{'url':'xxx.com', 'location':'oregon'}

KeyError generally means the key doesn't exist in the dictionary collection.
For example :
>>> mydoc1=dict(url='xxx.com', location='oregon')
>>> mydoc2=dict(wrongkey='yyy.com', location='oregon')
>>> mylist=[]
>>> mylist.append(mydoc1)
>>> mylist.append(mydoc2)
>>> print mylist[0]['url']
xxx.com
>>> for i in range(0, 2):
... print(mylist[i]['url'])
...
xxx.com
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
KeyError: 'url'
>>>
Here, mydoc2 doesn't have a key called 'url', hence the "KeyError" is being raised for the second element in the list.
So,are you sure 'url' exist in first two records. can you print the contents of "fromDB" and make
sure that first two records has 'url' key.
>>> print mylist
[{'url': 'xxx.com', 'location': 'oregon'}, {'wrongkey': 'yyy.com', 'location': 'oregon'}]