We Keep Coding

How can I import JSON files

I have a problem. I have several JSON files. I do not want to create manually Collections and import these files. I found this question Bulk import of .json files in arangodb with python, but unfortunately I got an error [OUT] AttributeError: 'Database' object has no attribute 'collection'.
How can I import several JSON files and import them fully automatically via Python in Collections?
from pyArango.connection import *
conn = Connection(username="root", password="")
db = conn.createDatabase(name="test")
a = db.collection('collection_name') # <- here is the error
for x in list_of_json_files:
with open(x,'r') as json_file:
data = json.load(json_file)
a.import_bulk(data)
I also looked at the documentation from ArangoDB https://www.arangodb.com/tutorials/tutorial-python/

There is no "collection" method in db instance, which you try to call in your code on this line:
a = db.collection('collection_name') # <- here is the error
According to docs you should use db.createCollection method of db instance.
studentsCollection = db.createCollection(name="Students")

Importing module required by module

I'm writing a web app for a college assignment using Python/Flask and, to keep my app.py file neat, I have a function to query a DB stored in another file. This function uses pymysql and json modules and I can't manage to load these in a way that makes it work - I keep getting an attribute error saying pymysql is not defined.
I've tried putting import statements in my module file (DBjson.py), within the function contained in my module, and within app.py. This is my module code:
def fetchFromDB(host,port,dbname,user,password,query,jsonString=False):
import pymysql # these import statements are in the function in this example - one of several places I've tried them!
import json
conn = pymysql.connect(host, user=user,port=port, passwd=password, db=dbname)
cursorObject = conn.cursor(pymysql.cursors.DictCursor)
with cursorObject as cursor:
cursor.execute(query)
result = cursor.fetchall()
conn.close()
if jsonString == True:
try:
for i in range(len(result)):
result[i]['dateTime'] = result[i]['dateTime'].strftime('%Y-%m-%d %H:%M')
except:
pass
result = json.dumps(result)
return result
And the route from my app.py:
import pymysql
import json
#app.route('/')
def index():
wds = DBjson.fetchFromDB(host,port,dbname,user,password,weatherQuery)
bds = DBjson.fetchFromDB(host,port,dbname,user,password,bikesQuery)
return render_template('weatherDateTime.html', wds=wds, bds=bds)
Any help on how to make this work?
Thanks!
edit - I wrote a test script from which I can load my module and run my function no problem - I have my import statements at the start of the DBjson.py module file and outside of the function. Is this some quirk of Flask/scoping that I don't know about?
PS - Thanks for all the replies so far
import DBjson
query = "SELECT * FROM dublinBikesInfo WHERE dateTime LIKE (SELECT MAX(datetime) FROM dublinBikesInfo);"
#login details for AWS RDS DB
host="xyza"
port=3306
dbname="xyza"
user="xyza"
password="xyza"
a = DBjson.fetchFromDB(host,port,dbname,user,password,query)
print(a)

Hi in your code there is a indent error all the statements has to be inside of the function/method that you have created ex.
`def method():
#code here`
And also try to import the library’s before defining a function/method in the beginning of the page is also good!!
In your scenario please put all the function/method related statements inside of the function/method!!

Python is very unforgiving about indentation. Your module code isn't indented correctly.
The proper way to do this, would be:
def Function():
#your code indented in here

Python URLLib does not work with PyQt + Multiprocessing

A simple code as such:
import urllib2
import requests
from PyQt4 import QtCore
import multiprocessing
import time
data = (
['a', '2'],
)
def mp_worker((inputs, the_time)):
r = requests.get('http://www.gpsbasecamp.com/national-parks')
request = urllib2.Request("http://www.gpsbasecamp.com/national-parks")
response = urllib2.urlopen(request)
def mp_handler():
p = multiprocessing.Pool(2)
p.map(mp_worker, data)
if __name__ == '__main__':
mp_handler()
Basically, if i import PyQt4, and i have a urllib request (i believe this is used in almost all web extraction libraries such as BeautifulSoup, Requests or Pyquery. it crashes with a cryptic log on my MAC)

This is exactly True. It always fails on Mac, I have wasted rows of days just to fix this. And honestly there is no fix as of now. The best way is to use Thread instead of Process and it will work like a charm.
By the way -
r = requests.get('http://www.gpsbasecamp.com/national-parks')
and
request = urllib2.Request("http://www.gpsbasecamp.com/national-parks")
response = urllib2.urlopen(request)
do one and the same thing. Why are you doing it twice?

This may be due _scproxy.get_proxies() not being fork-safe on Mac.
This is raised here https://bugs.python.org/issue33725#msg329926
_scproxy has been known to be problematic for some time, see for instance Issue31818. That issue also gives a simple workaround: setting urllib's "no_proxy" environment variable to "*" will prevent the calls to the System Configuration framework.
This is something that urllib may be attempting to do causing failure when multiprocessing.
There is a workaround and that is to set the environmental variable no-proxy to *
Eg. export no_proxy=*

How to iterate/parse Tweepy get_user object

In playing around with Tweepy I notice that the 'status' variable returned from a call to get_user is <tweepy.models.Status object at 0x02AAE050>
Sure, I can call get_user.USER.status, but how can I grab that information from the get_user call? i.e. I want to loop through user.getstate() and if I find an object which requires further iteration, loop through that too
I've searched high/low of answers, but my newness to Python is causing problems, which I'm pretty sure are easy to solve if I knew the right questions to ask.
Thanks for any pointer here...
# -*- coding: utf-8 -*-
import sys
import tweepy
import json
from pprint import pprint
api = tweepy.API()
def main():
print "Starting."
user = api.get_user('USER',include_entities=1)
print "================ type ================="
print type(user)
print "================ dir ================="
print dir(user)
print "================ user ================="
#
# We can see 'status': <tweepy.models.Status object at 0x02AAE050>, .......but how do I "explode" that automagically?
#
pprint ((user).__getstate__())
print "================ user.status ================="
pprint ((user).status.__getstate__())
print "================= end ================="
if __name__ == "__main__":
main()
I was able to get the intended behaviour by using jsonpickle, using the following code.
import jsonpickle
.
.
.
user = api.get_user('USERNAME',include_entities=1)
pickled = jsonpickle.encode(user)
print(json.dumps(json.loads(pickled), indent=4, sort_keys=True)) #you could just print pickled, but this makes it pretty
I'm still really interested to understand what I'm missing in understanding how to detect and expand that status object.

try this:
api = tweepy.API(auth,parser=tweepy.parsers.JSONParser())
user = api.get_user('USER',include_entities=1)
now you can user user['status'] and others easily

Django - importing dynamically

I'm trying to do a dynamic import of a python module in django. I have two different apps that I want to import from, and I want to replace these import statements:
from app1.forms import App1ProfileForm
from app2.forms import App2ProfileForm
I am dynamically able to create the strings App1ProfileForm and App2ProfileForm and then instantiate them like so:
globals()[form]()
I tried following some of the instructions in this post: Dynamically import class by name for static access
and so I tried doing this:
theModule = __import__("app1.forms.App1ProfileForm")
but I'm getting an error that says No module named App1ProfileForm
EDIT:::
Ok I tried this code:
theModule = __import__("app1")
print theModule
theClass = getattr(theModule,'forms')
print theClass
theForm = getattr(theClass,'App1ProfileForm')
print theForm
theForm.initialize()
but I get an error that type object 'App1ProfileForm' has no attribute 'initialize'

You don't want to do this. Imports are done when the relevant code is first executed - in the case of module-level imports, it's when the module itself is imported. If you're depending on something in the request, or some other run-time element, to determine what class you want, then this will not work.
Instead, just import them both, and get the code to choose which one you need:
from app1.forms import App1ProfileForm
from app2.forms import App2ProfileForm
forms = {'app1': App1ProfileForm,
'app2': App2ProfileForm}
relevant_form = forms[whatever_the_dependent_value_is]

I don't quite know how you're generting the string to import. I'll assume you generate the whole "path". Try this:
def import_from_strings(paths):
ret = []
for path in paths:
module_name, class_name = path.rsplit('.', 1)
module = __import__(module_name, globals(), locals(), [class_name], -1)
ret.append(getattr(module, class_name))
return ret

Aren't you trying to import a class, and not a module ? I'm not an expert, but I think you must import the module using __import__, then select it's App1ProfileForm class with something like yourmodule.App1ProfileForm

I figured it out. Here's how to do it:
theModule = __import__(module_name+".forms") # for some reason need the .forms part
theClass = getattr(theModule,'forms')
theForm = getattr(theClass,form_name)
then to initialize:
theForm() or theForm(request.POST)