I am trying to write a script to GET project data from Insightly and post to 10000ft. Essentially, I want to take any newly created project in one system and create that same instance in another system. Both have the concept of a 'Project'
I am extremely new at this but I only to GET certain Project parameters in Insightly to pass into the other system (PROJECT_NAME, LINKS:ORGANIZATION_ID, DATE_CREATED_UTC) to name a few.
I plan to add logic to only POST projects with a DATE_CREATED_UTC > yesterday, but I am clueless on how to setup the script to grab the JSON strings and create python variables (JSON datestring to datetime). Here is my current code. I am simply just printing out some of the variables I require to get comfortable with the code.
import urllib, urllib2, json, requests, pprint, dateutil
from dateutil import parser
import base64
#Set the 'Project' URL
insightly_url = 'https://api.insight.ly/v2.1/projects'
insightly_key =
api_auth = base64.b64encode(insightly_key)
headers = {
'GET': insightly_url,
'Authorization': 'Basic ' + api_auth
}
req = urllib2.Request(insightly_url, None, headers)
response = urllib2.urlopen(req).read()
data = json.loads(response)
for project in data:
project_date = project['DATE_CREATED_UTC']
project_name = project['PROJECT_NAME']
print project_name + " " + project_date
Any help would be appreciated
Edits:
I have updated the previous code with the following:
for project in data:
project_date = datetime.datetime.strptime(project['DATE_CREATED_UTC'], '%Y-%m-%d %H:%M:%S').date()
if project_date > (datetime.date.today() - datetime.timedelta(days=1)):
print project_date
else:
print 'No New Project'
This returns every project that was created after yesterday, but now I need to isolate these projects and post them to the other system
Here is an example of returning a datetime object from a parsed string. We will use the datetime.strptime method to accomplish this. Here is a list of the format codes you can use to create a format string.
>>> from datetime import datetime
>>> date_string = '2014-03-04 22:30:55'
>>> format = '%Y-%m-%d %H:%M:%S'
>>> datetime.strptime(date_string, format)
datetime.datetime(2014, 3, 4, 22, 30, 55)
As you can see, the datetime.strptime method returns a datetime object.
Related
I am building a project to import crypto prices in json format that I can later parse out.
Now I have been researching and I haven't found a sure way to combine these two statements together so I can do one call instead of two. I want to do BTC and ETH like you see in one "req_params" I just dont know what the right syntax is a lot of the sites have confusing material.
import requests
import json
from datetime import datetime,timedelta
base_url = 'https://api.kucoin.com/api/v1/market/candles'
start_timestmp = datetime.strptime("2021-11-11T11:11:11","%Y-%m-%dT%H:%M:%S")
end_timestmp = start_timestmp + timedelta(minutes=1)
# How do I combine these two stamens into one call ?
req_params = {'symbol':'BTC-USDT','type':'1min',
'startAt':int(datetime.timestamp(start_timestmp)),
'endAt': int(datetime.timestamp(end_timestmp))
}
req_params2 = {'symbol':'ETH-USDT','type':'1min',
'startAt':int(datetime.timestamp(start_timestmp)),
'endAt': int(datetime.timestamp(end_timestmp))
}
url_response = requests.get(url=base_url,params=req_params)
url_response2 = requests.get(url=base_url,params=req_params2)
response_json = url_response.json()
response_json2 = url_response2.json()
# put the response in json format
closing_price = float(response_json['data'][0][2])
print(response_json)
print(response_json2)
print(closing_price)
I am trying to convert a datetime from an API that stores datetime values as UTC. I need to convert the datetime to my local time 'Pacific/Auckland'
The API I am using is Sunrise-Sunset https://sunrise-sunset.org/api
The specific location I am requesting is Christchurch, New Zealand https://sunrise-sunset.org/search?location=christchurch
import requests
api_url = 'https://api.sunrise-sunset.org/json?lat=-43.525650&lng=172.639847&formatted=0'
response = requests.get(api_url)
if response.status_code == 200:
sunset_today = response.json()['results']['sunset']
print(sunset_today) # outputs '2021-09-26T06:31:41+00:00'
I have searched StackOverflow and Google extensively, but cannot seem to find a solution that fits my needs.
The question I am asking is
How can I convert the UTC value to my local datetime ('Pacific/Auckland')?
FYI, I don't want to add bloat to the application, but from previous (unsuccessful) attempts at solving this problem I have already installed the tzlocal and pytz packages.
I am writing my application in Django 3.2.7 and have adjusted my settings.py TIME_ZONE = 'Pacific/Auckland'
Edit
When trying to convert the string to a datetime I get the following error.
time data '2021-09-26T06:31:41+00:00' does not match format '%Y-%m-%dT%H:%M:%S %Z'
sunset_today = response.json()['results']['sunset']
format = '%Y-%m-%dT%H:%M:%S %Z'
parsed_date = datetime.strptime(sunset_today, format)
print(parsed_date)
# ERROR: time data '2021-09-26T06:31:41+00:00' does not match format '%Y-%m-%dT%H:%M:%S %Z'*
To convert timezone aware string to python datetime easier to use fromisoformat, since you are getting ISO formatted string from API anyway:
import datetime
sunset_today = response.json()['results']['sunset']
parsed_date = datetime.datetime.fromisoformat(sunset_today)
# 2021-09-26 06:31:41+00:00
I solved the problem using the dateutil and pytz library
import requests
import pytz
from dateutil import parser
api_url = 'https://api.sunrise-sunset.org/json?lat=-43.525650&lng=172.639847&formatted=0'
response = requests.get(api_url)
nz_zone = pytz.timezone('Pacific/Auckland')
if response.status_code == 200:
sunset_today = response.json()['results']['sunset']
converted_date = parser.parse(sunset_today).astimezone(nz_zone)
print(converted_date) # outputs 2021-09-26 19:31:41+13:00
I am trying to use Tableau calculated field to use my python script, which gets JSON data. My ultimate goal is to get this data into tableau in a tabular format.
I have read JSON is easier to get into tableau opposed to dataframe object.
I currently use it in Spyder program. and execute this to get my data.
print (get1D ("2019-02-02", "2019-02-05", "Tableau", "Limits"))
In my calculated field I get the error: "Error Parsing Number" on
.format(status_code))
error msg:
Any help would be appreciated on getting this data into tableau. Here is my full script.
SCRIPT_INT(
import time
import requests
import json
import pandas as pd
import re
import urllib3
import math
from io import StringIO
from datetime import datetime, date,timedelta
from pandas.tseries.offsets import BDay
from urllib.parse import urlencode
from flask import json
def call_api(url, request_dict, post):
if post:
header = {'content-type':'application/json'}
resp = requests.post(url, data=json.dumps(request_dict), auth=('user', 'pass'), headers = header, verify=False)
else:
url = url + urlencode(request_dict)
resp = requests.get(url, auth=('user', 'pass'), verify=False)
status_code = resp.status_code
if status_code == 401:
raise ValueError("There is an error with the connection.\nLogin failed. \nNot authorized. Please check your credentials and try again.\nStatus code {}".format(status_code))
elif status_code == 404:
raise ValueError("There is an error with the connection.\nCould not connect to the server.\nStatus code {}".format(status_code))
elif status_code == 200:
pass
else:
raise ValueError("There is an error with the connection.\nStatus code {}".format(status_code))
return resp
def getData (startDate, endDate, nodeName, Type, Id):
request_dict = [{
"hierarchy": "Tableau",
"nodeName": nodeName,
"FilterId": Type,
"Id": Id ,
}]
url = "https://sampleurl/startDate={0}&endDate={1}"
startDate = datetime.strptime(startDate, '%Y-%m-%d')
startDate = startDate.strftime ('%Y%m%d')
endDate = datetime.strptime(endDate, '%Y-%m-%d')
endDate = endDate.strftime ('%Y%m%d')
url = url.format(startDate, endDate)
resp = call_api(url, request_dict, True)
return resp.json ()
def get1D(startDate, endDate, nodeName, Type):
return getData (startDate, endDate, nodeName, Type, 1)
)
Take a look at Tableau's guide to Authoring Python Calculations in Tableau.
In general, the format needs to be as such:
SCRIPT_INT("import xyz foo=_arg1+_arg2 return foo", SUM([A number]), SUM([Another Number])
From what I can see, you'll need to add quotation marks to your calculated field, replace any field variables that need to be passed with _argX, add commas to the calculated field, then a list of fields to pass as those args.
Take note that anytime you see the error, "The calculation contains errors," in the Tableau Calculated field window that the problem is within the Tableau Calculated field (Formatting/compiling) and not necessarily with the underlying Python. (The error you're seeing is a red herring. The Tableau Calculated Field interpreter is seeing the "." as a decimal and expecting to receive a number behind it.) In the calculated field window, Tableau will have no way of checking the underlying Python - it only passes it to TabPy as a string. The opposite is true as well - seeing "This calculation is valid" in the Tableau Calculated Field window doesn't necessarily mean that the Python script will return properly.
Hope this helps.
Edit in response to comment:
Below is an example using the code you provided in your question.
Replaced double quotations within Python script with single quotations. In this way, Tableau will be able to tell its own double quotations apart. (Tableau is the same as Python in that it treats double and single quotes the same.)
Replaced the input parameters for getData() with the _arg1 through _arg4.
Passed [Start Date], [End Date], [Node Name], and [Type] as args after the Python script is passed as a string. (These get implanted into the string as _arg1 through _arg4. (ATTR() might not be the right aggregation method here - you'll have to experiment.)
The Calculated field will now at least compile, however, I cannot guarantee that it will execute on the Python side or that it will accomplish what you're trying to do.
I'm not sure how get1D() is going to react here. You may have to put your _arg1 through _arg4 as parameters there too. There is some work that will need to be done - and maybe even a reformat of the code to accept Tableau args.
Please do read the TabPy documentation for more clarification on its use than I'm able to provide here. Also, here is a good blog post. When used properly, its immensely powerful.
Good luck!
SCRIPT_INT(
"import time
import requests
import json
import pandas as pd
import re
import urllib3
import math
from io import StringIO
from datetime import datetime, date,timedelta
from pandas.tseries.offsets import BDay
from urllib.parse import urlencode
from flask import json
def call_api(url, request_dict, post):
if post:
header = {'content-type':'application/json'}
resp = requests.post(url, data=json.dumps(request_dict), auth=('user', 'pass'), headers = header, verify=False)
else:
url = url + urlencode(request_dict)
resp = requests.get(url, auth=('user', 'pass'), verify=False)
status_code = resp.status_code
if status_code == 401:
raise ValueError('There is an error with the connection.\nLogin
failed. \nNot authorized. Please check your credentials and try
again.\nStatus code {}'.format(status_code))
elif status_code == 404:
raise ValueError('There is an error with the connection.\nCould not
connect to the server.\nStatus code {}'.format(status_code))
elif status_code == 200:
pass
else:
raise ValueError('There is an error with the connection.\nStatus
code {}'.format(status_code))
return resp
def getData (startDate, endDate, nodeName, Type, Id):
request_dict = [{
'hierarchy': 'Tableau',
'nodeName': nodeName,
'FilterId': Type,
'Id': Id ,
}]
url = 'https://sampleurl/startDate={0}&endDate={1}'
startDate = datetime.strptime(startDate, '%Y-%m-%d')
startDate = startDate.strftime ('%Y%m%d')
endDate = datetime.strptime(endDate, '%Y-%m-%d')
endDate = endDate.strftime ('%Y%m%d')
url = url.format(startDate, endDate)
resp = call_api(url, request_dict, True)
return resp.json ()
def get1D(startDate, endDate, nodeName, Type):
return getData (_arg1, _arg2, _arg3, _arg4, 1)"
,
ATTR([Start Date]),ATTR([End Date]),ATTR([Node Name], ATTR([Type]
)
Adding this answer for posterity and to simplify my previous answer and resulting thread.
The Python script presented in the question is intending to load data directly into Tableau. TabPy is meant to operate on data already existing within Tableau, as opposed to being a source if ingestion.
A necessity here is to place the output of the Python script into an intermediary location where it can then be connected to by Tableau.
I'm working on creating scripts using python, mongodb and the pymongo module to fetch certain aspects of the Twitter API and store them in a mongo database. I've written some scripts to do different things: access the search API, access the user_timeline, and more. However, I have been just getting to know all of the tools that I'm working with and it's time for me to go back and make it more efficient. Thus, right now I'm working on adding functions and classes to my scripts. Here is one of my scripts without functions or classes:
#!/usr/local/bin/python
import twitter
import datetime
from datetime import date, timedelta, datetime
import pymongo
from pymongo import Connection
# Twitter handle that we are scraping mentions for
SCREEN_NAME = '#twitterapi'
# Connect to the database
connection = Connection()
db = connection.test
collection = db.twitterapi_mentions # Change the name of this database
t = twitter.Twitter(domain='search.twitter.com')
# Fetch the information from the API
results = []
for i in range(2):
i+=1
response = t.search(q=SCREEN_NAME, result_type='recent', rpp=100, page=i)['results']
results.extend(response)
# Create a document in the database for each item taken from the API
for tweet in results:
id_str = tweet['id_str']
twitter_id = tweet['from_user']
tweetlink = "http://twitter.com/#!/%s/status/%s" % (twitter_id, id_str)
created_at = datetime.strptime(tweet['created_at'], "%a, %d %b %Y %H:%M:%S +0000")
date = created_at.date().strftime("%m/%d/%y")
time = created_at.time().strftime("%H:%M:%S")
text = tweet['text']
identifier = {'id' : id_str}
entries = {'id' : id_str, 'tweetlink' : tweetlink, 'date' : date, 'time' : time, 'text' : text, 'twitter_id':twitter_id }
collection.update(identifier, entries, upsert = True)
These scripts have been working well for me, but I have to run the same script for multiple twitter handles. For instance I'll copy the same script and change the following two lines:
SCREEN_NAME = '#cocacola'
collection = db.cocacola_mentions
Thus I'm getting mentions for both #twitterapi and #cocacola. I've thought a lot about how I can make this into a function. The biggest problem that I've run into is finding a way to change the name of the collection. For instance, consider this script:
#!/usr/local/bin/python
import twitter
import datetime
from datetime import date, timedelta, datetime
import pymongo
from pymongo import Connection
def getMentions(screen_name):
# Connect to the database
connection = Connection()
db = connection.test
collection = db.screen_name # Change the name of this database
t = twitter.Twitter(domain='search.twitter.com')
# Fetch the information from the API
results = []
for i in range(2):
i+=1
response = t.search(q=screen_name, result_type='recent', rpp=100, page=i) ['results']
results.extend(response)
# Create a document in the database for each item taken from the API
for tweet in results:
id_str = tweet['id_str']
twitter_id = tweet['from_user']
tweetlink = "http://twitter.com/#!/%s/status/%s" % (twitter_id, id_str)
created_at = datetime.strptime(tweet['created_at'], "%a, %d %b %Y %H:%M:%S +0000")
date = created_at.date().strftime("%m/%d/%y")
time = created_at.time().strftime("%H:%M:%S")
text = tweet['text']
identifier = {'id' : id_str}
entries = {'id' : id_str, 'tweetlink' : tweetlink, 'date' : date, 'time' : time, 'text' : text, 'twitter_id':twitter_id }
collection.update(identifier, entries, upsert = True)
getMentions("#twitterapi")
getMentions("#cocacola")
If I use the above script then all of the data is stored in the collection "screen_name" but I want it to be stored in the screen name that is passed through. Ideally, I want #twitterapi mentions to be in a "twitterapi_mentions" collection and I want #cocacola mentions to be in a "cocacola_mentions" collection. I believe that using the Collection class of pymongo might be the answer and I've read the documentation but can't seem to get it to work. If you have other suggestions of how I should make this script more efficient they would be incredibly appreciated. Otherwise, please excuse any mistakes I've made, as I said, I'm new to this.
Use getattr to retrieve the attribute by string name:
collection = getattr(db, screen_name)
I'd go with:
collection = db[screen_name]
I think it's more straightforward.
Is there any facility in Django for doing currency conversions? Obviously, rates change day by day but I'm somewhat hopeful that the locale module has some sort of web-service based converter.
There's a snippet here that handles the formatting: http://www.djangosnippets.org/snippets/552/ But I need to localize the values first.
Probably more elegant ways to do this, but it works.
currency_in = 'USD'
currency_out = 'NOK'
import urllib2
req = urllib2.urlopen('http://finance.yahoo.com/d/quotes.csv?e=.csv&f=sl1d1t1&s='+currency_in+currency_out+'=X')
result = req.read()
# result = "USDNOK=X",5.9423,"5/3/2010","12:39pm"
Then you can split() the result for the modifier.
# Install google-currency package
# pip install google-currency
>>> from google_currency import convert
>>> convert('usd', 'bdt', 1)
Output:
{"from": "USD", "to": "BDT", "amount": "85.30", "converted": true}
You could use django-money app for currency conversions in Django-based projects.
It works with different rate sources and provides an interface to perform conversions & money localization:
>>> # After app setup & adding rates to the DB
>>> from djmoney.money import Money
>>> from djmoney.contrib.exchange.models import convert_money
>>> value = Money(100, 'EUR')
>>> converted = convert_money(value, 'USD')
>>> converted
<Money: 122.8184375038380800 USD>
>>> str(converted)
US$122.82
Formats are easily customizable, you could find the documentation on the project page.
You can use django-money to handle money values including the conversion of currencies as shown below. *The example below converts 100 USD to ... EUR:
# "views.py"
from django.http import HttpResponse
from app.models import MyModel
from djmoney.contrib.exchange.models import convert_money
from djmoney.money import Money
def test(request):
print(convert_money(Money(100, 'USD'), 'EUR')) # Here
return HttpResponse("Test")
You can see my answer and django-money doc which explain more about how to convert currencies.