I'm trying to perform a custom query in Python via an ajax call.
The frontend sends the the start time and end time data in unix time eg 1548417600000.
I then convert to (ISO) time (I think?) in Python as that is what MongoDB prefers afaik.
Document example:
{
"_id" : ObjectId("5c125a185dea1b0252c5352"),
"time" : ISODate("2018-12-13T15:09:42.536Z"),
}
PyMonogo doesn't return anything however, despite knowing that there should be thousands of results.
#login_required(login_url='/login')
def querytimerange(request):
print("Expecto Patronum!!")
if request.method == 'POST':
querydata = lambda x: request.POST.get(x)
colname = querydata('colname')
startdate = querydata('start')
enddate = querydata('end')
startint = int(startdate)
endint = int(enddate)
dtstart = datetime.utcfromtimestamp(startint/1000.0)
iso_start = str(dtstart.isoformat())
print(iso_start)
dtend = datetime.utcfromtimestamp(endint/1000.0)
iso_end = str(dtend.isoformat())
print(iso_end)
collection = db[colname]
data = collection.find({"time": {"$gt": iso_start,"$lt": iso_end}})
for a in data:
print(a)
return JsonResponse({"ok": "ok"})
else:
return JsonResponse({"ok": "no"})
So yeah, I think I'm struggling to get the format of the dates right.
After converting from Unix time, the date is in a str like this:
2019-01-20T04:00:00 &
2019-01-25T12:00:00.
Not sure if that's correct, but that should by isoformat afaik?
Main goal is to use it in an aggregation pipeline.
{
"$match": {
"time":{
"date": {
"$gt":startdate,
"$lt":enddate
}
}
}
},
I'm using PyMongo Driver on my Django app.
Thanks!
Related
I have a MongoDB collection which is having updatedtimestamp column. I am passing from and to date using an external yaml file. But updated timestamp column is in the UNIX timestamp.
Any suggestions on this? Or anyway to bring the latest records based on updatedtimestamp which is captured in unixtimestamp?
data format
id:1223344444444
createTimestamp:1640845784796
updateTimestamp:1640845784796
deleteTimestamp:null
uuid:"abcdefgh"
YAML file
collection_name:
incremental: true
date_field: updateTimestamp
start_date: 2022-01-01
end_date: 2022-01-02
hours_interval: 24
Incremental query
# Build interval dates
initial_dt_t = datetime.combine(day, datetime.min.time()) + timedelta(
minutes=lower_limit
)
final_dt_t = datetime.combine(day, datetime.min.time()) + timedelta(
minutes=upper_limit
)
initial_dt = mktime(initial_dt_t.timetuple())
final_dt = mktime(final_dt_t.timetuple())
print("initial-dt")
print(initial_dt_t)
print(final_dt_t)
print("unix")
print(initial_dt)
print(final_dt)
# Connect to MongoDB
mongo_client = MongoClient(MONGODB_URI)
mongo_db = mongo_client[MONGODB_NAME]
collection = mongo_db[COLLECTION_NAME]
# Build aggregation pipeline for MongoDB
or_list = []
print("coming to query")
# Check if there is a single field
if "date_field" in config.keys():
or_list.append(
{
config["date_field"]: {
"$gte": initial_dt,
"$lt": final_dt,
}
}
)
# Check if there are multiple date fields
if "date_fields" in config.keys():
for date_field in config["date_fields"]:
or_list.append(
{
config["date_field"]: {
"$gte": initial_dt,
"$lt": final_dt,
}
}
)
# Get data from MongoDB
query = {"$or": or_list}
# Keep track of batch number
batch_number = 0
print("chunk")
data = pd.DataFrame(list(collection.find(query)))
print("dataframe")
print(data)
I have a collection in my MongoDB with records that have an attribute "timestamp" which is stored as type datetime
I want to use FastAPI to query and return all records which are within a given range of dates that I am passing on as query parameters.
db = client["tasks"]
async def list_activities(
lower_date: str = "", upper_date: str = ""
):
start_date = datetime.strptime(lower_date, "%d-%m-%Y")
end_date = datetime.strptime(upper_date, "%d-%m-%Y")
activities = (
await db["activities"]
.find({"timestamp": {"$gte": start_date, "$lt": end_date}})
.to_list(1000)
)
return activities
On running this with some parameters for lower and upper dates, I'm being returned an empty list
However I wrote another program to test this
MONGODB_URL = os.getenv("MONGODB_URL")
mc = pymongo.MongoClient(MONGODB_URL)
tgdb = mc["tasks"]
tgcol = tgdb["activities"]
recordno = 1
for x in tgcol.find(
{"timestamp": {"$gte": "2022-01-31 00:00:00", "$lt": "2022-02-06 00:00:00"}}
):
print("Record No: " + str(recordno))
recordno += 1
print(x)
This on the other hand returns all the records within the date range just fine.
Any clue as to what I'm doing wrong with the first piece of code?
I m working on a stock prediction project. This is how I want:
To show all the stocks available in Nifty50, Nifty100 or so and then the user will select the stock to predict the high and low price of a stock on next day only.
I m using Django.
What I have done till now:
I m able to display a list of stock.
def index(request):
api_key = 'myAPI_Key'
url50 = 'https://archives.nseindia.com/content/indices/ind_nifty50list.csv'
url100 = 'https://archives.nseindia.com/content/indices/ind_nifty100list.csv'
url200 = 'https://archives.nseindia.com/content/indices/ind_nifty200list.csv'
sfifty = requests.get(url50).content
shundred = requests.get(url100).content
stwohundred = requests.get(url200).content
nifty50 = pd.read_csv(io.StringIO(sfifty.decode('utf-8')))
nifty100 = pd.read_csv(io.StringIO(shundred.decode('utf-8')))
nifty200 = pd.read_csv(io.StringIO(stwohundred.decode('utf-8')))
nifty50 = nifty50['Symbol']
nifty100 = nifty100['Symbol']
nifty200 = nifty200['Symbol']
context = {
'fifty': nifty50,
'hundred': nifty100,
'twohundred': nifty200
}
return render(request, 'StockPrediction/index.html', context)
What I want:
I want to get the live data of all stocks open, high,LTP,Change, Volume.by mean of live data is that it will change as per stock values will change.
Please Help!
You must combine Ajax/Jquery like code below to periodically get data and update values in DOM :
(function getStocks() {
$.ajax({
type: "GET",
url: "url to your view",
success: function (data) {
// here you can get data from backend and do changes like
// changing color by the data coming from your view.
}
}).then(function() { // on completion, restart
setTimeout(getStocks, 30000); // function refers to itself
});
})();
But be careful about making too requests, you must choose proper interval right in this line setTimeout(getStocks, "proper interval");
And in your view you should put queries into a JSON format something like this :
return JsonResponse({'stocks': stocks})
here stocks must be in json format.
I'm having this requirement to get data out from Pardot Saleforce objects using Python API.
Can someone please share any available snippets to get data from all the Pardot objects(tables) using Python.
I am working on a Pardot sync solution using pypardot4 (kudos to Matt for https://github.com/mneedham91/PyPardot4), which involves retrieving data through the API (v4).
Here are some snippets for Visitors API, but you can use the same for almost any Pardot APIs (except Visit...):
from pypardot.client import PardotAPI
# ... some code here to read API config ...
email = config['pardot_email']
password = config['pardot_password']
user_key = config['pardot_user_key']
client = PardotAPI(email, password, user_key)
client.authenticate()
# plain query
data = client.visitors.query(sort_by='id')
total = data['total_results']
# beware - max 200 results are returned, need to implement pagination using offset query paramerter
# filtered query
data = client.visitors.query(sort_by='id', id_greater_than=last_id)
Also I have used some introspection to iterate through API config data I have set up like this:
apiList = config['apiList']
# loop through the apis, and call their query method
for api in apiList:
api_name = api['apiName']
api_object_name = api['clientObject']
api_object = getattr(client, api_object_name)
method = 'query'
if api.get('noSortBy', False) == False:
data = getattr(api_object, method)(created_after=latest_sync, sort_by='created_at')
else:
# The API is not consistent, the sort_by criteria is not supported by all resources
data = getattr(api_object, method)(created_after=latest_sync)
And a snippet from the apiList config JSON:
"apiList":[
{
"apiName": "campaign",
"clientObject": "campaigns"
},
{
"apiName": "customField",
"clientObject": "customfields"
},
{
"apiName": "customRedirect",
"clientObject": "customredirects"
},
{
"apiName": "emailClick",
"clientObject": "emailclicks",
"noSortBy": true
},
...
Notice the noSortBy field and how it's handled in the code.
Hope this helps!
I would like to get Dividend and Split from the python module to Bloomberg API (blapi) for some companies in the US (I am using a Screening to extract these companies). I am using the python module blapi :
import blpapi
# Connect the bloomberg platform
sessionOptions = blpapi.SessionOptions()
sessionOptions.setServerHost(bloomberg_host)
sessionOptions.setServerPort(bloomberg_port)
session = blpapi.Session(sessionOptions)
# Get the dividend and Split
refDataService = session.getService("//blp/refdata")
request = refDataService.createRequest("HistoricalDataRequest")
request.getElement("securities").appendValue("AAPL US Equity")
request.getElement("fields").appendValue("DVD_HIST_ALL")
request.set("periodicityAdjustment", "ACTUAL")
request.set("periodicitySelection", "DAILY")
request.set("startDate", "20140101")
request.set("endDate", "20141231")
request.set("maxDataPoints", 1)
But I get the following amswer :
HistoricalDataResponse = {
securityData = {
security = "AAPL US Equity"
eidData[] = {
}
sequenceNumber = 0
fieldExceptions[] = {
fieldExceptions = {
fieldId = "DVD_HIST_ALL"
errorInfo = {
source = "951::bbdbh5"
code = 1
category = "BAD_FLD"
message = "Not valid historical field"
subcategory = "NOT_APPLICABLE_TO_HIST_DATA"
}
}
}
fieldData[] = {
}
}
}
Looking at the documentation (blpapi-developers-guide) I see multiple request possibility (Reference Data Service, Market Data Service, API Field Information Service) but none of them explain how to get the dividend/split. I don't know which Service and which Request to use.
From the terminal these Dividend and Split and registered under the tag CACT if you use a screening and DVD if you look for the dividend/split of a currently loaded stock (I can loop over the companies I want in my code in worse case).
If someone knows how to do it you will illuminate my day!