I'm building a website with Django and among other things I want to display the latest news about a certain topic. I have a python script in the back-end that I would like to program to retrieve the latest news once every 1 hour for example. In the meantime I want to be able to display the most recently retrieved news. I'm doing this in order to avoid that the script is being activated every time someone opens my website.
My script is in news.py:
import pprint
import requests
import datetime
import pandas as pd
import dateutil.parser
secret = "********"
url = 'https://newsapi.org/v2/everything?'
quote = 'Amazon'
parameters1 = {
'q': quote,
'pageSize': 100,
'sortby': 'publishedAt',
'apiKey': secret,
}
response1 = requests.get(url, params=parameters1)
response_json1 = response1.json()
text_combined1 = []
for i in response_json1['articles']:
if i['content'] != None:
case = {'Source': i['source']['name'], 'Title': i['title'], 'url': i['url'],
'Published on:': dateutil.parser.parse(i['publishedAt']).strftime('%Y-%m-%d'), 'Image': i['urlToImage']}
text_combined1.append(case)
data_amazon = pd.DataFrame.from_dict(text_combined1)
news1 = data_amazon.iloc[0]
news2 = data_amazon.iloc[1]
news3 = data_amazon.iloc[2]
My views.py looks like this:
from django.shortcuts import render
from .news import *
def dashboard(request):
content = {'data': data, 'news1': news1, 'news2': news2, 'news3': news3}
return render(request, 'dashboard.html',
content)
I'm new to web development but my understanding as of now is that every time someone sends a request to my webpage that script would be run, which would result in delay in the display of the news and most likely access denial to the news.api due to too many requests.
Thank you in advance!
A good way to do this is with Celery. It will let you schedule tasks in Django.
You can read more about it here, and see some other options as well.
Set up a scheduled job?
Related
I'm learning Django on the fly so bear with me. My goal here is for a user to be able to upload a file, the server script process it and send the results on each iteration to the client live.
My current structure is: User uploads a file (csv), this is read via views.py then renders the blank html view, JS in the html creates a new websocket, and well this is where I'm not able to pass data outside of consumers.py or process the data.
I'm also not sure the structure of view > JS script > consumers > ? is the best route for processing files but I've not found much documentation on file uploads and channels.
views.py
from django.shortcuts import render
import pandas as pd
def someFunc(request):
if request.method == "POST":
global csvUpload
csvUpload = pd.read_csv(request.FILES['filename'])
return render(request, 'app/appPost.html')
I render the view here first so that the blank page displays prior to the script running.
appPost.html JS script
var socket = new WebSocket('ws://localhost:8000/ws/app_ws/')
socket.onmessage = function(event){
var data = JSON.parse(event.data);
console.log(data);
var para = document.createElement("P");
para.innerText = data.message;
document.body.appendChild(para);
}
consumers.py
from channels.generic.websocket import WebsocketConsumer
from asgiref.sync import async_to_sync
class WSConsumer(WebsocketConsumer):
def connect(self):
self.accept()
self.render()
async_to_sync(self.add_group)('appDel_updates_group')
Ultimately I'd then like to pass what's in this alarm file to the websocket connection.
app_alarm.py
from requests.api import request
import pandas as pd
import requests as r
def app_process(csvUpload):
csvUpload=pd.read_csv(request.FILES['filename'])
csvFile = pd.DataFrame(csvUpload)
colUsernames = csvFile.usernames
for user in colUsernames:
req = r.get('https://reqres.in/api/users/2')
t = req.json()
data = t['data']['email']
message = user + " " + data
channel_layer = get_channel_layer()
async_to_sync(channel_layer.group_send)(
'appDel_updates_group',
{'type': 'render', 'message': message}
)
How do you do something after you render the view? (Django)
This seems somewhat relevant, I'm going to see about creating a thread to run the file processing only after the render is done in views.
Edit: Threading allowed me to render the page then pass the data over to the alarm.py file for processing. This has solved the issue I was having by adding the following code into the views.py:
x = threading.Thread(target=app_alarm.sub_process, args=[csvUpload])
x.setDaemon(False)
x.start()
return render(request, 'efax/appPost.html')
This allows views to grab the file using request.FILES and pass it to the script before the page was rendered without waiting on the completion of the processing. That would have rendered the entire point of channels in this situation pointless. Hopefully this helps someone.
Copying from the original post edit:
Threading allowed me to render the page then pass the data over to the alarm.py file for processing. This has solved the issue I was having by adding the following code into the views.py:
x = threading.Thread(target=app_alarm.sub_process, args=[csvUpload])
x.setDaemon(False)
x.start()
return render(request, 'efax/appPost.html')
This allows views to grab the file using request.FILES and pass it to the script before the page was rendered without waiting on the completion of the processing. That would have rendered the entire point of channels in this situation pointless. Hopefully this helps someone.
I hope all you guys be fine.
I was trying to modify multiple products in Woocommerce with python(Restful Api WooCommerce)
here is the code I used:
import sys
import json
from woocommerce import API
from wordpress_xmlrpc import Client, WordPressPost
from wordpress_xmlrpc.methods.posts import GetPosts, NewPost
from wordpress_xmlrpc.methods.users import GetUserInfo
wcapi = API(
url="https://...",
consumer_key="ck_...",
consumer_secret="cs_...",
wp_api = True,
version="wc/v3",
query_string_auth=True # Force Basic Authentication as query string true and using under HTTPS
)
data = {"id":"1995","regular_price":"1775000","sale_price":"1685000","stock_quantity":"6"},
{"id":"4673","regular_price":"2300000","sale_price":"2045000","stock_quantity":"15"}
print(wcapi.put("products/bulk",data).json())
but when I run the code, I got below error:
{'code': 'rest_no_route', 'message': 'No route was found matching the URL and request method', 'data': {'status': 404}}
product IDs are available.
before trying to make some changes on bulk, I was able to modify a single product with the below code:
print(wcapi.put("products/1995",data).json())
my current Woocommerce version 3.9 - & - current python version 3.8.1
i was not able to find the solution or why is that happening. how to fix it?
also how can i modify multiple variable Woocommerce products?
Thanks
finally, the problem got solved.
I just wanted to post it here for the future use of others who may face such this problem.
here is the change which I made in the code:
data = {
"update":
[
{
"id":"1995",
"regular_price":"1775000",
"sale_price":"1685000",
"stock_quantity":"6"},
{
"id":"4673",
"regular_price":"2300000",
"sale_price":"2045000",
"stock_quantity":"15"}
]
}
# Bulk edit of Woocommerce products - here we have 2 sample products
print(wcapi.put("products/batch",data).json())
if you have a better way or think should notice any other thing for working with woocommerce in python, happy to have your post below in this post:-)
I want to have a continuous conversation with watson chatbot.
current situation :
The chatbox does not remember the status of the conversation you previously sent.
I installed the Django framework on the server and created a watson.py file to load the workspace and work with KakaoTalk, a Korean chat application.
The chatbot's conversation flow that I want is as follows.
User: I want to make a reservation
Chatbot: When is the meeting date?
User: tomorrow
Chatbot: How is your meeting time?
User: 14:00
We need your help very much.
watson.py
import json
from watson_developer_cloud import ConversationV1
from .models import Test
from . import views
import simplejson
conversation = ConversationV1(
username = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
password = "xxxxxxxxxxxxxxxxxxxxxx",
version = '2017-05-26' )
workspace_id = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' #workspace
def test(return_str):
result = ''
except Exception as err:
pass
response = conversation.message(
workspace_id = workspace_id,
message_input = {'text': return_str},
)
for i in range(len(response['output']['text'])):
result += response['output']['text'][i] +'\n'
return result
views.py
import json
from django.shortcuts import render
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from .models import Jidum, Test
from . import watson
# Create your views here.
def keyboard(request):
return JsonResponse({
'type':'text',
})
#csrf_exempt
def message(request):
message = ((request.body).decode('utf-8'))
return_json_str = json.loads(message)
return_str = return_json_str['content']
return JsonResponse({
'message': {
'text': watson.test(return_str),
},
'keyboard': {
'type':'text',
},
})
As you can see, has a lot examples inside Watson Developer Cloud for use each API from IBM Developers. Take a look in one example using Watson Conversation.
When you wanna use the same conversation for multiple requests (message), you need to include context object from the previous response.
But remember, you need to create the conversation flow inside your workspace.
For example:
import json
from watson_developer_cloud import ConversationV1
#########################
# message
#########################
conversation = ConversationV1(
username='YOUR SERVICE USERNAME',
password='YOUR SERVICE PASSWORD',
version='2017-04-21')
# replace with your own workspace_id
workspace_id = '0a0c06c1-8e31-4655-9067-58fcac5134fc'
# this example don't include
response = conversation.message(workspace_id=workspace_id, message_input={
'text': 'What\'s the weather like?'})
print(json.dumps(response, indent=2))
# This example include the context object from the previous response.
# response = conversation.message(workspace_id=workspace_id, message_input={
# 'text': 'turn the wipers on'},
# context=response['context']) //example
# print(json.dumps(response, indent=2))
See the Official API Reference using Python.
As #SayuriMizuguchi mentions the main issue is with not maintaining the context object.
Based on your sample above, you would do something like:
context = {}
def test(input_string):
except Exception as err:
pass
response = conversation.message(
workspace_id = workspace_id,
message_input = {'text': return_str},
context = context
)
result = '\n'.join(response['output']['text'])
context = response['context']
return result
Then to run that as an example:
response = test('I want to make a reservation')
print(response)
response = test('tomorrow')
print(response)
I am making a Bitcoin/Ethereum price ticker webpage in Bottle for Python for my company and I want to refresh the page when new prices are available. I am pulling the price data from an API endpoint available through the company. I have hidden the URL to this for security purposes.
templates.py
from bottle import run, get, route, template
import requests
main_api = #url to company's api
def isDataValid(json_data):
if "status" in json_data:
return True
else:
return False
def returnPrices(coin, curr):
url = main_api + coin + curr
json_data = requests.get(url).json()
prices = {}
if isDataValid(json_data) == True:
buy_price = str(json_data["data"]["buy_price"])
sell_price = str(json_data["data"]["sell_price"])
prices = [buy_price, sell_price]
else:
prices = ["Error"]
return prices
#route('/')
def index():
pricesBTC = returnPrices('BTC','USD')
pricesETH = returnPrices('ETH','USD')
btc_buy_price = pricesBTC[0]
btc_sell_price = pricesBTC[1]
eth_buy_price = pricesETH[0]
eth_sell_price = pricesETH[1]
return template('index', btc_buy_price = btc_buy_price, btc_sell_price = btc_sell_price, eth_buy_price = eth_buy_price, eth_sell_price = eth_sell_price)
run(reLoader = True, debug = True)
So how do I refresh the page everytime prices change? I think the prices for ETH and BTC don't change at the same time, so I might have to refresh whenever either of them change. Thank you.
It is not possible without some browser-side JavaScript.
For example, you create additional Bottle endpoint that provides updated data as JSON, and an in-browser script polls that data via AJAX and update the respective html page elements.
As far as JS concerned, there are too many ways to implement this functionality, from simple JQuery to JS frameworks like Angular and Vue.js.
I'm working on connecting a python script to a ServiceNow ticketing environment. Thankfully, ServiceNow has documentation on how to create a ticket from a python script, see documentation here:
http://wiki.servicenow.com/index.php?title=Python_Web_Services_Client_Examples#gsc.tab=0
Here's the script I'm using to create a ticket:
#!/usr/bin/python
from SOAPpy import SOAPProxy
import sys
def createincident(params_dict):
# instance to send to
instance='demo'
# username/password
username='itil'
password='itil'
# proxy - NOTE: ALWAYS use https://INSTANCE.service-now.com, not https://www.service-now.com/INSTANCE for web services URL from now on!
proxy = 'https://%s:%s#%s.service-now.com/incident.do?SOAP' % (username, password, instance)
namespace = 'http://www.service-now.com/'
server = SOAPProxy(proxy, namespace)
# uncomment these for LOTS of debugging output
#server.config.dumpHeadersIn = 1
#server.config.dumpHeadersOut = 1
#server.config.dumpSOAPOut = 1
#server.config.dumpSOAPIn = 1
response = server.insert(impact=int(params_dict['impact']), urgency=int(params_dict['urgency']), priority=int(params_dict['priority']), category=params_dict['category'], location=params_dict['location'], caller_id=params_dict['user'], assignment_group=params_dict['assignment_group'], assigned_to=params_dict['assigned_to'], short_description=params_dict['short_description'], comments=params_dict['comments'])
return response
values = {'impact': '1', 'urgency': '1', 'priority': '1', 'category': 'High',
'location': 'San Diego', 'user': 'fred.luddy#yourcompany.com',
'assignment_group': 'Technical Support', 'assigned_to': 'David Loo',
'short_description': 'An incident created using python, SOAPpy, and web
services.', 'comments': 'This a test making an incident with python.\nIsn\'t
life wonderful?'}
new_incident_sysid=createincident(values)
print "Returned sysid: "+repr(new_incident_sysid)
However, I cannot find any good documentation on the process to resolve the ticket that I just created using the API. When I run the above script, I get the ticket number as well as the sys_id.
Any help would be appreciated.
Thanks.
Apparently thats an old request, but i landed here searching, and apprently 400 other people too, so here is my solution :
Use the server.update(sys_id="...",state="..",...) to modify your record and set the right values to "resolve" it.
For this method only the sys_id parameter is mandatory, it's up to you for the other fields of your form.
P.S. : you can find the API here - https://.service-now.com/incident.do?WSDL