What I'm trying to do:
Android Application (ADMIN) that gets job Title from user and fetches all the jobs related to it using Scrapy (Python) which are saved to database through API.
Android Application (CLIENT) that fetches all the data from database through API.
Problem:
I'm stuck on how to connect my python Script with Ktor.
Some Code for more clarity:
import spiders.Linkedin as linkedinSpider
linkedinSpider.main(numberOfPages=1, keywords="ios developer", location="US")
This piece of Code works fine and helps me to fetch and store data into database and returns True if saved successfully else False. I just need to call this function and all the work is done for me.
Similarly,
This is how i get data from the user through API from ADMIN Side using KTOR. Also this works fine.
// Fetchs New Jobs.
get("/fetch") {
val numberOfPages = call.request.queryParameters["pages"]?.toInt() ?: 1
val keyword = call.request.queryParameters["keyword"]
val location = call.request.queryParameters["location"]
if (numberOfPages < 1) {
call.respond(
status = HttpStatusCode.BadRequest,
message = Message(message = "Incorrect Page Number.")
)
} else {
call.respond(
status = HttpStatusCode.PK,
message = Message(message = "Process Completed.")
)
}
}
This is how i wanted the code to work logically.
// Fetch New Jobs.
get("/fetch") {
val numberOfPages = call.request.queryParameters["pages"]?.toInt() ?: 1
val keyword = call.request.queryParameters["keyword"]
val location = call.request.queryParameters["location"]
if (numberOfPages < 1) {
call.respond(
status = HttpStatusCode.BadRequest,
message = Message(message = "Incorrect Page Number.")
)
} else {
// TODO: Call Python Function Here and Check the Return Type. Something Like This.
if(linkedinSpider.main(numberOfPages=numberOfPages, keywords=keyword, location=location)) {
call.respond(
status = HttpStatusCode.OK,
message = Message(message = "Process Completed.")
)
} else {
call.respond(
status = HttpStatusCode.InternalServerError,
message = Message(message = "Something Went Wrong.")
)
}
}
}
Also these are two different individual projects. Do i need to merge them or something as when i tried it says no python interpreter found in intelliJ which i'm using as an IDE for Ktor Development. Also I tried to configure python interpreter but it seems of no use as it is not able to reference the variables or the python files.
Edit 1
This is what i tried
val processBuilder = ProcessBuilder(
"python3", "main.py"
)
val exitCode = processBuilder.start().waitFor()
if (exitCode == 0) {
println("Process Completed.")
} else {
println("Something Went Wrong.")
}
}
The exitCode i am getting is 2. But when i run the below code it works with exitCode 0.
ProcessBuilder("python3", "--version")
Edit 2
After This the code is working but not terminating and i get no output
This is the kotlin file i made to simulate the problem
package com.bhardwaj.routes
import java.io.File
fun main() {
val processBuilder = ProcessBuilder(
"python3", "main.py", "1", "android", "india"
)
processBuilder.directory(File("src/main/kotlin/com/bhardwaj/"))
val process = processBuilder.start()
val exitCode = process.waitFor()
if (exitCode == 0) {
val output = String(process.inputStream.readBytes())
print("Process Completed -> $output")
} else {
val output = String(process.errorStream.readBytes())
print("Something went wrong -> $output")
}
}
If i run same commands in terminal it works.
python3 src/main/kotlin/com/bhardwaj/main.py 1 "android" "india"
After i run same commands in the Terminal the output is ->
Edit 3
When I stopped default Scrapy logging, the code worked. It seems there is some limit to output stream in Process Builder. Here is what i did.
def __init__(self, number_of_pages=1, keywords="", location="", **kwargs):
super().__init__(**kwargs)
logging.getLogger('scrapy').setLevel(logging.WARNING)
This worked flawlessly in Development mode. But there is something more i want to discuss.
When I deployed the same on Heroku and made a get request that gives me 503 service unavailable after 30-40 sec later. But after another 20-30 sec i get my data into database. It's seems to be very confusing why this is happening.
For the flow of the program and clarity -
When i make get request to ktor, it request scrapy(A python program) to scrape data and store it in JSON file. After all the process done it make a request to another endpoint to store all the data to database and after all this it returns and respond the user with the particular status code it should give as output using call.respond in ktor.
Is the issue is due to another request made by the python program to the endpoint or it is related to heroku that we can handle one process at a time. Because there is no issue on development mode i.e. localhost url
When I stopped default Scrapy logging, the code worked. It seems there is some limit to output stream in Process Builder. Here is what i did.
def __init__(self, **kwargs):
super().__init__(**kwargs)
logging.getLogger('scrapy').setLevel(logging.WARNING)
Related
I try to write a script to get transactions of my ethereum address as they happen.
This script here works when monitoring a contractAddress, but if i want to monitor my own address or non contract address, it dont get any transactions
def handle_event(event):
print(event)
print(Web3.toJSON(event))
def log_loop(event_filter, poll_interval):
while True:
for event in event_filter.get_new_entries():
handle_event(event)
time.sleep(poll_interval)
def main():
event_filter = web3.eth.filter({"address": trackAddress})
#get_block = web3.eth.get_block('latest')
#block_filter = web3.eth.filter('latest')
log_loop(event_filter, 2)
if __name__ == '__main__':
main()
As far as I know, using the filter command, you can only monitor events that occur on the contract. This option will not work in your case.
I'll write the JavaScript code as I'm more familiar with it, but you can adapt it to python. I would probably do something like this:
async function checkBlock(address) {
const block = await web3.eth.getBlock('latest');
console.log(`Checking new block ${block.number}`);
for (let txHash of block.transactions) {
const tx = await web3.eth.getTransaction(txHash);
if (address === tx.to.toLowerCase()) {
console.log(`New transaction found. Block - ${block.number}`);
console.log(`Transaction: ${tx}`);
}
}
}
You can insert this function into setInterval and call it approximately every ~15 seconds. Why 15 seconds? On average, a new block appears just every 15 seconds.
Another option is using the ethers.js library. You can subscribe directly to blocks and request transaction information for a new block. But I don't know if such a library exists for python.
You can also do something like this with:
const topicSets = [
utils.id("Transfer(address,address,uint256)"),
null,
[
null,
hexZeroPad(address, 32)
]
]
provider.on(topicSets, (log, event) => {
// Emitted any token is sent TO your address
})
In this case, the handler will be triggered every time you receive ERC-20 tokens to your address.
I recently encountered an issue that I have not been able to solve, despite calling the Bloomberg helpdesk and researching thoroughly the internet for similar cases.
In short, I am using the official Python blpapi from Bloomberg (https://github.com/msitt/blpapi-python) and now am experiencing some connectivity issue: I cannot leave a session opened.
Here is the code I am running: https://github.com/msitt/blpapi-python/blob/master/examples/SimpleHistoryExample.py
I simply added a "while True loop" and a "time.sleep" in it so that I can keep the session open and refresh my data every 30 seconds (this is my use case).
This use to run perfectly fine for days, however, since last Friday, I am now getting those log messages:
22FEB2021_08:54:18.870 29336:26880 WARN blpapi_subscriptionmanager.cpp:7437 blpapi.session.subscriptionmanager.{1} Could not find a service for serviceCode: 90.
22FEB2021_08:54:23.755 29336:26880 WARN blpapi_platformcontroller.cpp:377 blpapi.session.platformcontroller.{1} Connectivity lost, no connected endpoints.
22FEB2021_08:54:31.867 29336:26880 WARN blpapi_platformcontroller.cpp:344 blpapi.session.platformcontroller.{1} Connectivity restored.
22FEB2021_08:54:32.731 29336:26880 WARN blpapi_subscriptionmanager.cpp:7437 blpapi.session.subscriptionmanager.{1} Could not find a service for serviceCode: 90.
which goes on and on and on, along with those responses as well:
SessionConnectionDown = {
server = "localhost:8194"
}
ServiceDown = {
serviceName = "//blp/refdata"
servicePart = {
publishing = {
}
}
}
SessionConnectionUp = {
server = "localhost:8194"
encryptionStatus = "Clear"
compressionStatus = "Uncompressed"
}
ServiceUp = {
serviceName = "//blp/refdata"
servicePart = {
publishing = {
}
}
}
I still can pull the data from the bloomberg API: I see the historical data request results just fine. However:
Those service/session status messages messes up my code (I could still ignore them)
For some reason the connect/reconnect also messes my Excel BBG in the background and prevent me from using the BBG excel add-in at all! I now have those "#N/A Connection" outputs in all of my workbooks using bloomberg formulas.
screenshot from excel
Has anyone ever encountered such cases? If yes, please do not hesitate to share your experience, any help is more than appreciated!
Wishing you all a great day,
Adrien
I cannot comment yet so I will try to "answer" it. I use blpapi everyday and pull data all day. I am a BBG Anywhere user and never have any session issues. If you log in from a different device it will kill your session for the Python app. Once you log back in where the python app is running it will connect again.
Why do you have another while loop and sleep to keep the session alive? You should create a separate session and always call it to run your request. You should not need any "keep alive" code inside the request. Just don't call session.stop(). This is what I ended up doing after much trial and error from not knowing what to do.
I run my model using Excel, trying to move away from any substantial code in Excel and use it as a GUI until I can migrate to a custom GUI. I also have BDP functions in my Excel and they work fine.
import blpapi
# removed optparse because it is deprecated.
from argparse import ArgumentParser
SERVICES = {}
def parseCmdLine():
parser = ArgumentParser(description='Retrieve reference data.')
parser.add_argument('-a',
'--ip',
dest='host',
help='server name or IP (default: %(default)s)',
metavar='ipAddress',
default='localhost')
parser.add_argument('-p',
dest='port',
type=int,
help='server port (default: %(default)s)',
metavar='tcpPort',
default=8194)
args = parser.parse_args()
return args
def start_session():
"""Standard session for synchronous refdata requests. Upon creation
the obj is held in SERVICES['session'].
Returns:
obj: A session object.
"""
args = parseCmdLine()
# Fill SessionOptions
sessionOptions = blpapi.SessionOptions()
sessionOptions.setServerHost(args.host)
sessionOptions.setServerPort(args.port)
# Create a Session
session = blpapi.Session(sessionOptions)
# Start a Session
session.start()
SERVICES['session'] = session
return SERVICES['session']
def get_refDataService():
"""Create a refDataService object for functions to use. Upon creation
it is held in SERVICES['refDataService'].
Returns:
obj: refDataService object.
"""
# return the session and request because requests need session.send()
global SERVICES
if 'session' not in SERVICES:
start_session()
session = SERVICES['session']
# Check for SERVICES['refdata'] not needed because start_session()
# is called by this function and start_session() is never called on its own.
session.openService("//blp/refdata")
refDataService = session.getService("//blp/refdata")
SERVICES['refDataService'] = refDataService
session = SERVICES['session']
refDataService = SERVICES['refDataService']
print('get_refDataService called. Curious when this is called.')
return session, refDataService
# sample override request
def ytw_oride_muni(cusip_dict):
"""Requests the Price To Worst for a dict of cusips and Yield To Worst values.
The dict must be {'cusip' : YTW}. Overrides apply to each request, so this
function is designed for different overrides for each cusip. Although they could
all be the same as that is not a restriction.
Returns: Single level nested dict
{'cusip Muni': {'ID_BB_SEC_NUM_DES': 'val', 'PX_ASK': 'val1', 'YLD_CNV_ASK': 'val2'}}
"""
session, refDataService = get_refDataService()
fields1 = ["ID_BB_SEC_NUM_DES", "PX_ASK", "YLD_CNV_ASK"]
try:
values_dict = {}
# For different overrides you must send separate requests.
# This loops and creates separate messages.
for cusip, value in cusip_dict.items():
request = refDataService.createRequest("ReferenceDataRequest")
# append security to request
request.getElement("securities").appendValue(f"{cusip} Muni")
# append fields to request
request.getElement("fields").appendValue(fields1[0])
request.getElement("fields").appendValue(fields1[1])
request.getElement("fields").appendValue(fields1[2])
# add overrides
overrides = request.getElement("overrides")
override1 = overrides.appendElement()
override1.setElement("fieldId", "YLD_CNV_ASK")
override1.setElement("value", f"{value}")
session.sendRequest(request)
# Process received events
while(True):
# We provide timeout to give the chance to Ctrl+C handling:
ev = session.nextEvent(500)
# below msg.messageType == ReferenceDataResponse
for msg in ev:
if msg.messageType() == "ReferenceDataResponse":
if msg.hasElement("responseError"):
print(msg)
if msg.hasElement("securityData"):
data = msg.getElement("securityData")
num_cusips = data.numValues()
for i in range(num_cusips):
sec = data.getValue(i).getElement("security").getValue()
try:
des = data.getValue(i).getElement("fieldData").getElement("ID_BB_SEC_NUM_DES").getValue()
except:
des = None
try:
ptw = data.getValue(i).getElement("fieldData").getElement("PX_ASK").getValue()
except:
ptw = None
try:
ytw = data.getValue(i).getElement("fieldData").getElement("YLD_CNV_ASK").getValue()
except:
ytw = None
values = {'des': des, 'ptw': ptw, 'ytw': ytw}
# Response completly received, so we could exit
if ev.eventType() == blpapi.Event.RESPONSE:
values_dict.update({sec: values})
break
finally:
# Stop the session
# session.stop()
return values_dict
I am trying to setup push notifications using the django-push-notifications, I have copied most of the code from the example. I have a button that calls the enablePushNotifications() function:
export function enablePushNotifications() {
console.log("enabling push notifications...");
if (!'serviceWorker' in navigator){
console.log("Push notifications not supported");
return;
}
navigator.serviceWorker.ready.then(
(registration)=>{
registration.pushManager.subscribe({
userVisibleOnly: true
}).then(
(sub)=>{
console.log(sub);
let endpointParts = sub.endpoint.split("/");
let registration_id = endpointParts[endpointParts.length - 1];
let data = {
'browser': loadVersionBrowser(navigator.userAgent).name.toUpperCase(),
'p256dh': btoa(String.fromCharCode.apply(null, new Uint8Array(sub.getKey('p256dh')))),
'auth': btoa(String.fromCharCode.apply(null, new Uint8Array(sub.getKey('auth')))),
'registration_id': registration_id,
'endpoint': sub.endpoint,
};
request(
"/api/subscribe",
data
)
}
).catch(
(error)=>console.error(error)
)
}
)
}
When I open the networking tab I don't see anything strange about the passed properties or headers. Specifically the browser value is correct. The implementation for the /api/subscribe URL is as follows:
import push_notifications.models as pushmodels
def registerPush(request):
data = requestFormatting.get_data(
request
)
device = pushmodels.WebPushDevice(
name=request.user.username,
active=True,
user=request.user,
registration_id=data["registration_id"],
p256dh=data["p256dh"],
auth=data["auth"],
browser=data["browser"]
)
device.save()
device.send_message("hello")
Calling this function raises a PushError: Push failed: 401 Unauthorized. I have tested on both chrome and firefox, both give the same error. I specify my FCM_API_KEY in settings, nothing else. I do not currently use VAPID, though I plan to in the future. I have tried various variations of the code, but nothing seems to work. The documentation is unclear on how to actually initialize devices, and I only picked a WebPushDevice objects since it seems to contain similar attributes as the data provided in the example.
This is my first time trying to use web push, so a little help would be much appreciated!
I am using the new Twilio Programmable Voice SDK with Swift and Python. I have begun with the respective quick starter projects, and for the most part things work. I'm able to get a valid access token, I'm able to successfully create a call, and I am even able to receive the call. The problem is on the caller side of the house.
When I try to make a call via the Swift SDK the call is disconnected before it ever starts to ring on the other end.
I have read in the Twilio docs that the client.calls.create function will immediately return a status of complete if you do not handle the status_callback event. I have tried to add this but whenever I do I get an error saying the the key status_callback is not a valid parameter for the client.calls.create function. Also, I cannot find any examples anywhere of actually how to handle the call status.
My question is what in the wold am I doing wrong here? Any help would be greatly appreciated.
Here is my Python code
#app.route('/outgoing', methods=['GET', 'POST'])
def outgoing():
account_sid = os.environ.get("ACCOUNT_SID", ACCOUNT_SID)
api_key = os.environ.get("API_KEY", API_KEY)
api_key_secret = os.environ.get("API_KEY_SECRET", API_KEY_SECRET)
from_number = [HARD_CODED_PHONE_NUMBER_FOR_CALLER]
to_number = [HARD_CODED_PHONE_NUMBER_FOR_RECIPIENT]
client = Client(api_key, api_key_secret, account_sid)
call = client.calls.create(url='http://twimlets.com/holdmusic?Bucket=com.twilio.music.ambient', to=to_number, from_=from_number)
# return str(call.sid)
resp = twilio.twiml.Response()
resp.say("Thank you for calling");
return str(resp)
Here is my relevant iOS code. Please bear in mind that this is NOT my full source. I have only provided what should be necessary in this event. My full source does include handling the registry and invite delegates. I also did not include my source that shows/hides my active call UI as there are no problems with that. This is simply to show how I am placing a call and receiving the call complete delegate.
class VoiceManager: NSObject, PKPushRegistryDelegate, TVONotificationDelegate, TVOCallDelegate, AVAudioPlayerDelegate {
//MARK: - Singleton
static let sharedManager = VoiceManager()
//MARK: - Private Constants
private let baseURLString = [MY_WEBAPP_ENDPOINT]
private let accessTokenEndpoint = "/accessToken"
//MARK: - Private Variables
private var deviceTokenString:String?
private var callInvite: TVOCallInvite?
private var call: TVOCall?
private var status: VoiceStatus = .idle
//MARK: - Getters
private func fetchAccessToken() -> String? {
guard let accessTokenURL = URL(string: baseURLString + accessTokenEndpoint) else {
return nil
}
return try? String.init(contentsOf: accessTokenURL, encoding: .utf8)
}
func placeCall(withParameters params: VoiceParameters, completion: #escaping (_ success: Bool, _ error: VAError?) -> Void) {
if (call != nil) {
call?.disconnect()
completion(false, .phoneCallInProgress)
status = .callEnded
hideActiveCallUI()
} else {
guard let accessToken = fetchAccessToken() else {
completion(false, .phoneAccessTokenFetchFailed)
return
}
guard let paramsDict = params.toDictionary() else {
completion(false, .phoneAccessTokenFetchFailed)
return
}
playOutgoingRingtone(completion: { [weak self] in
if let strongSelf = self {
strongSelf.call = VoiceClient.sharedInstance().call(accessToken, params: [:], delegate: strongSelf) //NOTE: The params here are not necessary as the phone numbers for now are hard coded on the server
if (strongSelf.call == nil) {
strongSelf.status = .callEnded
completion(false, .phoneCallFailed)
return
} else {
strongSelf.status = .callConnecting
self?.showActiveCallUI(withParameters: params)
completion(true, nil)
}
}
})
}
}
// MARK: TVOCallDelegate
func callDidConnect(_ call: TVOCall) {
NSLog("callDidConnect:")
self.call = call
status = .inCall
routeAudioToSpeaker()
}
func callDidDisconnect(_ call: TVOCall) {
NSLog("callDidDisconnect:")
playDisconnectSound()
self.call = nil
status = .callEnded
hideActiveCallUI()
}
func call(_ call: TVOCall, didFailWithError error: Error) {
NSLog("call:didFailWithError: \(error)");
self.call = nil
status = .callEnded
hideActiveCallUI()
}
}
Twilio developer evangelist here.
Your Swift code says that your phone numbers are hard coded on the server right now. The issue, as Robert has said too, is that you are using the REST API to generate a call when you get a callback from Twilio to your /outbound endpoint.
What is actually happening is that when you generate the call in Swift on the device that starts the call for the application. Twilio then makes an HTTP request to your /outbound endpoint to see what to do with that call. So, instead of generating a new call with the REST API, you need to respond with TwiML to tell Twilio what to do with the call next.
In this case, it sounds like you are trying to dial straight onto another number. For that, you should try the following response:
#app.route('/outgoing', methods=['GET', 'POST'])
def outgoing():
from_number = [HARD_CODED_PHONE_NUMBER_FOR_CALLER]
to_number = [HARD_CODED_PHONE_NUMBER_FOR_RECIPIENT]
resp = twilio.twiml.Response()
with resp.dial(callerId=from_number) as r:
r.number(to_number)
return str(resp)
Let me know if that helps.
Note: I have also responded in the ticket you created with Twilio Support.
Please check your account debugger for the whole pile of error notifications you are getting. Here's an example:
An attempt to retrieve content from
https://voiceapp-twilio.herokuapp.com/outgoing
https://voiceapp-twilio.herokuapp.com/outgoing returned the HTTP
status code 500.
The Python web server returned an error message, which includes this:
TypeError: create() got an unexpected keyword argument
'status_events' // Werkzeug Debugger
It looks like a code error in your Python outgoing() function. Most notably, you are attempting to use the REST API to create a new call, when you should actually be returning TwiML. You should be returning TwiML that includes the Dial verb to create the outgoing call leg.
I am creating a small application to test how GAE Channel API works. I think I have done all as it's described in the documentation but when I launch it, it shows an error in FireFox error log about syntax in the beginning and then another repeating error that an element wasn't found that.
Here is the first error info:
Source: http://127.0.0.1:8080/_ah/channel/dev?command=connect&channel=channel-773698929-185804764220139124118
Line 1, symbol 1
Here is the url where my javascript code tries to connect repeatedly and it raises the second error:
http://127.0.0.1:8080/_ah/channel/dev?command=poll&channel=channel-2071442473-185804764220139124118&client=1
I get the token through a JSON request with jQuery $.get. Then I run this code to get the token and open the channel. The error begins to show just when I run socket = channel.open(handler):
var response = JSON.parse(data);
var token = response.token.toString();
channel = new goog.appengine.Channel(token);
var handler = {
'onopen': onOpened,
'onmessage': onMessage,
'onerror': function() {
},
'onclose': function() {
}
};
socket = channel.open(handler);
Here is the server side code in Python to open the channel:
class OpenChannel(webapp.RequestHandler):
def get(self):
user = users.get_current_user()
token = channel.create_channel(user.user_id())
serialized = json.dumps({'token': token})
self.response.headers['Content-Type'] = "application/json"
self.response.out.write(serialized)
What's my error and what can I do? Thanks!
It seems that Channel API works on localhost different way than on GAE hosting. I uploaded it to the cloud and it works well now. Though it looks like it working fine on the local computer, it shows permanent JS error repeating in the error log.
You could try removing the handler argument and adding the handlers as methods of the socket object i.e. socket.onopen = function() {}; etc. That worked for me. But you are right. According to this, you should be able to get this working by using the handler argument. Hmm.