I am interested in getting real-time data using the Gdax (Coinbase) WebSocket. I'm a total noob so I am inspecting the example Gdax posted in their documentation:
import gdax, time
class myWebsocketClient(gdax.WebsocketClient):
def on_open(self):
self.url = "wss://ws-feed.gdax.com/"
self.products = ["LTC-USD"]
self.message_count = 0
print("Lets count the messages!")
def on_message(self, msg):
self.message_count += 1
if 'price' in msg and 'type' in msg:
print ("Message type:", msg["type"],
"\t# {}.3f".format(float(msg["price"])))
def on_close(self):
print("-- Goodbye! --")
wsClient = myWebsocketClient()
wsClient.start()
print(wsClient.url, wsClient.products)
while (wsClient.message_count < 500):
print ("\nmessage_count =", "{} \n".format(wsClient.message_count))
time.sleep(1)
wsClient.close()
The output is:
...
Message type: received # 50.78.3f
Message type: open # 50.78.3f
Message type: done # 51.56.3f
Message type: received # 51.59.3f
Message type: open # 51.59.3f
Message type: done # 51.51.3f
Message type: done # 51.17.3f
Message type: done # 51.66.3f
Kernel died, restarting
I have a few question regarding this code and output:
What does the message type (received, open, done, match) mean, which type is used for doing calculations, and why are some types skipped?
Why does running the code always ends in 'Kernel died, restarting'?
The documentation states that this code is for illustration only. Does that mean that this isn't a proper way of getting real-time data in order to do stuff with it?
If you know some good articles or books that can teach a noob how to work with WebSockets, I would love to hear about them!
1) see the full documentation for each message here
2) Everything I find related to that issue stems from environment set up. Whether that is library dependencies not being properly installed or other environmental factors.
3) It's a proper way to set up a connection to the WebSocket but they don't provide any error handling or other logic. It's usually to cover themselves legally and reduce expectations for the code they provide (i.e. when someone receives error's similar to this they aren't liable to fix, update, help, etc)
Python interpreter (3.6.2 64bit win) crashed on close() for me too.
Here is a fix (from https://github.com/danpaquin/gdax-python/issues/152):
client.stop = True
client.thread.join()
client.ws.close()
I just added these in the on_close method and no more crashes (so far).
ps: the issue linked say that it should be fixed in the latest gdax version but the latest pip gdax (1.0.6) still crashed.
There are rather a few pitfalls to be aware of in building a full, real time order book (level 3) that I cannot document here but you may be interested to learn GDAX now offer a level 2 (almost real-time) update which will send you the updated prices for the order book. Probably much simpler to implement.
Related
In Pulsar Python Client, there is subscriber option unacked_messages_timeout_ms to set the interval after which the unacked messages will be redelivered.
What is the equivalent of that in Pulsar Go Client ?
Python
py_consumer = client.subscribe(
topic='my-topic',
subscription_name="py-subscriber",
unacked_messages_timeout_ms=10000,
consumer_type=pulsar.ConsumerType.Shared
)
Golang
go_consumer, err := client.Subscribe(
pulsar.ConsumerOptions{
Topic: "my-topic",
SubscriptionName: "go-subscriber",
Type: pulsar.Shared,
unacked_messages_timeout_ms ????
})
I could not find anything here: https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#ConsumerOptions
if it s not there, how to configure the re-delivery interval and what is the default value ?
Same question asked in Github Issues too: https://github.com/apache/pulsar-client-go/issues/608
The "unacked messages timeout" is kind of a deprecated feature that was introduced long time ago.
More recently we have added the concept of "negative acks" to provide the application an easy way to handle failure in the processing of a message.
Since the Go client was written when negative acks were already available, we decided to not introduce the deprecated feature in there.
I've been trying to get an image to post to ROS (using Python/rospy), and while I think I have the method right, I'm having a hard time confirming it. Using
rosrun image_view image_view image:=(topic)
doesn't seem to show anything. I've also tried rqtbag, but I don't really know how that thing works, other than it doesn't show things published, anyways.
A few notes before pasting my current code:
The code I use right now is based off of code I have gotten to work previously. I've used a similar setup to post text images to ROS, and those output fairly reliably.
This is slightly modified here. Most of this code is part of an on_message function, since this all runs through MQTT when implemented. (The logic is acquire image on one system -> encode it -> transfer to other system -> decode -> publish to ROS.)
I'm using Python 2.7 on Ubuntu, and ROS Indigo.
Without further ado, my current code for publishing:
rospy.init_node('BringInAnImage', log_level = rospy.INFO)
def convert(messagepayload):
t = open('newpic.bmp','w')
t.write(messagepayload)
t.close()
def on_message(client, userdata, msg):
img = base64.b64decode(msg.payload)
convert(img)
time.sleep(5)
source = cv2.imread('newpic.bmp') #this should be a mat file
# talk to ROS
bridge = CvBridge()
pub2 = rospy.Publisher('/BringInAnImage', Image, queue_size = 10)
pub2.publish(bridge.cv2_to_imgmsg(source, "bgr8"))
print "uh..... done??"
I'm using a basic listening function to try and see what is going on (this is within a different script I execute in a separate terminal):
def listener():
rospy.init_node('listener', anonymous=True)
rospy.Subscriber("/BringInAnImage", Image, callback)
rospy.spin()
if __name__ == '__main__':
listener()
The callback just prints out that the image was received.
How to check if something is published on topic xyz
To check if a message is really published, you can use the rostopic command.
Run the following in a terminal to print everything that is published on the specified topic. This is the easiest way to check if there is something published.
rostopic echo <topic_name>
See the ROS wiki for more useful things rostopic can do.
Why is the image not received by image_view?
While you are doing it basically right, your images will not be received by any subscriber for a not so obvious but fatal problem in your code: You are using the publisher (pub2) immediately after initializing it. Subscribers need some time to register to the new publisher and will not be ready before you publish the image (see also this answer).
➔ Do not initialize a publisher just before you need it but do it right in the beginning, when initializing the node.
I'm trying to use websockets with Django for small parts of my application.
Trying the first example to broadcast a message with django-websocket-redis
from ws4redis.publisher import RedisPublisher
redis_publisher = RedisPublisher(facility='foobar', broadcast=True)
redis_publisher.publish_message('Hello World')
I'm actually receiving the message into subscribed clients but I'm getting this error:
wrong number of arguments for 'set' command
[...]
Exception location my_virtualenv/local/lib/python2.7/site-packages/redis/connection.py in read_response, line 344
(traced from the publish_message() call)
My versions:
Django==1.6.2
django-websocket-redis==0.4.0
redis==2.9.1
Can someone help me to debug that ?
Looks like it's a bug.
Fix:
in ws4redis.redis_store.RedisStore in publish_message, change
self._connection.set(channel, message, ex=expire)
to
self._connection.setex(channel, expire, message)
the redis SET command does not take a 3rd argument. What I believe was meant was to set a value that expires after a number of seconds, which is the redis SETEX command. The py-redis setex method is called like setex(name, time, value).
This resolves the "Wrong number of argument for 'set'" error.
ref: https://github.com/jrief/django-websocket-redis/pull/30
I finally set the expiration time to 0 as a workaround
WS4REDIS_EXPIRE = 0
This prevents ws4redis to store anything in redis.
Fixed since 0.4.1
I am in the process of upgrading an older legacy system that is using Biztalk, MSMQs, Java, and python.
Currently, I am trying to upgrade a particular piece of the project which when complete will allow me to begin an in-place replacement of many of the legacy systems.
What I have done so far is recreate the legacy system in a newer version of Biztalk (2010) and on a machine that isn't on its last legs.
Anyway, the problem I am having is that there is a piece of Python code that picks up a message from an MSMQ and places it on another server. This code has been in place on our legacy system since 2004 and has worked since then. As far as I know, has never been changed.
Now when I rebuilt this, I started getting errors in the remote server and, after checking a few things out and eliminating many possible problems, I have established that the error occurs somewhere around the time the Python code is picking up from the MSMQ.
The error can be created using just two messages. Please note that I am using sample XMls here as the actual ones are pretty long.
Message one:
<xml>
<field1>Text 1</field1>
<field2>Text 2</field2>
</xml>
Message two:
<xml>
<field1>Text 1</field1>
</xml>
Now if I submit message one followed by message two to the MSMQ, they both appear correctly on the queue. If I then call the Python script, message one is returned correctly but message two gains extra characters.
Post-Python message two:
<xml>
<field1>Text 1</field1>
</xml>1>Te
I thought at first that there might have been scoping problems within the Python code but I have gone through that as well as I can and found none. However, I must admit that the first time that I've looked seriously at Python code is this project.
The Python code first peeks at a message and then receives it. I have been able to see the message when the script peeks and it has the same error message as when it receives.
Also, this error only shows up when going from a longer message to a shorter message.
I would welcome any suggestions of things that might be wrong, or things I could do to identify the problem.
I have googled and searched and gone a little crazy. This is holding an entire project up, as we can't begin replacing the older systems with this piece in place to act as a new bridge.
Thanks for taking the time to read through my problem.
Edit: Here's the relevant Python code:
import sys
import pythoncom
from win32com.client import gencache
msmq = gencache.EnsureModule('{D7D6E071-DCCD-11D0-AA4B-0060970DEBAE}', 0, 1, 0)
def Peek(queue):
qi = msmq.MSMQQueueInfo()
qi.PathName = queue
myq = qi.Open(msmq.constants.MQ_PEEK_ACCESS,0)
if myq.IsOpen:
# Don't loose this pythoncom.Empty thing (it took a while)
tmp = myq.Peek(pythoncom.Empty, pythoncom.Empty, 1)
myq.Close()
return tmp
The function calls this piece of code. I don't have access to the code that calls this until Monday, but the call is basically:
msg= MSMQ.peek()
2nd Edit.
I am attaching the first half of the script. this basically loops around
import base64, xmlrpclib, time
import MSMQ, Config, Logger
import XmlRpcExt,os,whrandom
QueueDetails = Config.InQueueDetails
sleeptime = Config.SleepTime
XMLRPCServer = Config.XMLRPCServer
usingBase64 = Config.base64ing
version=Config.version
verbose=Config.verbose
LogO = Logger.Logger()
def MSMQToIAMS():
# moved svr cons out of daemon loop
LogO.LogP(version)
svr = xmlrpclib.Server(XMLRPCServer, XmlRpcExt.getXmlRpcTransport())
while 1:
GotOne = 0
for qd in QueueDetails:
queue, agency, messagetype = qd
#LogO.LogD('['+version+"] Searching queue %s for messages"%queue)
try:
msg=MSMQ.Peek(queue)
except Exception,e:
LogO.LogE("Peeking at \"%s\" : %s"%(queue, e))
continue
if msg:
try:
msg = msg.__call__().encode('utf-8')
except:
LogO.LogE("Could not convert massege on \"%s\" to a string, leaving it on queue"%queue)
continue
if verbose:
print "++++++++++++++++++++++++++++++++++++++++"
print msg
print "++++++++++++++++++++++++++++++++++++++++"
LogO.LogP("Found Message on \"%s\" : \"%s...\""%(queue, msg[:40]))
try:
rv = svr.accept(msg, agency, messagetype)
if rv[0] != "OK":
raise Exception, rv[0]
LogO.LogP('Message has been sent successfully to IAMS from %s'%queue)
MSMQ.Receive(queue)
GotOne = 1
StoreMsg(msg)
except Exception, e:
LogO.LogE("%s"%e)
if GotOne == 0:
time.sleep(sleeptime)
else:
gotOne = 0
This is the full code that calls MSMQ. Creates a little program that watches MSMQ and when a message arrives picks it up and sends it off to another server.
Sounds really Python-specific (of which I know nothing) rather then MSMQ-specific. Isn't this just a case of a memory variable being used twice without being cleared in between? The second message is shorter than the first so there are characters from the first not being overwritten. What do the relevant parts of the Python code look like?
[[21st April]]
The code just shows you are populating the tmp variable with a message. What happens to tmp before the next message is accessed? I'm assuming it is not cleared.
I'm attempting to write a quick load-test script for our ejabberd cluster that simply logs into a chat room, posts a couple of random messages, then exits.
We had attempted this particular test with tsung, but according to the authors, the muc functionality did not make it into this release.
pyxmpp seems to have this functionality, but darned if I can figure out how to make it work. Here's hoping someone has a quick explanation of how to build the client and join/post to the muc.
Thanks!
Hey I stumbled over your question a few times, while trying the same thing.
Here is my answer:
Using http://pyxmpp.jajcus.net/svn/pyxmpp/trunk/examples/echobot.py as a quickstart, all you have to do is import the MUC-Stuff
from pyxmpp.jabber.muc import MucRoomState, MucRoomManager
And once your Client is connected, you can connect to your room:
def session_started(self):
"""Handle session started event. May be overriden in derived classes.
This one requests the user's roster and sends the initial presence."""
print u'SESSION STARTED'
self.request_roster()
p=Presence()
self.stream.send(p)
print u'ConnectToParty'
self.connectToMUC()
def connectToMUC(self):
self.roomManager = MucRoomManager(self.stream);
self.roomHandler = MucRoomHandler()
self.roomState = self.roomManager.join(
room=JID('room#conference.server.domain'),
nick='PartyBot',
handler=self.roomHandler,
history_maxchars=0,
password = None)
self.roomManager.set_handlers()
To send a message, all you have to do is call self.roomState.send_message("Sending this Message")
To do stuff, inherit from MucRoomHandler and react on events. Notice the "set_handlers()" to roomManager though, it's is important, otherwise callbacks will not be called..