So I wrote my scraper and passed a C# class (card) into IronPython, which it then happily loaded with what I think is binary image data into a byte[] like so:
imageurl = "http://blabla.com/Image.ashx?id=" + card.Id + "&type=card"
imageresult = urllib2.urlopen(imageurl).read()
if imageresult == '':
print 'Could not find image for ' + card.Title
card.AddImage(imageresult) # AddImage(byte[])
I then persist this and pull it from the database with NHibernate and tried to pull it back with this on my MVC front end:
var ms = new MemoryStream(card.Image);
var image = Image.FromStream(ms); // ***Parameter is not valid.***
Had I just written this out to a file instead of a C# object with Python, I'm fairly sure this would work. My question is, is there a good way to tell what the conversions will look like between IronPython and CLR data types? My binary sucks, I'm just not real sure what to do about it, in this case.
It looks like byte[] cannot be persisted. Check out this question:
NHibernate MappingException: no persister for byte[]
it may be that you're not getting the same message because it's happening in IronPython not in C#, due to C#'s type checking.
Here's another link a possible solution by serializing byte[] as ASCII:
http://guildsocial.web703.discountasp.net/dasblogce/2009/04/03/NHibernateMappingToBinaryData.aspx
Here's a snippet from that post:
return new ASCIIEncoding().GetString(bytes);
public static string ConvertByteArrayToString(byte[] bytes)
{
try
{
return new ASCIIEncoding().GetString(bytes);
}
catch (Exception)
{
return "";
}
}
DrNewman hit the nail basically on the head. The problem was the string format it ended up in coming out of iron python. Rather than deal with converting it to the correct format in python then brining it back with C#, I just elected to hand that part off to the C#. What I ended up doing was calling AddImage from iron python and just passing the url. Then doing the last step of the scraping (the image) with C#
Related
In my project, I am using WebRTC to connect between 2 client using the aiortc package.
I am using this example code and it works, but it seems I can't send non-string data in the data channel.
This is what I send in the data channel (modified the code in the start function in client.js file):
dc.onopen = function() {
dataChannelLog.textContent += '- open\n';
dcInterval = setInterval(function() {
let message = new DataObject(/*All the parameters*/);
dataChannelLog.textContent += '> ' + message + '\n';
dc.send(message);
}, 100);
};
Where DataObject is a class I created that contains the data I want to send.
The Python client receives [object Object] as a string. I expected it will send the bytes representing the object that I can convert back in Python to a normal class.
I know that a workaround for this is to convert the object to a string-format (like JSON), but I prefer not to do it because I am sending the objects very frequently (and every object contains a large array in it) and I am sure it will lead to a performance issues.
So my question is, how can I send the object through the data channel without converting to a string¿
EDIT: If it helps, I can use an array instead of an object to represent my data. But again, it is still sent and received as a string.
You need some sort of serializer function to convert your Javascript object into a stream of bytes. Those bytes don't have to be readable as text. You can't just send a Javascript object.
The built-in robust and secure serializer is, of course JSON.stringify(). As you've pointed out JSON is a verbose format.
To avoid using JSON, you'll need to create your own serializer in Javascript and deserializer in Python. Those will most likely be custom code for your particular object type. For best results, you'll copy the attributes of your object, one by one, into a Uint8Array, then send that.
You didn't tell us anything about your object, so it's hard to help you further.
If this were my project I'd get everything working with JSON and then do the custom serialization as a performance enhancement.
Thanks o-jones for the detailed answer.
In my case it was fairly simple, because I was able to represent all my data as an array.
The main issue I had is that I didn't know the send function has an "overload" that accepts bytes array…
After realizing that, I created a Float32Array in Javascript to hold my data and send it.
And in the Python side, I read this data and converted it to a float array using struct.unpack function.
Something like that:
Javascript side:
dc.onopen = function() {
dataChannelLog.textContent += '- open\n';
dcInterval = setInterval(function() {
let dataObj = new DataObject(/*All the parameters*/);
let data = new Float32Array(3); // Build an array (in my case 3 floats array)
// Insert the data into the array
data[0] = dataObj.value1;
data[1] = dataObj.value2;
data[2] = dataObj.value3;
// Send the data
dc.send(data);
}, 100);
};
Python side:
import struct
def message_received(message: str | bytes) -> None:
if isinstance(message, str):
return # Don't handle string messages
# Read 3 floats from the bytes array we received.
data = struct.unpack('3f', message)
I created a Python script which creates the following map (illustration):
map<uint32_t, string> tempMap = {{2,"xx"}, {200, "yy"}};
and saved it as map.out file (a binary file).
When I try to read the binary file from C++, it doesn't copy the map, why?
map<uint32_t, string> tempMap;
ifstream readFile;
std::streamsize length;
readFile.open("somePath\\map.out", ios::binary | ios::in);
if (readFile)
{
readFile.ignore( std::numeric_limits<std::streamsize>::max() );
length = readFile.gcount();
readFile.clear(); // Since ignore will have set eof.
readFile.seekg( 0, std::ios_base::beg );
readFile.read((char *)&tempMap,length);
for(auto &it: tempMap)
{
/* cout<<("%u, %s",it.first, it.second.c_str()); ->Prints map*/
}
}
readFile.close();
readFile.clear();
It's not possible to read in raw bytes and have it construct a map (or most containers, for that matter)[1]; and so you will have to write some code to perform proper serialization instead.
If the data being stored/loaded is simple, as per your example, then you can easily devise a scheme for how this might be serialized, and then write the code to load it. For example, a simple plaintext mapping can be established by writing the file with each member after a newline:
<number>
<string>
...
So for your example of:
std::map<std::uint32_t, std::string> tempMap = {{2,"xx"}, {200, "yy"}};
this could be encoded as:
2
xx
200
yy
In which case the code to deserialize this would simply read each value 1-by-1 and reconstruct the map:
// Note: untested
auto loadMap(const std::filesystem::path& path) -> std::map<std::uint32_t, std::string>
{
auto result = std::map<std::uint32_t, std::string>{};
auto file = std::ifstream{path};
while (true) {
auto key = std::uint32_t{};
auto value = std::string{};
if (!(file >> key)) { break; }
if (!std::getline(file, value)) { break; }
result[key] = std::move(value);
}
return result;
}
Note: For this to work, you need your python program to output the format that will be read from your C++ program.
If the data you are trying to read/write is sufficiently complicated, you may look into different serialization interchange formats. Since you're working between python and C++, you'll need to look into libraries that support both. For a list of recommendations, see the answers to Cross-platform and language (de)serialization
[1]
The reason you can't just read (or write) the whole container as bytes and have it work is because data in containers isn't stored inline. Writing the raw bytes out won't produce something like 2 xx\n200 yy\n automatically for you. Instead, you'll be writing the raw addresses of pointers to indirect data structures such as the map's internal node objects.
For example, a hypothetical map implementation might contain a node like:
template <typename Key, typename Value>
struct map_node
{
Key key;
Value value;
map_node* left;
map_node* right;
};
(The real map implementation is much more complicated than this, but this is a simplified representation)
If map<Key,Value> contains a map_node<Key,Value> member, then writing this out in binary will write the binary representation of key, value, left, and right -- the latter of which are pointers. The same is true with any container that uses indirection of any kind; the addresses will fundamentally differ between the time they are written and read, since they depend on the state of the program at any given time.
You can write a simple map_node to test this, and just print out the bytes to see what it produces; the pointer will be serialized as well. Behaviorally, this is the exact same as what you are trying to do with a map and reading from a binary file. See the below example which includes different addresses.
Live Example
You can use protocol buffers to serialize your map in python and deserialization can be performed in C++.
Protocol buffers supports both Python and C++.
I have obtained the OHLC values from the iqoption and trying to find out a way to use it with MT5.
Here is how I got the values:
import time
from iqoptionapi.stable_api import IQ_Option
I_want_money=IQ_Option("email","password")
goal="EURUSD"
print("get candles")
print(I_want_money.get_candles(goal,60,111,time.time()))
The above code library is here: iqoptionapi
The line: I_want_money.get_candles(goal,60,111,time.time()) output json as : Output of the command
Now I am getting json in the output so it work like an API, I guess so.
Meanwhile, I try to create a Custom Symbol in MT5 as iqoption. Now I just wanted to add the data of the OHLC from the API to it, so that it will continue fetching data from the Iqoption and display the chart on the chart window for the custom symbol iqoption.
But I am not able to load it in the custom symbol. Kindly, help me.
Edited
This is the code for live streaming data from the iqoption:
from iqoptionapi.stable_api import IQ_Option
import logging
import time
logging.basicConfig(level=logging.DEBUG,format='%(asctime)s %(message)s')
I_want_money=IQ_Option("email","password")
I_want_money.start_candles_stream("EURUSD")
thread=I_want_money.collect_realtime_candles_thread_start("EURUSD",100)
I_want_money.start_candles_stream("USDTRY")
thread2=I_want_money.collect_realtime_candles_thread_start("USDTRY",100)
time.sleep(3)
#Do some thing
ans=I_want_money.thread_collect_realtime.items()
for k, v in ans:
print (k, v)
I_want_money.collect_realtime_candles_thread_stop(thread)
I_want_money.stop_candles_stream("EURUSD")
I_want_money.collect_realtime_candles_thread_stop(thread2)
I_want_money.stop_candles_stream("USDTRY")
Ok, you need to
1. receive the feed from the broker(I hope you succeeded)
2. write it into a file
** (both - python) **
3. read and parse it
4. add it to the history centre/marketWatch
** (both - mt5) **
So, you receive data as a string after
I_want_money.get_candles(goal,60,111,time.time())
this string might be json or json-array.
The important question is of course the path you are going to put the data. An expert in MQL45 can access only two folders (if not applying dll):
C:\Users\MY_NAME_IS_DANIEL_KNIAZ\AppData\Roaming\MetaQuotes\Terminal\MY_TERMINAL_ID_IN_HEX_FORMAT\MQL4\Files
and
C:\Users\MY_NAME_IS_DANIEL_KNIAZ\AppData\Roaming\MetaQuotes\Terminal\Common\Files
in the latter case you need to open a file with const int handle=FileOpen(,|*| FILECOMMON);
In order to parse json, you can use jason.mqh https://www.mql5.com/en/code/13663 library (there are few others) but as far as i remember it has a bug: it cannot parse array of objects correctly. In order to overcome that, I would suggest to write each tick at a separate line.
And the last, you will recieve data from your python application at random time, and write it into Common or direct folder. The MT5 robot will read it and delete. Just to avoid confusion, it could be better to guarantee that a file has a unique name. Either random (random.randint(1,1000)) or milliseconds from datetime can help.
So far, you have python code:
receivedString = I_want_money.get_candles(goal,60,111,time.time())
filePath = 'C:\Users\MY_NAME_IS_DANIEL_KNIAZ\AppData\Roaming\MetaQuotes\Terminal\MY_TERMINAL_ID_IN_HEX_FORMAT\MQL4\Files\iqoptionfeed'
fileName = os.path.join(filePath,"_"+goal+"_"+str(datetime.now())+".txt")
file = open(fileName, "w")
for string_ in receivedString:
file.write(string_)
file.close()
In case you created a thread, each time you receive an answer from the thread you write such a file.
Next, you need that data in MT5.
The easiest way is to loop over the existing files, make sure you can read them and read (or give up if you cannot) and delete after reading, then proceed with the data received.
The easiest and faster way is to use 0MQ of course, but let us do it without dll's.
In order to read the files, you need to setup a timer that can work as fast as possible, and let it go. Since you cannot make a windows app sleeping less then 15.6ms, your timer should sleep this number of time.
string path;
int OnInit()
{
EventSetMillisecondTimer(16);
path="iqoptionfeed\\*";
}
void OnDeinit(const int reason) { EventKillTimer(); }
string _fileName;
long _search_handle;
void OnTimer()
{
_search_handle=FileFindFirst(path,_fileName);
if(_search_handle!=INVALID_HANDLE)
{
do
{
ResetLastError();
FileIsExist(_fileName);
if(GetLastError()!=ERR_FILE_IS_DIRECTORY)
processFile(path+_fileName);
}
while(FileFindNext(_search_handle,_fileName));
FileFindClose(_search_handle);
}
}
this piece of code loops the folder and processes each file it managed to find.
Now reading the file (two functions) and processing the message inside it:
void processFile(const string fileName)
{
string message;
if(ReadFile(fileName,message))
processMessage(message,fileName);
}
bool ReadFile(const string fileName,string &result,const bool common=false)
{
const int handle = FileOpen(fileName,common?(FILE_COMMON|FILE_READ):FILE_READ);
if(handle==INVALID_HANDLE)
{
printf("%i - failed to find file %s (probably doesnt exist!). error=%d",__LINE__,fileName,GetLastError());
return(false);
}
Read(handle,result);
FileClose(handle);
if(!FileDelete(fileName,common?FILE_COMMON:0))
printf("%i - failed to delete file %s/%d. error=%d",__LINE__,fileName,common,GetLastError());
return(true);
}
void Read(const int handle,string &message)
{
string text="";
while(!FileIsEnding(handle) && !IsStopped())
{
text=StringConcatenate(text,FileReadString(handle),"\n");
}
//printf("%i %s - %s.",__LINE__,__FUNCTION__,text);
message=text;
}
And the last but not the least: process the obtained file.
As it was suggested above, it has a json formatted tick for each new tick, separated by \r\n.
Our goal is to add it to the symbol. In order to parse json, jason.mqh is an available solution but you can parse it manually of course.
void processMessage(const string message,const string fileName)
{
string symbolName=getSymbolFromFileName(fileName);
if(!SymbolSelect(symbolName,true))
{
if(!CustomSymbolCreate(symbolName))
return;
}
string lines[];
int size=StringSplit(message,(ushort)'\n',lines);
for(int i=0;i<size;i++)
{
if(StringLen(lines[i])==0)
continue;
CJAVal jLine(jtUNDEF,NULL);
jLine.Deserialize(lines[i]);
MqlTick mql;
//here I assume that you receive a json file like " { "time":2147483647,"bid":1.16896,"ask":1.16906,"some_other_data":"someOtherDataThatYouMayAlsoUse" } "
mql.time=(datetime)jLine["time"].ToInt();
mql.bid=(double)jLine["bid"].ToDbl();
mql.ask=(double)jLine["ask"].ToDbl();
ResetLastError();
if(CustomTicksAdd(symbolName,mql)<0)
printf("%i %s - failed to upload tick: %s %s %.5f %.5f. error=%d",__LINE__,__FILE__,symbolName,TimeToString(mql.time),mql.bid,mql.ask,GetLastError());
}
}
string getSymbolFromFileName(const string fileName)
{
string elements[];
int size=StringSplit(fileName,(ushort)'_',elements);
if(size<2)
return NULL;
return elements[1];
}
Do not forget to add debugging info and request for GetLastError() is for some reason you get errors.
Can this work in a back tester? Of course not. Fist, OnTimer() is not supported in MQL tester. Next, you need some history record in order to make it running. If you do not have any history - Nobody can help you unlees a broker can give it to you; the best idea could be to start collecting and storing it right now, and when the project is ready (maybe another couple months), you will have it ready and be able to test and optimize the strategy with the available dataset. You can apply the collected set into tester (MQL5 is really the next step in algo trading development compared to MQL4), either manually or with something like tickDataSuite and its Csv2Fxt.ex4 file that makes HST binary files that the tester can read and process; anyway that is another question and nobody can tell you if your broker stores their data somewhere to provide it to you.
After second-reading what you wrote (and edited) I can see you want:
a symbol synchronized with iqoption [ through your proxy / remotely ]
The symbol could be used for backtesting
The symbol could be used for on-screen live/strategy/indicator run
That implies operations outside strategy/indicator which MT platforms do not allow in an automated manner - you can achieve it manually by providing a data package, parsing it to CSV and importing to custom symbol creator. Well documented here.
Unfortunately, you choose a platform that by-design stands for self-contained strategies and indicators, more for beginners than professionals taking it seriously.
Refer to the link I provided and see for yourself. The official doc states you can create a custom symbol via mql ref, yet even though they state, in the foreword, it allows 3rd party providers - it's not referenced anywhere else and does not show any integration possibilities.
custom indicators
custom symbol properties
I'm writing Python code for IOS automation using Appium, but get stuck on this issue:
the check point is check some cells in tableView, when changing the device location setting (the cells will be different, both number and text), if the text and image appears in cell correctly, and the image matches the text, like football icon matches "football", I try to find webdriver api to capture the snapshot for the cell only, but all the methods I found are for the driver, it just can capture the whole screen.
so we found a way to check this: first we capture the correct screenshot for the cells manually, then running the automation script, use the correct cell_screenshot to check if it appears in the screen
now my workmates has the java code for this:
import org.sikuli.api.DesktopScreenRegion;
import org.sikuli.api.ImageTarget;
import org.sikuli.api.ScreenRegion;
import org.sikuli.api.Target;
public class SnapShot {
static DesktopScreenRegion sr = new DesktopScreenRegion();
static Target image;
static ScreenRegion result;
static String fileURL;
private static String getImageURL(String fileURL){
return "screenshots/" + fileURL;
}`enter code here`
public static boolean imageExists(String fileURL, Double similar, int timeout){
timeout = timeout * 1000;
image = new ImageTarget(new File(getImageURL(fileURL)));
image.setMinScore(similar);
result = sr.wait(image, timeout);
if (result == null){
System.out.println("Can not find image");
return false;
}
return true;
}
}
this code can use imageExists to judge if the image can be found in another
I search for sikuli api for python, but cannot find anything, looks like there's java api only
now I'm stuck at this, could anyone help? Thanks a lot!
Sikuli supports Jython as well. In fact that is default the scripting language for Sikuli. So you can write similar logic in Jython and then call that script from Python(if at all you can). I have seen users using Java to call sikuli script. Refer here for Sikuli details.
I'm trying to encode an object in a Python script and set it as a cookie so I can read it with client-side JavaScript.
I've run into problems every way I've tried to do this. Generally, the cookie is formatted in a way that makes JSON.parse() break.
My current script:
cookie = Cookie.SimpleCookie()
data = {"name": "Janet", "if_nasty": "Ms. Jackson"}
cookie['test'] = json.dumps(data)
self.response.headers.add_header("Set-Cookie", cookie.output(header=''))
... which results in
test="{\"name\": \"janet\"\054 \"if_nasty\": \"Ms. Jackson\"}"
on the client.
I don't really want to introduce a hack-y solution to replace instances of commas when they appear. Any ideas how I can pass complex data structures (both by setting and reading cookies) with Python?
I also wanted to read a cookie (that had been set on the server) on the client. I worked around the issue by base64 encoding the JSON String, however there are a few small gotchas involved with this approach as well.
1: Base64 strings end with 0-2 equal signs, and these were being converted into the string \075. My approach is to revert those characters into equal characters on the client.
2: The base64 string is being enclosed in double quote characters in the cookie. I remove these on the client.
Server:
nav_json = json.dumps(nav_data)
nav_b64=base64.b64encode(nav_json)
self.response.set_cookie('nav_data', nav_b64)
Client:
var user_data_base64= $.cookie('nav_data');
// remove quotes from around the string
user_data_base64 = user_data_base64.replace(/"/g,"");
// replace \075 with =
user_data_base64 = user_data_base64.replace(/\\075/g,"=");
var user_data_encoded=$.base64.decode( user_data_base64 );
var user_data = $.parseJSON(user_data_encoded);
I am using 2 jquery plugins here:
https://github.com/carlo/jquery-base64
and
https://github.com/carhartl/jquery-cookie
Note: I consider this a hack: It would be better to re-implement the python code that encodes the cookie in javascript, however this also has the downside that you would need to notice and port and changes to that code.
I have now moved to a solution where I use a small html file to set the cookie on the client side and then redirect to the actual page requested. Here is a snippet from the JINJA2 template that I am using:
<script type="text/javascript">
var nav_data='{% autoescape false %}{{nav_data}}{% endautoescape %}';
$.cookie('nav_data', nav_data, { path: '/' });
window.location.replace("{{next}}")
</script>
Note 2: Cookies are not ideal for my use case and I will probably move on to Session or Local Storage to reduce network overhead (although my nav_data is quite small - a dozen characters or so.)
On the Python side:
json.dumps the string
escape spaces - just call .replace(' ', '%20')
Call urllib.parse.quote_plus() then write the string to the cookie
On the JavaScript side:
read the cookie
pass it through decodeURIComponent()
JSON.parse it
This seems to be the cleanest way I've found.
not sure a cookie is the best way of doing this? see the getting started guide for info rendering data to the client