CFDs are complex instruments and come with a high risk of losing money rapidly due to leverage. 66 % of retail investor accounts lose money when trading CFDs with this provider. You should consider whether you understand how CFDs work and whether you can afford to take the high risk of losing your money.

[ANNOUNCEMENT] Darwin Quotes API

Darwin Quotes API

Our version 1.0 of the Darwin Quotes API is finally here!
The Darwin Quotes API allows reception of real time quotes for the requested Darwins in https stream over a POST request. An API client can send a request to POST /quotes over this API for darwins and as soon as there is an update in the quote of the darwin, a message is send to the client with the timestamp of the quote update. Notice that this update is like a tick update... many updates can be sent over a minute.

Notice that this POST request is valid for almost an hour. Once this time-period finishes the stream is closed. Therefore, additional calls might be required every hour to maintain quote updates, at the moment. If there is no quote update over a product then no quote update is sent over the stream.

Example of Python client for doing one call (just replace the bearer token with your one:

from requests import Request, Session

s = Session()

def streaming(symbols):
    url = "https://api.darwinex.com/quotes/1.0/quotes"
    bearerToken = ""
    separator = "\",\""
    payload = "{\"productNames\": [ \"" + separator.join(symbols) + "\" ]}"
    headers = {'connection': 'keep-alive', 'content-type': 'application/json',
               'Authorization': 'Bearer ' + bearerToken}
    req = Request("POST",url,headers=headers,data=payload).prepare()
    resp = s.send(req, stream=True, verify=False)
    for line in resp.iter_lines():
        if line:
            yield line

def read_stream():
    for line in streaming(['DWC.4.20', 'DWF.4.19']):
        print(line)

read_stream()

All application clients and users have been granted access to this API now :wink: Let us know if you have any issues with this new method :wink:

8 Likes

Simply amazing! Thanks!

1 Like

Hi, everybody.

I don’t have much time these days but finally, I can test the
Darwin Quotes API and I do with JavaScript (Node.js).

First of all, I was curious about this mode of real-time implementation with a POST request open while chunks of data are sent, these type of “stream event handlers” I usually do it with Node.js and the easy async environment.

Things that I saw:

  • The chunks of data are usually received every second, but sometimes the latency goes up and it takes more than 7 seconds on every chunk.

  • After various consecutive requests when the previous one is closed, like 10 minutes approx, there was an error: “request failed with status code 504 Gateway Timeout.” I don’t know if there any type of limit. Later I was able to return to make requests.

  • Effectively if the Darwin doesn’t have open positions, there is no response.

  • It’s possible to make a request of several Darwins at a time, and this is cool.

The code:
I run this in Node.js 10, the only dependency is Axios (to make the requests), the output is like the cool gif.
This code makes the POST request to API quotes, print the chunks with data and every time the stream is closed it makes another POST request to still receive the chunks again.


const axios = require('axios');
const api = axios.create({
  baseURL: 'https://api.darwinex.com/quotes/1.0',
  headers: {'Authorization': 'YOUR API TOKEN HERE'}
});

async function quotes() {
  const config = {
    method: 'post',
    url: `/quotes`,
    responseType: 'stream',
    data: {
      productNames: ['DWC.4.20']
    }
  }
  return api.request(config);
}

run();
async function run () {
  try{
    console.log('Darwinex API runner: DarwinQuotesAPI');
    apiResponseToText(await quotes());
  } catch (error) {
    console.error(error);
  }
}

// DOCS for manage streams: https://nodejs.org/api/stream.html#stream_event_data
function apiResponseToText (r){
  console.log('Path: '+r.request.path);
  const stream = r.data;
  let count = 0;
  console.time('timerStreamEnd');
  // This event seems to be triggered every second 
  stream.on('data', (chunk) => {
    // chunk is an ArrayBuffer
    console.log("chunk n: "+count++);
    const chunkTimestamp = JSON.parse(chunk.toString()).timestamp;
    console.log(new Date(chunkTimestamp));
    console.log(chunk.toString());
  });
  stream.on('end', () => {
    console.log('end');
    console.timeEnd('timerStreamEnd');
    run();
  });
}

5 Likes

Thanks, zicrox! Good work!

There might be sometimes with higher load in the system which certainly could derive into some latencies. We expect to be improving this performance with the time. And certainly, we also need to finetune a bit more the times to avoid the Gateway Timeout. At the moment, this timeout is defined for 60 seconds for a request so it looks like in that case the request was not closed before the 60 seconds. Nevertheless, we expect to increase this 59/60 seconds timeouts with the time.

2 Likes

I have some concerns, in no particular order:

  • productsRequest allows any user to subscribe to all products by supplying an empty array. I can see too many users using this indiscriminately, thus hogging bandwidth. Consider requiring explicit symbols.

  • I do believe streaming is superior to ad hoc price snapshot requests, but it would be nice to have the option to use the latter, simply because not everyone knows how to stream. EDIT: Also, not everyone needs a stream either.

  • By comparison, Oanda’s streaming API provides a continuous stream without expiration. It also supplies a heartbeat every five seconds, so the cue to reconnect is when 10 seconds have passed without any messages received. I’m not sure which model is superior, but this model might be worth consideration.

1 Like

What is the recommended max number of subscribed products?
Noticed that after first successful connection to > 3000 active darwin’s i am like banned for a while

There’s throttling for API access. Maybe you got throttled instead of banned?

Yes, sure @yhlasx, “like… banned”

Another possibility is that you exceeded a rate limit. Anyways, I don’t know what you did.

For example, I want to stream 418 darwins an do:

curl -k -X POST "https://api.darwinex.com/quotes/1.0/quotes" -H "accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer 3de36b5e-c454-394d-833c-98eff53baa12" -d "{ \"productNames\": [ \"AAN.4.2\", \"ACG.4.7\", \"ACT.4.5\", \"ACY.4.17\", \"AEJ.4.6\", \"AET.4.17\", \"AEW.4.21\", \"AEY.4.13\", \"AGD.4.9\", \"AHU.4.3\", \"AJH.4.21\", \"AJY.4.7\", \"AKO.4.5\", \"ALN.4.3\", \"ANI.4.6\", \"APZ.4.6\", \"AQA.4.11\", \"AQU.4.9\", \"ARI.4.18\", \"ARL.4.1\", \"ASY.4.6\", \"ATL.4.2\", \"AVG.4.19\", \"AWD.4.21\", \"AWO.4.4\", \"AXF.4.23\", \"AXG.4.21\", \"AYR.4.7\", \"AZG.4.20\", \"BAD.4.18\", \"BAH.4.20\", \"BAX.4.5\", \"BBO.4.10\", \"BDJ.4.2\", \"BEB.4.22\", \"BGO.4.13\", \"BGR.4.7\", \"BIF.4.2\", \"BIQ.4.11\", \"BLS.4.16\", \"BMO.4.8\", \"BNE.4.4\", \"BOT.4.17\", \"BRS.4.10\", \"BSG.4.14\", \"BTI.4.8\", \"BUX.4.4\", \"BZZ.4.3\", \"CAA.4.18\", \"CAM.4.8\", \"CCD.4.6\", \"CCV.4.19\", \"CHM.4.17\", \"CIB.4.22\", \"CMZ.4.6\", \"COB.4.16\", \"COI.4.4\", \"COJ.4.2\", \"CQG.4.18\", \"CRA.4.11\", \"CRF.4.21\", \"CRJ.4.11\", \"CSB.4.4\", \"CVL.4.10\", \"CYN.4.16\", \"DBD.4.17\", \"DBK.4.8\", \"DBU.4.23\", \"DCY.4.17\", \"DDY.4.16\", \"DEK.4.6\", \"DHD.4.10\", \"DIO.4.17\", \"DIT.4.20\", \"DJA.4.18\", \"DJC.4.5\", \"DJS.4.22\", \"DJT.4.12\", \"DMB.4.22\", \"DMR.4.1\", \"DNK.4.16\", \"DNZ.4.1\", \"DRN.4.18\", \"DSR.4.11\", \"DUS.4.10\", \"DWF.4.19\", \"DYH.4.18\", \"DZQ.4.24\", \"EAZ.4.19\", \"ECB.4.3\", \"ECO.4.17\", \"EDF.4.13\", \"EDJ.4.7\", \"EFO.4.15\", \"EHH.4.2\", \"EJA.4.14\", \"EKD.4.17\", \"ELD.4.10\", \"ELR.4.7\", \"EMM.4.13\", \"ENK.4.13\", \"ENM.4.16\", \"ENT.4.19\", \"ENW.4.10\", \"EPO.4.13\", \"ETT.4.22\", \"EVX.4.16\", \"EXA.4.12\", \"EYS.4.7\", \"EZN.4.2\", \"FAU.4.10\", \"FAV.4.15\", \"FBX.4.19\", \"FCL.4.14\", \"FCQ.4.20\", \"FDW.4.19\", \"FER.4.24\", \"FET.4.22\", \"FHG.4.22\", \"FIR.4.6\", \"FIV.4.3\", \"FMT.4.7\", \"FOU.4.11\", \"FPN.4.1\", \"FVG.4.6\", \"FVV.4.24\", \"FWB.4.16\", \"FZK.4.17\", \"GDS.4.7\", \"GFA.4.21\", \"GFN.4.13\", \"GFV.4.1\", \"GNI.4.20\", \"GOB.4.17\", \"GOM.4.6\", \"GPO.4.1\", \"GQC.4.10\", \"GTJ.4.10\", \"GUJ.4.2\", \"GUS.4.15\", \"GXW.4.19\", \"GYK.4.1\", \"GZZ.4.14\", \"HBB.4.5\", \"HBD.4.20\", \"HHN.4.24\", \"HIE.4.1\", \"HKM.4.22\", \"HLX.4.16\", \"HMP.4.12\", \"HON.4.7\", \"HSM.4.24\", \"HWH.4.2\", \"HWR.4.15\", \"HXH.4.16\", \"HYD.4.15\", \"IAF.4.10\", \"ICX.4.2\", \"IDX.4.2\", \"IFE.4.23\", \"IGU.4.22\", \"IGX.4.1\", \"IIZ.4.17\", \"IJD.4.8\", \"IJI.4.15\", \"IKJ.4.5\", \"IKQ.4.1\", \"IMO.4.17\", \"ING.4.10\", \"INI.4.16\", \"IOS.4.3\", \"IPA.4.23\", \"IRP.4.12\", \"ISA.4.20\", \"ITS.4.8\", \"IYK.4.16\", \"JDD.4.12\", \"JDP.4.19\", \"JLK.4.1\", \"JME.4.17\", \"JMN.4.14\", \"JNA.4.22\", \"JNE.4.1\", \"JNI.4.11\", \"JUB.4.14\", \"JVO.4.7\", \"JXF.4.6\", \"JXN.4.13\", \"JZG.4.8\", \"JZO.4.7\", \"KCC.4.23\", \"KCG.4.8\", \"KDS.4.3\", \"KDT.4.3\", \"KGL.4.1\", \"KGX.4.1\", \"KLY.4.10\", \"KPO.4.16\", \"KTB.4.4\", \"KTE.4.5\", \"KWC.4.19\", \"KXX.4.7\", \"LDH.4.7\", \"LDZ.4.20\", \"LED.4.17\", \"LEL.4.21\", \"LIF.4.10\", \"LIV.4.20\", \"LJS.4.1\", \"LJX.4.5\", \"LOK.4.22\", \"LOM.4.24\", \"LQH.4.13\", \"LRB.4.6\", \"LSC.4.18\", \"LSH.4.2\", \"LSN.4.10\", \"LSW.4.18\", \"LUF.4.4\", \"LUG.4.21\", \"LVS.4.20\", \"LYK.4.9\", \"LYM.4.17\", \"LYS.4.4\", \"LYT.4.24\", \"LZL.4.23\", \"LZM.4.13\", \"MAW.4.11\", \"MBI.4.6\", \"MCB.4.19\", \"MDO.4.5\", \"MED.4.3\", \"MET.4.3\", \"MJO.4.18\", \"MLW.4.8\", \"MMD.4.13\", \"MOA.4.8\", \"MOI.4.6\", \"MSF.4.10\", \"MSJ.4.1\", \"MSX.4.4\", \"MVH.4.12\", \"MVP.4.12\", \"MWX.4.10\", \"MXF.4.15\", \"MYY.4.2\", \"MZS.4.7\", \"NAF.4.9\", \"NAO.4.2\", \"NAP.4.24\", \"NBR.4.22\", \"NER.4.14\", \"NFP.4.12\", \"NGK.4.6\", \"NMU.4.19\", \"NNY.4.7\", \"NRZ.4.8\", \"NSC.4.6\", \"NSY.4.9\", \"NTP.4.6\", \"NTR.4.12\", \"NUN.4.3\", \"NVS.4.13\", \"NVY.4.12\", \"NWM.4.6\", \"NXG.4.8\", \"NXN.4.7\", \"NYJ.4.4\", \"OCO.4.6\", \"OIT.4.5\", \"OJH.4.19\", \"ONE.4.1\", \"ONZ.4.20\", \"OXI.4.11\", \"OXX.4.19\", \"OYE.4.21\", \"PAG.4.1\", \"PFD.4.16\", \"PFH.4.8\", \"PFQ.4.4\", \"PFZ.4.8\", \"PGH.4.1\", \"PGL.4.15\", \"PHT.4.19\", \"PLF.4.1\", \"PLL.4.23\", \"PMZ.4.8\", \"PNB.4.4\", \"PNY.4.24\", \"PPL.4.22\", \"PPR.4.8\", \"PPW.4.12\", \"PPX.4.15\", \"PRB.4.9\", \"PRG.4.23\", \"PSM.4.8\", \"PTC.4.12\", \"PTD.4.23\", \"PUH.4.19\", \"PUL.4.6\", \"PUP.4.13\", \"PXC.4.12\", \"PZK.4.12\", \"QAN.4.10\", \"QFE.4.1\", \"QGV.4.5\", \"QJW.4.17\", \"QPT.4.10\", \"QSF.4.20\", \"QTA.4.10\", \"QTG.4.18\", \"QXW.4.20\", \"QYR.4.10\", \"RAQ.4.12\", \"RAR.4.21\", \"RAT.4.2\", \"RBQ.4.23\", \"RDK.4.6\", \"RDM.4.3\", \"RDZ.4.12\", \"RGT.4.14\", \"RGX.4.17\", \"RHA.4.11\", \"RIH.4.18\", \"RKK.4.7\", \"RMJ.4.11\", \"RQQ.4.17\", \"RST.4.10\", \"RVL.4.6\", \"RVU.4.12\", \"RXC.4.17\", \"RXO.4.24\", \"RZV.4.23\", \"RZX.4.11\", \"SAG.4.3\", \"SAJ.4.9\", \"SBR.4.15\", \"SBY.4.17\", \"SBZ.4.4\", \"SCQ.4.4\", \"SCR.4.5\", \"SCS.4.5\", \"SDI.4.24\", \"SEH.4.7\", \"SES.4.3\", \"SFO.4.14\", \"SFR.4.18\", \"SGP.4.4\", \"SGS.4.6\", \"SHC.4.13\", \"SHJ.4.6\", \"SJS.4.4\", \"SKI.4.11\", \"SKJ.4.22\", \"SMG.4.14\", \"SMH.4.6\", \"SQA.4.22\", \"SQR.4.18\", \"SRI.4.1\", \"SSU.4.11\", \"STG.4.4\", \"STW.4.6\", \"SUG.4.2\", \"SUU.4.22\", \"SUV.4.7\", \"SVN.4.22\", \"SWD.4.8\", \"SYE.4.15\", \"SYW.4.3\", \"SZB.4.23\", \"TDK.4.4\", \"TEA.4.17\", \"THK.4.19\", \"THW.4.1\", \"TIV.4.17\", \"TKR.4.4\", \"TLP.4.7\", \"TOC.4.8\", \"TOS.4.6\", \"TSB.4.5\", \"TTE.4.9\", \"TTL.4.24\", \"TUL.4.23\", \"TVU.4.21\", \"TWK.4.24\", \"UHF.4.6\", \"UKM.4.12\", \"UNU.4.15\", \"UPP.4.5\", \"USB.4.7\", \"USO.4.14\", \"UUV.4.18\", \"UYZ.4.23\", \"VCC.4.5\", \"VEG.4.15\", \"VFM.4.21\", \"VIW.4.6\", \"VNA.4.22\", \"VNB.4.1\", \"VNM.4.2\", \"VNY.4.1\", \"VOT.4.24\", \"VSA.4.19\", \"VSL.4.4\", \"VZN.4.23\", \"VZU.4.18\", \"WAJ.4.16\", \"WBX.4.16\", \"WGH.4.9\", \"WMG.4.5\", \"WNL.4.6\", \"WOI.4.2\", \"WSB.4.21\", \"WSS.4.13\", \"WWO.4.4\", \"WWT.4.14\", \"XAF.4.22\", \"XEN.4.17\" ]}"

Then I get:

{"productName":"TUL.4.23","quote":112.5675756817879,"timestamp":1547049583639}
{"productName":"EKD.4.17","quote":83.52388325357528,"timestamp":1547049583639}
{"productName":"JDD.4.12","quote":781.1113341243696,"timestamp":1547049583640}
{"productName":"PFH.4.8","quote":46.86906101667438,"timestamp":1547049583640}
{"productName":"QPT.4.10","quote":82.76281584501356,"timestamp":1547049583640}
{"productName":"FVG.4.6","quote":90.26246006009767,"timestamp":1547049583640}
{"productName":"LSC.4.18","quote":122.12946637931437,"timestamp":1547049583640}
{"productName":"ICX.4.2","quote":166.19151629463454,"timestamp":1547049583640}
{"productName":"PMZ.4.8","quote":194.23066705346113,"timestamp":1547049583640}
{"productName":"IFE.4.23","quote":117.6163593153524,"timestamp":1547049583640}
{"productName":"AQA.4.11","quote":158.85879463858558,"timestamp":1547049583640}
#...etc

A total of 15706 lines in this case, recived in 1 min.

After this successful connection I get <504 Gateway Time-out>
Then have to wait aprox. one hour, maybe minutes, maybe more to get another one

Am I doing something wrong? Is it because of throttling? Exceeded rate limit?

I think it is because there is a timeout limit on the server side about how much time the server will dedicate per query, and when it is exceeded, connection gets interrupted? To cut it short, I think it is because you are asking for quotes of too many darwins.

Edit: From googling

As mentioned, a 504 Gateway Timeout Error means that a server that is upstream to one that you (the client ) are connecting to did not receive a “timely” response from another server further along upstream.

From what I understood, the server that handles your connection requests data from someplace else where it doesn’t get a response in a timely manner (probably because your query is too big and it takes time) and thus returns you this error

I thought the same, because of that my question about a maximum recommended number of subscriptions.
That was not throttling, I got that error message when it really was.
The curl command works well but after first run “gets trapped”.

Solved adding -m 58, so it gets out by itself without wait to the server for close the connection. (I supose)

curl -m 58 -k -X POST "https:...

I have been looking into the issues you were having with these requests and it looks like there were errors with requests involving hundreds of products. There were several connections already closed between the streaming service and the API gateway and the client … and having some inconsistencies due to these errors. I have changed some parameters to improve it a bit … until we allow these streamings for more than only a minute.

At the moment the rate limit for this product quotes API is 20 requests / minute for all application client. In the case of users using direct access token at the moment, if amongst all these users reach the 20 request / minute then other clients might begin to be rejected. But … as far as I can see in the stats, this limit hasn’t been reached. The requests weren’t working due to some other internal issues like the one mentioned in the beginning.

Nevertheless, we will provide another Websocket API for subscribing / unsubscribing to product quote updates. We are still working on it. For this API I wouldn’t recommend to ask for too many product quote updates eventhough we are not limiting this number at the moment.