Article

Use mitmdump to Capture Refinitiv Real-Time - Optimized Content

Jirapongse Phuriphanvichai
Developer Advocate Developer Advocate

Use mitmdump to capture Refinitiv Real-Time - Optimized content

Introduction

In the previous article: Use Fiddler to Capture Refinitiv Real-Time - Optimized Content, we introduced a Fiddler tool which is a web debugger proxy server application and can be used to capture HTTP(S) and Web Socket traffic. Although Fiddler is easy to use, it may not be a suitable tool in some scenarios. For example, Fiddler is a UI application so it requires a lot of resources to run. Moreover, after starting Fiddler, the program registers itself as the system proxy for Microsoft Windows Internet Services (WinInet), the HTTP layer used by Internet Explorer, Microsoft Office, and many other products. Therefore, in addition to traffic from the Refinitiv Real-Time Optimized, developers also see traffics from other products, such as web browsers, captured by Fiddler. Furthermore, this may cause unexpected behaviors in other products, such as VPN connections. 

 For this reason, in this article, we introduce mitmdump to capture HTTP(S) and WebSocket traffics. mitmdump is a command line tool so it requires less memory than Fiddler. Moreover, mitmdump doesn't interfere with the Windows system proxy so it will not affect other products. 

mitmdump

mitmdump (Man In The Middle Dump) is a command-line tool in the mitmproxy package. It is a command line version of mitmproxy which is a free and open source interactive HTTPS proxy. It can be run on various operating systems including Windows, Linux, and macOS. Moreover, it can be run as a Docker image. To install mitmproxy, please refer to the mitmproxy official website. In this article, we only focus on the mitmdump tool on Windows. 

Certificate Files

When the first time mitmdump is run, the certificate files will be created in the config directory (~/.mitmproxy or %USERPROFILE%/.mitmproxy by default). The files created by mitmdump in the .mitmproxy directory are as follows:

mitmproxy-ca.pem  The certificate and the private key in PEM format.
mitmproxy-ca-cert.pem  The certificate in PEM format. Use this to distribute on most non-Windows platforms.
mitmproxy-ca-cert.p12  The certificate in PKCS12 format. For use on Windows.
mitmproxy-ca-cert.cer  Same file as .pem, but with an extension expected by some Android devices.

By default, the mitmdump uses this certificate to intercept encrypted connections. Moreover, these certificate files are required by the Refinitiv Real-Time Optimized applications when connecting through the mitmdump as a proxy.  The examples for connecting to the Refinitiv Real-Time Optimized via a proxy server are available in GitHub

  • To run the Python example, the mitmproxy-ca-cert.pem file must be specified in the --cert_file argument
  • To run the CSharp example, double click to the mitmproxy-ca-cert.p12 file to import the certificate to the Trusted Root Certification Authorities
  • To run the Java example, use the keytool command to import the mitmproxy-ca-cert.pem  file to the Java Key Store file (cacerts) which is in the JAVA_HOME\jre\lib\security\cacerts directory
    	
            
C:\Program Files\Java\jdk1.8.0_131\jre\lib\security>keytool -import -alias mitmproxy -file mitmproxy-ca-cert.pem -keystore cacerts  -storepass changeit

 

Then, the examples can be run with --proxy_hostname and --proxy_port arguments to connect to the mitmdump as a proxy.

 

Usages

This section lists usages of the mitmdump command from simpler to more complex. 

  1. Run the mitmdump without any arguments

    The mitmdump will be run with the default values. It will run as a regular proxy server listening at http://*:8080. It uses certificate files from the config directory (~/.mitmproxy or %USERPROFILE%/.mitmproxy). After running, it will display the full request URL with response status code for client  connections.
    	
            

C:\Program Files (x86)\mitmproxy\bin>mitmdump

Proxy server listening at http://*:8080

127.0.0.1:45223: clientconnect

127.0.0.1:45223: POST https://api.refinitiv.com/auth/oauth2/v1/token

 

              << 200 OK 2.56k

127.0.0.1:45225: clientconnect

127.0.0.1:45225: GET https://api.refinitiv.com/streaming/pricing/v1/?transport=websocket

              << 200 OK 708b

127.0.0.1:45227: clientconnect

127.0.0.1:45227: GET https://amer-3-t3.streaming-pricing-api.refinitiv.com/WebSocket

              << 101 Switching Protocols 0b

127.0.0.1:45227 -> WebSocket 1 message -> amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket

127.0.0.1:45227 <- WebSocket 1 message <- amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket

127.0.0.1:45227 -> WebSocket 1 message -> amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket

127.0.0.1:45227 <- WebSocket 1 message <- amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket

...

2.    Run with  the --list-port, or -p argument to change the listening port 

By default, mitmdump is listening on TCP 8080. This TCP port can be changed via the --list-port or -p argument. 

 

    	
            

C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe -p 8888

Proxy server listening at http://*:8888

With the above command, mitmdump is listening on TCP 8888 so the application needs to use that port to connect to mitmdump.

 

3.    Run with the --flow-detail argument to change the detail log level

The default log level of mitmdump is 1 which displays the full request URL with response status code. This can be changed via the --flow-detail argument. The maximum level is 3 (very verbose) which displays the full request URL with response status code, HTTP headers, full response content, the content of WebSocket and TCP messages. 

    	
            

C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe --flow-detail 3

Proxy server listening at http://*:8080

127.0.0.1:15129: clientconnect

127.0.0.1:15129: POST https://api.refinitiv.com/auth/oauth2/v1/token

 

    Host: api.refinitiv.com

    User-Agent: python-requests/2.18.4

    Accept-Encoding: gzip, deflate

    Accept: application/json

    Connection: keep-alive

    Content-Length: 127

    Content-Type: application/x-www-form-urlencoded

...

    grant_type:                 password

    takeExclusiveSignOnControl: True

    scope:                      trapi

 

 << 200 OK 2.36k

    Date: Fri, 08 Feb 2019 07:43:58 GMT

    Content-Type: application/json

    Transfer-Encoding: chunked

    Connection: keep-alive

    Content-Encoding: gzip

..

 

    {

        "access_token": "..."

...

127.0.0.1:15133 <- WebSocket 1 message <- amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket

 

    [{"Type":"Ping"}]

 

127.0.0.1:15133 -> WebSocket 1 message -> amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket

 

    {"Type": "Pong"}

...

4.    Run with the --save-stream-file or -w argument and stream_websocket option to save stream flows to a file

By default, the log is displayed on the console. To save the content flow both HTTP(S) and WebSocket to a file, the following command can be used:

    	
            
C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe --w content.log --flow-detail 3 --set stream_websocket=true

The content in the file is encoded. To view the content, the following command can be used:

    	
            
C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe -r content.log --flow-detail 3

5.    Pipe the mitmdump's output to another program to add timestamps to the output

The output from mitmdump doesn't have timestamps so to get timestamps, the output can be piped to another application to add timestamps to the output. The following is a Python 3 sample script which adds timestamps to the mitmdump's output and optionally creates a log file with timestamps.

    	
            

import sys

import re

from datetime import datetime

 

fileObject = None

try:

   if len(sys.argv) == 2:

      fileObject=open(sys.argv[1],"w") 

   for line in sys.stdin:

      if line.strip() != "":

        if re.match("^(?:[0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]*", line.strip()):

            line = "\n"+datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3] + ": " + line        

        print(line)

        if fileObject:

            fileObject.write(line)    

            fileObject.flush()

except:

   if fileObject:

      fileObject.close()

The output from mitmdump must be piped to this script.

    	
            
C:\Program Files (x86)\mitmproxy\bin>mitmdump --flow-detail 3 | python LogWithTime.py log.txt

The content is the log.txt file is:

    	
            

Proxy server listening at http://*:8080

 

2021-02-08 17:15:51.477: 127.0.0.1:58603: clientconnect

 

2021-02-08 17:15:54.601: 127.0.0.1:58603: POST https://api.refinitiv.com/auth/oauth2/v1/token

    Host: api.refinitiv.com

    User-Agent: python-requests/2.18.4

    Accept-Encoding: gzip, deflate

    Accept: application/json

    Connection: keep-alive

    Content-Length: 127

...

    grant_type:                 password

    takeExclusiveSignOnControl: True

    scope:                      trapi

 << 200 OK 2.57k

    Date: Fri, 08 Feb 2021 10:15:54 GMT

    Content-Type: application/json

    Transfer-Encoding: chunked

    Connection: keep-alive

    Content-Encoding: gzip

    X-Amzn-Trace-Id: Root=1-5c5d56d9-a15a8ee094b8d03e69a1a590

    X-Tr-Requestid: 309479a4-2892-4b59-ab54-6403e8c2856a

    {

...

2021-02-08 17:16:18.618: 127.0.0.1:58607 <- WebSocket 1 message <- amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket

    [{"Type":"Ping"}]

 

2021-02-08 17:16:18.622: 127.0.0.1:58607 -> WebSocket 1 message -> amer-3-t3.streaming-pricing-api.refinitiv.com:443/WebSocket

    {"Type": "Pong"}

Summary

mitmdump is a useful tool which can be used as a web proxy to capture content from REST API and web socket. Therefore, it can be used to capture content from RDP.  Mitmdump is a console application which is easy to use and supports multiple platforms including Windows, Linux, macOs, and Docker. This article shows how to use mitmdump to capture content from the Refinitiv Real-Time Optimized. The application needs to use mitmdurmp as a proxy when connecting to the Refinitiv Real-Time Optimized. The examples for connecting to the Refinitiv Real-Time Optimized via a proxy server are available in the GitHub. The output from mitmdump can be used to investigate issues, such an invalid credential, incorrect content, and disconnection.  

References

  1. Fiddler Proxy
  2. mitmproxy
  3. mitmproxy certificates
  4. Article.RDPRT.WebSocket.MarketPriceEdpGwExamplesWithProxy
  5. Use Fiddler to Capture Refinitiv Real-Time - Optimized Content