Use mitmdump to capture EDP-RT content

Introduction

In the previous article: Use Fiddler to Capture EDP-RT Content, we introduced a Fiddler tool which is a web debugger proxy server application and can be used to capture HTTP(S) and Web Socket traffic. Although Fiddler is easy to use, it may not be a suitable tool in some scenarios. For example, Fiddler is a UI application so it requires a lot of resources to run. Moreover, after starting Fiddler, the program registers itself as the system proxy for Microsoft Windows Internet Services (WinInet), the HTTP layer used by Internet Explorer, Microsoft Office, and many other products. Therefore, in addition to traffic from EDP-RT, developers also see traffics from other products, such as web browsers, captured by Fiddler. Furthermore, this may cause unexpected behaviors in other products, such as VPN connections. 

 For this reason, in this article, we introduce mitmdump to capture HTTP(S) and WebSocket traffics. mitmdump is a command line tool so it requires less memory than Fiddler. Moreover, mitmdump doesn't interfere with the Windows system proxy so it will not affect other products. 

mitmdump

mitmdump (Man In The Middle Dump) is a command-line tool in the mitmproxy package. It is a command line version of mitmproxy which is a free and open source interactive HTTPS proxy. It can be run on various operating systems including Windows, Linux, and macOS. Moreover, it can be run as a Docker image. To install mitmproxy, please refer to the mitmproxy official website. In this article, we only focus on the mitmdump tool on Windows. 

Certificate Files

When the first time mitmdump is run, the certificate files will be created in the config directory (~/.mitmproxy or %USERPROFILE%/.mitmproxy by default). The files created by mitmdump in the .mitmproxy directory are as follows:

mitmproxy-ca.pem  The certificate and the private key in PEM format.
mitmproxy-ca-cert.pem  The certificate in PEM format. Use this to distribute on most non-Windows platforms.
mitmproxy-ca-cert.p12  The certificate in PKCS12 format. For use on Windows.
mitmproxy-ca-cert.cer  Same file as .pem, but with an extension expected by some Android devices.

By default, the mitmdump uses this certificate to intercept encrypted connections. Moreover, these certificate files are required by the EDP-RT applications when connecting through the mitmdump as a proxy.  The examples for connecting to EDP-RT via a proxy server are available in the GitHub

  • To run the Python example, the mitmproxy-ca-cert.pem file must be specified in the --cert_file argument
  • To run the CSharp example, double click to the mitmproxy-ca-cert.p12 file to import the certificate to the Trusted Root Certification Authorities
  • To run the Java example, use the keytool command to import the mitmproxy-ca-cert.pem  file to the Java Key Store file (cacerts) which is in the JAVA_HOME\jre\lib\security\cacerts directory

     
    C:\Program Files\Java\jdk1.8.0_131\jre\lib\security>keytool -import -alias mitmproxy -file mitmproxy-ca-cert.pem -keystore cacerts  -storepass changeit

     

Then, the examples can be run with --proxy_hostname and --proxy_port arguments to connect to the mitmdump as a proxy.

 

Usages

This section lists usages of the mitmdump command from simpler to more complex. 

  1. Run the mitmdump without any arguments

    The mitmdump will be run with the default values. It will run as a regular proxy server listening at http://*:8080. It uses certificate files from the config directory (~/.mitmproxy or %USERPROFILE%/.mitmproxy). After running, it will display the full request URL with response status code for client connections.

     
    C:\Program Files (x86)\mitmproxy\bin>mitmdump
    Proxy server listening at http://*:8080
    127.0.0.1:45223: clientconnect
    127.0.0.1:45223: POST https://api.edp.thomsonreuters.com/auth/oauth2/beta1/token
    
                  << 200 OK 2.56k
    127.0.0.1:45225: clientconnect
    127.0.0.1:45225: GET https://api.edp.thomsonreuters.com/streaming/pricing/v1/?transport=websocket
                  << 200 OK 708b
    127.0.0.1:45227: clientconnect
    127.0.0.1:45227: GET https://amer-3.pricing.streaming.edp.thomsonreuters.com/WebSocket
                  << 101 Switching Protocols 0b
    127.0.0.1:45227 -> WebSocket 1 message -> amer-3.pricing.streaming.edp.thomsonreuters.com:443/WebSocket
    127.0.0.1:45227 <- WebSocket 1 message <- amer-3.pricing.streaming.edp.thomsonreuters.com:443/WebSocket
    127.0.0.1:45227 -> WebSocket 1 message -> amer-3.pricing.streaming.edp.thomsonreuters.com:443/WebSocket
    127.0.0.1:45227 <- WebSocket 1 message <- amer-3.pricing.streaming.edp.thomsonreuters.com:443/WebSocket
    ...
  2. Run with  the --list-port, or -p argument to change the listening port 

    By default, mitmdump is listening on TCP 8080. This TCP port can be changed via the --list-port or -p argument. 

     
    C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe -p 8888
    Proxy server listening at http://*:8888

    With the above command, mitmdump is listening on TCP 8888 so the application needs to use that port to connect to mitmdump.

     

  3. Run with the --flow-detail argument to change the detail log level

    The default log level of mitmdump is 1 which displays the full request URL with response status code. This can be changed via the --flow-detail argument. The maximum level is 3 (very verbose) which displays the full request URL with response status code, HTTP headers, full response content, the content of WebSocket and TCP messages. 

     
    C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe --flow-detail 3
    Proxy server listening at http://*:8080
    127.0.0.1:15129: clientconnect
    127.0.0.1:15129: POST https://api.edp.thomsonreuters.com/auth/oauth2/beta1/token
    
        Host: api.edp.thomsonreuters.com
        User-Agent: python-requests/2.18.4
        Accept-Encoding: gzip, deflate
        Accept: application/json
        Connection: keep-alive
        Content-Length: 127
        Content-Type: application/x-www-form-urlencoded
    ...
        grant_type:                 password
        takeExclusiveSignOnControl: True
        scope:                      trapi
    
     << 200 OK 2.36k
        Date: Fri, 08 Feb 2019 07:43:58 GMT
        Content-Type: application/json
        Transfer-Encoding: chunked
        Connection: keep-alive
        Content-Encoding: gzip
    ..
    
        {
            "access_token": "..."
    ...
    127.0.0.1:15133 <- WebSocket 1 message <- amer-3.pricing.streaming.edp.thomsonre
    uters.com:443/WebSocket
    
        [{"Type":"Ping"}]
    
    127.0.0.1:15133 -> WebSocket 1 message -> amer-3.pricing.streaming.edp.thomsonre
    uters.com:443/WebSocket
    
        {"Type": "Pong"}
    ...

     

  4. Run with the --save-stream-file or -w argument and stream_websocket option to save stream flows to a file

    By default, the log is displayed on the console. To save the content flow both HTTP(S) and WebSocket to a file, the following command can be used:
    C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe --w content.log --flow-detail 3 --set stream_websocket=true

    The content in the file is encoded. To view the content, the following command can be used:

    C:\Program Files (x86)\mitmproxy\bin>mitmdump.exe -r content.log --flow-detail 3
  5. Pipe the mitmdump's output to another program to add timestamps to the output

    The output from mitmdump doesn't have timestamps so to get timestamps, the output can be piped to another application to add timestamps to the output. The following is a Python 3 sample script which adds timestamps to the mitmdump's output and optionally creates a log file with timestamps.
    import sys
    import re
    from datetime import datetime
    
    fileObject = None
    try:
       if len(sys.argv) == 2:
          fileObject=open(sys.argv[1],"w") 
       for line in sys.stdin:
          if line.strip() != "":
            if re.match("^(?:[0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]*", line.strip()):
                line = "\n"+datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3] + ": " + line        
            print(line)
            if fileObject:
                fileObject.write(line)    
                fileObject.flush()
    except:
       if fileObject:
          fileObject.close()
    

    The output from mitmdump must be piped to this script.

    C:\Program Files (x86)\mitmproxy\bin>mitmdump --flow-detail 3 | python LogWithTime.py log.txt

    The content is the log.txt file is:

    Proxy server listening at http://*:8080
    
    2019-02-08 17:15:51.477: 127.0.0.1:58603: clientconnect
    
    2019-02-08 17:15:54.601: 127.0.0.1:58603: POST https://api.edp.thomsonreuters.com/auth/oauth2/beta1/token
        Host: api.edp.thomsonreuters.com
        User-Agent: python-requests/2.18.4
        Accept-Encoding: gzip, deflate
        Accept: application/json
        Connection: keep-alive
        Content-Length: 127
    ...
        grant_type:                 password
        takeExclusiveSignOnControl: True
        scope:                      trapi
     << 200 OK 2.57k
        Date: Fri, 08 Feb 2019 10:15:54 GMT
        Content-Type: application/json
        Transfer-Encoding: chunked
        Connection: keep-alive
        Content-Encoding: gzip
        X-Amzn-Trace-Id: Root=1-5c5d56d9-a15a8ee094b8d03e69a1a590
        X-Tr-Requestid: 309479a4-2892-4b59-ab54-6403e8c2856a
        {
    ...
    2019-02-08 17:16:18.618: 127.0.0.1:58607 <- WebSocket 1 message <- amer-3.pricing.streaming.edp.thomsonreuters.com:443/WebSocket
        [{"Type":"Ping"}]
    
    2019-02-08 17:16:18.622: 127.0.0.1:58607 -> WebSocket 1 message -> amer-3.pricing.streaming.edp.thomsonreuters.com:443/WebSocket
        {"Type": "Pong"}

Summary

mitmdump is a useful tool which can be used as a web proxy to capture content from REST API and web socket. Therefore, it can be used to capture content from EDP.  Mitmdump is a console application which is easy to use and supports multiple platforms including Windows, Linux, macOs, and Docker. This article shows how to use mitmdump to capture content from EDP-RT. The application needs to use mitmdurmp as a proxy when connecting to EDP-RT. The examples for connecting to EDP-RT via a proxy server are available in the GitHub. The output from mitmdump can be used to investigate issues, such an invalid credential, incorrect content, and disconnection.  

References

  1. Fiddler Proxy
  2. mitmproxy
  3. mitmproxy certificates
  4. Article.EDPRT.WebSocket.MarketPriceEdpGwExamplesWithProxy
  5. Use Fiddler to Capture EDP-RT Content