20170217

SYNPROXY To The Rescue: PFSense TCP Solution for ERR_CONNECTION_TIMED_OUT

Objective:

  • Establish TCP connection to a server in a different subnet and interface using a PFSense Firewall.
  • Allow for SSH, HTTP and HTTPS connections from a LAN client  to a FreeNAS box.
  • Keep mind intact and don't lose any sleep.

Summary:

  • If you are experiencing slow connections that never seem to complete the TCP handshake, make sure your firewall and the rules/chains are correctly configured for the type of connection you are attempting.
  • After a couple of hours (to be honest more like 4 hours or troubleshooting 1 hour of blogging), I couldn't figure out why I kept receiving TCP TIME_WAIT and CLOSE_WAIT for a simple HTTP/HTTPS connection to my FreeNAS box.
    • Simple topology
      • 2 interfaces and 2 networks.
      • Create a rule to allow for network A (LAN) to talk to network B (FreeNAS).
      • TCP connections from a LAN client to FreeNAS box.
    • Complicated results
      • Created a simple rule for LAN to FreeNAS with Any Protocol, Any Source, Any Destination, and Any Port.
      • Unable to connect to FreeNAS via HTTP/HTTPS
      • Successfully able to connect via SSH.
      • Successfully able to ping between LAN client and FreeNAS box.
      • Unable to connect to FreeNAS HTTP/HTTPs port over netcat:
        • netcat <FreeNAS IP> 443 
      • OK check the logs, seeing a lot log entries for ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, and TIME_WAIT from LAN client to FreeNAS box.
      • Check chrome and inspect the traffic, same info as the logs. Good.
    • Identified the issue, but don't understand it enough
      • At this point I can see that there is a TCP connection issue, the connection goes as far as ESTABLISHED but there is an issue once the LAN client goes into the FIN_WAIT_1 state.
      • Now the problems start, no ACK from FreeNAS box and LAN client goes into FIN_WAIT_2, again no ACK from FreeNAS box  now the LAN client goes into TIME_WAIT.
      • At this point while client is in TIME_WAIT, any other attempts to connect to the FreeNAS box are going to be blocked because now the wait time needs to expire.
        • The wait time is calculated as 2 times the Maximum Segment Lifetime
        • MSL on PFSense 30 seconds so that means 2*30 = 60 seconds
          • Use the following command to determine MSL on PFSense
            • # sysctl net.inet.tcp.msl
            • output > net.inet.tcp.msl: 30000
      • After TIME_WAIT expires, the connection goes into CLOSE_WAIT and browser reports ERR_CONNECTION_TIMED_OUT
    • Solution
      • I didn't have one at first and could not determine how to solve this problem.
      • Early in the day I recalled an Advanced Feature inside the PFSense Rules from  PFSense documentation about Asymmetric Routing and Firewall Rules.
      • There is an option to select the handling of State Type to be used for the firewall rule.
        • State types - the pfSense software offers multiple options for state handling.
          • Keep state - Works with all protocols. Default for all rules.
          • Sloppy state - Works with all protocols. Less strict state tracking, useful in cases of asymmetric routing.
          • Synproxy state - Proxies incoming TCP connections to help protect servers from spoofed TCP SYN floods. This option includes the functionality of keep state and modulate state combined.
          • None - Do not keep any state entries for this traffic. This is very rarely desirable, but is available because it can be useful under some limited circumstances.
        • The Keep state is the default set to the rule used in this project, but as far as documentation and what this state does, I found nothing useful. 
        • Fortunately Sloppy and Synproxy states are documented extensively and I was able to determine a solution now.
        • The Synproxy state handling is the solution used to sucessufully establish HTTP/HTTPS connections to the FreeNAS box from the LAN client!
          • Here is quick summary of what Synproxy does:
            • The LAN Client connects to the Synproxy server (PFSense box).
            • The Synproxy server estbalishes the TCP connection to the FreeNAS box on behalf of the LAN client via spoofing LAN client's SYN responses.
            • Synproxy server and FreeNAS server continue the handshake and once a connection is opened to the FreeNAS box, the Synproxy server hands it off to the LAN client to communicate with the FreeNAS box.
          • The downside
            • The PFSense box becomes a proxy for these TCP connections and this could have potential performance impact in much larger environments.
      • Overall, a terrific learning experience and really good knowledge for me to share to all of you. Thanks.


     Really Really Good References
    https://www.pfsense.org/about-pfsense/features.html
    https://doc.pfsense.org/index.php/Asymmetric_Routing_and_Firewall_Rules
    https://en.wikipedia.org/wiki/Maximum_segment_lifetime
    https://benohead.com/tcp-about-fin_wait_2-time_wait-and-close_wait/
    http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html
    http://rhelblog.redhat.com/tag/synproxy/
    https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY
    https://www.cyberciti.biz/tips/freebsd-lists-open-internet-unix-domain-sockets.html
    Posted by at
    Labels: , , , , , , , , , ,