• Establish TCP connection to a server in a different subnet and interface using a PFSense Firewall.
  • Allow for SSH, HTTP and HTTPS connections from a LAN client  to a FreeNAS box.
  • Keep mind intact and don't lose any sleep.


  • If you are experiencing slow connections that never seem to complete the TCP handshake, make sure your firewall and the rules/chains are correctly configured for the type of connection you are attempting.
  • After a couple of hours (to be honest more like 4 hours or troubleshooting 1 hour of blogging), I couldn't figure out why I kept receiving TCP TIME_WAIT and CLOSE_WAIT for a simple HTTP/HTTPS connection to my FreeNAS box.
    • Simple topology
      • 2 interfaces and 2 networks.
      • Create a rule to allow for network A (LAN) to talk to network B (FreeNAS).
      • TCP connections from a LAN client to FreeNAS box.
    • Complicated results
      • Created a simple rule for LAN to FreeNAS with Any Protocol, Any Source, Any Destination, and Any Port.
      • Unable to connect to FreeNAS via HTTP/HTTPS
      • Successfully able to connect via SSH.
      • Successfully able to ping between LAN client and FreeNAS box.
      • Unable to connect to FreeNAS HTTP/HTTPs port over netcat:
        • netcat <FreeNAS IP> 443 
      • OK check the logs, seeing a lot log entries for ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, and TIME_WAIT from LAN client to FreeNAS box.
      • Check chrome and inspect the traffic, same info as the logs. Good.
    • Identified the issue, but don't understand it enough
      • At this point I can see that there is a TCP connection issue, the connection goes as far as ESTABLISHED but there is an issue once the LAN client goes into the FIN_WAIT_1 state.
      • Now the problems start, no ACK from FreeNAS box and LAN client goes into FIN_WAIT_2, again no ACK from FreeNAS box  now the LAN client goes into TIME_WAIT.
      • At this point while client is in TIME_WAIT, any other attempts to connect to the FreeNAS box are going to be blocked because now the wait time needs to expire.
        • The wait time is calculated as 2 times the Maximum Segment Lifetime
        • MSL on PFSense 30 seconds so that means 2*30 = 60 seconds
          • Use the following command to determine MSL on PFSense
            • # sysctl net.inet.tcp.msl
            • output > net.inet.tcp.msl: 30000
      • After TIME_WAIT expires, the connection goes into CLOSE_WAIT and browser reports ERR_CONNECTION_TIMED_OUT
    • Solution
      • I didn't have one at first and could not determine how to solve this problem.
      • Early in the day I recalled an Advanced Feature inside the PFSense Rules from  PFSense documentation about Asymmetric Routing and Firewall Rules.
      • There is an option to select the handling of State Type to be used for the firewall rule.
        • State types - the pfSense software offers multiple options for state handling.
          • Keep state - Works with all protocols. Default for all rules.
          • Sloppy state - Works with all protocols. Less strict state tracking, useful in cases of asymmetric routing.
          • Synproxy state - Proxies incoming TCP connections to help protect servers from spoofed TCP SYN floods. This option includes the functionality of keep state and modulate state combined.
          • None - Do not keep any state entries for this traffic. This is very rarely desirable, but is available because it can be useful under some limited circumstances.
        • The Keep state is the default set to the rule used in this project, but as far as documentation and what this state does, I found nothing useful. 
        • Fortunately Sloppy and Synproxy states are documented extensively and I was able to determine a solution now.
        • The Synproxy state handling is the solution used to sucessufully establish HTTP/HTTPS connections to the FreeNAS box from the LAN client!
          • Here is quick summary of what Synproxy does:
            • The LAN Client connects to the Synproxy server (PFSense box).
            • The Synproxy server estbalishes the TCP connection to the FreeNAS box on behalf of the LAN client via spoofing LAN client's SYN responses.
            • Synproxy server and FreeNAS server continue the handshake and once a connection is opened to the FreeNAS box, the Synproxy server hands it off to the LAN client to communicate with the FreeNAS box.
          • The downside
            • The PFSense box becomes a proxy for these TCP connections and this could have potential performance impact in much larger environments.
      • Overall, a terrific learning experience and really good knowledge for me to share to all of you. Thanks.

    Really Really Good References


    10GbE LAN: FreeNAS, VMware ESXi, Windows 10 and $100 Budget

    1. Create a 10Gb Ethernet LAN for the purpose of...I have no real purpose.
    2. Spend no more than $100.
    3. Only 3 computers will be on 10GbE but can only use 1 PCIe x16 port (version 2 and 3).
    4. Use fiber to connect to 1 server not collocated with other 2 systems.
    5. Dual port NIC supported by FreeNAS, VMware ESXi, and Windows 10.
    6. NIC must be able to support a variety of SFP+ Multi Mode and Single Mode optics.
    7. Nice to have: MacOS and Linux support over thunderbolt adapter(future projects).
    8. Ideally would like to have NICs with well documented support and use.
    • After a week or two of ebay searches and research on the internet, I came across a few NICs 
      • Mellanox ConnectX (affordable, but not supported FreeNAS)
      • Chelsio T Series (pricey, 100% compatible)
      • Intel X520 (expensive, unable to determine FreeNAS support) 
      • SolarFlare (best price, 100% compatible)
    • Linux and Windows support for all NICs listed above were highly likely but the FreeNAS support was the issue
      • https://www.freebsd.org/relnotes/9-STABLE/hardware/support.html#ethernet
    • Chelsio is the most common choice for FreeNAS builds, it is listed in the hardware requirements and mentioned a lot in the FreeNAS forums.
    • Decided to go with Solarflare:
      • Solarflare is not mentioned very much in FreeNAS forums and I was almost hesitant to use these NICs because of the lack of documented use I was able to come across. The deciding factor was a youtube video and blog I came across by Scott Schweitzer
      • I was immediately sold after looking at the list of systems with driver support  for Solarflare NICs.
        • Systems Supported
          • Apple Mac OS X
          • FreeBSD
          • Linux
          • Solaris
          • VMware
          • Windows
          • XenServer
      • Another plus is that HP uses Solarflare adapters.
      • Potential Prices of used Solarflare adapters
        • Here are the different series of adapters available
          • SFN5000 (Best Price, 100% Compatible) Around $30-$50
          • SFN6000 (Good Price, 100% Compatible) Around $40-$80
          • SFN7000 (Over budget) Over $100
          • SFN8000 (Over budget) Over $100
      • Purchased three SFN5000 series dual 10GbE NIC for a great price of $80 USD total! 
        • This purchase came with four SFP+ MM Optics.
        • Tested and updated all NICs with the latest firmware
      • Last purchase was for two Cisco Twinax SFP+ DAC cable for $20 USD.
    • So with some luck and lots of research, I was able to deploy a 10GbE LAN for $100 USD.  Enjoy!