Hi there, hoping to find some help with a naive networking question.
I recently bought my first firewall appliance, installed Opnsense and am going to use it with my ISP modem in bridge mode, but while I’m learning I added it to my existing LAN with a 192.168.0.0/24 address assigned to the WAN port by my current DHCP. On the firewall’s LAN port I set up a 10.0.0.0/24 network and am starting to build up my services. So far so good, but there’s one thing I can’t get to work: I can’t port forward the firewall’s WAN IP to a service on the firewall’s LAN network and I can’t figure out why.
To illustrate, I would like laptop with IP 192.168.0.161 to be able to reach service on 10.0.0.22:8888 by requesting firewall WAN IP 192.168.0.136:8888.
Private IPs and bogons are permitted on the WAN interface and I have followed every guide I can find for the port forwarding, but the closest I have come to this working is a “connection reset” browser error.
Hope my question is clear and isn’t very dumb. Thanks for the help or any explanation why I might be struggling to get this to work. Am I missing something obvious?
UPDATE The thread is all over the place, but I have made some progress:
- RDR rule gets triggered when requesting 192.168.0.136:8888 from 192.168.0.123
- Apache logs show
2024-02-09T17:39:17.056208857Z 192.168.0.123 - - [09/Feb/2024:17:39:17 +0000] "GET / HTTP/1.1" 200 161 - a tcpdump (in spoiler below) on the apache container looks inconspicuous to my untrained eye, with the exception of checksum errors in some packets from the docker container (172.20.0.2). The last five lines, after the second GET request (why is there a second GET request?) appear in tcpdump after a delay of about five seconds.
tcpdump
192.168.0.123.54120 > 172.20.0.2.80: Flags [S], cksum 0xfdc5 (correct), seq 4106772895, win 64240, options [mss 1460,sackOK,TS val 1485594466 ecr 0,nop,wscale 7], length 0
17:45:14.918207 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.20.0.2.80 > 192.168.0.123.54120: Flags [S.], cksum 0x6d68 (incorrect -> 0x2fd7), seq 3999845366, ack 4106772896, win 65160, options [mss 1460,sackOK,TS val 1469298770 ecr 1485594466,nop,wscale 7], length 0
17:45:14.924098 IP (tos 0x0, ttl 62, id 63128, offset 0, flags [DF], proto TCP (6), length 52)
192.168.0.123.54120 > 172.20.0.2.80: Flags [.], cksum 0x5b30 (correct), ack 3999845367, win 502, options [nop,nop,TS val 1485594472 ecr 1469298770], length 0
17:45:14.924102 IP (tos 0x0, ttl 62, id 63129, offset 0, flags [DF], proto TCP (6), length 134)
192.168.0.123.54120 > 172.20.0.2.80: Flags [P.], cksum 0x70f5 (correct), seq 4106772896:4106772978, ack 3999845367, win 502, options [nop,nop,TS val 1485594472 ecr 1469298770], length 82: HTTP, length: 82
GET / HTTP/1.1
Host: 192.168.0.136:8888
User-Agent: curl/7.74.0
Accept: */*
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Index of /</title>
</head>
<body>
<h1>Index of /</h1>
<ul></ul>
</body></html>
17:45:14.924119 IP (tos 0x0, ttl 64, id 34500, offset 0, flags [DF], proto TCP (6), length 52)
172.20.0.2.80 > 192.168.0.123.54120: Flags [.], cksum 0x6d60 (incorrect -> 0x5ad1), ack 4106772978, win 509, options [nop,nop,TS val 1469298776 ecr 1485594472], length 0
17:45:14.924407 IP (tos 0x0, ttl 64, id 34501, offset 0, flags [DF], proto TCP (6), length 364)
172.20.0.2.80 > 192.168.0.123.54120: Flags [P.], cksum 0x6e98 (incorrect -> 0x0a74), seq 3999845367:3999845679, ack 4106772978, win 509, options [nop,nop,TS val 1469298776 ecr 1485594472], length 312: HTTP, length: 312
HTTP/1.1 200 OK
Date: Fri, 09 Feb 2024 17:45:14 GMT
Server: Apache/2.4.58 (Unix)
Content-Length: 161
Content-Type: text/html;charset=ISO-8859-1
17:45:14.929077 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 40)
192.168.0.123.54120 > 172.20.0.2.80: Flags [R], cksum 0x1833 (correct), seq 4106772978, win 0, length 0
17:45:15.138862 IP (tos 0x0, ttl 62, id 63130, offset 0, flags [DF], proto TCP (6), length 134)
192.168.0.123.54120 > 172.20.0.2.80: Flags [P.], cksum 0x701e (correct), seq 4106772896:4106772978, ack 3999845367, win 502, options [nop,nop,TS val 1485594687 ecr 1469298770], length 82: HTTP, length: 82
GET / HTTP/1.1
Host: 192.168.0.136:8888
User-Agent: curl/7.74.0
Accept: */*
17:45:15.138872 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
172.20.0.2.80 > 192.168.0.123.54120: Flags [R], cksum 0xb48d (correct), seq 3999845367, win 0, length 0
17:45:19.995097 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.20.0.1 tell 172.20.0.2, length 28
17:45:19.995161 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.20.0.2 tell 172.20.0.1, length 28
17:45:19.995164 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.20.0.2 is-at 02:42:ac:14:00:02, length 28
17:45:19.995164 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.20.0.1 is-at 02:42:b8:07:c2:99, length 28```
UPDATE 2 I see the exact same behaviour with a second VM and apache directly installed on it instead of in a docker container.
UPDATE 3
Thank you everybody for coming up with ideas. And thank you most of all to @[email protected]: The culprit was the Filter rule association in my Port Forward settings which I had as Add associated filter rule but needs to be Pass. As soon as that is set, everything works.
The full solution is a NAT Port forwarding rule with filter rule “pass”, an outbound NAT rule for hairpinning, and everything related to reflection turned off in Settings > Advanced. It’s that easy! 😵💫
You must log in or # to comment.
Post a screenshot of your NAT > Port Forward rule if you can, that will be the easiest way to help I think
The docker01 alias is a host alias with 10.0.0.22 and there’s an apache test container running on port 8888.
I have created a pass any in rule on WAN (just until I figure out what’s wrong)
In firewall > settings > advanced, I have set “reflection for port forwards” and “automatic outbound Nat for reflection” although I’m not sure if that is needed.
Is there any other info I can provide?
Your filter rule association is set to ‘rule’. What is that associated rule, and do things work if you change it to ‘pass’?
https://www.reddit.com/r/opnsense/comments/puty62/correct_option_for_filter_rule_association_when/
Son of a gun!!! Thank you so much! I spent HOURS changing every setting except this one and actually came to the conclusion that it must be something to do with my ISP’s modem or DNS or something.
The rule is the “associated filter rule” OPNsense automatically creates (interfaces are WAN and LAN) and it triggers as a “pass” just fine when I send a request. (I’m attaching another screenshot from the live log below.)
You don’t happen to have a clue WHY this rule breaks everything?
Associated filter rule
Live log with associated filter rule active (leads to
curl: (56) Recv failure: Connection reset by peer)
Here’s some more: From behind the firewall (i.e. from a 10.0.0.x IP) the port forward works (which would be a reflection, I suppose?).
From in front of the firewall, I get “connection reset”, which I interpret as somewhat working but then breaking somewhere else. Does that make sense?
Instead of connecting with a web browser, can you try using curl or telnet just to check if you’re getting through at the TCP/IP connection level?
Further digging: The request reaches the docker container, which returns
200 OK.my-apache-app | 2024-02-09T12:53:22.925676854Z 192.168.0.123 - - [09/Feb/2024:12:53:22 +0000] "GET / HTTP/1.1" 200 161What is going on here? Do I need some rules in the other direction, on top of “Automatic outbound NAT rule generation”?
Like this?
~$ curl 192.168.0.136:8888 curl: (56) Recv failure: Connection reset by peerAnd here’s what this request looks like in the firewall log:
The rules and the log looks find to me, so I suspect something else is going on. Since you’re seeing the request hit apache.
Have you tried another browser just to see if that’s the issue?
Please take a look at my updated original post. I have added some information and a tcpdump.
I’m very uncertain about your network topology. Why is WAN 192.168.x.x?
I am trying to learn in a safe environment without breaking my existing network. It’s not actually a WAN, except from the firewall’s point of view.
I am no expert, but maybe something about 192.168 not being broadcast across wan as it is restricted to local…i coukd be wrong
Can you please elaborate? Who’s restricting 192.168.0.x? It’s not actually WAN, right? It’s just a local network I connected the firewall to.
Again, I’m not a network expert. i just wondered if to the firewall your WAN side connection (not Actually WAN) had some baked in rules to drop 192.168 from that side?? If that is not it does your port forward specify IP range to forward from?
Also, would your firewall need dns pointing to 192.168.0.1 upstream? rather than to 8.8.8.8. In case it needs to find addresses on your upstream network when talking back? I’m throwing stuff at the wall here
And I’m happy to see what sticks!
Pointing DNS to 192.168.0.1 doesn’t change anything, and I’m anyway able to talk out from behind the firewall to the 192.168 net, so that would mean that address resolution works in that direction, no?
I do agree, though, that it seems like the responses are not making their way back correctly, as I can see the requests coming in and replied to in the apache logs.
I appreciate you taking a look. It does indeed have standard rules to drop private networks (192.168, 10.0 and so on), but I have them disabled.
The forward specifies range 8888-8888 and translates it to 8888.
Some modems have DHCP on by default so there is a second firewall that needs to be disabled.
WAN -> modem with DHCP -> router with DHCP.
At least for Comcast, there is a option to disable NAT/Firewall in case you have your own hardware. Some modems on networks like AT&T can’t disable it if I recall.
Woooow totally spaced. You know all that hahaha!
Is this the fabled double-NAT I’ve heard so much about?
My post title was going to be “firewall noob vs. double NAT”, but I’m too much of a noob to tell if that’s where the problem is. 😅
Edit: plus, is it actually a double NAT if I try to port forward into 10.0.0.x from 192.168.0.x? I’m only crossing one NAT, no?
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters DNS Domain Name Service/System HTTP Hypertext Transfer Protocol, the Web IP Internet Protocol NAT Network Address Translation TCP Transmission Control Protocol, most often over IP
5 acronyms in this thread; the most compressed thread commented on today has 8 acronyms.
[Thread #498 for this sub, first seen 9th Feb 2024, 07:05] [FAQ] [Full list] [Contact] [Source code]
There is an opensense option that blocks rfc1918 traffic on the wan. your wan ip addeess range matches rfc1918. I do not remember where to disable it exactly, but this is the error.
I wrote it in reply to another comment, but the traffic reaches the service on 10.0.0.22:8888. The problem seems to be with the return path,
i.e. Hairpin NAT, but I don’t know what it is.edit: scratch that, it’s not hairpinning.
Do you mean these options under Interfaces > WAN? I have them disabled after they did show up as a block in the log.
