I've been successful with this for ubuntu, CentOS, debian and fedora.  I can't get it to work with red hat.  I am trying to buiild a gre ipsec tunnel between this red had box and a Cisco router.  Enclosed are the configs for both and ping tests between the two sites.  From the outputs you will see that I am able to get isakmp working (phase 1), but not IPSEC (phase 2).  From the debugs you see that when each site is trying to ping the other site's tunnel address, each is encrypting their packets, and the packets are reaching the other end, BUT neither end is answering the esp (encryption) packets that are recieved.  And no traffic can pass thru either.
The Cisco error message "IPSEC SA failed identity check" indicates that the acls (for the "interesting traffic" ) is not mirrored properly.  I checked that and they are mirrored so it has to be something else.
I am using racoon to encrypt a GRE tunnel.
NOTE: The ipsec-tools are not the absolute latest version.  Won't know until it is checked again in two days.  If that is a show stopper please let me know now.  Otherwise I prefer to use the version that is already installed and don't want to upgrade it if I don't have to.
The legend for the enclosed documents:
router IP is 12.127.x.x
router loopback IP (used as tunnel source) is 10.5.5.x
router GRE tunnel interface is
red hat server ip is
There is an asa in front ot the server and I verified that the ipsec protocals are allowed thru.  In fact I am allowing ALL IP from the router thru the asa to the server.  So the asa is really a nat wall for the server and the server static IP is 192.168.5.x
red hat server tunnel interface is

Please help.
ping-tests-both-dir.txt (6 KB)
ping tests (text/plain)
redhat-config.txt (6 KB)
rh config (text/plain)
rtr-config-and-outputs.txt (11 KB)
router config and outputs (text/plain)

asked 09/17/2011 11:11

mrkent's gravatar image

mrkent ♦♦

31 Answers:
First, why GRE?

IPSEC does already provide tunnels, a IP tunnel in an IP/SEC tunnel?
If you need to traverse NAT on the way, you can use NAT-T ..

Maybe you need to look into openswan, that might provide simpler tools.
noci's gravatar image


Need GRE because eventually we may want to pass multicast and ospf between sites.

Already committed to ipsec-tools.  It's what was used in the past few installations.  Perhaps if there is more time, and on the next project where there might be time for a learning curve.  For now, I need to get what I have here working.

What's frustrating is that I've done this configuration several times with other distributions.  Can't get the red hat system to work.  Maybe it is not the latest ipsec-tools package?
mrkent's gravatar image


Or... given the parameters I listed in my opening question, perhaps you could shoot me a sample configuration in openwan that would do what I'm looking for?
mrkent's gravatar image


wrt. GRE I have no need for it, so never used it. But Mcast is definitly one of its uses.  (otherwise it will only complicate the setup).
Stick with the tools you have so far, if they are familiar to you. (I had the impression this was a totaly new setup).
This is the source for ipsec tools, (with release notes per version, etc.)  So you can verify  (The original kame project is now defunct).

You don't mention which redhat, Centos is a copy from redhat so if Redhat 4 fails Centos 4 should fail too ...
CentOS is the source distribution from Redhat with all redhat logo material removed. (some management tooling is removed too as that is
redhat private, and only makes sense for their distribution model).

So you need to setup the layers, ipsec being the bottom layer need to work FIRST..., so you need both phases to be successful
to be able to use it.
Phase one negotiates the IKE setup., Phase 2 negotiates the tunnel key, so if that doesn't complete you are in trouble.
Check for typo's in keys, addresses ..
noci's gravatar image


This line from the ethernet config is quite curious:
 inet addr:  Bcast:  Mask:

For I address with netmask 255.255.252 I would expect the broadcast to be: and not
So there is a problem with your eth0 for a start.

Here is an ipcalc output:

$ ipcalc
Address:          11000000.10101000.000001 01.00001000
Netmask: = 22   11111111.11111111.111111 00.00000000
Wildcard:            00000000.00000000.000000 11.11111111
Network:       11000000.10101000.000001 00.00000000
HostMin:          11000000.10101000.000001 00.00000001
HostMax:        11000000.10101000.000001 11.11111110
Broadcast:        11000000.10101000.000001 11.11111111
Hosts/Net: 1022                  Class C, Private Internet
noci's gravatar image


Double checked the ifconfig and yes you are right.  The netmask and broadcast is how you have it.

Not working.  Both sides encrypt and both sides do receive the encrypted packets from the other end.  But neither decrypts!
Phase 1 is working, I can see the tunnel is up isakmp -wise.

What is frustrating is I used this exact configuration with a Centos machine and that machine works!!
But not this red hat machine.

I also tried the nat_traversal option in the racoon config.  No worky.
Tried to replace "require" with "unique:<id number>" in the setkey config.  No worky.

What could be different?

mrkent's gravatar image


And, BTW, I do get SAP and SPD entries listed when I do a setkey -DP command.
But still no tunnel ping and no traffic passes thru.
mrkent's gravatar image


I'm also looking into Nat-T because my server is behind a nat firewall.  Though for now the firewall is letting all packet thru between the router and the server, it is natting the server's address to a public IP.  So I'm looking into the configuration changes in the racoon.conf file to make that happen.  I hope this works.  Are there other things in the kernel I need to change or check to see if my system supports Nat traversal?

Any other advice please.
mrkent's gravatar image


If NAT-T is used, you alos need access to port UDP/4500  to make it work, and port forward port 4500 to your internal system, if the other side need to contact your system.
noci's gravatar image


I have it turned on at both locations.  The cisco router does it automatically at one end and at the other end I have the configuration changes in the racoon.conf file (i think I have it right)  enclosed here.
Anything else?  Do I know for sure the kernel supports nat-t? How would I know?

But also that may not be enough, right?  Because at the firewall itself, where all IP is allowed to pass thru between the router IP and the server IP, is the one doing the natting  from 192.168... to 64.15.. (public).  And I don't have full access to it.  Something tells me I also have to set it for ipsec pass-thru???  Or tell someone how to do it.  It's not enough that it's passing IP between the two, right?
Please advise.
nat-t update to racoon.conf file
mrkent's gravatar image


Normally that shouldn't be a problem, (It's IP traffic) but it might also be your provider... Or firewall rules limiting traffic to UDP/TCP/ICMP only?

Can you test with any previously working setup on that location.?
There is no "ping" program that verifies if esp etc. works.
If you have a DSL modem you are more or less fried, if you have an Ethernet connection you might be able to plugin something else (laptop to test or so).
noci's gravatar image


We are going to take the FW out of the picture and see if the tunnel comes up.

BTW, I tried this with just using GRE in case there was some issue or corruption with IPSEC.  Still did not work.

How do I know an interface is configured correctly?
mrkent's gravatar image


GRE uses a separate protocol (47) just like IPSEC, so maybe the firewall is only allowing UDP/TCP.

You need to know the setup of the firewall..., measure on both sides of the firewall and try various traffic and see what is passed on and what not. (The same holds for ISP's btw. )
noci's gravatar image


I got a chance to view the FW config.  Its acl was allowing protocols 47, 50 and 51 along with udp 500 and 4500 so it seems it should pass it thru.

Noci, you mentioned openswan early on.  If I had the opportunity to load it on this server, could it coexist with the ipsec-tools (racoon) package?  If there was time, I would like to compare the two seemingly competing packages.

And, for this particular network, could you tell me what the openswan configuration would be?  For racoon, it was simply a matter of three files... racoon.conf, setkey.conf and psk.txt.  Would you know what the comparable openswan config for this system is?
Thank you
mrkent's gravatar image


You can have both installed at the same time, but not running at the same time.
And to be honest the freeswan project had 2 successors openswan as mentioned and strongswan.
Both have the same source, but both developers had a different view on what was the most important missing feature.
for strongswan mainly X509 certificates, ease of use,;
for openswan interconnection with others, encryption architecture;

because of their common architecture openswan & strongswan cannot be installed at the same time.
noci's gravatar image


Thank you.  Based on that, openswan would be preferable.

Do you mind getting me started on that config with the basics and with the information I have.  Or is that too much to put on paper here?
mrkent's gravatar image


I can guide you through it. It a rather straight forward but you can try for yourself.

the default configuration allows for a directory where you can drop the per connection configuration.
The secrets are in a separate file, ipsec.secrets.
noci's gravatar image


Openswan has been loaded on the server.

But before I continue that, I tried one more test with racoon, this time taking out the GRE component and just doing IPSEC across the path.  It seems to work!  So it tells me it's one of three things happening.
1.  My matching spadd entries in the setkey.conf file were wrong.  Even though it followed the same exact steps as another system (CentOS) that worked with GRE/IPSEC.
2.  I configured the tun0 interface incorrectly.  Even though it indicated it was "up"  And the "ip link" command does indicate that it is GRE.
3.  This RHEL box does not support GRE.  Though ip_gre module does exist.

What could it be?

I have to fix this part first before going to openswan because if GRE fails with racoon, it will fail with openswan.

Any ideas on the GRE interface problem?

mrkent's gravatar image


Still working with the admin of the linux RHEL box.  Some questions:

Does there need to be a patch to the kernel in order for it to support NAT traversal?
Same question for the GRE protocol.
Does order matter in the list of spadd entries in the setkey.conf file?
Can multiple virtual interfaces conflict with this particular tunnel interface, even if they are "ifconfig down"?

Any other advice going forward?

mrkent's gravatar image


GRE needs to be enabled in the kernel

probably multicast if needed
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y    ?? a lot of different address/routing or not
CONFIG_IP_PIMSM_V1=y                                ?? pim v1 needed
CONFIG_IP_PIMSM_V2=y                                ?? pim v2 needed?
CONFIG_NF_CT_PROTO_GRE=m             (Netfilter Connection Tracking & NAT support)

IPSEC needs to be enabled

According this article the code is in 2.6.6 & later.
(You can check if there is natt code in the transformation sources.
in the kernel tree look for the net/key/af_key.c file, if natt structures are assign to & used then natt is in the kernel.

You will need the Connection tacking & nat modules for gre.

I never realy used racoon so I don't known if order is important on the
At the time freeswan was available, and racoon started development so i chose freeswan and moved on to openswan.

All methods are mentioned with pro/con  in this presentation:
noci's gravatar image


Wow, thanks for the info.  And I love that presentation.  Thanks!

I will forward this on to the admin...

So is it possible that GRE is configured but not all the correct parameters have been checked off?  For example, it seems that I have GRE because 'modprobe ip_gre' seemed to have created the module and it is seen wehn you do a 'lsmod'.  Also these outputs when you do an ip link and ifconfig:
##ip link

 tun0@eth0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue
    link/gre #this address# peer #far end address#


tun0      Link encap:UNSPEC  HWaddr C1-B4-23-04-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:#this tun addr#  P-t-P:#far end tun addr#  Mask:
          UP POINTOPOINT RUNNING NOARP  MTU:1300  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37 errors:579 dropped:0 overruns:0 carrier:579
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:3996 (3.9 KiB)##ip link

Same stupid question regarding the IPSEC.  Could have been configured but some important parameters not checked off?  IPSEC does seem to work when I do it without the GRE.

 And for NAT-T I looked up the kernel version 'uname -r' and it is 2.6.18.

I'm going to look at lowering the MTU and also turning off the "Don't Fragmetn" bit to see what happens.
Let me know what you think about all this.
mrkent's gravatar image


Turning off the don't fragment is a wise move.
you need extra space for, IP + GRE + IP + ESP TUNNEL header...

All the gre modules are:
 gre.ko                                  - used by ip_ge
 ip_gre.ko                              -
 nf_nat_proto_gre.ko              - nat module
 nf_conntrack_proto_gre.ko    - netfilter conntrack module

noci's gravatar image


Do I invoke those four modules with 'modprobe' for each one?  Like I did with 'modprobe ip_gre'?

And can you take a look at how I plan to do this?  iptables is the only way I can think of and for clearing the DF bit is it something like this?
iptables -t mangle -A POSTROUTING -j DF --clear

and for adjusting the maximum segment size for outgoing packets is it something like this?
iptables -I FORWARD -s SS.SS.SS.SS/NM -d DD.DD.DD.DD/NM -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1200
  or whatever the appropriate value.

Thanks for all the help.
mrkent's gravatar image


gre is inseserted by modprobing ip_gre,
the other two need to be modprobed too.

Linux only uses DF for path_mtu detection, so disabling that should suffice.

Setting it to 1

echo 1  >/proc/sys/net/ipv4/ip_no_pmtu_disc

disable PMTU
echo 0   >/proc/sys/net/ipv4/ip_no_pmtu_disc

disables it.

(This can be configured using sysctl).
noci's gravatar image


The  MSS should not be modified.
You set the MTU where possible on an interface.
noci's gravatar image


FYI, the sys admin of the RHEL server informs me that the only module available in their repository is the ip_gre module.  The other modules listed are custom and not supported.

...I haven't been able to try the MTU, MSS and DF adjustments yet.  I'll continue tomorrow.
mrkent's gravatar image


the other modules are standard kernel modules (the name might be sleightly different, as i lifted the names from a 2.6.39 kernel...) If there is no nat on your machine that shouldn't be too big a problem though.

You anly can adjust the mtu on the tun0 device.
noci's gravatar image


Server is RHEL 5.6.  MIght explain why some modules are not readily available?
Troubleshooting continues tomorrow.
mrkent's gravatar image


No luck.  I am going to switch to openswan now that I have time.  I will open a new question on setting up openswan.  Perhaps you can find it and work with me.  You've already outperformed the maximum points I can give you on this one, and I thank you.

One last thing before I move on.  I'm not totally convinced my kernel is configured to support NAT-T.  Even though I do see udp 4500 used in tcpdumps, the server just never seems to respond to it with another udp 4500.  Racoon is generating the sleep timers every 10 seconds in udp 4500, but I don't know that the server itself is actually processing NAT-T.  You mentioned look in the kernel tree and look at the net/key/af_key.c file.  I can't find it.  How do I find it and what do I look for in it?

It is linux 2.6.18, but how do I know it is processing NAT-T?

Thanks again.
mrkent's gravatar image


OK i'll look for an openswan Q
noci's gravatar image


Thank you!!
mrkent's gravatar image


Your answer
[hide preview]

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments


Asked: 09/17/2011 11:11

Seen: 736 times

Last updated: 10/08/2011 06:47