About the Site

This weblog is edited and run by members of reallyenglish, a company offering a total English learning solution based in London, Beijing, Shanghai and Tokyo. Visit our corporate site to know more about what we do.

Notes are posted by members from various cultural and geographical backgrounds, and the topics range from education, business and international communication to software development, the internet culture, and more.


Masatomo Nakano http://twitter.com/masatomon /m/mt-static/support/assets_c/userpics/userpic-2-100x100.png simonl davida jeremyw Go Kameda gavin b No name tomoyukis


Recent Comments

  • Tomoyuki Sakurai: we've been using vether(4) successfully since 4.7. http://old.nabble.com/kernel-6318:-arp-cache-problem-bridge%284%29-with-vether%284%29-td27635451.html (the bug read more
  • RofR: Hi everyone Where about to setup serveral OBSD 4.9 or read more
  • Tomoyuki Sakurai: the above configuration has been working fine since 4.6. read more
  • Jean Aumont: Hi Tomoyuki, If you capture the packets with tcpdump and read more
  • Tomoyuki Sakurai: you will not see OSPF packets on physical interface. > read more
  • Jean Aumont: Hi everyone, I have been trying to pass OSPF in read more
  • Anonymous: without any other information, all i can say is doing read more
  • Jean Aumont: I have a similar set-up, but my 2 end points read more
  • illouca: thank you for the patch !! read more
  • Go Kameda: Glad to hear that - im afraid i cannot really read more

Tomoyuki Sakurai Archives

We’ve been creating our own release for some time. Every new release or bugfix release, I just ssh to build servers and “make”. Here is what I wrote to release the latest version of OpenBSD and FreeBSD. Most of defaults are reasonably defined for many, but please grep “changeme” the Makefiles below and change the values. See release(8) for OpenBSD, release(7) for FreeBSD.


The Makefile does the same thing documented at 5.4 - Building a Release in the FAQ.

Create /etc/src.supfile to update src. Change the tag to which release you want to build. In this example, it’s OPENBSD49.

*default release=cvs tag=OPENBSD_4_9
*default delete use-rel-suffix
*default umask=002
*default base=/var/db
*default prefix=/usr

Install csup by “pkgadd csup”. Make sure to create a slice for /usr/obj and change DEVOBJ. Also, double check RELEASEROOTDIR has enough space (1GB is sufficient for 4.9) for all release files.

RELEASE_ROOT_DIR?=  /usr/release
DESTDIR?=           ${RELEASE_ROOT_DIR}/dest
CVSUP_HOST?=        changeme.cvsup.example.org
SRCDIR?=            /usr/src
OBJDIR?=            /usr/obj
# XXX this Makefile assumes ${OBJDIR} is on a dedicated slice
DEV_OBJ?=           /changeme/dev/rwd0g
SUPFILE?=           /etc/src.supfile
CONFIG?=            GENERIC

all: update patch buildkernel buildworld release checkflist make-index

    pkg_add csup
    mkdir -p ${DESTDIR} ${RELEASEDIR}
    @echo "make init finished successfully"
    @echo "please create ${SUPFILE} for your own."
    @echo "make sure to create base dir for csup (*default base=/var/db)"

    /usr/local/bin/csup -h ${CVSUP_HOST} -gsL2 ${SUPFILE}

.for P in ${PATCHES_FILES}
    (cd ${SRCDIR} && patch ${PATCHFLAGS} < ${P})

    (cd ${SRCDIR}/sys/arch/`uname -m`/conf && config ${CONFIG})
    (cd ${SRCDIR}/sys/arch/`uname -m`/conf/../compile/${CONFIG} && make clean && make depend && make && make install)

buildworld: clean-obj make-obj
    (cd ${SRCDIR}/etc && env DESTDIR=/ make distrib-dirs)
    (cd ${SRCDIR} && make build)


    umount ${OBJDIR} || true
    newfs  ${DEV_OBJ}
    mount ${OBJDIR}

    test -d ${DESTDIR} && mv ${DESTDIR} ${DESTDIR}.old && rm -rf ${DESTDIR}.old &
    mkdir -p ${DESTDIR} ${RELEASEDIR}

release: clean-destdir
    (cd ${SRCDIR}/etc  && make release DESTDIR=${DESTDIR} RELEASEDIR=${RELEASEDIR})

    (cd ${SRCDIR}/distrib/sets && export DESTDIR=${DESTDIR} && sh checkflist)

    (cd ${RELEASEDIR} && /bin/ls -1 >index.txt)


As our FreeBSD nodes are mix of several release branch, the Makefile supports multiple releases. It also supports uploading the images to server, possibly useful for pxeboot server. Install git and subversion by running “pkg_add -r subversion git”. Then, “make init”.

# FreeBSD release(8) Makefile
# to initilize, "make init".
# * create all directories
# * checkout all sources
# "make all" creates release ISO and FTP directories for all
# "make update-stable release-stable" creates one for -STABLE
#   http://www.freebsd.org/doc/en/articles/releng/index.html (outdated but still useful)
#   release(7)
#   /usr/src/release/Makefile 

RELEASE_DIR?=       /usr/home/release
CHROOT_DIR?=        ${RELEASE_DIR}/chroot
SVNROOT?=       svn://svn.freebsd.org/base
EXTPORTSDIR?=       /usr/ports
SYSCTL=         /sbin/sysctl
ARCH!=          uname -m
NCPU!=          ${SYSCTL} -n kern.smp.cpus
MAKE_JOBS_NUMBER!=  echo ${NCPU} \* 2 | bc
PXE_HOST?=      changem.pxe.example.org

.if defined(DEBUG)
SVN_FLAGS=  --quiet
GIT_FLAGS=  --quiet

GIT!=           which git
SVN!=           which svn
.if !defined(GIT)
    @echo "git not found in PATH." 1>&2
    @echo "please install devel/git" 1>&2 && exit 1
.if !defined(SVN)
    @echo "svn not found in PATH." 1>&2
    @echo "please install devel/svn" 1>&2 && exit 1

PORTS_GIT_URL=      git://gitorious.org/freebsd/freebsd-ports.git

# release the latest RELEASE branches.
# to release -STABLE, "make update-stable release-stable"
all:    update release

# let's make release faster
# * copy local ports (EXTPORTSDIR)
# * do not fetch distfiles every build (RELEASEDISTFILES)
# * create ISO (MAKE_ISOS)
# * use svn (SVNROOT, SVNBRANCH)
# * use multiple cores (WORLD_FLAGS, KERNEL_FLAGS)
# * do not build doc (NODOC)
# * use FTP passive (FTP_PASSIVE_MODE)
    ${INSTALL} -d ${CHROOT_DIR}/releng/${RELEASE_MAJOR}.${V}
    make -C ${RELEASE_DIR}/sources/releng/${RELEASE_MAJOR}.${V}/src \
        -j${MAKE_JOBS_NUMBER} \
    make -C ${RELEASE_DIR}/sources/releng/${RELEASE_MAJOR}.${V}/src/release \
        release \
        CHROOTDIR=${CHROOT_DIR}/releng/${RELEASE_MAJOR}.${V} \
        SVNROOT=${SVNROOT} \
        SVNBRANCH=releng/${RELEASE_MAJOR}.${V} \
    make -C ${RELEASE_DIR}/sources/stable/${RELEASE_MAJOR}/src \
        -j${MAKE_JOBS_NUMBER} \
    make -C ${RELEASE_DIR}/sources/stable/${RELEASE_MAJOR}/src/release \
        -j ${MAKE_JOBS_NUMBER} \
        release \
        SVNROOT=${SVNROOT} \
        SVNBRANCH=stable/${RELEASE_MAJOR} \

init:   create-dirs checkout checkout-stable checkout-head

    ${INSTALL} -d ${RELEASE_DIR}/sources
    ${INSTALL} -d ${RELEASE_DIR}/sources/releng
    ${INSTALL} -d ${RELEASE_DIR}/sources/stable
    ${INSTALL} -d ${RELEASE_DIR}/portstrees
    ${INSTALL} -d ${RELEASE_DIR}/chroot
    ${INSTALL} -d ${RELEASE_DIR}/conf
    ${INSTALL} -d ${RELEASE_DIR}/conf/boot

    ${INSTALL} -d ${RELEASE_DIR}/sources/releng/${RELEASE_MAJOR}.${V}
    (cd ${RELEASE_DIR}/sources/releng/${RELEASE_MAJOR}.${V} && \
        ${SVN} checkout ${SVN_FLAGS} \
        ${SVNROOT}/releng/${RELEASE_MAJOR}.${V} src)

    ${INSTALL} -d ${RELEASE_DIR}/sources/stable/${RELEASE_MAJOR}
    (cd ${RELEASE_DIR}/sources/stable/${RELEASE_MAJOR} && \
        ${SVN} checkout ${SVN_FLAGS} \
        ${SVNROOT}/stable/${RELEASE_MAJOR} src)

    if [ -d ${RELEASE_DIR}/sources/releng/${RELEASE_MAJOR}.${V}/src ]; then \
        (cd ${RELEASE_DIR}/sources/releng/${RELEASE_MAJOR}.${V}/src && \
            ${SVN} ${SVN_FLAGS} update); \
    else \
        echo "please make checkout first" 1>&2 && exit 1; \

    if [ -d ${RELEASE_DIR}/sources/stable/${RELEASE_MAJOR}/src ]; then \
        (cd ${RELEASE_DIR}/sources/stable/${RELEASE_MAJOR}/src && \
            ${SVN} ${SVN_FLAGS} update); \
    else \
        echo "please make checkout-stable first" 1>&2 && exit 1; \

    (cd ${RELEASE_DIR}/portstrees && \
        ${GIT} clone ${GIT_FLAGS} ${PORTS_GIT_URL})

    if [ -d ${RELEASE_DIR}/portstrees/freebsd-ports ]; then \
        (cd ${RELEASE_DIR}/portstrees/freebsd-ports && \
            ${GIT} pull ${GIT_FLAGS}); \
    else \
        @echo "please make clone-ports first" 1>&2 && exit 1; \

# two targets to get HEAD sources.
# no release-head target because we don't need it.
# also, building HEAD on non-HEAD is not supported.
# use snapshot or build HEAD on your own host.
    (cd ${RELEASE_DIR}/sources && \
        ${SVN} checkout ${SVN_FLAGS} ${SVNROOT}/head head)

    (cd ${RELEASE_DIR}/sources/head && ${SVN} ${SVN_FLAGS} update)

    # XXX [KNOWN BUG] gzipped mfsroot doesn't work
    if [ -f ${RELEASE_DIR}/chroot/releng/${RELEASE_MAJOR}.${RELEASE_MINOR_VERSIONS}/R/cdrom/disc1/boot/mfsroot.gz ]; then \
        gunzip ${RELEASE_DIR}/chroot/releng/${RELEASE_MAJOR}.${RELEASE_MINOR_VERSIONS}/R/cdrom/disc1/boot/mfsroot.gz ; \
    ssh ${PXE_HOST} \
        rm -rf /tftproot/pub/FreeBSD/${ARCH}/${RELEASE_MAJOR}.${RELEASE_MINOR_VERSIONS}-RELEASE
    scp -r \
        ${RELEASE_DIR}/chroot/releng/${RELEASE_MAJOR}.${RELEASE_MINOR_VERSIONS}/R/cdrom/disc1 \
    # copy loader.conf for pxeboot
    scp ${RELEASE_DIR}/conf/boot/* \

iPhone 4 is able to make VPN connection and send all traffic through it, which is useful when you use public WiFi and/or need intranet resources.

Installing OpenBSD 4.9

See the FAQ. You need to install the kernel and userland sources as well to build npppd and PIPEX kernel.

Building PIPEX kernel and installing npppd

At the time of this writing, npppd is not installed by default. npppd requires "option PIPEX" in kernel config file, which is not enabled by default. Follow the instruction in /usr/src/usr.sbin/npppd/HOWTO_PIPEX_NPPPD.txt after you extract the source code. Note that amd64 users need to patch npppd at the moment. The patch below was from the author. Patch at your own risk.

> cd && fetch http://journal.reallyenglish.com/2011/05/13/privsep.diff.txt
> cd /usr/src/usr.sbin/npppd/npppd
> sudo patch < ~/privsep.diff.txt



interface_list: tun1
interface.tun1.ip4addr: $ip.add.re.ss
pool.dyna_pool: $ip.add.re.ss/$subnet
lcp.mru: 1400
ipcp.dns_primary: $dns.add.re.ss


Choose authentication method. CSV file is good for testing.

auth.method: mschapv2 chap
auth.local.realm_list: local
auth.local.realm.acctlist: /etc/npppd/npppd-users.csv
realm.local.concentrate: tun1

Create a CSV file, /etc/npppd/npppd-users.csv which looks like:

user,secret,$ip.add.re.ss,,memo for user

If you have working RADIUS server,

auth.radius.realm_list: radius
auth.radius.realm.server.secret: $radius_secret
realm.radius.concentrate: tun1

Note that you need clear text password or NT password in the RADIUS backend when mschapv2 is used.

VPN protocol

Choose VPN protocol.

pptpd.enabled: true

Connecting to npppd

Run npppd in debug mode.

# npppd -d
2011-05-21 17:54:03:NOTICE: Starting npppd pid=30068 version=5.0.0                                               
2011-05-21 17:54:03:NOTICE: Load configuration from='/etc/npppd/npppd.conf' successfully.
2011-05-21 17:54:03:WARNING: write() failed in in_route0 on RTM_ADD : File exists
2011-05-21 17:54:03:INFO: tun1 Started ip4addr=
2011-05-21 17:54:03:INFO: pool name=default dyn_pool=[] 
2011-05-21 17:54:03:INFO: Added 1 routes for new pool addresses
2011-05-21 17:54:03:INFO: Loading pool config successfully.
2011-05-21 17:54:03:INFO: realm name=radius(radius) Loaded configuration timeout=9 nserver=1
2011-05-21 17:54:03:INFO: Listening /var/run/npppd_ctl (npppd_ctl)
2011-05-21 17:54:03:INFO: l2tpd Listening (L2TP LNS) [L2TP]
2011-05-21 17:54:03:INFO: l2tpd Listening [::]:1701/udp (L2TP LNS) [L2TP]
2011-05-21 17:54:03:INFO: pptpd Listening (PPTP PAC) [PPTP]
2011-05-21 17:54:03:INFO: pptpd Listening (PPTP PAC)
2011-05-21 17:54:03:INFO: tun1 is using ipcp=default(1 pools).

When everything works, you will see:

2011-05-21 17:58:00:NOTICE: ppp id=1 layer=base logtype=TUNNELSTART user="user" duration=3sec layer2=PPTP layer2from= auth=MS-CHAP-V2  ip= iface=tun1
2011-05-21 17:58:00:NOTICE: ppp id=1 layer=base Using pipex=yes

UPDATE: Ivan Ristic implemented SecDisableBackendCompression in trunk. With this directive, you don’t need the hack below. See my post to the list.

One of cryptic warnings found in ModSecurity audit.log is

  [msg "ModSecurity does not support content encodings"]

Clearly, it says ModSecurity doesn’t support content encodings. That is, it cannot inspect gzipped content. This is problematic especially when ModSecurity is running on reverse proxy. The question is, “how can I inspect, then?” There are some very useful advice from the developer in the list archive. Bad news is, it doesn’t work. Some claims it worked, but they skip deflate process by SetEnvIfNoCase, which is not an option for me.

My goal is:

  • Compress by Content-Type, not by file extension because sometime file extension is not available (/foo.php?imageid=1234). You cannot use SetEnvIfNocase because it cannot access to response headers.
  • Do not compress images, video and other pre-compressed files (compress all except a few others).
  • Do not compress before ModSecurity inspect the content (the goal).

After hours and days to find why it doesn’t work, I finally found the problem. In modextfilter.c, it looks for ext filter by f->frec->name, which is not the name of ext filter but FilterProvider’s name when it should have looked for it by “nodeflate” in my case.

583     /* look for the user-defined filter */
584     ctx->filter = find_filter_def(f->r->server, f->frec->name);

I’m not an apache guru in any way, I might be wrong.

Three tricks:

  • Even though configtest doesn’t complain, ExtFilterDefine is only valid in server config context
  • DO NOT “SetOutputFilter DEFLATE”. It compresses all contents, ignoring nodeflate
  • Define two ext filters

Here is the WORKING config.

    LoadModule ext_filter_module libexec/apache22/mod_ext_filter.so
    LoadModule filter_module libexec/apache22/mod_filter.so
    # Netscape 4.x has some problems...
    BrowserMatch ^Mozilla/4 gzip-only-text/html
    # Netscape 4.06-4.08 have some more problems
    BrowserMatch ^Mozilla/4\.0[678] no-gzip
    # IE is ok, but looked like Netscape, so we reset it
    BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

    # We need to force compression since we will remove the
    # request Content-Encoding/TE headers which mod_deflate
    # looks for.
    SetEnvIfNoCase Accept-Encoding gzip force-gzip
    SetEnvIfNoCase TE gzip force-gzip

    # Disable compression from backend
    RequestHeader unset Accept-Encoding
    RequestHeader unset TE

    # Make sure caching still works
    Header append Vary User-Agent env=!dont-vary

    # XXX be sure to define ExtFilterDefine in server config context!
    # not in virtual host, not directory, etc
    ExtFilterDefine nodeflate mode=output \
        cmd=/usr/bin/true \
    # XXX workaround a bug in mod_ext_filter.
    # it looks up extfilter by FilterProvider's name, not user defined ext filter.
    # it complains "couldn't find definition of filter 'compress'"
    # Define the same name of FilterProvider.
    ExtFilterDefine compress mode=output \
        cmd=/usr/bin/true \

    # Setup the filter to compress the response if we saw
    # an Accept-Encoding/TE header and marked it as force-gzip
    FilterDeclare compress CONTENT_SET
    FilterProvider compress DEFLATE env=force-gzip =1

    # Overwrite the "deflate" filter with the "nodeflate"
    # for various content types so they will not be compressed.
    # This is done as a hack because there is no way to
    # use more than one condition with FilterProvider above.
    FilterProvider compress nodeflate Content-Type $image/
    FilterProvider compress nodeflate Content-Type $application/
    FilterProvider compress nodeflate Content-Type $video/
    FilterProvider compress deflate Content-Type $application/javascript
    FilterProvider compress deflate Content-Type $application/json
    FilterProtocol compress "change=yes"

    FilterDeclare  modsec CONTENT_SET
    FilterProvider modsec modsecurity_out env=modsec-ignore !=1

    FilterChain modsec compress

I tested this on FreeBSD, so, use “cmd=/bin/true” on Linux and others (“/bin/sh” should work and is more portable, if you prefer).

When Apache 2.4 is released, it will be much simpler. See “Expression parser for Apache”.

Almost all serious project require redundancy, reallyenglish is no exception. carp(4), developed by OpenBSD project, is a free implementation of VRRP, which is patented by Cisco. carp(4) has load balancing based on L2 and L3, but here, I'm going to focus on its fail over feature.

 vr0 |         vr0 |
 .---+---. vr3 .---+---.
 | core1 +-----+ core2 |
 `---+---'     `---+---'
 vr1 |          vr1|

Firstly, enable IPv4 forwarding. Also, enable preemption to failover all carp interfaces altogether when one interface fails.

# sysctl net.inet.ip.forwarding=1
# echo net.inet.ip.forwarding=1 >> /etc/sysctl.conf
# sysctl net.inet.carp.preempt=1
# echo net.inet.carp.preempt=1 >> /etc/sysctl.conf

Using carp(4) is fairly simple. Here is what hostname.if(5) looks like.

/etc/hostname.vr0:     up
/etc/hostname.vr1:     up
/etc/hostname.vr3:     inet
/etc/hostname.carp0: vhid 1 carpdev vr0 pass mypass
/etc/hostname.carp1: vhid 2 carpdev vr1 pass mypass
/etc/hostname.pfsync0: syncdev vr3

Bring up the interfaces. You can also use ifconfig(8), but I always prefer startup script to make sure that the configuration will not be lost.

# sh /etc/netstart vr0
# sh /etc/netstart vr1
# sh /etc/netstart vr3
# sh /etc/netstart carp0
# sh /etc/netstart carp1
# sh /etc/netstart pfync0

If you insists on using ifconfig(8):

# ifconfig vr0 up
# ifconfig vr1 up
# ifconfig vr3 inet
# ifconfig carp0 create
# ifconfig carp1 create
# ifconfig carp0 vhid 1 carpdev vr0 pass mypass
# ifconfig carp1 vhid 2 carpdev vr0 pass mypass
# ifconfig pfsync0 syncdev vr3

Repeat almost same configuration on core2. The difference is each carp interface has advskew keyword and vr3 for pfsync device has different IP address. A router with higher advskew value becomes slave.

/etc/hostname.vr3:     inet
/etc/hostname.carp0: vhid 1 carpdev vr0 pass mypass advskew 100
/etc/hostname.carp1: vhid 2 carpdev vr1 pass mypass advskew 100

Now you have redundant firewalls. Easy!

As explained in the earlier entry, we use OSPF for internal routing. How do I establish OSPF neighbor? Because ospfd(8) cannot be enabled actively on carp'ed interface, you need real interface that connects OSPF cloud and core routers. Additionally, real interface cannot preempt carp interface. That means when the real interface fails, you cannot failover. To workaround this problem, you need two OSPF-enabled routers and two links on each core. Things get messy from now on.

           udav1   vr1    sis0      /
-----.      .------.     .-------. / 
     +------+  r1  +-----+ core1 |<
     |      `------'     `-------' \
     |     udav0|vr2\   / sis1  \   \
     |          |    \ /         \   carp1
OSPF |          |     X        pfsync0
     |          |    / \         /   carp0
     |     udav0|vr1/   \ sis0  /   /
     |      .------.     .-------. / 
     +------|  r2  +-----+ core2 |<
-----'      `------'     `-------' \
           udav1   vr2     sis1     \

In this setup, when one of the link from core to OSPF cloud fails, traffic will be routed via the link between r1 and r2. When carp'ed interface fails, preemption kicks in and ospfd(8) on core1 will remove the routes and r1 will not route traffic to core1. claudio@, one of "the networking guy" in OpenBSD admitted it's not optimal. Hopefully, it will be fixed in the future release.

ALIX 2D3 board, which is used as r1 and r2, has only three ports. And one port is used for diskless boot in this testing. So, i added two USB ethernet adopters, deteced as udav(4). Make sure the right patch goes to the right port. Each link between the routers has a /30 subnet. If confused, draw a diagram on paper.

Let's enable OSPF on all interface on every router, but not on real interfaces used for carp. ospfd.conf(5) looks like:

# core1 and core2
auth-type crypt
auth-md 1 "mypass"
auth-md-keyid 1

area {
  # link to r1/r2
  interface sis0 { metric 10 }
  interface sis1 { metric 20 }
  # you can omit "passive" on carp interface. it's always passive anyway
  interface carp1 { passive }
  interface carp0 { passive }

# r1 and r2
auth-type crypt
auth-md 1 "mypass"
auth-md-keyid 1

area {
  # link to core
  interface vr1 { metric 10 }
  interface vr2 { metric 20 }
  interface udav0 { }
  interface udav1 { }

Run ospfd(8) on every router. The output of ospfctl(8) on core1 looks like this.

# ospfctl sh int
Interface   Address            State  HelloTimer Linkstate  Uptime    nc  ac
carp0 DOWN   -          master     00:00:00   0   0
carp1 DOWN   -          master     00:00:00   0   0
sis1     BCKUP  00:00:00   active     1d06h19m   1   1
sis0     BCKUP  00:00:00   active     1d06h19m   1   1

# ospfctl sh ne
ID              Pri State        DeadTime Address         Iface     Uptime  10  FULL/DR      00:00:33     sis1      1d06h20m  1   FULL/DR      00:00:32     sis0      1d06h20m

Now the fun (and boring) part. While pinging from a host in OSPF cloud to the other end of core, pull out the cable randomly. You can shutdown one of the router, too. Nothing should happen, other than normal ping output. If it doesn't work, tcpdump(8) is your friend. You can manually fail over the master by:

# ifconfig -g carp carpdemote 10

This setup requires no additional package and works out of the box. If you need more redundancy, i.e. redundancy at application level, than this setup, OpenBSD also provides relayd(8). If you need redundant DHCP servers, modified version of dhcpd(8) has synchronization support. OpenBSD makes redundant networking fairly easy.

Happy redundancy!

reallyenglish.com is geographically diverse. The HQ in UK, a few branches in Asia, maybe more in the future. Except for few exceptions, the employees expect universal services, such as source code repository, project management system and wiki. All of them must be protected by strong encryption. IPsec is a standard protocol designed for encrypting packets from one place to another. We extensively use IPsec between our offices.

As a sysadmin, whenever someone says "the Internet is down", I say "no, it isn't". The problem would be caused by networking issue, misconfiguration by someone, maybe his own fault or whatever. The Internet is fine. But, partially, he is right. The Internet is the network of networks. Different network is managed by different entity. As such, links between networks
cut off for various reason every day somewhere on the Internet. You just don't know it is happening. The Internet as a whole works somehow thanks to "dynamic routing".

What dynamic routing does is, in short, "if there are multiple paths to the destination, use the best path" and "tell neighbors about your directly connected networks so that others can understand the whole network map". Without dynamic routing, you have to teach every possible route to every single router. For our intranet, we're using OSPF.

Okay, for robust intranetworking, all you need is multiple meshed IPsec tunnels with OSPF. IPSec is a standard and mature protocol, and so is OSPF. Sounds easy? Not quite. What I don't really like about IPsec is, it wasn't designed with routing in mind. As Security Association is statically configured, you cannot specify multiple path to a destination. Moreover, it doesn't handle multicast packet. OSPF is a routing protocol and uses multicast for its communication.

One more thing to make this problem complicated is proprietary products, Netscreen security devices to be specific, we're still using. If you stick with a specific vendor, you will not have interoperability problem, but you're successfully vendor locked-in. What you can do is what that particular vendor implemented. If they don't, you cannot do anything with it. Anyway, I strongly prefer open standards and open platforms. The choice in this case is OpenBSD. However, there wasn't documentation about how to implement meshed, dynamically routed IPsec with OpenBSD and ScreenOS.

IPsec doesn't like multicast pakcet. Okay, use GRE, Generic Routing Encapsulation. By encapsulating multicast packet into GRE unicast packet, IPsec happily forward the packet to its peer. When the link to the peer goes down, ospfd(8) marks the link as unavailable and use alternative path to the destination. Sounds simple, isn't it? Again, not quite. You have to understand all the network layers involved. There would be 6 endpoints, or IP addresses, involved in this setup. IPsec endpoints, GRE tunnel endpoints and a /30 network for OSPF communication.

Let's configure OpenBSD first. Make sure both packet forwarding and GRE are enabled. Also, you better to disable pf(4) during the testing. You can enable it again after everything works. OpenBSD 4.6 and later enable pf by default.

# sysctl net.inet.gre.allow=1
# sysctl net.inet.ip.forwarding=1
# pfctl -d

Next, create GRE tunnel.

# ifconfig gre0 create
# ifconfig gre0 tunnel myglobal.ip.addre.ss peerglobal.ip.addre.ss (outer header)
# ifconfig gre0 inet mygre.ip.addre.ss peergre.ip.addre.ss netmask 0xffffffff link0 up (inner header)

The packets between mygre.ip.addre.ss and peergre.ip.addre.ss will be encapsulated by GRE unicast packets with the global IP addresses of the both endpoints as source and destination header. GRE packets are not encrypted, yet.

Then, configure IPsec. OpenBSD's newer ipsec.conf(5) makes IPsec configuration much easier than ever. The security policy would be "encrypt all packets between the endpoints with their global address". Due to performance limitation of the hardware, we use SHA1 and 3DES.

# vi /etc/ipsec.conf
mypeer = "peerglobal.ip.addre.ss"
me = "myglobal.ip.addre.ss"
mypassword = "password"
ike esp from $me to $mypeer peer $mypeer \ main auth hmac-sha1 enc 3des group modp1024 \ srcid $me/32 dstid $mypeer/32 \ psk $mypassword

Start isakmpd and tell it to read security policy from /etc/ipsec.conf.

# isakmpd -K
# ipsecctl -f /etc/ipsec.conf

NS 5GT's config is not so straight forward, but almost same you did above on OpenBSD.

set interface tunnel.1 zone untrust
set interface tunnel.1 ip mygre.ip.addre.ss/30 (inner header)
set interface tunnel.1 tunnel encap gre
set interface tunnel.1 tunnel local-if untrust dst-ip global.ip.addre.ss (outer GRE header, GRE endpoints)
set interface tunnel.1 tunnel keep-alive (optional, but useful to see GRE packet on the wire)
set ike gateway mypeer address peerglobal.ip.addre.ss main outgoing-interface "untrust" preshare password sec-level compatible
set vpn mypeer gateway jp proposal "nopfs-esp-3des-sha" 
set vpn mypeer bind interface tunnel.1
set vpn mypeer proxy-id local-ip global.ip.addre.ss/32 remote-ip peerglobal.ip.adre.ss/32 ANY

The gotcha was, I thought it wasn't possible for OpenBSD and 5GT to talk to each other in a setup like this because OpenBSD's GRE tunnel is point-to-point, 5GT's is a normal /30 subnet. I even didn't give it a try during the initial testing. In fact, it is. Now, you can ping the peer using GRE /30 IP address each other.

# ping peergre.ip.addre.ss
PING peergre.ip.addre.ss (peergre.ip.addre.ss): 56 data bytes
64 bytes from peergre.ip.addre.ss: icmp_seq=0 ttl=64 time=277.665 ms

You'll see ICMP packets on the GRE interface.
# tcpdump -tnei gre0 -c 10 icmp 
tcpdump: listening on gre0, link-type NULL
mygre.ip.addre.ss > peergre.ip.addre.ss: icmp: echo reply
peergre.ip.addre.ss > mygre.ip.addre.ss: icmp: echo request

Make sure the traffic is encrypted. Replace vr0 with your egress interface's name.
# ipsecctl -s all
flow esp in from myglobal.ip.addre.ss to peerglobal.ip.addre.ss peer peerglobal.ip.addre.ss srcid myglobal.ip.addre.ss/32 dstid peerglobal.ip.addre.ss/32 type use
flow esp out from peerglobal.ip.addre.ss to myglobal.ip.addre.ss peer peerglobal.ip.addre.ss srcid peerglobal.ip.addre.ss/32 dstid myglobal.ip.addre.ss/32 type require

esp tunnel from myglobal.ip.addre.ss to peerglobal.ip.addre.ss spi 0x1275cf8c auth hmac-sha1 enc aes
esp tunnel from peerglobal.ip.addre.ss to myglobal.ip.addre.ss spi 0x20eec763 auth hmac-sha1 enc aes

# tcpdump -tni vr0 -c 10
tcpdump: listening on vr0, link-type EN10MB
esp myglobal.ip.addre.ss > peerglobal.ip.addre.ss spi 0x61a862f2 seq 45586 len 372
esp peerglobal.ip.addre.ss > myglobal.ip.addre.ss spi 0x84cd621d seq 45973 len 132
Now that both IPsec and GRE tunnel worked, the last thing you need is enabling OSPF. ospfd.conf is very simple. Replace vr1 with your internal network interface.

# vi /etc/ospfd.conf
auth-md 1 $password
auth-type crypt
auth-md-keyid 1
area {
 interface gre0 { }
 interface vr1 { passive }

Start ospfd(8) in debug and verbose mode.

# ospfd -dv
... lots of debug information ...

Do the same thing on 5GT.

set vr trust-vr protocol ospf enable
set interface tunnel.1 protocol ospf area
set interface tunnel.1 protocol ospf authentication md5 mypassword key-id 1
set interface tunnel.1 protocol ospf authentication active-md5-key-id 1
set interface tunnel.1 protocol ospf enable
set interface trust protocol ospf area
set interface trust protocol ospf passive
set interface trust protocol ospf enable

Now you'll see OSPF neighbor relationship is established.

# ospfctl show neighbor
ID                      Pri State      DeadTime Address                Iface     Uptime
peergre.ip.addre.ss     1   FULL/P2P   00:00:30 peergre.ip.addre.ss    vr1       00:00:23

run "ospfctl show rib" to see RIB and "ospfctl show fib" to see FIB. It should show you the network at the peer's end. If it doesn't work for you, make sure IPsec flow is established,  OSPF packet is flowing between both hosts. To make a meshed IPsec, simply repeat this procedure. Tedious? Well, for that matter, you have to replace all the proprietary boxes with OpenBSD and use some kind of automated configuration management system.

Hello world!

The first entry from reallyenglish to the world. I hope you'll find something useful here.