Life is a State of Mind

Man, what a tricky, undocumented, annoying configuration… but it works.

We have a few Sun T2000 and T5200 boxes running Solaris 10 at work. We want to run multiple zones (Sun’s virtualization technology) on each physical box; and the virtual servers need to be spread out over several different VLANs to isolate traffic for firewall rules.

So, configuring the Extreme switch to tag multiple VLANs to the port our Sun T2000 is connected through was trivial:

On the Extreme switch (we already had dev-tier and data-tier VLANs configured on 103 and 110 subnets):

configure vlan "dev-tier" delete port 4:11
configure vlan "dev-tier" add ports 4:11 tagged
configure vlan "data-tier" add ports 4:11 tagged

The configuration changes on the Solaris global zone have been far trickier, mostly because Sun does not expect you to create virtual servers on virtual networks, so their documentation is considerably lacking in this area. Here is a brief outline of what I did to get it all working.

On the Sun global zone, as root, set the host up for multiple default routers, which sounds very wrong, but it has to be done because the local zones (virtual servers) on Solaris inherit the network stack from the global zone, they cannot individually modify the network stack:

vi /etc/defaultrouter
    # Global and Dev zones gateway:
    192.168.110.1
    # Data zones gateway:
    192.168.103.1

vi /etc/netmasks
    192.168.103.0   255.255.255.0
    192.168.110.0   255.255.255.0

vi /etc/hosts
    192.168.103.2    datazone1.domain.net   datazone1
    192.168.103.3    globalzoned
    192.168.110.2    globalzone.domain.net  globalzone
    192.168.110.3    devzone2.domain.net    devzone2

vi /etc/resolv.conf
    (make sure your domains are all listed in the search fields, and name server is correct)

mv /etc/hostname.e1000g /etc/old_hostname.e1000g
echo globalzone >  /etc/hostname.e1000g110000
echo globalzoned  >  /etc/hostname.e1000g103000

zoneadm -z datazone1 halt
zoneadm -z devzone2 halt
zonecfg -z devzone2
info net
select net address=192.168.110.3
set physical=e1000g110000
# this changes it from e1000g to the VLAN device name
end
commit
exit
zonecfg -z datazone1
info net
select net address=192.168.103.2
set physical=e1000g103000
# this changes it from e1000g to the other VLAN device name
end
commit
exit
reboot

ifconfig -a
# you should only see the VLAN devices e1000g110000 and e1000g103000 now
netstat -rnv
# you should see multiple default routes
zoneadm -z datazone1 boot
zoneadm -z devzone2 boot
ifconfig -a
# now you should see all the VLAN devices and additional interfaces on each of them
ifconfig e1000g103000 down

That very last command, downing the e1000g103000 VLAN interface is the one that still perplexes me, there must be a way to configure Solaris to plumb a VLAN interface, but not actually bring it up… I have not figured it out yet.

This gives us a virtual host called datazone1 that sits on the 103 VLAN and a virtual host called devzone2 that sits on the same 110 VLAN as the global zone server itself; all over one shared physical category 6 ethernet cable and using only one network switch port and one host based network interface card.