KVM LXC DNS UL 14.04
This post is about how to configure Domain Name Services (DNS) for a private network of KVM virtual machines running on an Ubuntu 14.04 64-bit Desktop Edition on a Lenovo W520 Mobile Workstation laptop with 32Gb RAM. The goal of this work was to find a way to provide DNS at both the KVM host level, and also in each of the KVM guests making no changes to the NM-controlled dnsmasq-base that Network Manager uses by default.
Objectives
BEGIN UPDATE 2014-09-07
This page is a copy of the original page here. An option was to simply update that page with the added information for using LXC in the setup but it's enough of a separate topic to warrant separate, albeit partially redundant content from the original post. What has been added in this page is how to integrate LXC networking into the below-described Ubuntu Desktop 14.04 64-bit KVM lab and testing environment. The approach that is used builds on the focus of this blog on OpenvSwitch (OVS) which is the networking solution used here for LXC inclusion in this setup. The LXC containers, when the network is configured as shown below, will attach to existing OVS switches already in use by KVM and on the same network ranges, allowing full ethernet connectivity between all the KVM guests, all the LXC containers, and the host (and of course www internet connectivity for all of the preceding!). The details of the solution as modified for LXC are enumerated below.
No modifications to Ubuntu Network-Manager, it continues to function as designed for desktop use (VPN, wired ethernet, wifi, bluetooth tether, etc.)
All functionality of the desktop that depends on dnsmasq-base and Network Manager is left intact, functioning as in the base Ubuntu install.
All networking for the KVM guests and the LXC containers is provided by OpenvSwitch (and see my post here for extending OpenvSwitch to VirtualBox).
All KVM guests and LXC containers obtain DHCP services, when needed, from isc-dhcp-server provided over bind9 ("named").
The OpenvSwitch solution devised here detects the type of www WAN and ip range and builds the internet connectivity at boot time automatically.
END UPDATE 2014-09-07
Configuring DNS for a set of KVM guests running on an Ubuntu Desktop KVM host presents some additional design considerations compared to say, using Ubuntu Server Edition as the KVM host, because of the existence on the Desktop Edition of the NetworkManager, which uses dnsmasq-base for DNS resolution, among other things. The Network Manager has come a long way since say, Ubuntu 9, and it has become very useful and robust for the desktop, imho. Therefore, since this is a laptop that is used for many purposes, not just hosting KVM guests, but also for work over a corporate VPN, accessing internet content, etc., therefore a major objective was to implement the DNS for the KVM guests without affecting NetworkManager, i.e. a goal was to leave NetworkManager working exactly as designed, with little or no changes to NM-dnsmasq.
Another objective was to locate the master DNS server inside one of the KVM guests. The reason for this objective is that this group of KVM guests is intended to be a self-contained lab that other employees of the company, and interested persons as well, could use on their own desktop or laptop machine. In other words, this set of KVM guests should be portable, and so the DNS master server must be in one of the KVM guests so that it travels along with the KVM images and XML files.
BEGIN UPDATE 2014-09-07
There may be some changes coming to this blog about the idea of having the master DNS server in a KVM guest. With the new interest in LXC and exploration of possiblities there, some change may be made to that strategy. It's under consideration. For now it remains the strategy of this blog post for this laptop setup.
END UPDATE 2014-09-07
Note also that "vmem.org" is an existing domain, and so a local authoritative-only DNS server was needed so that "vmem.org" servers could be defined in the confines of this local laptop network, and not from resolution to DNS servers for the actual "www.vmem.org" servers.
The last main objective was to design the DNS so that new entries or updates to DNS for the KVM guests only needed to be made in one location, not in multiple locations. For example, it is desired to be able to login to the KVM guests using their hostname, and not their IP addresses. This could be accomplished most siimply by just adding the KVM guests to the /etc/hosts file of the KVM host. But this approach has several design drawbacks. One drawback is that providing DNS to both the KVM host and the KVM guests would involve maintaining both the DNS server inside the KVM guest as well as the /etc/hosts on the KVM host. The second drawback is that it was stated previously that it is desired that the DNS reside in the KVM guest, and that the DNS be fully portable and travel with the KVM guest images and XML files. A DNS could probably be designed, say, using dnsmasq inside a KVM guest, to query /etc/hosts on the KVM host, but this would not be portable, and would tie the KVM guests to the original KVM host for DNS resolution. Therefore, /etc/hosts was ruled out.
The design that was adopted was to install a bind9 (aka "named") master DNS server in one of the KVM guests (running Oracle Enterprise Linux 6.5) and then create a bind9 slave DNS server on the Ubuntu 14.04 KVM host. This design provides a DNS server that travels with the set of KVM guests, but also provides DNS resolution at the KVM host level. For example, to connect to a KVM guest using the hostname and not the IP address, before ANY of the KVM guests are running, we would need /etc/hosts on the KVM host or a DNS server on the KVM host (because the KVM guest DNS server is not running yet) hence the need for a slave DNS server at the KVM host.
The design used therefore is as follows:
A DNS master running in the oracle651.vmem.org KVM guest
A DNS slave running in the vmem1.vmem.org KVM host which is updated with changes from the DNS master
A dnsmasq-base instance running in the vmem1.vmem.org host to support Ubuntu 14.04 Network Manager to handle VPN and internet DNS
Primary DNS Server Details
Information on the primary DNS server is shown below. This server began life as a pure Oracle Enterprise Linux 6.5 UEK kernel running in a KVM guest server, but now runs a Centos 6 kernel custom-built for SCST as described here.
BEGIN UPDATE 2014-09-07
The SCST install on the OEL 6.5 UEK KVM guest was redone to use the UEK kernel source for the custom SCST kernel. The stopgap solution of running the CentOS 6 kernel modified for SCST ontop of OEL 6.5 packages was just that - a quick stopgap because the kernel sources were not immediately available at the inception of that work. So a few days later, I created a new guide here explaining how to compile a custom Oracle UEK kernel for SCST using UEK source code obtained free from here. Note that although access to Oracle Linux Edelivery site is free, it will be necessary to create a free Oracle Technology Network (OTN) account as described here. An OTN account is a very useful thing to have anyway, so worthwhile to register to make available the extensive resources hosted by Oracle Corporation.
END UPDATE 2014-09-07
Aside from the custom kernel, the server is OEL 6.5. The instructions in this post should work fine albeit with possibly some slight modifications here and there, on any CentOS 6 / RHEL 6 / OEL 6 server.
[root@oracle651 ~]# uname -a
Linux oracle651.vmem.org 2.6.32-scst #1 SMP Mon Aug 11 15:55:43 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@oracle651 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)
[root@oracle651 ~]# cat /etc/oracle-release
Oracle Linux Server release 6.5
[root@oracle651 ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 52:54:00:66:26:A0
inet addr:10.207.39.74 Bcast:10.207.39.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe66:26a0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:443 errors:0 dropped:0 overruns:0 frame:0
TX packets:259 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:44522 (43.4 KiB) TX bytes:36703 (35.8 KiB)
[root@oracle651 ~]#
Slave DNS Server Details
Details of the slave DNS server are shown below. The laptop normally operates on wireless networking, but can also operate on wired network just as well using the solution described in this post. The wlan0 information is shown here. If the laptop were plugged into a wired connection it would acquire an IP address on this same home network (192.168.1.x).
A note about listings. I have listed the code for the relevant files for configuring the networking environment and DNS on the KVM host (Lenovo W520 laptop). However, for the KVM host, I have also attached the relevant files to this blog post (see end of post).
gstanden@vmem1:~$ uname -a
Linux vmem1.vmem.org 3.13.0-34-generic #60-Ubuntu SMP Wed Aug 13 15:45:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
gstanden@vmem1:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.1 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.1 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
gstanden@vmem1:~$ ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 68:a3:c4:e6:98:ed
inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::6aa3:c4ff:fee6:98ed/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:54337 errors:0 dropped:0 overruns:0 frame:0
TX packets:42990 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:51549615 (51.5 MB) TX bytes:8859803 (8.8 MB)
gstanden@vmem1:~$ ifconfig sw1
sw1 Link encap:Ethernet HWaddr 92:6b:0e:6e:a8:4f
inet addr:10.207.39.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::64fd:4bff:fe23:1fdd/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:413 errors:0 dropped:0 overruns:0 frame:0
TX packets:819 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:64440 (64.4 KB) TX bytes:74489 (74.4 KB)
gstanden@vmem1:~$
OpenvSwitch Details
The interface "sw1" shown above is an OpenvSwitch. This switch provides the management network which connects the KVM guests to all the other KVM guests on that network, and also connects the KVM guests to the KVM host, providing full ssh connectivity host-guest and guest-host. Connectivity between the KVM guests and the internet is provided by the iptables rules contained in the script below.
One might ask, "Why not just attach wlan0 (or eth0) to the OpenvSwitch directly?"
The answer is that attaching wlan0 or eth0 directly to the OpenvSwitch typically interrupts networking and causes direct overlap between the territory of Network Manager and OpenvSwitch. Several posts on the internet go so far as to rip out Network Manager entirely so that there is no overalp.
As stated previously, one of the goals of this implementation was to implement all the DNS and DHCP services for the KVM guests without affecting Network Manager (and it's use of dnsmasq-base). Thus, these iptables rules are used to provide internet access to the KVM guests without actually attaching the eth0 or wlan0 interfaces directly to the OpenvSwitch.
These iptables rules are taken from the post here by Jean-Jacques Sarton. The switch sw1 is defined and built at bootup in a script "/home/gstanden/crt_ovs_sw1.sh" from the following file as shown below. This script is run from the Upstart job file at /etc/init/my-network-up.conf at boot time to configure the OpenvSwitch and related networking as shown below. The crt_ovs_sw1.sh script owes much to the reference previously mentioned here by Jean-Jacques Sarton. The work described by Jean-Jacquest was modified here for OpenvSwitch from the original (linux bridge) code.
UPDATE 2014-08-25 12:47 PM CDT: Some testing revealed there were some additional configuration changes and improvements that were needed to handle the case of (1) no internet connected interface available, and (2) the case of connection to corporate VPN. The new and improved "/home/gstanden/crt_ovs_sw1.sh" file is shown below. Changes were also made to programmatically set $EXTIF at KVM host boot time instead of using a hard-coded value.
gstanden@vmem1:~$ cat crt_ovs_sw1.sh
#!/bin/bash
# Requires use of Upstart Script /etc/init/my-network-up.conf to ensure interfaces are up before running.
tunctl -t s1
tunctl -t s2
tunctl -t s3
tunctl -t s4
tunctl -t s5
tunctl -t s6
ip link set s1 up
ip link set s2 up
ip link set s3 up
ip link set s4 up
ip link set s5 up
ip link set s6 up
ovs-vsctl add-br sw1
ovs-vsctl add-port sw1 s1
ovs-vsctl add-port sw1 s2
ovs-vsctl add-port sw1 s3
ovs-vsctl add-port sw1 s4
ovs-vsctl add-port sw1 s5
ovs-vsctl add-port sw1 s6
ip link set up dev sw1
ip addr add 10.207.39.1/24 dev sw1
ip route replace 10.207.39.0/24 dev sw1
# GLS 20140825 Get active external interface dynamically at boot. Tested & works with {wlan0, eth0, bnep0} on NM-manager Ubuntu 14.04.1 Desktop x86_64.
# GLS 20140825 Interface "bnep0" is Blackberry Z30 OS10 Bluetooth Tether.
### BEGIN Get Active EXTIF Dynamcially. ###
function GetInterface
{
ifconfig|egrep -B1 'inet addr'|egrep -A1 'wlan0|eth0|bnep0'|sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f1 -d':'
}
function GetIP
{
ifconfig|egrep -B1 'inet addr'|egrep -A1 'wlan0|eth0|bnep0'|sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f2 -d':'
}
### END Get Active EXTIF Dynamically. ###
echo ' IP: '$(GetIP)
echo 'Interface: '$(GetInterface)
INTIF="sw1"
EXTIF=$(GetInterface)
# EXTIF="wlan0"
echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F
# set forwarding and nat rules
iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
service isc-dhcp-server start
gstanden@vmem1:~$
The above script creates an OpenvSwitch that can be shown with the command "sudo ovs-vsctl show".
Note that this setup also has two storage networks, 10.207.40.x and 10.207.41.x, which connect the KVM Oracle guests to this KVM SCST iSCSI Linux SAN to provide true virtualized multipath via two additional OpenvSwitch switches. The scripts for those networks are similar to the above. No DNS resolution is used for the storage networks.
DNS is provided only for the server management network 10.207.39.x which is used to ssh to the servers, scp files between them, etc. The files used to build these OpenvSwitch switches are attached to this blog.
The "/home/gstanden/crt_ovs_sw1.sh" script is run by an Ubuntu Upstart script /etc/init/my-network-up.conf at bootup of the KVM host (a Lenovo W520 laptop in this case) as shown below. This script ensures that the network interfaces are up before the "/home/gstanden/crt_ovs_sw1.sh" script runs.
gstanden@vmem1:~$ cat /etc/init/my-network-up.conf
# 'my-network-up.conf' - My custom upstart events
#
# These are the scripts that run when a network appears.
description "My custom upstart events"
start on net-device-up # Start a daemon or run a script
stop on net-device-down # (Optional) Stop a daemon, scripts already self-terminate.
script
# Shell script go here, including optionally if/then and tests.
/home/gstanden/crt_ovs_sw1.sh 2>&1 > /home/gstanden/crt_ovs_sw1.log
/home/gstanden/crt_ovs_sw2.sh 2>&1 > /home/gstanden/crt_ovs_sw2.log
/home/gstanden/crt_ovs_sw3.sh 2>&1 > /home/gstanden/crt_ovs_sw3.log
end script
gstanden@vmem1:~$
The finished OpenvSwitch switches are shown below with the ovs-vsctl utility. OpenvSwitch "sw1" is the management network. The OpenvSwitch "sw2" and "sw3" provide multipath iSCSI storage networks. As mentioned above, these switches and related networking are all built at each bootup.
gstanden@vmem1:~$ sudo ovs-vsctl show
[sudo] password for gstanden:
2fc24710-34b5-4aa2-a32d-4e7bcb1afa1a
Bridge "sw2"
Port "sw2"
Interface "sw2"
type: internal
Port "t3"
Interface "t3"
Port "t1"
Interface "t1"
Port "t4"
Interface "t4"
Port "t5"
Interface "t5"
Port "t2"
Interface "t2"
Bridge "sw1"
Port "sw1"
Interface "sw1"
type: internal
Port "s5"
Interface "s5"
Port "s4"
Interface "s4"
Port "s3"
Interface "s3"
Port "s1"
Interface "s1"
Port "s2"
Interface "s2"
Bridge "sw3"
Port "w2"
Interface "w2"
Port "w3"
Interface "w3"
Port "w1"
Interface "w1"
Port "w4"
Interface "w4"
Port "sw3"
Interface "sw3"
type: internal
Port "w5"
Interface "w5"
ovs_version: "2.0.1"
gstanden@vmem1:~$
DHCP is provided to the KVM guest on the 10.207.39.x network by isc-dhcp-server. That configuration is shown here and here in another post on this site.
KVM Guest Master DNS Installation
First the DNS was installed in a KVM guest chosen among the patch of KVM guests (all on the same 10.207.39.x network) to be the DNS server. The "oracle651.vmem.org" server was chosen because it is already in use as the SCST iSCSI Linux SAN, and as such, must be started before all of the other KVM guests on this network to provide the iSCSI LUNs to the other servers (the other KVM guests are all Oracle database servers, which use ASM, and therefore need SCST to present multipathed iSCSI LUNs). Much of the configuration shown below is based on the guide at ostechnix.
Install bind9 in the KVM guest oracle651.vmem.org as shown below.
[root@oracle651 ~]# yum install bind* -y
The /etc/named.conf file is shown below, with bolded sections indicating parts that customized for this network and setup.
[root@oracle651 etc]# cat /etc/named.conf
//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
options {
listen-on port 53 { 127.0.0.1; 10.207.39.74; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query { localhost; 10.207.39.0/24; };
recursion yes;
notify yes;
dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;
/* Path to ISC DLV key */
bindkeys-file "/etc/named.iscdlv.key";
managed-keys-directory "/var/named/dynamic";
};
logging {
channel default_debug {
file "data/named.run";
severity dynamic;
};
};
zone "." IN {
type hint;
file "named.ca";
};
zone "vmem.org" IN {
type master;
file "fwd.vmem.org";
allow-update { none; };
};
zone "39.207.10.in-addr.arpa" IN {
type master;
file "rev.vmem.org";
allow-update {none; };
};
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
[root@oracle651 etc]#
The zone files used are shown below. First, here is the forward zone lookup file (fwd.vmem.org) as shown below. One thing to notice is that there are "A" records for both of the nameservers in this file, in addition to the "NS" records for each of the namerservers (master and slave). It is important to have these "A" records for the DNS servers in this setup, including for resolution of the slave DNS server.
[root@oracle651 named]# pwd
/var/named
[root@oracle651 named]# ls -lrt fwd.vmem.org
-rw-r--r-- 1 root root 634 Aug 20 16:11 fwd.vmem.org
[root@oracle651 named]# cat fwd.vmem.org
$TTL 86400
$ORIGIN vmem.org.
@ IN SOA oracle651.vmem.org. postmaster.vmem.org. (
201408202001 ;Serial
60 ;Refresh
1800 ;Retry
604800 ;Expire
86400 ;Minimum TTL
)
@ IN NS oracle651 ; Master DNS Server
@ IN NS vmem1 ; Slave DNS Server
oracle651 IN A 10.207.39.74 ; KVM guest SCST iSCSI Linux SAN
oracle631 IN A 10.207.39.72 ; KVM guest Oracle Dataguard Primary (512e)
oracle632 IN A 10.207.39.76 ; KVM guest Oracle Dataguard Standby (4K )
oracle635 IN A 10.207.39.90 ; DNS master-slave propagation test IP
vmem1 IN A 10.207.39.1 ; KVM host (Ubuntu 14.04 laptop)
[root@oracle651 named]#
Next is shown the reverse lookups file as shown below.
[root@oracle651 named]# pwd
/var/named
[root@oracle651 named]# ls -lrt rev.vmem.org
-rw-r--r-- 1 root root 467 Aug 20 16:11 rev.vmem.org
[root@oracle651 named]# cat rev.vmem.org
$TTL 86400
@ IN SOA oracle651.vmem.org. postmaster.vmem.org. (
201408201611 ;Serial
3600 ;Refresh
1800 ;Retry
604800 ;Expire
86400 ;Minimum TTL
)
@ IN NS oracle651.vmem.org.
@ IN NS vmem1.vmem.org.
oracle651 A 10.207.39.74
vmem1 A 10.207.39.1
39.207.10.in-addr.arpa IN NS oracle651.vmem.org. ; Master DNS
39.207.10.in-addr.arpa IN NS vmem1.vmem.org. ; Slave DNS
1 IN PTR vmem1.vmem.org. ; KVM host (Ubuntu 14.04 laptop)
72 IN PTR oracle631.vmem.org. ; KVM guest Oracle Dataguard Primary
74 IN PTR oracle651.vmem.org. ; KVM guest Oracle Dataguard Standby
76 IN PTR oracle632.vmem.org. ; KVM guest SCST iSCSI Linux SAN
90 IN PTR oracle635.vmem.org. ; DNS master-slave propagation test IP
[root@oracle651 named]#
Configure Bind Service
Start the bind service as shown below and use set bind to start on boot as shown below.
[root@oracle651 named] # service named start
[root@oracle651 named] # chkconfig named on
In this setup, iptables is set to off as shown below, so there is no need to configure iptables for this setup.
[root@oracle651 named] # chkconfig iptables off
[root@oracle651 named] # service iptables stop
However, if using iptables, be sure to also do the steps in the next section "set iptables rules".Te
Set Iptables Rules
My servers have been set to iptables off, but if iptables is on, use the guide here at ostechnix to configure rules to allow communication to the DNS server. Those rules are reproduced here for convenience as shown below. The were not used on this implementation because as mentioned, this setup has iptables off.
Note that a secure implementation would not likely allow simply turning iptables off, and the rules shown below would need to be implemented.
[root@masterdns ~]# vi /etc/sysconfig/iptables # Firewall configuration written by system-config-firewall # Manual customization of this file is not recommended. *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -p udp -m state --state NEW --dport 53 -j ACCEPT -A INPUT -p tcp -m state --state NEW --dport 53 -j ACCEPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT
Test Master DNS Server
Test the Master DNS service as described here at ostechnix. The tests are reproduced here for convenience.
Check DNS Configuration File
Check the /etc/named.conf file and the /etc/named.rfc1912.zones files as shown below. The output shown below is correct expected output.
[root@oracle651 named]# named-checkconf /etc/named.conf
[root@oracle651 named]#
[root@oracle651 named]# named-checkconf /etc/named.rfc1912.zones
[root@oracle651 named]#
Check Zone Files
Check the zone files as shown below. The output shown is the correct expected output. Note that there is some disagreement in the literature about the syntax for the reverse zone check. This reverse zone file passes both forms of the reverse zone check syntax with no errors, as shown below.
Here is the results for the forward zone file check, as shown below.
[root@oracle651 named]# named-checkzone vmem.org /var/named/fwd.vmem.org
zone vmem.org/IN: loaded serial 3839706385
OK
[root@oracle651 named]#
Here is the result for the first published syntax of the reverse zone check as shown below.
[root@oracle651 named]# named-checkzone vmem.org /var/named/rev.vmem.org
zone vmem.org/IN: loaded serial 3839705995
OK
[root@oracle651 named]#
Here is the result of the alternate published syntax of the reverse zone check as shown below.
[root@oracle651 named]# named-checkzone 39.207.10.in-addr.arpa rev.vmem.org
zone 39.207.10.in-addr.arpa/IN: loaded serial 3839705995
OK
[root@oracle651 named]#
Test DNS Servers
Here are the results of the DNS server tests, dig by master DNS hostname, as shown below.
[root@oracle651 named]# dig oracle651.vmem.org
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> oracle651.vmem.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60922
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1
;; QUESTION SECTION:
;oracle651.vmem.org. IN A
;; ANSWER SECTION:
oracle651.vmem.org. 86400 IN A 10.207.39.74
;; AUTHORITY SECTION:
vmem.org. 86400 IN NS vmem1.vmem.org.
vmem.org. 86400 IN NS oracle651.vmem.org.
;; ADDITIONAL SECTION:
vmem1.vmem.org. 86400 IN A 10.207.39.1
;; Query time: 0 msec
;; SERVER: 10.207.39.74#53(10.207.39.74)
;; WHEN: Wed Aug 20 23:10:11 2014
;; MSG SIZE rcvd: 102
[root@oracle651 named]#
Here are the results of the DNS server tests, dig by master DNS IP address, as shown below.
[root@oracle651 named]# dig -x 10.207.39.74
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> -x 10.207.39.74
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52536
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2
;; QUESTION SECTION:
;74.39.207.10.in-addr.arpa. IN PTR
;; ANSWER SECTION:
74.39.207.10.in-addr.arpa. 86400 IN PTR oracle651.vmem.org.
;; AUTHORITY SECTION:
39.207.10.in-addr.arpa. 86400 IN NS vmem1.vmem.org.
39.207.10.in-addr.arpa. 86400 IN NS oracle651.vmem.org.
;; ADDITIONAL SECTION:
oracle651.vmem.org. 86400 IN A 10.207.39.74
vmem1.vmem.org. 86400 IN A 10.207.39.1
;; Query time: 1 msec
;; SERVER: 10.207.39.74#53(10.207.39.74)
;; WHEN: Wed Aug 20 23:13:27 2014
;; MSG SIZE rcvd: 141
[root@oracle651 named]#
Master DNS Server nslookup
Do an nslookup of the master DNS server as shown below.
[root@oracle651 named]# nslookup oracle651.vmem.org
Server: 10.207.39.74
Address: 10.207.39.74#53
Name: oracle651.vmem.org
Address: 10.207.39.74
[root@oracle651 named]#
KVM Host Slave DNS Installation
Install the required packages on the Ubuntu 14.04 KVM host using apt-get as shown below. Note that "haveged" package is an entropy generator which is not absolutely necessary but which is good to have and helps speed things up when certificates are being generated.
gstanden@vmem1:~$ sudo apt-get install bind9 bind9utils bind9-doc haveged
Configure Slave DNS Server
The configuration on Ubuntu differs from the configuration of bind9 on RHEL 6 / CentOs 6 / OEL 6. The files are in different directory locations, and named.conf is not edited directly. Instead, the named.conf.options file is edited, as shown below. Key sections to be set are in bold. Some revisions to the original /etc/bind/named.conf.options file where made 2014-08-25 which are explained in the comments to this blog post where I make a comment to my own blog.
Note that more work is needed to make this nameserver comply with security best practices. This configuration file addresses all lookup and connectivity needs, but has not yet been tweaked for security best practice compliance.
gstanden@vmem1:~$ cd /etc/bind
gstanden@vmem1:/etc/bind$ ls -lrt named.conf.options
-rw-r--r-- 1 root bind 964 Aug 20 16:09 named.conf.options
gstanden@vmem1:/etc/bind$ cat named.conf.options
options {
directory "/var/cache/bind";
// If there is a firewall between you and nameservers you want
// to talk to, you may need to fix the firewall to allow multiple
// ports to talk. See http://www.kb.cert.org/vuls/id/800113
// If your ISP provided one or more IP addresses for stable
// nameservers, you probably want to use them as forwarders.
// Uncomment the following block, and insert the addresses replacing
// the all-0's placeholder.
// GLS 20140825 forwarders uncommented and set to 127.0.1.1 (NM-controlled dnsmasq-base DNS) for lookups on NM-controlled VPN.
// GLS 20140825 Company VPN connectivity was broken unless forwarders was set to 127.0.1.1 (NM-controlled dnsmasq-base DNS).
forwarders {
127.0.1.1;
};
//========================================================================
// If BIND logs error messages about the root key being expired,
// you will need to update your keys. See https://www.isc.org/bind-keys
//========================================================================
dnssec-validation auto;
recursion yes;
allow-transfer { none; };
allow-notify { 10.207.39.74; };
auth-nxdomain no; # conform to RFC1035
listen-on-v6 { any; };
};
gstanden@vmem1:/etc/bind$
The /etc/bind/named.conf file is shown below for reference only. This file is not edited directly. It is left at the defaults.
gstanden@vmem1:/etc/bind$ cat named.conf
// This is the primary configuration file for the BIND DNS server named.
//
// Please read /usr/share/doc/bind9/README.Debian.gz for information on the
// structure of BIND configuration files in Debian, *BEFORE* you customize
// this configuration file.
//
// If you are just adding zones, please do that in /etc/bind/named.conf.local
include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
gstanden@vmem1:/etc/bind$
The /etc/bind/named.conf.local file is edited to include the zone files as shown below, relevant added sections shown in bold.
gstanden@vmem1:/etc/bind$ cat named.conf.local
//
// Do any local configuration here
//
// Consider adding the 1918 zones here, if they are not used in your
// organization
//include "/etc/bind/zones.rfc1918";
zone "vmem.org" {
type slave;
masters { 10.207.39.74; };
file "fwd.vmem.org";
};
zone "39.207.10.in-addr.arpa" {
type slave;
masters { 10.207.39.74; };
file "rev.vmem.org";
};
gstanden@vmem1:/etc/bind$
Verify File /etc/hosts
The /etc/hosts file should look similar to that shown below. This is the standard auto-generated /etc/hosts file which is the format it will have after a fresh installation of Ubuntu Destkop 14.04 as shown below.
gstanden@vmem1:/etc/bind$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 vmem1
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
gstanden@vmem1:/etc/bind$
Edit File /etc/nsswitch.conf
The /etc/nsswitch.conf file should be edited on the hosts line to use DNS as the first lookup method, and files and other methods as secondary methods, as shown below, line to be edited in bold. Notice that on the "hosts" line that "dns" is the first entry. This is correct. The default is for "files" to be the first entry. This DNS method is using DNS and not /etc/hosts so "dns" is listed as the first method, as shown below.
gstanden@vmem1:/etc/bind$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: compat
group: compat
shadow: compat
hosts: dns files mdns4_minimal [NOTFOUND=return]
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nis
gstanden@vmem1:/etc/bind$
Verify File /etc/NetworkManager/NetworkManager.conf
Verify that the file /etc/NetworkManager/NetworkManager.conf uses "dnsmasq" as the dns option. This is the default setting for Network Manager. This default setting will be kept unchanged. Here it is necessary only to verify it is stil set to the default, as shown below.
gstanden@vmem1:/etc/bind$ cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile,ofono
dns=dnsmasq
[ifupdown]
managed=false
gstanden@vmem1:/etc/bind$
Edit File /etc/network/interfaces
Editing of this file will in turn set necessary attributes of the /etc/resolv.conf file. The required entries are shown below. The configuration of this file shown below is for DHCP addressing of the KVM host, and the "dns-domain" line will add "search vmem.org" to the /etc/resolv.conf file (as shown in the next step). The changes will take effect when the KVM host is rebooted or networking is restarted. If a static IP is desired for the KVM host, additional configuration will be necessary in this file. See additional references for static IP addressing on Ubuntu 14.04 on the web in that case.
UPDATE: 2014-08-25 "dns-nameserver 127.0.0.1" was added to /etc/network/interfaces to handle the case of the KVM host (Lenovo W520 laptop) starting up with NO internet-connected interface available (eth0, wlan0, bnep0) all disconnected. It was found that under that circumstance /etc/resolv.conf would get set to "nameserver 127.0.1.1" which is the NM-controlled dnsmasq-base DNS server. This caused lookups of KVM guests on my private network to break. By setting "dns-nameserver 127.0.0.1" explicitly, resolution was always available for my private network.
gstanden@vmem1:~$ cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback
dns-domain vmem.org
dns-nameserver 127.0.0.1
gstanden@vmem1:~$
That this resolution succeeds is understood clearly from the output of netstat and also from this post here at Stephane Graber's blog which explains that 127.0.0.1 always resolves to dnsmasq. However, it appears that is true except when there is a bind9 DNS running, in which case the "named" service of bind9 runs on 127.0.0.1, as shown below from the netstat output. Therefore, this configuration ensures that lookups for the private network will not fail, and also that internet lookups will be forwarded to 127.0.1.1 (dnsmasq) because in the /etc/bind/named.conf.options recursion is set to yes so it finds dnsmasq and uses it for external resolutions, but preferentially hits named at 127.0.0.1 first to get the resolution for the private network. The output below helps to show the results from some different configurations that might be possible.
### Settings of selected files for Configuration Tests ###
gstanden@vmem1:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
search vmem.org
gstanden@vmem1:~$ cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback
dns-domain vmem.org
### Configuration 1 Test ###
gstanden@vmem1:~$ cat /etc/bind/named.conf.options | egrep 'recursion|listen-on'
recursion no;
listen-on-v6 { no; };
gstanden@vmem1:~$ nslookup oracle631
;; Got recursion not available from 127.0.0.1, trying next server
;; connection timed out; no servers could be reached
gstanden@vmem1:~$ sudo netstat -ulnp | grep :53
udp 0 0 0.0.0.0:5353 0.0.0.0:* 787/avahi-daemon
udp 0 0 192.168.122.1:53 0.0.0.0:* 4452/named
udp 0 0 10.207.41.1:53 0.0.0.0:* 4452/named
udp 0 0 10.207.40.1:53 0.0.0.0:* 4452/named
udp 0 0 10.207.39.1:53 0.0.0.0:* 4452/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 4452/named
udp 0 0 127.0.1.1:53 0.0.0.0:* 3039/dnsmasq
udp 0 0 192.168.122.1:53 0.0.0.0:* 2995/dnsmasq
udp6 0 0 :::5353 :::*
### End Configuration 1 Test ###
### Configuration 2 Test ###
gstanden@vmem1:~$ sudo vi /etc/bind/named.conf.options
gstanden@vmem1:~$ cat /etc/bind/named.conf.options | egrep 'recursion|listen-on'
recursion no;
listen-on-v6 { any; };
gstanden@vmem1:~$ sudo service bind9 restart
* Stopping domain name service... bind9 waiting for pid 4452 to die [ OK ]
* Starting domain name service...bind9 [ OK ]
gstanden@vmem1:~$ sudo netstat -ulnp | grep :53
udp 0 0 0.0.0.0:5353 0.0.0.0:* 787/avahi-daemon
udp 0 0 192.168.122.1:53 0.0.0.0:* 4768/named
udp 0 0 10.207.41.1:53 0.0.0.0:* 4768/named
udp 0 0 10.207.40.1:53 0.0.0.0:* 4768/named
udp 0 0 10.207.39.1:53 0.0.0.0:* 4768/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 4768/named
udp 0 0 127.0.1.1:53 0.0.0.0:* 3039/dnsmasq
udp 0 0 192.168.122.1:53 0.0.0.0:* 2995/dnsmasq
udp6 0 0 :::5353 :::* 787/avahi-daemon
udp6 0 0 :::53 :::* 4768/named <-- "listen-on-v6 { any; };
gstanden@vmem1:~$ nslookup oracle631
;; Got recursion not available from 127.0.0.1, trying next server
Server: ::1
Address: ::1#53 <-- because of "listen-on-v6 {any; }" in /etc/bind/named.conf.options file;
Name: oracle631.vmem.org
Address: 10.207.39.72 <--lookup succeeds
### End Configuration 2 Test ###
### Configuration 3 Test ###
gstanden@vmem1:~$ sudo vi /etc/bind/named.conf.options
gstanden@vmem1:~$ cat /etc/bind/named.conf.options | egrep 'recursion|listen-on'
recursion yes;
listen-on-v6 { any; };
gstanden@vmem1:~$ sudo service bind9 restart
* Stopping domain name service... bind9 waiting for pid 4694 to die [ OK ]
* Starting domain name service... bind9 [ OK ]
gstanden@vmem1:~$ nslookup oracle631 <-- Clean error-free lookup achieved.
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: oracle631.vmem.org
Address: 10.207.39.72
gstanden@vmem1:~$
Based on this testing, the configuration of "recursion yes" and "listen-on-v6 { any; };" is used.
Verify File /etc/resolv.conf
Verify that the /etc/resolv.conf file is configured as shown below. Note that this file is auto-generated by Network Manager, and will change automatically, for example when logging on to a VPN etc. But in the default case, this file should have only the nameserver shown, namely 127.0.0.1, and should show, in this case, the vmem.org domain. The file also needs to show the "search vmem.org" line as well. As mentioned, both of these are set in the /etc/network/interfaces file.
gstanden@vmem1:/etc/bind$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search vmem.org
gstanden@vmem1:/etc/bind$
Start bind9 Service
Start the bind9 service as shown below.
gstanden@vmem1:/etc/bind$ sudo service bind9 start
* Starting domain name service... bind9
[ OK ]
gstanden@vmem1:/etc/bind$
Verify Zone File Transfer to Slave DNS
Upon start of the slave DNS server, the zone files should have been transferred to the slave DNS server automatically, as shown below. These files are binary files so it is not possible to "cat" them or otherwise look at them. The best way forward is to proceed to testing the lookups and DNS server.
gstanden@vmem1:/etc/bind$ cd /var/cache/bind
gstanden@vmem1:/var/cache/bind$ ls -lrt
total 12
-rw-r--r-- 1 bind bind 720 Aug 20 10:21 managed-keys.bind
-rw-r--r-- 1 bind bind 555 Aug 20 23:14 rev.vmem.org
-rw-r--r-- 1 bind bind 413 Aug 20 23:55 fwd.vmem.org
gstanden@vmem1:/var/cache/bind$
Test Slave DNS Server
Check DNS Configuration Files
Check the DNS configuration files as shown below.
gstanden@vmem1:~$ named-checkconf /etc/bind/named.conf
gstanden@vmem1:~$
gstanden@vmem1:~$ named-checkconf /etc/bind/zones.rfc1918
gstanden@vmem1:~$
Check DNS Zone Files
As mentioned above, the zone files on the Ubuntu slave DNS server are in binary format and cannot be checked directly using methods such as "cat". They are located as shown below. Modification times can be checked.
gstanden@vmem1:/etc/bind$ cd /var/cache/bind
gstanden@vmem1:/var/cache/bind$ ls -lrt
total 12
-rw-r--r-- 1 bind bind 720 Aug 20 10:21 managed-keys.bind
-rw-r--r-- 1 bind bind 555 Aug 20 23:14 rev.vmem.org
-rw-r--r-- 1 bind bind 413 Aug 20 23:55 fwd.vmem.org
gstanden@vmem1:/var/cache/bind$
Test DNS Servers
Run dig test on the slave DNS FQDN from slave DNS server as shown below.
gstanden@vmem1:~$ dig vmem1.vmem.org
; <<>> DiG 9.9.5-3-Ubuntu <<>> vmem1.vmem.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44494
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;vmem1.vmem.org. IN A
;; ANSWER SECTION:
vmem1.vmem.org. 86400 IN A 10.207.39.1
;; AUTHORITY SECTION:
vmem.org. 86400 IN NS oracle651.vmem.org.
vmem.org. 86400 IN NS vmem1.vmem.org.
;; ADDITIONAL SECTION:
oracle651.vmem.org. 86400 IN A 10.207.39.74
;; Query time: 2 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Aug 22 08:13:18 CDT 2014
;; MSG SIZE rcvd: 113
gstanden@vmem1:~$
Run dig test on the master DNS FQDN from slave DNS server as shown below.
gstanden@vmem1:~$ dig oracle651.vmem.org
; <<>> DiG 9.9.5-3-Ubuntu <<>> oracle651.vmem.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55870
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;oracle651.vmem.org. IN A
;; ANSWER SECTION:
oracle651.vmem.org. 86400 IN A 10.207.39.74
;; AUTHORITY SECTION:
vmem.org. 86400 IN NS oracle651.vmem.org.
vmem.org. 86400 IN NS vmem1.vmem.org.
;; ADDITIONAL SECTION:
vmem1.vmem.org. 86400 IN A 10.207.39.1
;; Query time: 1 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Aug 22 08:13:57 CDT 2014
;; MSG SIZE rcvd: 113
gstanden@vmem1:~$
Run dig test on master DNS IP address from slave DNS server as shown below.
gstanden@vmem1:~$ dig -x 10.207.39.74
; <<>> DiG 9.9.5-3-Ubuntu <<>> -x 10.207.39.74
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23435
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;74.39.207.10.in-addr.arpa. IN PTR
;; ANSWER SECTION:
74.39.207.10.in-addr.arpa. 86400 IN PTR oracle651.vmem.org.
;; AUTHORITY SECTION:
39.207.10.in-addr.arpa. 86400 IN NS vmem1.vmem.org.
39.207.10.in-addr.arpa. 86400 IN NS oracle651.vmem.org.
;; ADDITIONAL SECTION:
vmem1.vmem.org. 86400 IN A 10.207.39.1
oracle651.vmem.org. 86400 IN A 10.207.39.74
;; Query time: 1 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Aug 22 08:21:35 CDT 2014
;; MSG SIZE rcvd: 152
gstanden@vmem1:~$
Run dig test on slave DNS IP address from slave DNS server as shown below.
gstanden@vmem1:~$ dig -x 10.207.39.1
; <<>> DiG 9.9.5-3-Ubuntu <<>> -x 10.207.39.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41789
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;1.39.207.10.in-addr.arpa. IN PTR
;; ANSWER SECTION:
1.39.207.10.in-addr.arpa. 86400 IN PTR vmem1.vmem.org.a.39.207.10.in-addr.arpa.
;; AUTHORITY SECTION:
39.207.10.in-addr.arpa. 86400 IN NS vmem1.vmem.org.
39.207.10.in-addr.arpa. 86400 IN NS oracle651.vmem.org.
;; ADDITIONAL SECTION:
vmem1.vmem.org. 86400 IN A 10.207.39.1
oracle651.vmem.org. 86400 IN A 10.207.39.74
;; Query time: 2 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Aug 22 08:22:10 CDT 2014
;; MSG SIZE rcvd: 168
gstanden@vmem1:~$
Slave DNS Server Lookup
Perform an nslookup of the servers in both forward and reverse ah shown below, testing both short name and FQDN as shown below.
gstanden@vmem1:/var/cache/bind$ nslookup oracle651
Server: 127.0.1.1
Address: 127.0.1.1#53
Non-authoritative answer:
Name: oracle651.vmem.org
Address: 46.30.212.99
gstanden@vmem1:/var/cache/bind$ nslookup oracle631
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle631.vmem.org
Address: 10.207.39.72
gstanden@vmem1:/var/cache/bind$ nslookup oracle632
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle632.vmem.org
Address: 10.207.39.76
gstanden@vmem1:/var/cache/bind$ nslookup oracle651
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle651.vmem.org
Address: 10.207.39.74
gstanden@vmem1:/var/cache/bind$ nslookup 10.207.39.72
Server: 127.0.1.1
Address: 127.0.1.1#53
72.39.207.10.in-addr.arpa name = oracle631.vmem.org.
gstanden@vmem1:/var/cache/bind$ nslookup 10.207.39.74
Server: 127.0.1.1
Address: 127.0.1.1#53
74.39.207.10.in-addr.arpa name = oracle651.vmem.org.
gstanden@vmem1:/var/cache/bind$ nslookup 10.207.39.76
Server: 127.0.1.1
Address: 127.0.1.1#53
76.39.207.10.in-addr.arpa name = oracle632.vmem.org.
gstanden@vmem1:/var/cache/bind$ nslookup 10.207.39.90
Server: 127.0.1.1
Address: 127.0.1.1#53
90.39.207.10.in-addr.arpa name = oracle635.vmem.org.
gstanden@vmem1:/var/cache/bind$ nslookup oracle635
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle635.vmem.org
Address: 10.207.39.90
gstanden@vmem1:/var/cache/bind$ nslookup oracle635.vmem.org
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle635.vmem.org
Address: 10.207.39.90
gstanden@vmem1:/var/cache/bind$
Shutdown the KVM guest oracle651 master DNS server and retry the DNS lookups above to be sure that lookups are using the slave DNS as shown below. The virsh command shows that oracle651 (oracle651.vmem.org) is down so it cannot possibly be returning DNS lookups.
gstanden@vmem1:/var/cache/bind$ virsh -c qemu:///system list
Id Name State
----------------------------------------------------
gstanden@vmem1:/var/cache/bind$ nslookup oracle631
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle631.vmem.org
Address: 10.207.39.72
gstanden@vmem1:/var/cache/bind$ nslookup oracle632
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle632.vmem.org
Address: 10.207.39.76
gstanden@vmem1:/var/cache/bind$ nslookup oracle635
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle635.vmem.org
Address: 10.207.39.90
gstanden@vmem1:/var/cache/bind$ nslookup oracle651
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle651.vmem.org
Address: 10.207.39.74
gstanden@vmem1:/var/cache/bind$ nslookup oracle631.vmem.org
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle631.vmem.org
Address: 10.207.39.72
gstanden@vmem1:/var/cache/bind$ nslookup 10.207.39.72
Server: 127.0.1.1
Address: 127.0.1.1#53
72.39.207.10.in-addr.arpa name = oracle631.vmem.org.
gstanden@vmem1:/var/cache/bind$ nslookup 10.207.39.74
Server: 127.0.1.1
Address: 127.0.1.1#53
74.39.207.10.in-addr.arpa name = oracle651.vmem.org.
gstanden@vmem1:/var/cache/bind$ nslookup 10.207.39.76
Server: 127.0.1.1
Address: 127.0.1.1#53
76.39.207.10.in-addr.arpa name = oracle632.vmem.org.
gstanden@vmem1:/var/cache/bind$
Test Zone File Propagation
Start oracle651 primary nameserver back up and make an edit to the forward and reverse lookup files, increment the "Serial" and restart bind9 on the master DNS after making these changes to the zone files on the master DNS and verify on the slave DNS that lookup files were also updated (updating the Serial should initiate and immediate push to the slave DNS of the changes) as shown below.
[root@oracle651 named]# pwd
/var/named
[root@oracle651 named]#
[root@oracle651 named]# ls -lrt *.vmem.org
-rw-r--r-- 1 root root 657 Aug 21 00:08 fwd.vmem.org
-rw-r--r-- 1 root root 862 Aug 21 00:09 rev.vmem.org
[root@oracle651 named]# vi fwd.vmem.org
[root@oracle651 named]# cat fwd.vmem.org
$TTL 86400
$ORIGIN vmem.org.
@ IN SOA oracle651.vmem.org. postmaster.vmem.org. (
201408210008 ;Serial
60 ;Refresh
1800 ;Retry
604800 ;Expire
86400 ;Minimum TTL
)
@ IN NS oracle651 ; Master DNS Server
@ IN NS vmem1 ; Slave DNS Server
oracle651 IN A 10.207.39.74 ; KVM guest SCST iSCSI Linux SAN
oracle631 IN A 10.207.39.72 ; KVM guest Oracle Dataguard Primary (512e)
oracle632 IN A 10.207.39.76 ; KVM guest Oracle Dataguard Standby (4K )
oracle635 IN A 10.207.39.90 ; DNS master-slave propagation test IP
vmem1 IN A 10.207.39.1 ; KVM host (Ubuntu 14.04 laptop)
[root@oracle651 named]# vi fwd.vmem.org
[root@oracle651 named]# cat fwd.vmem.org
$TTL 86400
$ORIGIN vmem.org.
@ IN SOA oracle651.vmem.org. postmaster.vmem.org. (
201408220845 ;Serial
60 ;Refresh
1800 ;Retry
604800 ;Expire
86400 ;Minimum TTL
)
@ IN NS oracle651 ; Master DNS Server
@ IN NS vmem1 ; Slave DNS Server
oracle651 IN A 10.207.39.74 ; KVM guest SCST iSCSI Linux SAN
oracle631 IN A 10.207.39.72 ; KVM guest Oracle Dataguard Primary (512e)
oracle632 IN A 10.207.39.76 ; KVM guest Oracle Dataguard Standby (4K )
oracle635 IN A 10.207.39.95 ; DNS master-slave propagation test IP
vmem1 IN A 10.207.39.1 ; KVM host (Ubuntu 14.04 laptop)
[root@oracle651 named]# cat rev.vmem.org
$TTL 86400
@ IN SOA oracle651.vmem.org. postmaster.vmem.org. (
201408210008 ;Serial
3600 ;Refresh
1800 ;Retry
604800 ;Expire
86400 ;Minimum TTL
)
@ IN NS oracle651.vmem.org.
@ IN NS vmem1.vmem.org.
oracle651 A 10.207.39.74 ; Required A record for NS
vmem1 A 10.207.39.1 ; Required A record for NS
39.207.10.in-addr.arpa IN NS oracle651.vmem.org. ; Master DNS
39.207.10.in-addr.arpa IN NS vmem1.vmem.org. ; Slave DNS
1 IN PTR vmem1.vmem.org.a ; KVM host
72 IN PTR oracle631.vmem.org. ; KVM guest Oracle Dataguard Primary
74 IN PTR oracle651.vmem.org. ; KVM guest Oracle Dataguard Standby
76 IN PTR oracle632.vmem.org. ; KVM guest iSCSI Linux SAN
90 IN PTR oracle635.vmem.org. ; KVM master-slave DNS propagation test IP
[root@oracle651 named]# vi rev.vmem.org
[root@oracle651 named]# cat rev.vmem.org
$TTL 86400
@ IN SOA oracle651.vmem.org. postmaster.vmem.org. (
201408220845 ;Serial
3600 ;Refresh
1800 ;Retry
604800 ;Expire
86400 ;Minimum TTL
)
@ IN NS oracle651.vmem.org.
@ IN NS vmem1.vmem.org.
oracle651 A 10.207.39.74 ; Required A record for NS
vmem1 A 10.207.39.1 ; Required A record for NS
39.207.10.in-addr.arpa IN NS oracle651.vmem.org. ; Master DNS
39.207.10.in-addr.arpa IN NS vmem1.vmem.org. ; Slave DNS
1 IN PTR vmem1.vmem.org.a ; KVM host
72 IN PTR oracle631.vmem.org. ; KVM guest Oracle Dataguard Primary
74 IN PTR oracle651.vmem.org. ; KVM guest Oracle Dataguard Standby
76 IN PTR oracle632.vmem.org. ; KVM guest iSCSI Linux SAN
95 IN PTR oracle635.vmem.org. ; KVM master-slave DNS propagation test IP
[root@oracle651 named]#
[root@oracle651 named]# ls -lrt *.vmem.org
-rw-r--r-- 1 root root 657 Aug 22 08:46 fwd.vmem.org
-rw-r--r-- 1 root root 862 Aug 22 08:47 rev.vmem.org
[root@oracle651 named]#
Restart "named" DNS service on Master DNS server as shown below.
[root@oracle651 named]# service named restart
Stopping named: . [ OK ]
Starting named: [ OK ]
[root@oracle651 named]#
Check modification date of zone files on slave DNS server as shown below (before and after). Zone files have been updated automatically.
gstanden@vmem1:/var/cache/bind$ ls -lrt
total 12
-rw-r--r-- 1 bind bind 720 Aug 21 11:53 managed-keys.bind
-rw-r--r-- 1 bind bind 803 Aug 21 22:19 rev.vmem.org
-rw-r--r-- 1 bind bind 413 Aug 21 22:36 fwd.vmem.org
gstanden@vmem1:/var/cache/bind$ ls -lrt
total 12
-rw-r--r-- 1 bind bind 720 Aug 21 11:53 managed-keys.bind
-rw-r--r-- 1 bind bind 803 Aug 22 09:02 rev.vmem.org
-rw-r--r-- 1 bind bind 413 Aug 22 09:02 fwd.vmem.org
gstanden@vmem1:/var/cache/bind$
Verify correct updated IP address is in use by slave and master DNS servers using nslookup with specification of which DNS server to use.
gstanden@vmem1:/var/cache/bind$ nslookup oracle635.vmem.org vmem1.vmem.org
Server: vmem1.vmem.org
Address: 10.207.39.1#53
Name: oracle635.vmem.org
Address: 10.207.39.95
gstanden@vmem1:/var/cache/bind$ nslookup oracle635.vmem.org oracle651.vmem.org
Server: oracle651.vmem.org
Address: 10.207.39.74#53
Name: oracle635.vmem.org
Address: 10.207.39.95
gstanden@vmem1:/var/cache/bind$
Configure Private Network on NM-dnsmasq
It had been observed that sometimes, i.e. "intermittently and unpredictably", the DNS resolution on the KVM host Ubuntu 14.04.1 laptop would sometimes return a "non-authoritative" DNS lookup of the non-local "vmem.org" rather than my local "vmem.org" running on the laptop. This made the internet resolution on the laptop for my VMs occassionally unreliable because it was using dnsmasq internet resolution instead of the local Authoritative (bind9) DNS.
Here are example of how the issue manifested itself as shown below.
gstanden@vmem1:~$ nslookup oracle651
Server: 127.0.1.1
Address: 127.0.1.1#53
Non-authoritative answer:
Name: oracle651.vmem.org
Address: 46.30.212.99
gstanden@vmem1:~$ nslookup oracle651
Server: 127.0.1.1
Address: 127.0.1.1#53
Name: oracle651.vmem.org
Address: 10.207.39.74
gstanden@vmem1:~$
It might be guessed that this is because there are two DNS servers running on the Ubuntu laptop, (1) NM-dnsmasq (dnsmasq-base) and (2) bind9, and that guess would seem to be correct, because if bind9 is stopped, the resolution of "oracle651" will ALWAYS be "46.30.212.99". Once bind9 is restarted, it will sometimes return "46.30.212.99" and sometimes return "10.207.39.74" (the second one is of course the desired "internal network" ip address).
Using netstat it can be seen that both NM-dnsmasq and bind9 ("named") are both listening on port 53, so there is some sort of random effect at work where sometimes "named" satisfies the lookup request, and sometimes "dnsmasq". Sometimes the lookup was not using the desired DNS bind9 nameserver.
gstanden@vmem1:~$ sudo netstat -ulnp | grep :53
[sudo] password for gstanden:
udp 0 0 0.0.0.0:5353 0.0.0.0:* 769/avahi-daemon: r
udp 0 0 192.168.122.1:53 0.0.0.0:* 1514/named
udp 0 0 10.207.41.1:53 0.0.0.0:* 1514/named
udp 0 0 10.207.40.1:53 0.0.0.0:* 1514/named
udp 0 0 10.207.39.1:53 0.0.0.0:* 1514/named <--Want lookup to use this everytime !
udp 0 0 192.168.1.12:53 0.0.0.0:* 1514/named
udp 0 0 127.0.1.1:53 0.0.0.0:* 3331/dnsmasq
udp 0 0 192.168.122.1:53 0.0.0.0:* 2813/dnsmasq
udp 0 0 127.0.0.1:53 0.0.0.0:* 1514/named
udp6 0 0 :::5353 :::* 769/avahi-daemon: r
udp6 0 0 :::53 :::* 1514/named
gstanden@vmem1:~$
It turns out this can be fixed by using a configuration setting on dnsmasq so that "named" (bind9) local authoritative DNS will ALWAYS return the nslookup request and indeed any type of similar lookup (dig, ssh, etc.) on "vmem.org" local KVM guests. The required fix needed was found in these posts thanks to Sokratis Galiatsis at his Techie in IT blog here, and also at the Dnsmasq setup page at thekelleys.org.uk page in the "Using Special Servers" subsection. Below is the configuration file that was needed to fix this problem and force the lookup for "vmem.org" and "10.207.39.x" to always go to bind9 "named" listening on 10.207.39.1:53 as shown below and to NOT use NM-dnsmasq. The filed called "local" may not exist by default. If not create it and add the required private network lookup configuration to tell dnsmasq to defer this lookup to the authoritative local nameserver for the specific domain and network, in this case "vmem.org" and "10.207.39.x" as shown below.
gstanden@vmem1:~$ sudo ls -lrt /etc/NetworkManager/dnsmasq.d/local
-rw-r--r-- 1 root root 72 Aug 21 23:51 /etc/NetworkManager/dnsmasq.d/local
gstanden@vmem1:~$ sudo cat /etc/NetworkManager/dnsmasq.d/local
server=/vmem.org/10.207.39.1
server=/39.207.10.in-addr.arpa/10.207.39.1
gstanden@vmem1:~$
Restart NetworkManager or just reboot the laptop. Now the lookups will ALWAYS use bind9 and give the correct desired LOCAL bind9 lookup!