Dynamic config with LUA
Setup VM
========
Login with root user
$ ssh root@<ip>
Add user 'cloud'
$ sudo bash
$ adduser cloud
$ passwd cloud
Password : cloud
Add user cloud to sudo list
$ sudo echo "cloud ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers
exit
Login with cloud user
$ ssh cloud@<ip>
Stop firewall for Centos-7
$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld
$ sudo chkconfig firewalld off
$ sudo service status firewalld
Disable SElinux
$ sudo setenforce 0
$ getenforce
Compiling HAProxy with Lua
==============================
Install LUA 5.3.1
HAProxy 1.6\1.7 requires Lua 5.3. Lua 5.3 offers some features which make easy the
integration. Lua 5.3 is young, and some distros do not distribute it. Luckily,
Lua is a great product because it does not require exotic dependencies, and its
build process is really easy.
Install Dependencies: http://jackatlinux.blogspot.com/2013/11/installing-lua-and-luarocks-in-centos.html
- Install gcc
$ sudo yum install -y gcc gcc-c++ kernel-devel
The compilation process for linux is easy:
- download the source tarball
wget http://www.lua.org/ftp/lua-5.3.1.tar.gz
- untar it
tar xf lua-5.3.1.tar.gz
- enter the directory
cd lua-5.3.1
- build the library for linux
http://www.lua.org/manual/5.3/readme.html
If you're running Linux and get compilation errors, make sure you have installed the readline development package (which is probably named libreadline-dev or readline-devel).
sudo yum install -y readline-devel
make linux
- install it:
sudo make INSTALL_TOP=/usr install [This is to overwrite system default lua 5.1.0 to 5.3.0]
sudo make INSTALL_TOP=/opt/lua-5.3.1 install [This is used by HAProxy from /opt location]
LUA-cjson
=========
The LUA-cjson is a library required for any json parsing in LUA code.
$ wget http://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz
$ tar -xzvf lua-cjson-2.1.0.tar.gz
$ cd lua-cjson-2.1.0
vi Makefile
LUA_VERSION = 5.3
PREFIX = /usr
$ vi lua-cjson-2.1.0-1.rockspec
dependencies = {
"lua >= 5.3"
}
$ vi lua-cjson.spec
%define luaver 5.3
#sudo make install
Console logs -
$ cp cjson.so //usr/lib/lua/5.3
$ chmod 755 //usr/lib/lua/5.3/cjson.so
$ sudo cp /usr/lib/lua/5.3/cjson.so /opt/lua-5.3.1/lib/cjson.so
$ sudo cp /usr/lib/lua/5.3/cjson.so /opt/lua-5.3.1/lib/lua/5.3/cjson.so
Install HAProxy 1.7\1.6 with LUA
============================
HAProxy builds with your favourite options, plus the following options for
embedding the Lua script language:
- download the source tarball
wget http://www.haproxy.org/download/1.7/src/snapshot/haproxy-ss-20160518.tar.gz
- untar it
tar -xzvf haproxy-ss-20160518
- enter the directory
cd haproxy-ss-20160518
- build HAProxy:
To fix openssl errors
sudo yum install -y openssl-devel
make TARGET=linux \
USE_DL=1 \
USE_LUA=1 \
USE_OPENSSL=1 \
LUA_LIB=/opt/lua-5.3.1/lib \
LUA_INC=/opt/lua-5.3.1/include
[ backup - if wanna install in /usr]
make TARGET=linux \
USE_DL=1 \
USE_LUA=1 \
USE_OPENSSL=1 \
LUA_LIB=/usr/lib/lua/5.3/ \
LUA_INC=/usr/include
- install it:
sudo make PREFIX=/opt/haproxy-1.7 install
[ backup - if wanna install in /usr]
sudo make PREFIX=/usr install
Configure HAProxy
================
- copy binary file to sbin
sudo cp /opt/haproxy-1.7/sbin/haproxy /usr/sbin/
- copy init file
sudo cp /home/cloud/haproxy/haproxy-1.7-dev3/examples/haproxy.init /etc/init.d/haproxy
sudo chmod 755 /etc/init.d/haproxy
- Create these directories and the statistics file for HAProxy to record in.
sudo mkdir -p /etc/haproxy
sudo mkdir -p /run/haproxy
sudo mkdir -p /var/lib/haproxy
sudo touch /var/lib/haproxy/stats
- Then add a new user for HAProxy.
sudo useradd -r haproxy
- After the installation you can double check the installed version number with the following
$ sudo haproxy -v
HA-Proxy version 1.7-dev3-27b639d 2016/05/17
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>
- Configure the load balancer
sudo vi /etc/haproxy/haproxy.cfg
Add the config
HAProxy Script
$ cat /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
lua-load /usr/sbin/lua-choose-backend.lua
# turn on stats unix socket
stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 2s
log 127.0.0.1 local0
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main
bind *:7000
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
use_backend static if url_static
default_backend backend1
frontend http-in
mode http
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ {%[ssl_c_verify],%{+Q}[ssl_c_s_dn],%{+Q}[ssl_c_i_dn]}\ %{+Q}r
bind *:443 ssl crt /etc/haproxy/server.pem ca-file /etc/haproxy/ca.crt verify required
use_backend %[lua.choose_backend]
#default_backend backend1
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
option forwardfor
# With HAProxy Enterprise HAPEE, this is for dynamic disk write for map updates
#dynamic-update
# update id cdpload.map url https://10.128.41.10:80/cdpload.map delay 30s timeout 5s retries 3 map
frontend ssl-in
mode tcp
bind *:8443 ssl crt /etc/haproxy/server.pem ca-file /etc/haproxy/ca.crt verify required
default_backend backendssh
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if HTTP
listen stats
bind *:8181
stats enable
stats uri /
stats realm Haproxy\ Statistics
stats auth cloud:cloud
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4321 check
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend backend1
balance roundrobin
server app1 1.0.0.1:80 check
server app2 1.0.0.2:80 check
backend backend2
balance roundrobin
server app3 1.0.0.3:80 check
server app4 1.0.0.4:80 check
backend backendssh
mode tcp
balance roundrobin
server app1 1.0.0.1:80 check
server app2 1.0.0.2:80 check
server app3 1.0.0.3:80 check
server app4 1.0.0.4:80 check
timeout server 2h
LUA Script
/usr/sbin/lua-choose-backend.lua
-- Cloud map
cloudmetadata = Map.new("/home/cloud/cloudmetadata.map", Map.str);
clouddynamicdata = Map.new("/home/cloud/clouddynamicdata.map", Map.str);
local function split( inSplitPattern, outResults )
if not outResults then
outResults = {}
end
local theStart = 1
local theSplitStart, theSplitEnd = string.find( self, inSplitPattern, theStart )
while theSplitStart do
table.insert( outResults, string.sub( self, theStart, theSplitStart-1 ) )
theStart = theSplitEnd + 1
theSplitStart, theSplitEnd = string.find( self, inSplitPattern, theStart )
end
table.insert( outResults, string.sub( self, theStart ) )
return outResults
end
local function find_backend(cn)
--local json = require('cjson')
core.log(core.info, cn)
-- backend1
--[[
local clouddynamic = clouddynamicdata:lookup(cn);
if(clouddynamic == nil) then
core.log(core.info, "clouddynamic nil")
else
core.log(core.info, clouddynamic)
return clouddynamic
end
]]--
local metadata = cloudmetadata:lookup(cn);
if(metadata == nil) then
core.log(core.info, "metadata nil")
end
core.log(core.info, metadata)
-- 10
core.set_map("/home/cloud/clouddynamicdata.map", cn, 14)
-- this is in-memory write won't get flushed to disk
-- to flush to disk, run external service which would periodically poll /home/cloud/clouddynamicdata.map and write to disk map.
return metadata
end
function choose_backend(txn)
local arg1 = txn.f:ssl_c_s_dn("CN")
backend = find_backend (arg1)
return backend
end
core.register_fetches("choose_backend", choose_backend)
Setup logging in HAproxy
==========================
Step 1: In Global Section of haproxy.cfg put the value log 127.0.0.1 local0 .Like given below
global
log 127.0.0.1 local0
Step 2: Create new haproxy configuration file in /etc/rsyslog.d . Here we are keeping the log in localhost or in other words we should say HAproxy server
Note:
local0.=info -/var/log/haproxy.log defines the http log will be saved in haproxy.log
local0.notice -/var/log/haproxy-status.log defines the Server status like start,stop,restart,down,up etc. will be saved in haproxy-status.log
UDPServerRun 514 means opening UDP port no. 514 to listen haproxy messages
sudo vi /etc/rsyslog.d/haproxy.conf
$ModLoad imudp
$UDPServerRun 514
$template Haproxy,"%msg%\n"
local0.=info -/var/log/haproxy.log;Haproxy
local0.notice -/var/log/haproxy-status.log;Haproxy
### keep logs in localhost ##
local0.* ~
Step 3: Now restart the HAproxy service
$ sudo service rsyslog restart
$ sudo service https restart
After restarting the haproxy service two logs will be created itself i.e haproxy.log and haproxy-status.log
Install netcat (nc)
$ sudo yum install -y nc
$ sudo tail -f /var/log/haproxy.log
Start HAProxy
==============
- Copy SSL files
/etc/haproxy/server.pem
/etc/haproxy/ca.crt
/usr/sbin/lua-choose-backend.lua
/home/cloud/cloudmetadata.map
$ cat cloudmetadata.map
backend1 10
backend2 2
- Start
sudo service haproxy start
Dynamic map
==============
1. Turn on stat socket
$ sudo vi /etc/haproxy/haproxy.cfg
$ stats socket /var/lib/haproxy.sock mode 600 level admin
2. Install socat
$ sudo yum install socat
3.
$ echo "show map /home/cloud/cloudmetadata.map" | sudo socat /var/run/haproxy.sock stdio
0x16cf160 backend1 10
0x16cf300 backend2 2
$ echo "set map /home/cloud/cloudmetadata.map backend2 14" | sudo socat /var/run/haproxy.sock stdio
$ echo "show map /home/cloud/cloudmetadata.map" | sudo socat /var/run/haproxy.sock stdio
0x8de070 backend1 10
0x8de0a0 backend2 14
$ cat /home/cloud/cloudmetadata.map
backend1 10
backend2 2
The in-memory state of /home/cloud/cloudmetadata.map is dynamically set by set map cmd or set_map LUA API
The on-disk /home/cloud/cloudmetadata.map remains un-altered, means dynamic changes are not persistent and would vanish on HAProxy reload.
To make the dynamic changes persistence, run a separate external service to also write the dynamic changes to disk map /home/cloud/cloudmetadata.map
This would ensure changes to persist as well as keep the disk write load and latency limited to external service and not to HAProxy.
Dynamic map - Unix socket commands
Dynamic map - Sudo before socat
References
LUA DOcumentation with HAProxy