Hi,
In a previous; post we covered the installation of satellite in LAN. It is usefull if you have to monitor machines from distant sites. Generaly these sites are accessible with your local network or through a VPN but sometimes, the site is far away and is only accessible via internet connection. In most case, it’s located behind a firewall.
We will try to describe how we could set a secured connexion between distant satellite and our central monitoring server using SSH connection.
It will looks like something like that:
First thing to do is to open an SSH access on both sides. In my case, I used an IPCOP firewall but it’s quite similar if you have an endian firewall or something else… The main idea is to forward the SSH port 22 to the Central centreon server on one side and to your Satelitte on the other side.
So, on Central IPCOP server side I’ll have two settings:
And external access allowed on port 22.
Port forwarding on TCP port 22 to port 22 on my internal machine 192.168.3.22 only allowed from 10.10.10.10 sender.
On the other side, on Satelittes IPCOP I’ll have also:
And external access allowed on port 22.
Port forwarding on TCP port 22 to port 22 on my internal machine 192.168.0.166 only allowed from 11.11.11.11 sender.
In my example, I’ve restricted the access to my own IP and I highly recommend you proceed like that. It’s more secure (and prevent jackass to try to connect to your system).
To verify that your SSH access is enabled, try to connect with SSH on your satellite to the central Centreon and vice versa (use the external IP adress as destination).
Now that the SSH port have been openned, let’s start with the satellite install.
Satelitte install
The installation process is quite similar to our previous satellite install except some IP port which will be different.
Here is a spreadsheet for the different login and passwords that will be used for the installation process (some of them are used in further post):
Account role | Login | Password |
Nagios | nagios | nagaccess03 |
User account | activlan | remoteaccess03 |
Web Nagios | nagiosadmin | nagwebaccess03 |
This part is done on the distant site:
Insert you CD, boot from it and proceed for a minimal install.
For the partition scheme let the system choose one for you (I recommend using LVM because in case of “out of disk space” it will be easier to extend disk).
When asked, disable the automatic update.
DO NOT INSTALL ANY ROLE EXCEPT SSH.
System update
Once it’s installed, proceed with a system update
sudo aptitude update sudo aptitude full-upgrade -y
Switch to fixed IP address
sudo nano /etc/network/interfaces
Please verify that you are working with fixed IP address. If not, please change your settings. You should see something like this at the end of the file (the value should be different for you):
auto eth0 iface eth0 inet static address 192.168.0.166 netmask 255.255.255.0 network 192.168.0.0 broadcast 192.168.0.255 gateway 192.168.0.254
REBOOT
Compilation component install
sudo aptitude install build-essential -y sudo aptitude install linux-headers-$(uname -r) -y
The headers should be already present.
If you are using a virtual machine, it’s time to install the appropriate tools (it’s up to you).
System
sudo aptitude install sysutils policykit ntp -y
Web and php5
sudo aptitude install apache2 apache2-mpm-prefork libapache2-mod-php5 -y sudo aptitude install php5 php5-mysql -y sudo aptitude install php-pear php5-ldap php5-snmp php5-gd -y
RRDTool
sudo aptitude install rrdtool librrds-perl -y
Perl
sudo aptitude install libconfig-inifiles-perl libcrypt-des-perl libdigest-hmac-perl libdigest-sha1-perl libgd-gd2-perl -y
SNMP
sudo aptitude install snmp snmpd libnet-snmp-perl libsnmp-perl -y
GD-Libraries
sudo aptitude install libgd2-xpm libgd2-xpm-dev libpng12-dev -y
Other needed library
sudo aptitude install libldap2-dev libssl-dev libsnmp-dev -y
Now we are ready to install Nagios.
User install
PS: use “nagaccess03” as password for nagios account.
sudo useradd -m -s /bin/bash nagios sudo passwd nagios sudo groupadd nagcmd sudo usermod -G nagios,nagcmd nagios sudo usermod -a -G nagios,nagcmd www-data
Nagios Core
cd /usr/local/src sudo wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.2.tar.gz sudo tar xvzf nagios-3.2.2.tar.gz cd nagios-3.2.2 sudo ./configure --prefix=/usr/local/nagios --with-nagios-user=nagios --with-nagios-group=nagios --with-command-group=nagcmd --enable-nanosleep --enable-event-broker sudo make all sudo make install sudo make install-init sudo make install-commandmode sudo make install-config sudo update-rc.d -f nagios defaults cd ..
Nagios plugins 1.4.15
sudo wget http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz sudo tar xvfz nagios-plugins-1.4.15.tar.gz cd nagios-plugins-1.4.15 sudo ./configure --prefix=/usr/local/nagios --with-nagios-user=nagios --with-nagios-group=nagios sudo make sudo make install cd /usr/local/src
Ndoutils 1.4b9
sudo wget http://prdownloads.sourceforge.net/sourceforge/nagios/ndoutils-1.4b9.tar.gz sudo tar xvfz ndoutils-1.4b9.tar.gz cd ndoutils-1.4b9 sudo ./configure --prefix=/usr/local/nagios/ --enable-mysql --disable-pgsql --with-ndo2db-user=nagios --with-ndo2db-group=nagios sudo cp ./config/ndo2db.cfg-sample /usr/local/nagios/etc/ndo2db.cfg sudo cp ./config/ndomod.cfg-sample /usr/local/nagios/etc/ndomod.cfg sudo chown nagios:nagios /usr/local/nagios/etc/ndo*
It appears that with large monitoring system, ndoutils use a lot of IO and slows down drasticly the system. It’s better to installe the patched version.
cd /usr/local/src sudo wget -r -k -np http://svn.centreon.com/trunk/Patch-NDOutils/ cd svn.centreon.com/trunk/Patch-NDOutils sudo bash ./configure --prefix=/usr/local/nagios/ --enable-mysql --disable-pgsql --with-ndo2db-user=nagios --with-ndo2db-group=nagios sudo make sudo cp src/ndomod-3x.o /usr/local/nagios/bin/ndomod.o sudo cp src/ndo2db-3x /usr/local/nagios/bin/ndo2db sudo chmod 775 /usr/local/nagios/bin/ndo2db sudo chmod 755 /usr/local/nagios/bin/ndomod.o sudo chown nagios:nagios /usr/local/nagios/bin/ndo2db sudo chown nagios:nagios /usr/local/nagios/bin/ndomod.o
Nagios web interface
I also recommend installing Nagios web interface as it could help you comparing your centreon checks and nagios exported configuration.
Use “nagwebaccess03” as password
cd /usr/local/src/nagios-3.2.2 sudo make install-webconf sudo htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
SNMP settings
Please edit /etc/default/snmpd
sudo nano /etc/default/snmpd
and change these sections:
#SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid 127.0.0.1' SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid 127.0.0.1 192.168.0.166'
Replace 192.168.0.166 by your IP address.
#TRAPDRUN=no TRAPDRUN=yes
Edit snmpd configuration file:
sudo nano /etc/snmp/snmpd.conf
and change this section:
# sec.name source community #com2sec paranoid default public com2sec readonly default public com2sec readonly 127.0.0.1 public com2sec readonly 192.168.0.0/24 public #com2sec readwrite default private
Edit ndomod configuration file:
sudo nano /usr/local/nagios/etc/ndomod.cfg
find and replace these sections:
#instance_name=default instance_name=Central
output_type=tcpsocket #output_type=unixsocket
output=127.0.0.1 #output=/usr/local/nagios/var/ndo.sock
#reconnect_warning_interval=15 reconnect_warning_interval=900
#config_output_options=2 config_output_options=3
Edit sudoers grant file:
sudo nano /etc/sudoers
and add nagios user account after root account:
nagios ALL=NOPASSWD: /etc/init.d/nagios restart nagios ALL=NOPASSWD: /etc/init.d/nagios stop nagios ALL=NOPASSWD: /etc/init.d/nagios start nagios ALL=NOPASSWD: /etc/init.d/nagios reload nagios ALL=NOPASSWD: /usr/sbin/nagiostats nagios ALL=NOPASSWD: /usr/sbin/nagios *
Create missing folder:
sudo mkdir -p /var/lib/centreon/centplugins/ sudo chown nagios:nagios /var/lib/centreon/centplugins/
The following steps have to be done on CENTRAL CENTREON SERVER
If you didn’t generate ssh key yet, proceed as follow.
Log you on Centreon Central server and switch to nagios account.
su nagios cd ~ ssh-keygen
Here is an example of produced output (just press ENTER to all questions):
Generating public/private rsa key pair. Enter file in which to save the key (/home/nagios/.ssh/id_rsa): Created directory '/home/nagios/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/nagios/.ssh/id_rsa. Your public key has been saved in /home/nagios/.ssh/id_rsa.pub. The key fingerprint is: 17:f1:f4:f3:2a:de:94:eb:36:8a:82:84:d3:cb:cf:ee nagios@act-centreon-test The key's randomart image is: +--[ RSA 2048]----+ | . . | | + . | | . . o | | . o | | o S . .| | o o . o | | + o . + | | +.. o +o. | | +E.. o++. | +-----------------+
After that, copy your public key to the satellite:
ssh-copy-id -i ~/.ssh/id_rsa.pub nagios@{SATELITTE_FIREWALL_EXTERNAL_IP_ADDRESS}
Use nagaccess03 for accessing your satellite (or use your own password).
Verify that you could access to the satellite without password prompt:
ssh nagios@{SATELITTE_FIREWALL_EXTERNAL_IP_ADDRESS}
Exit to came back to your central centreon server.
If it does not work, check your firewall settings. This must work, it’s a PRE REQUIREMENT for going further.
From central centreon
Synchronize commands with the satellite (only if you are using the same architecture on both side 32bits or 64bits):
scp /usr/local/nagios/libexec/* nagios@{SATELITTE_FIREWALL_EXTERNAL_IP_ADDRESS}:/usr/local/nagios/libexec scp /usr/local/nagios/bin/ndomod.o nagios@{SATELITTE_FIREWALL_EXTERNAL_IP_ADDRESS}:/usr/local/nagios/bin
if you encounter access denied errors like:
scp: /usr/local/nagios/libexec/check_dhcp: Permission denied
Delete files from satellite and resync again with scp.
Optionnaly, if you use rsyslog module in Centreon, redirect your logs:
Edit rsyslog configuration file
sudo nano /etc/rsyslog.conf
Add at the end of the file log events redirection inside your database:
*.=notice;mail.*;\ *.=crit;*.=err;\ *.=warning @@{127.0.0.1}
We use 127.0.0.1 instead of the real IP address. It will be redirected through SSH tunnel. We used the double @ for TCP port (single @ will use UDP access).
Edit 50-default.conf to correct the path
sudo nano /etc/rsyslog.d/50-default.conf
At the end of the file, comment the xconsole section:
#daemon.*;mail.*;\ # news.err;\ # *.=debug;*.=info;\ # *.=notice;*.=warn |/dev/xconsole
Optionnaly intall NRPE module:
cd /usr/local/src sudo wget http://downloads.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.12/nrpe-2.12.tar.gz sudo tar xzfv nrpe-2.12.tar.gz cd nrpe-2.12/ sudo ./configure sudo make all sudo make install-plugin
SSH Tunelling
The main idea is to redirect through SSH the NDO messages (on TCP port 5668) and for thoose using rsyslog the system messages on our main console (on TCP 514).
Configure our main ssh config file:
sudo nano /etc/ssh/ssh_config
At the end of the file please edit like this:
Host activlan Hostname 11.11.11.11 User nagios CheckHostIP no LocalForward 5668 127.0.0.1:5668 LocalForward 514 127.0.0.1:514 Compression yes
If you are not planning to redirect Rsyslog messages, remove the ‘LocalForwad 514’ line.
You have to generate your SSH key as root. It will be used by our ssh tunelling daemon.
sudo ssh-keygen
Here is an exemple of resulting output:
Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 16:44:58:0e:da:f4:19:df:04:df:e4:ba:21:a7:ce:df root@centreon-dejan The key's randomart image is: +--[ RSA 2048]----+ | o+= ... . | | +.= + + + | | . . = . o o | | . . | | S . + | | . + o | | . . | | o . | | o.. E | +-----------------+
Then copy the public key to the distant server:
sudo ssh-copy-id -i /root/.ssh/id_rsa.pub nagios@11.11.11.11
Replace 11.11.11.11 by your external IP adress. Enter the nagios password and your key should be copied.
Try to connect to the distant server like that:
sudo ssh nagios@11.11.11.11
You should be logged in without password.
Install autossh on your satellite:
sudo aptitude install autossh
create your launching script:
sudo nano /etc/init.d/autossh-startup.sh
and put this line:
autossh -N activlan&
make it executable and autostart:
sudo chmod +x /etc/init.d/autossh-startup.sh sudo update-rc.d autossh-startup.sh defaults
Reboot your satellite.
Your ssh tunnel should be active.
Connect to the web interface on central Centreon server and proceed as follow:
Add a new pooler:
Duplicate the ndomod.cfg configuration
Edit it like this:
Duplicate the nagios.cfg configuration
Edit it like this:
Add your new host (which is your satellite):
Go to ‘Configuration / Nagios’ and export like this your settings:
Check on both Central and Satellite that you did not have any error message:
Then you could export your settings:
You should see very soon something like this:
What about using the autossh from the Central Centreon and dialing to the Satellite?
If the Satellite is hacked it would not a easily endanger the the Central Centreon
Anyways your blog is a fantastic ressource about Nagios, monitoring and Ubuntu.
Thank you very much!
Hi iiome,
To be honest, I have not think about this situation. I left of the principle that I initiate the ssh tunneling from the source to the destination (and mainly because all tutorials I’ve found on internet used the tunneling in this way but may be I’m wrong).
In the tutorial, I recommend to filter the ssh entry on both side (central server and satellite) this way you will minimize the hacking risk. I’ve tried a few minutes unfaltering IP address on the ssh port and a few minutes later I’ve been attacked from North-Korea with “brut force” attack (tons of invalid root loggin).
I could only advise to not use unfiltered access to your ssh access or maybe use a tutorial for hardening ssh connection (those filters could be also activated in configuration files).
Cheers
Hi, great tutorial, your effort is highly appreciated.
I’m having the issue that the command
autossh -N activlan
doe not work. It’s not complete.Just to be sure:
– ssh keys are exchanged, ssh login without password is succesfull
– I edited the /etc/ssh/ssh_conf on the centreon server
– Installed autossh on centreon and satellite
– tried the command
autossh -N activlan
on the satelliteI’m running Centos, shouldn’t make a difference I guess. What am I missing?
Grtz,
Roderick
I can create the tunnel now. That’s not the issue. The issue is how to start the tunnel just using the short commandline
autossh -N activlan
Hi mrcoffee11
I’m not familiar with CentOs but the idea should be something like that.
Create a bash script which will be launched when you start your server (on ubuntu, I’ve created a script here: /etc/init.d/autossh-startup.sh with the corresponding autossh command).
After that I made the script runnable (sudo chmod +x /etc/init.d/autossh-startup.sh) so it could be started when the server starts.
And finally I’ve declared this command to start whith the system (sudo update-rc.d autossh-startup.sh).
To summarize, the bash script declared “as a service” will launch the ssh tunnel when the computer start.
Regards
Hi Dejan,
I created an init script, works good.
You mention to use this command: autossh -N activlan&. This does not work for me, autossh complains that it’s not a correct command. I added the info to /etc/ssh/ssh_conf.
Grtz
Roderick