Thursday 18 September 2014

How to log bash (all users or specific user) history to syslog


There are two methods to get this thing done.

1. Trap method - drop the following snippet into either the per-user or system-wide bash profile (~/.bash_profile or /etc/profile, respectively)
unset HISTSIZE HISTFILESIZE export HISTTIMEFORMAT='%F %T' function log2syslog
{
   declare COMMAND
   COMMAND=$(fc -ln -0)
   logger -p local1.notice -t bash -i -- "${USER}:${COMMAND}"
}
trap log2syslog DEBUG


2. Prompt Method - this method logs by hacking the prompt command to call history and write to syslog.
PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -t "$USER[$$] $SSH_CONNECTION")'

Tuesday 16 September 2014

Process which using high swap memory in linux


Here the threshold is 40% . If any process using swap memory more than 40% we will print the process id, TotalSwap, SwapUsed, SwapUsed in percentage.

Note: Below script only works where the VmSwap line prints in /proc/processid/status file (Works in RHEL/Centos 6 2.6.32-279.2.1.el6.centos.plus.x86_64)

#!/bin/bash
##Author == Md. Mansoor
for id in `ps aux | sort -k 4 -k 3 -nr | head -n5 | awk '{print $2}'`
do
swapkb=`cat /proc/$id/status | grep -i swap | awk '{print $2}'`
swapusedMB=`echo $((swapkb/1024))`
total=`free -m | awk 'FNR == 4 {print $2}'`
foutyper=`bc <<< $total*40/100`
swapusedper=`bc -l <<< $swapusedMB/$total*100 | awk -F. '{print $1}'`
if [ $swapusedMB -gt $foutyper ]; then
user=`ps aux | grep $id | grep -v grep | awk '{print $1}'`
cmd=`cat /proc/$id/cmdline`
echo "Above 40% swap used ProcessID = $id TotalSwap = $total SwapUsed = $swapusedMB SwapUsedin% = $swapusedper% MB User = $user Command = $cmd"
fi
done


##Another Script:
##This works for all linux version including old kernel
##Server running lower than procpu-3.2.7-26.rpm package should instead of above use below script. Reference: https://rhn.redhat.com/errata/RHBA-2013-1338.html

#! /bin/bash
# Author: Md. Mansoor
for i in /proc/[0-9]*; do
  pid=$(echo $i | sed -e 's/\/proc\///g')
#echo $pid
  swap_pid=$(cat /proc/$pid/smaps | awk 'BEGIN{total=0}/^Swap:/{total+=$2}END{print total}')
total=`free -m | awk 'FNR == 4 {print $2}'`
swapusedMB=`echo $((swap_pid/1024))`
#echo "swap used in MB $swapusedMB MB"
swapusedper=`bc -l <<< $swapusedMB/$total*100 | awk -F. '{print $1}'`
#echo "$swapusedper in percentage"
foutyper=`bc <<< $total*10/100`
user=`ps aux | grep $pid | grep -v grep | awk '{print $1}'`
if [ $swapusedMB -gt $foutyper ]; then
cmd=`cat /proc/$pid/cmdline`
echo "Above 40% swap used ProcessID = $pid TotalSwap = $total SwapUsed = $swapusedMB SwapUsedin% = $swapusedper% MB User = $user Command = $cmd"
fi
done


Reference: http://www.quora.com/How-can-I-determine-which-process-is-contributing-to-paging-on-Linux

Wednesday 10 September 2014

Memcached Installation in RHEL/CentOS 5/6


cd /usr/local/src
mkdir -p /usr/local/libevent
wget
https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz
tar xzvf libevent-2.0.21-stable.tar.gz
cd libevent-2.0.21-stable
./configure --prefix=/usr/local/libevent
make
make install

cd /usr/local/src
mkdir -p /usr/local/memcached
wget http://memcached.org/latest
tar -zxvf memcached-1.x.x.tar.gz
cd memcached-1.x.x
./configure --prefix=/usr/local/memcached --with-libevent=/usr/local/libevent/
make
make install

cd /usr/local/memcached/bin
./memcached -d -u ukpdl -m 100 -p 10001

Close ticket like below:

Installed path of memcached  : /usr/local/memcached
Memcache running Port          : 10001
Memcache running User         : ukpdl
Memcache memory allocated    : 100MB

Amazon glacier Installation in RHEL/CentOS 6

1. Download Python & boto from below path

cd /usr/local/src/
https://github.com/boto/boto/downloads/boto-2.6.0.tar.gz
http://www.python.org/getit/releases/2.7.5/Python-2.7.5.tgz

2. Install python
yum install *gcc* [ for python support ]
yum install openssl-devel  [ to prevent this error : AttributeError: 'module' object has no attribute 'HTTPSConnection' ]
tar xzvf Python-x.x.x.x
cd Python.x.x.x.x
./configure --disable-ipv6
make
make install


3. Install boto
tar xzvf boto-x.x.x
cd boto-x.x.x
python setup.py install

4. Configure boto

##create below file and add the lines

vim ~/.boto
[Credentials]
aws_access_key_id=Enter here
aws_secret_access_key=Enter here

[glacier]
region=us-east-1
logfile=~/.glacier-cmd.log
loglevel=INFO
output=prin

[Boto]
https_validate_certificates = False


5. Export variables
#export LD_LIBRARY_PATH=/usr/local/lib/python2.7
#export PATH=/usr/local/lib/python2.7/site-packages/boto:$PATH


6. Check the command
#glacier valuts


7. glcier-cmd Install

Amazon Glacier CLI
==================

Command line interface for Amazon Glacier. Allows managing vaults, uploading
and downloading archives and bookkeeping of creat

#cd /usr/local/src
#yum install python-setuptools [ dependency  for glacier-cmd ]
#wget http://python-distribute.org/distribute_setup.py
#python distribute_setup.py
#easy_install pip
#wget https://github.com/uskudnik/amazon-glacier-cmd-interface/amazon-glacier-cmd-interface-master.zip
#unzip amazon-glacier-cmd-interface-master.zip
#cd amazon-glacier-cmd-interface-master
#python setup.py install

8. Create file for glacier-cmd and add the below lines

vim /etc/glacier-cmd.conf
[aws]
access_key=Enter here
secret_key=Enter here

[glacier]
region=us-east-1
logfile=~/.glacier-cmd.log
loglevel=INFO
output=print

9. Commands for glacier please refer below
/usr/local/src/amazon-glacier-cmd-interface-master/README.md
Reference: https://github.com/uskudnik/amazon-glacier-cmd-interface/blob/master/doc/Usage.rst

Puppet Master and Agent Install RHEL/CentOS 6


Master: alps.gai.net
Agent: gai-1234.gai.net (192.168.10.22)

Puppet dashboard : http://alps.gai.net:3000
Username: puppet
Password: puppet123

Pre-requisites: Configure NTP in Master & Agents


Step 1# Master Install & Configure:

##Install Puppet Repo
#rpm -ivh https://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-7.noarch.rpm

#yum install puppet-server

##Configure Master
#vim /etc/puppet/puppet.conf

[master]
certname=alps.gai.net
dns_alt_names=gateway.gai.net

Note:

dns_alt_names: A list of valid hostnames for the master, which will be embedded in its certificate. Defaults to the puppet master’s certname and puppet, which is usually fine. If you are using a non-default setting, set it before starting the puppet master for the first time.

Certname: The name to use when handling certificates. Defaults to the fully qualified domain name.
Default: (the system’s fully qualified domain name)

#/etc/init.d/puppetmaster start

Step 2# Agent Install & Configure
##Install Puppet Repo
#rpm -ivh https://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-7.noarch.rpm

#yum install puppet

##Configure Agent
#vim /etc/puppet/puppet.conf
[agent]
server=alps.gai.net
report=true
pluginsync=true
certname=gai-1234.gai.net
runinterval=60m

Here:
runinterval = for applying configuration with master (This will update configuration every 1hr with master, change based on your requirement)

#/etc/init.d/puppet start

Step 3##Creating & Sigining SSL Certificate for Agent in Master##

Note: The Master and Agent connections running in SSL hence we have to sign the agent certificate request in Master. Do the below steps for it.


Agent# puppet agent --test

Master#puppet cert list (#Once enter this command, you will see the agent FQDN)

##Sign the Agent certificate:##

Master#puppet cert sign xxxxx (Here append xxxxx as above command result client FQDN)

##Checking Agent###
Agent# puppet agent --test (This will provide as Applying configuration with some version number)


Step 4# Testing Master Agent:

In Master:
vim /etc/puppet/manifests/site.pp
##Add the below lines

class test_class {
file { "/tmp/testfile":
ensure => present,
mode => 644,
owner => root,
group => root
}
}

# tell puppet on which client to run the class
node 'gai-1234.gai.net' {
include test_class
}

In Agent:
#puppet agent --test

Now check /tmp/testfile will be created in Agent node.


Step 5# Installing Puppet Dashboard in Master server

Master#yum install mysql mysql-server puppet-dashboard
#cp /usr/share/puppet-dashboard/config/database.yml.example /usr/share/puppet-dashboard/config/database.yml

#vim /usr/share/puppet-dashboard/config/database.yml

#Modify the lines as below
production:
database: dashboard_production
username: dashboard
password: secret_password (Note: this password should be same as below mysql user password)
encoding: utf8
adapter: mysql

#cp /usr/share/puppet-dashboard/config/settings.yml.example /usr/share/puppet-dashboard/config/settings.yml

#vim /usr/share/puppet-dashboard/config/settings.yml

##Modify lines as below
ca_server: 'alps.gai.net'
ca_port: 8140
enable_inventory_service: true
inventory_server: 'alps.gai.net'
inventory_port: 8140
use_file_bucket_diffs: true
file_bucket_server: 'alps.gai.net'
file_bucket_port: 8140

##Creating Database & User & Tables
CREATE DATABASE dashboard CHARACTER SET utf8;
CREATE USER 'dashboard'@'localhost' IDENTIFIED BY 'secret_password';
GRANT ALL PRIVILEGES ON dashboard.* TO 'dashboard'@'localhost';
flush privileges;

#vim /etc/mysql/my.cnf
#Add below line
max_allowed_packet = 32M

##Once Dashboard has its database, it can create its tables, but this has to be done manually with the db:migrate rake task. For typical use with the production environment:

##Run the below commands in terminal
#rake RAILS_ENV=production db:migrate

##Testing That Dashboard is Working
#sudo -u puppet-dashboard ./script/server -e production (use the below service instead of this)
or
/etc/init.d/puppet-dashboard start
/etc/init.d/puppet-dashboard-workers start

browse and connect as http://alps.gai.net:3000

##Configuring Puppet

Using Dashboard for Reports
For Dashboard to receive reports, there are two requirements:
  • All agent nodes have to be configured to submit reports to the master.
  • The master has to be configured to send reports to Dashboard.
Make sure that all agents have reporting turned on:

# puppet.conf (on each agent)
[agent]
report = true

# puppet.conf (on puppet master)
[master]
reports = store, http
reporturl = http://dashboard.example.com:3000/reports/upload

#Restart services (Master)
#/etc/indi.d/puppetmaster restart
#/etc/indi.d/puppet-dashboard restart
#/etc/indi.d/puppet-dashboard-workers restart

#Restart services (Agent)
#/etc/indi.d/puppet restart

##Enabling Inventory for puppet aent

Master# vim /etc/puppet/auth.conf

#Go to end of file#
##Disable below lines
path /
auth any

#Add below lines
path /facts
auth any
method find, search
allow *

#Restart services
#/etc/init.d/puppetmaster stop
#/etc/init.d/puppetmaster start
#/etc/init.d/puppetmaster-dashboard stop
#/etc/init.d/puppetmaster-dashboard start
#/etc/init.d/puppetmaster-dashboard-workers stop
#/etc/init.d/puppetmaster-dashboard-workers start
Now browse: http://puppetmasterip:3000 check the inventory tab

Step 6# ##Running Puppet Master using apache instead of services (puppetmaster)
#yum install httpd httpd-devel mod_ssl ruby-devel rubygems gcc

#Install Rack/Passenger
#sudo gem install rack passenger
#sudo passenger-install-apache2-module
#sudo mkdir -p /usr/share/puppet/rack/puppetmasterd
#sudo mkdir /usr/share/puppet/rack/puppetmasterd/public /usr/share/puppet/rack/puppetmasterd/tmp
#sudo cp /usr/share/puppet/ext/rack/config.ru /usr/share/puppet/rack/puppetmasterd/
#sudo chown puppet:puppet /usr/share/puppet/rack/puppetmasterd/config.ru
#chown puppet:puppet /usr/share/puppet/rack/puppetmasterd/config.ru
#chown apache:apache /usr/share/puppet/rack/puppetmasterd/public -R
#chown apache:apache /usr/share/puppet/rack/puppetmasterd/tmp -R


#vim /etc/httpd/conf.d/puppetmaster.conf
#Add below lines
# This Apache 2 virtual host config shows how to use Puppet as a Rack
# application via Passenger. See
# http://docs.puppetlabs.com/guides/passenger.html for more information.

# You can also use the included config.ru file to run Puppet with other Rack
# servers instead of Passenger.
# #Below modules lines you will get while running above gem install rack passenger
LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-4.0.37/buildout/apache2/mod_passenger.so
<IfModule mod_passenger.c>
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.37
PassengerDefaultRuby /usr/bin/ruby
</IfModule>
# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
#RackAutoDetect Off
#RailsAutoDetect Off

Listen 8140

<VirtualHost *:8140>
SSLEngine On

# Only allow high security cryptography. Alter if needed for compatibility.
SSLProtocol All -SSLv2
SSLCipherSuite HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/alps.gai.net.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/alps.gai.net.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +ExportCertData

# These request headers are used to pass the client certificate
# authentication information on to the puppet master process
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

DocumentRoot /usr/share/puppet/rack/puppetmasterd/public

<Directory /usr/share/puppet/rack/puppetmasterd/>
Options None
AllowOverride None
# Apply the right behavior depending on Apache version.
<IfVersion < 2.4>
Order allow,deny
Allow from all
</IfVersion>
<IfVersion >= 2.4>
Require all granted
</IfVersion>
</Directory>

ErrorLog /var/log/httpd/alps_puppet_master_ssl_error.log
CustomLog /var/log/httpd/alps_puppet_master_ssl_access.log combined
</VirtualHost>

# Stop puppetmaster & restart http
#/etc/init.d/puppetmaster stop
#/etc/init.d/httpd restart

#Check listen
#netst -tupln | grep LISTEN | grep 8140

Step 7# ##Running Puppet Dashboard using apache instead of services (puppet-dashboard)

#cp /usr/share/puppet-dashboard/ext/passenger/dashboard-vhost.conf /etc/init.d/conf.d/
#vim /etc/httpd/conf.d/dashboard-vhost.conf
#Add below lines or modify
LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-4.0.37/buildout/apache2/mod_passenger.so
<IfModule mod_passenger.c>
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.37
PassengerDefaultRuby /usr/bin/ruby
</IfModule>
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
PassengerStatThrottleRate 120
Listen 3000
<VirtualHost *:3000>
ServerName puppet.gai.net
DocumentRoot /usr/share/puppet-dashboard/public/
<Directory /usr/share/puppet-dashboard/public/>
Options None
Order allow,deny
allow from all
</Directory>
ErrorLog /var/log/httpd/puppet.gai.net_error.log
LogLevel warn
CustomLog /var/log/httpd/puppet.gai.net_access.log combined
ServerSignature On
<Location "/">
Order allow,deny
Allow from 192.168.0.125 # your puppet master's IP
Satisfy any
AuthName "Puppet Dashboard"
AuthType Basic
AuthUserFile /etc/httpd/htpasswd
Require valid-user
</Location>

</VirtualHost>

##Stop puppet-dashboard services & restart http
#/etc/init.d/puppet-dashboard stop
#/etc/init.d/httpd restart


##Classes Examples:
#To Copy configuration file:
vim /etc/puppet/manifests/site.pp
class centos_6_repo_class {

file { "/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6":
owner => root,
group => root,
mode => 644,
source => "puppet:///modules/centos_6_repo/RPM-GPG-KEY-CentOS-6"
}

file { "/etc/yum.repos.d/CentOS-Base.repo":
owner => root,
group => root,
mode => 644,
source => "puppet:///modules/centos_6_repo/CentOS-Base.repo"
}

}

# tell puppet on which client to run the class
node 'gai-1234.gai.net' {
include centos_6_repo_class
}

To Install a package:
#vim /etc/puppet/manifests/site.pp
class ftp-0-17-54_install_class {
package { "ftp":
ensure => "0.17-54.el6"
}

}

node 'genistra.gai.net' {
include ftp-0-17-54_install_class
}
Pupper Agent for Windows:

Download from  https://downloads.puppetlabs.com/windows/puppet-3.4.3.msi


create file as C:\Program Files\Puppet Labs\Puppet Enterprise\puppet\conf\puppet.conf

add below lines

[agent]
    server = alps.gai.net
    certname = thinkpad4
        report=true
        pluginsync=true
        runinterval=2m

In server site.pp file add below lines

if $osfamily == 'windows' {
    File { source_permissions => ignore }
  }

class win-check {
file { "C:/temp/CentOS-Base.repo":
    ensure  => present,
    group => Administrators,
    source => "puppet:///modules/centos_6_repo/CentOS-Base.repo"
}

}

node 'thinkpad4' {
    include win-check
}


URLS Referred:



SVN Installation in RHEL/CentOS


#yum install subversion
 

#mkdir /backup/svnroot/
svnadmin create /backup/svnroot/testrepo

#vim /backup/svnroot/testrepo/conf/svnserve.conf
#Edit as below
passwd = /backup/svnroot/testrepo/conf/passwd
authz = /backup/svnroot/testrepo/conf/authz

vim /backup/svnroot/testrepo/conf/passwd
#Add below
mohamed.mansoor = secret123

vim /backup/svnroot/testrepo/conf/authz
#Add below
[testrepo:/]
mohamed.mansoor = rw

#Start svn as daemon
svnserve -d -r /backup/svnroot/

#svn ls svn://localhost/testrepo
username: mohamed.mansoor
password = secret123