CPU Statistics

                                                           CPU Statistics
There are several different ways to see the various CPU statistics. The most common is probably using the top command.

To start the top command you just type top at the command line:

The output from top is divided into two sections. The first few lines give a summary of the system resources including a breakdown of the number of tasks, the CPU statistics, and the current memory usage. Beneath these stats is a live list of the current running processes. This list can be sorted by 

PID, CPU usage, memory usage, and so on.

The CPU line will look something like this:

%Cpu(s): 30.8 us,  0.5 sy,  0.0 ni, 67.9 id,  0.6 wa,  0.0 hi,  0.2 si,  0.0 st

30.8 us - (user cpu time (or) % CPU time spent in user space) This tells us that the processor is spending 30.8% of its time running user space processes. A user space program is any process that doesn't belong to the kernel. Shells, compilers, databases, web servers, and the programs associated with the desktop are all user space processes. If the processor isn't idle, it is quite normal that the majority of the CPU time should be spent running user space processes.

67.9 id - (idle cpu time (or) % CPU time spent idle) Skipping over a few of the other statistics, just for a moment, the id statistic tell us that the processor was idle just over 67% of the time during the last sampling period. The total of the user space percentage - us, the niced percentage - ni, and the idle percentage - id, should be close to 100%. Which it is in this case. If the CPU is spending a more time in the other states then something is probably awry - see the Troubleshooting section below.

0.5 sy - (system cpu time (or) % CPU time spent in kernel space) This is the amount of time that the CPU spent running the kernel. All the processes and system resources are handled by the Linux kernel. When a user space process needs something from the system, for example when it needs to allocate memory, perform some I/O, or it needs to create a child process, then the kernel is running. In fact the scheduler itself which determines which process runs next is part of the kernel. The amount of time spent in the kernel should be as low as possible. In this case, just 0.5% of the time given to the different processes was spent in the kernel. This number can peak much higher, especially when there is a lot of I/O happening.

0.0 ni - (user nice cpu time (or) % CPU time spent on low priority processes)As mentioned above, the priority level a user space process can be tweaked by adjusting its niceness. The ni stat shows how much time the CPU spent running user space processes that have been niced. On a system where no processes have been niced then the number will be 0.

0.6 wa - (io wait cpu time (or) % CPU time spent in wait (on disk))Input and output operations, like reading or writing to a disk, are slow compared to the speed of a CPU. Although this operations happen very fast compared to everyday human activities, they are still slow when compared to the performance of a CPU. There are times when the processor has initiated a read or write operation and then it has to wait for the result, but has nothing else to do. In other words it is idle while waiting for an I/O operation to complete. The time the CPU spends in this state is shown by the wa statistic.

0.0 hi & 0.2 si - (hardware irq (or) % CPU time spent servicing/handling hardware interrupts) & (software irq (or) % CPU time spent servicing/handling software interrupts)These two statistics show how much time the processor has spent servicing interrupts. hi is for hardware interrupts, and si is for software interrupts. Hardware interrupts are physical interrupts sent to the CPU from various peripherals like disks and network interfaces. Software interrupts come from processes running on the system. A hardware interrupt will actually cause the CPU to stop what it is doing and go handle the interrupt. A software interrupt doesn't occur at the CPU level, but rather at the kernel level.

0.0 st - (steal time - - % CPU time in involuntary wait by virtual cpu while hypervisor is servicing another processor (or) % CPU time stolen from a virtual machine)This last number only applies to virtual machines. When Linux is running as a virtual machine on a hypervisor, the st (short for stolen) statistic shows how long the virtual CPU has spent waiting for the hypervisor to service another virtual CPU running on a different virtual machine. Since in the real-world these virtual processors are sharing the same physical processor(s) then there will be times when the virtual machine wanted to run but the hypervisor scheduled another virtual machine instead.

Introduction to mongoDB

Introduction to mongoDB
MongoDB is a document database that provides high performance, high availability, and easy scalability.
or
MongoDB is an open-source document database that provides high performance, high availability, and automatic scaling.

-1-Document Database

A record in MongoDB is a document, which is a data structure composed of field and value pairs. MongoDB documents are similar to JSON objects. The values of fields may include other documents, arrays, and arrays of documents.

A MongoDB document.

--The advantages of using documents are:

Documents (objects) map nicely to programming language data types.

Embedded documents and arrays reduce need for joins.
Dynamic schema makes polymorphism easier.

-2-High Performance

Embedding makes reads and writes fast.
Indexes can include keys from embedded documents and arrays.
Optional streaming writes (no acknowledgments).
i.e MongoDB provides high performance data persistence. In particular,

-Support for embedded data models reduces I/O activity on database system.
-Indexes support faster queries and can include keys from embedded documents and arrays.

-3-High Availability

Replicated servers with automatic master failover, 

i.e To provide high availability, MongoDB’s replication facility, called replica sets, provide:
--automatic failover.
--data redundancy.
A replica set is a group of MongoDB servers that maintain the same data set, providing redundancy and increasing data availability.

-4-Easy Scalability( or automatic scaling)
Automatic sharding distributes collection data across machines.
MongoDB provides horizontal scalability as part of its core functionality.
Eventually-consistent reads can be distributed over replicated servers. i.e Replica sets can provide eventually-consistent reads for low-latency high throughput deployments.

-5-Advanced Operation

With MongoDB Management Service (MMS) MongoDB supports a complete backup solution and full deployment monitoring.

#~~#Key MongoDB Features

MongoDB focuses on flexibility, power, speed, and ease of use:
--Flexibility
MongoDB stores data in JSON documents (which we serialize to BSON). JSON provides a rich data model that seamlessly maps to native programming language types, and the dynamic schema makes it easier to evolve your data model than with a system with enforced schemas such as a RDBMS.
--Power
MongoDB provides a lot of the features of a traditional RDBMS such as secondary indexes, dynamic queries, sorting, rich updates, upserts (update if document exists, insert if it doesn’t), and easy aggregation. This gives you the breadth of functionality that you are used to from an RDBMS, with the flexibility and scaling capability that the non-relational model allows.
--Speed/Scaling
By keeping related data together in documents, queries can be much faster than in a relational database where related data is separated into multiple tables and then needs to be joined later. MongoDB also makes it easy to scale out your database. Autosharding allows you to scale your cluster linearly by adding more machines. It is possible to increase capacity without any downtime, which is very important on the web when load can increase suddenly and bringing down the website for extended maintenance can cost your business large amounts of revenue.
--Ease of use
MongoDB works hard to be very easy to install, configure, maintain, and use. To this end, MongoDB provides few configuration options, and instead tries to automatically do the “right thing” whenever possible. This means that MongoDB works right out of the box, and you can dive right into developing your application, instead of spending a lot of time fine-tuning obscure database configurations.

#~~#Operations:

--MongoDB is a server process that runs on Linux, Windows and OS X. It can be run both as a 32 or 64-bit application. We recommend running in 64-bit mode, since MongoDB is limited to a total data size of about 2GB for all databases in 32-bit mode.

--The MongoDB process listens on port 27017 by default (note that this can be set at start time - please seemongod options for more information).

--Clients connect to the MongoDB process, optionally authenticate themselves if security is turned on, and perform a sequence of actions, such as inserts, queries and updates.

--MongoDB stores its data in files (default location is /data/db/), and uses memory mapped files for data management for efficiency.

--MongoDB can also be configured for data replication.

--Additionally the MongoDB Management Service (MMS) application for managing MongoDB clusters using a simple user interface. MMS provides backup and monitoring. MMS is available to all users in the cloud and on-premises as part of MongoDB Standard and Enterprise Subscriptions.

MongoDB Binary Installation on Linux(Red Hat Enterprise Linux, CentOS Linux, Fedora Linux, or a related system)

MongoDB Binary Installation on Linux(Red Hat Enterprise Linux, CentOS Linux, Fedora Linux, or a related system)
                Created By : Amit Gera DBA(MongoDB & MySQL & ORACLE)
Goal:MongoDB Standard Binary Installation on Linux 64 bit through rpm package.

Partition Requirement:-
/mongo_admin:-   Will contains mongo binaries and confirguration file
/mongo_backup:- Will contains mongo backup [i.e dumpfile or export files or import files]
/mongo_data:-     Will contains data files
/mongo_home:-   Will contains profiles , scripts etc
/mongo_logs:-     Will contains log files [i.e trace ,alert & audit logfiles]

Note:-Here I have use name mongoservice_script wrt service name ,As per guidelines we must set it to meaningful name i.e product service name

Prerequisites:-http://myora-dba-notes.blogspot.com/2014/08/mongodb-distribution-formats-different.html

#~~#Step 1:-Download mogodb packages for linux
a.Create downloading destination directory
cmd:-mkdir -p /mongo_admin/mongo_binaries/download
b.Start Downloading at destination directory
If internet is available then use this cmd:-
cd /mongo_admin/mongo_binaries/download/
curl -O  http://downloads.mongodb.org/linux/mongodb-linux-x86_64-2.6.3.tgz
wget -O /mongo_admin/mongo_binaries/download/mongodb-linux-x86_64-2.6.3.tgz  http://downloads.mongodb.org/linux/mongodb-linux-x86_64-2.6.3.tgz

Else download this from other sources and copy here

#~~#Step 2:-
a.Create group and create user in that group for mongo
cmds:-
groupadd mongo_grp
useradd -r -g mongo_grp mongo_user
b.Create directory mention in below configuration file of step 6
cmds:-
mkdir -p /mongo_data/datadir/
mkdir -p /mongo_logs/error_log
chown mongo_user.mongo_grp -R /mongo_data/
chown mongo_user.mongo_grp -R /mongo_logs/
c.Create configuration & binary directories
cmds:-
mkdir -p /mongo_admin/configuration_file
touch /mongo_admin/configuration_file/mongod.conf


#~~#Step 3:-Add following below listed entries into environment file and load it in envrinoment
step a. cmd:-
echo "myadmin="/mongo_admin/"" >> /etc/environment
echo "basedir="/mongo_admin/mongo_binaries/"" >> /etc/environment
echo "myconf="/mongo_admin/configuration_file"" >> /etc/environment
echo "mongo_HOME="/mongo_home/homedir"" >> /etc/environment
echo "myadmin_download="/mongo_admin/mongo_binaries/download"" >> /etc/environment
step b. Run the file with cmd:-source /etc/environment

#~~#Step 4:- Extract binaries 
cmd:-cd $basedir
cmd:- tar zxvf $myadmin_download/mongodb-linux-x86_64-2.6.3.tgz -C /mongo_admin/mongo_binaries/
cmd:- ln -s /mongo_admin/mongo_binaries/mongodb-linux-x86_64-2.6.3 binaries

#~~#Step 5:-Add following below listed entries into environment file and load it in envrinoment
step a. cmd:-
echo "PATH="$PATH:/mongo_admin/mongo_binaries/binaries/bin"" >> /etc/environment
step b. Run the file with cmd:-source /etc/environment

#~~#Step 6: Create configuration file
a. vi /mongo_admin/configuration_file/mongod.conf
b. write below listed information into configuration file and save
# mongod.conf
#where to log
logpath=/var/log/mongodb/mongod.log
logappend=true
# fork and run in background
fork=true
port=27017
dbpath=/mongo_data/datadir

#~~#Step 7:- Now we have to initializes the mongo data directory and creates the system tables that it should contains.
cmd:-cd $basedir/binaries/bin/
cmd:- mongod --dbpath /mongo_data/datadir/

#~~#Step8: There are many ways by which we can start mongo and here we invoke mongo.server script method that use System V-style run directories (that is, /etc/init.d and run-level specific directories),
##Generally, you start the mongod server in one of these ways:
a.Invoke mongod directly. This works on any platform.
b.Invoke mongod , which also tries to determine the proper options for mongod and then runs it with those options. This script is used on Unix and Unix-like systems.
c.Invoke mongod . This script is used primarily at system startup and shutdown on systems that use System V-style run directories (that is, /etc/init.d and run-level specific directories), where it usually is installed under the name mongo. The mongo.server script starts the server by invoking mongod_safe.
d.On Mac OS X, install a separate mongo Startup Item package to enable the automatic startup of mongo on system startup. The Startup Item starts the server by invoking mongo.server.

After initializing db ,Now we are proceding to add service in System V-style run directories

Execution steps:-
-i- First create mongod service start/stop/restartup script
a. vi $basedir/binaries/bin/mongoservice_script
b. write below listed information into mongoservice_script file and save

#!/bin/bash

# mongod - Startup script for mongod

# chkconfig: 35 85 15
# description: Mongo is a scalable, document-oriented database.
# processname: mongod
# config: /etc/mongod.conf
# pidfile: /var/run/mongodb/mongod.pid

. /etc/rc.d/init.d/functions

# things from mongod.conf get there by mongod reading it


# NOTE: if you change any OPTIONS here, you get what you pay for:
# this script assumes all options are in the config file.
CONFIGFILE="/mongo_admin/configuration_file/mongod.conf"
OPTIONS=" -f $CONFIGFILE"
SYSCONFIG="/etc/sysconfig/mongod"

# FIXME: 1.9.x has a --shutdown flag that parses the config file and
# shuts down the correct running pid, but that's unavailable in 1.8
# for now.  This can go away when this script stops supporting 1.8.
DBPATH=`awk -F= '/^dbpath[[:blank:]]*=[[:blank:]]*/{print $2}' "$CONFIGFILE"`
PIDFILE=`awk -F= '/^pidfilepath[[:blank:]]*=[[:blank:]]*/{print $2}' "$CONFIGFILE"`
mongod=${MONGOD-/mongo_admin/mongo_binaries/binaries/bin/mongod}

MONGO_USER=mongo_user
MONGO_GROUP=mongo_grp

if [ -f "$SYSCONFIG" ]; then
    . "$SYSCONFIG"
fi

# Handle NUMA access to CPUs (SERVER-3574)
# This verifies the existence of numactl as well as testing that the command works
NUMACTL_ARGS="--interleave=all"
if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
then
    NUMACTL="numactl $NUMACTL_ARGS"
else
    NUMACTL=""
fi

start()
{
  # Recommended ulimit values for mongod or mongos
  # See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
  #
  ulimit -f unlimited
  ulimit -t unlimited
  ulimit -v unlimited
  ulimit -n 64000
  ulimit -m unlimited
  ulimit -u 32000

  echo -n $"Starting mongod: "
  daemon --user "$MONGO_USER" "$NUMACTL $mongod $OPTIONS >/dev/null 2>&1"
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/mongod
}

stop()
{
  echo -n $"Stopping mongod: "
  killproc -p "$PIDFILE" -d 300 /usr/bin/mongod
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/mongod
}

restart () {
        stop
        start
}


RETVAL=0

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart|reload|force-reload)
    restart
    ;;
  condrestart)
    [ -f /var/lock/subsys/mongod ] && restart || :
    ;;
  status)
    status $mongod
    RETVAL=$?
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
    RETVAL=1
esac

exit $RETVAL


-ii) Copy server script and add in system V-style run directories
cmd:-cp $basedir/binaries/bin/mongoservice_script /etc/init.d/mongoservice_script
-iii) add in chmod +x /etc/init.d/mongoservice_script

-iv) Optional if there is any old services which we need to remove :-
cmd:-chkconfig --list | grep old_mongo_servicename
if present then remove with below command
cmd:- chkconfig –del old_mongo_servicename
-v) Adding system startup for /etc/init.d/mongoservice_script ...
cmd:- chkconfig --add mongoservice_script or chkconfig mongoservice_script on

#~~#Step 9: Now you can start mongo with
/etc/init.d/mongoservice_script start
or
/etc/init.d/mongoservice_script start

Note: We can shutdown with
/etc/init.d/mongoservice_script stop
or
/etc/init.d/mongoservice_script stop

MongoDB Distribution Formats / Different types for MongoDB Installation Methods

      MongoDB Distribution Formats / Different types for MongoDB Installation Methods

#~~#MongoDB Distribution Formats / Different types for MongoDB Installation Methods 
There are Two types of Binary Distribution Format for MongoDB Installation
a). Generic Binary Format(Zip arcives or compressed tar )
b). Native Format(RPM packages for Linux, DMG packages for Mac OS X or Ubuntu or Debian system, and PKG packages for Solaris.)

MariaDB Audit Plugin for MySQL

                                                      MariaDB Audit Plugin for MySQL
MariaDB Audit Plugin for MySQL:- The library is included in a tarball (server_audit-VERSION.tar.gz ) as 32-bit and 64-bit version for Linux for both, debug and non-debug versions of MariaDB and MySQL. I am using server_audit-1.1.7.tar.gz here.

Step1: Download MariaDB Audit Plugin for MySQL & untar the compressed tarball.
After untar we will get package of 32-bit and 64-bit version for Linux for both, debug and non-debug versions of MySQL.

Step2: Copy file that's meet the requirement and place at MySQL plugin dir mentioned in MySQL variable:-SHOW GLOBAL VARIABLES LIKE 'plugin_dir';
Note:Make sure for assign required ownership & privileges

Step 3: load plugin in MySQL:-
#~a~#The plugin can be loaded from the command-line as a start-up parameter, or it can be set in the configuration file (i.e.,
my.cnf or my.ini). Below is an excerpt from a configuration file, showing the relavent line to load this plugin. To use
this option from the command-line at start-up, just add a double-dash (e.g., pluginload).

[mysqld]
...
pluginload=server_audit=server_audit.so
...

#~b~#Another way to install this plug-in is to execute the INSTALL PLUGIN statement from within MySQL. You would need to
use an administrative account which has INSERT privilege for the mysql.plugin table. To do this, you would execute the
following within the mysql client or an equivalent client:
INSTALL PLUGIN server_audit SONAME 'server_audit.so';

Note:
--The variables that will be used by the plugin (see the Configuration section) will be unknown to the
server until the plugin has been loaded the first time. The database server will not start successfully if
these variables are set in the configuration file before the audit plugin has been loaded at least once
before.

--The UNINSTALL PLUGIN statement may be used to uninstall a plugin. For the auditing plugin, you might want to disable
this possibility. To do this, you could add the following line to the configuration file after the plugin is loaded once:
[mysqld]
...
pluginload=server_audit=server_audit.so
server_audit=FORCE_PLUS_PERMANENT
...

--Once you've added the option above to the server's configuration file and restarted the server, if someone tries then to
uninstall the audit plugin, an error message will be returned. Below is an example of this with the error message:
UNINSTALL PLUGIN server_audit;
ERROR 1702 (HY000):Plugin 'server_audit' is force_plus_permanent and can not be unloaded

Database Migration [ Oracle to MySQL ]

                            Database Migration [ Oracle to MySQL ]

#~~#Action Plan for Database Migration with best practices:
1.Prepare to Migrate
2.Test the Migration Process
3.Test the Migrated Test Database
4.Prepare and Preserve the Source Database
5.Migrate the Production Database
6.Tune and Adjust the New Production Database

#~~#Database Migration:

-1-Schema Migration Action Plan from Oracle to MySQL:
Migrate oracle schema into MySQL either manually or you can also use software for schema http://www.convert-in.com/ora2sql.htm but here if using third party utility then please make sure it  migrated into optimal state.This one is the most important step in migration and should be properly verified in order to avoid any unnecessary service outage in future.
There are also some third party tools that can help in complete migration,Here we have shared plan for doing manual migration freely without any cost.
Tools are :
i)  Oracle-to-MySQL
link:- http://dbconvert.com/convert-oracle-to-mysql-pro.php
ii) SQLways - Oracle to MySQL data migration
link:- http://www.ispirer.com/products/oracle-to-mysql-migration
any many more third party utitiles are there.
Note:There is one blog that I have found later after migration but i would like to list it here for your help its link is http://tkurek.blogspot.in/2013/04/migrate-oracle-to-mysql.html

Imp.Note:Will soon upload datatype & functioanlity comparision for Oracle and MySQL.

-2-Procedure/Function/Triggers Migration Action Plan from Oracle to MySQL:
It includes cnversion of follwing action items
i).Syntax conversion.
ii).Conversion for Oracle built-in functionality into MySQL built-in functionality if available else create user defined funtionality for the same behaviour.
iii).User Defined Method Conversion.

-3-Data Migration Action Plan from Oracle to MySQL:
a). Take dump from oracle using below listed format into dumpfiles
SET TRIMSPOOL ON
SET PAGESIZE  0
SET ECHO OFF
SET FEEDBACK  OFF
SET HEADING   OFF
Spool /path/filename.csv
query(i.e select Col1||','||','||Col2||','||Col3 from tablename; Note:Please make sure the output format of column in query should be as per MySQL format)
quit

b). Restore dumpfiles into MySQL using load command or mysql import utilities for dumpfile that we get from step 2.

SQLPLUS Parameters

SET TERM      OFF
-- TERM = ON will display on terminal screen (OFF = show in LOG only)

SET ECHO      ON
-- ECHO = ON will Display the command on screen (+ spool)
-- ECHO = OFF will Display the command on screen but not in spool files.
-- Interactive commands are always echoed to screen/spool.

SET TRIMOUT   ON
-- TRIMOUT = ON will remove trailing spaces from output

SET TRIMSPOOL ON
-- TRIMSPOOL = ON will remove trailing spaces from spooled output

SET HEADING   OFF
-- HEADING = OFF will hide column headings

SET FEEDBACK  OFF
-- FEEDBACK = ON will count rows returned

SET PAUSE     OFF
-- PAUSE = ON .. press return at end of each page

SET PAGESIZE  0  
-- PAGESIZE = height 54 is 11 inches (0 will supress all headings and page brks)

SET LINESIZE  80
-- LINESIZE = width of page (80 is typical)

SET VERIFY    OFF
-- VERIFY = ON will show before and after substitution variables

-- Start spooling to a log file
SPOOL C:\TEMP\MY_LOG_FILE.LOG
--
-- The rest of the SQL commands go here
--
SELECT * FROM TABLE_NAME;
SPOOL OFF