Attach:OVO_agent_architecture.gif

Source: http://www.softpanorama.org/Admin/HP_operations_manager/agents.shtml

OVO deploy

====================================================================

Executing remote commands from OVO server

====================================================================

Let's say that there is server host1, which has some of the following issues:

1) No working SSH & console ( Master of Disaster )

2) Full root fs

3) No swap

4) Missing root credentials due to some unknown/unwanted/unintended misconfiguration

5) You have to execute some generic commands on large amount of servers.

Then, you could use some OVO magic like remote execution ( you DO need root access on the OVO ):

ovdeploy -host <hostname> -cmd <command>

==============================

    script 

==============================

1. !/bin/sh

if [ $# -lt 2 ]; then

        echo "Usage: basename $0 [serverlist file] [command]"
        exit 1

fi

for server in `cat $1`; do

        echo "$server : \\c"
        ovdeploy -host $server -cmd "$2"

done

==============================

    script  end

============================== ==== Example: ./ovodepscript.sh nycomedservers "ioscan -fn | grep NO_HW | wc -l; ioscan -fnkC disk | grep NO_HW | wc -l"

osctrl09(OVO8):/root/home/hpeboyko (root) ./ovodepscript.sh nycomedservers uptime dedfc03.dekon.ap.altana : 9:35am up 226 days, 5:56, 0 users, load average: 4.52, 4.59, 4.62

dedfc04.dekon.ap.altana : 9:35am up 226 days, 5:35, 0 users, load average: 0.77, 0.83, 0.84

dedfc05.dekon.ap.altana : 9:35am up 226 days, 5:19, 0 users, load average: 0.01, 0.02, 0.02

dedme01.nycomed.local : 9:35am up 226 days, 14 mins, 0 users, load average: 0.07, 0.07, 0.07

dedme03.nycomed.local : 9:35am up 127 days, 18:21, 1 user, load average: 0.25, 0.28, 0.28

dedpr01.nycomed.local : 9:35am up 225 days, 23:54, 0 users, load average: 0.20, 0.17, 0.15

dedpr02.nycomed.local : 9:35am up 225 days, 21:35, 0 users, load average: 0.68, 0.71, 0.71

deidb01.dekon.ap.altana : 9:35am up 226 days, 48 mins, 0 users, load average: 0.03, 0.09, 0.10

desdp03.nycomed.local : 9:35am up 18 days, 17:01, 0 users, load average: 0.03, 0.03, 0.03

desdp04.nycomed.local : 9:35am up 17 days, 20:29, 0 users, load average: 0.18, 0.20, 0.20

desdp05.nycomed.local : 9:35am up 225 days, 1:04, 0 users, load average: 0.05, 0.04, 0.04

desdp06.nycomed.local : 9:35am up 224 days, 21:08, 0 users, load average: 0.04, 0.04, 0.04

osctrl09(OVO8):/root/home/hpeboyko (root) osctrl09(OVO8):/root/home/hpeboyko (root) osctrl09(OVO8):/root/home/hpeboyko (root) osctrl09(OVO8):/root/home/hpeboyko (root) ./ovodepscript.sh nycomedservers "uptime;date" dedfc03.dekon.ap.altana : 9:36am up 226 days, 5:57, 0 users, load average: 4.65, 4.61, 4.62 Mon Dec 19 09:36:42 MEZ 2011

dedfc04.dekon.ap.altana : 9:36am up 226 days, 5:36, 0 users, load average: 0.78, 0.82, 0.84 Mon Dec 19 09:36:43 MEZ 2011

dedfc05.dekon.ap.altana : 9:36am up 226 days, 5:20, 0 users, load average: 0.02, 0.02, 0.02 Mon Dec 19 09:36:44 MEZ 2011

dedme01.nycomed.local : 9:36am up 226 days, 15 mins, 0 users, load average: 0.07, 0.07, 0.07 Mon Dec 19 09:36:45 MET 2011

dedme03.nycomed.local : 9:36am up 127 days, 18:22, 1 user, load average: 0.35, 0.30, 0.29 Mon Dec 19 09:36:46 MET 2011

dedpr01.nycomed.local : 9:36am up 225 days, 23:55, 0 users, load average: 0.20, 0.17, 0.15 Mon Dec 19 09:36:47 MEZ 2011

dedpr02.nycomed.local : 9:36am up 225 days, 21:36, 0 users, load average: 0.69, 0.71, 0.72 Mon Dec 19 09:36:47 MEZ 2011

deidb01.dekon.ap.altana : 9:36am up 226 days, 49 mins, 0 users, load average: 0.03, 0.08, 0.10 Mon Dec 19 09:36:48 MEZ 2011

desdp03.nycomed.local : 9:36am up 18 days, 17:02, 0 users, load average: 0.02, 0.03, 0.03 Mon Dec 19 09:36:49 MEZ 2011

desdp04.nycomed.local : 9:36am up 17 days, 20:30, 0 users, load average: 0.18, 0.19, 0.20 Mon Dec 19 09:36:50 MEZ 2011

desdp05.nycomed.local : 9:36am up 225 days, 1:05, 0 users, load average: 0.05, 0.05, 0.04 Mon Dec 19 09:36:50 MEZ 2011

desdp06.nycomed.local : 9:36am up 224 days, 21:09, 0 users, load average: 0.04, 0.04, 0.04 Mon Dec 19 09:36:51 MEZ 2011

osctrl09(OVO8):/root/home/hpeboyko (root)

===================================== Examples === One line

for i in $(cat ./nsn.txt);do echo "Checking for dead processes";echo $i; ovdeploy -cmd "ps afuxwww | grep ' D '" -node $i;done

osctrl08:/root (root) cat .sh_history.hpekospa.201002260853

cd /root/home/hpekospa ll touch nsn.txt ll chown hpekospa.uxsup1 nsn.txt chown hpekospa:uxsup1 nsn.txt more nsn.txt super hcreport | grep hosting.emea.com touch nsn.txt ll vi nsn.txt vi nsn.txt for i in $(nsn.txt);do echo $i;done for i in $(cat ./nsn.txt);do echo $i;done for i in $(cat ./nsn.txt);do echo "Checking for dead processes";echo $i;done for i in $(cat ./nsn.txt);do echo "Checking for dead processes";echo $i; ovdeploy -cmd "ps afuxwww | grep ' D '" -node $i;done osctrl08:/root (root) cd home osctrl08:home (root) cd /root/home/hpekospa osctrl08:home/hpekospa (root) for i in $(cat ./nsn.txt);do echo "Checking for dead processes";echo $i; ovdeploy -cmd "ps auxwww | grep ' D '" -node $i;done Checking for dead processes viedvmw1.doam.hosting.emea.com root 24048 0.0 0.1 61312 1044 ? R 11:19 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes viedvmw2.doam.hosting.emea.com root 10258 0.0 0.1 61316 1056 ? S 11:19 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes viedvmw3.doam.hosting.emea.com root 6842 0.0 0.1 61328 1044 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes viedvmw4.doam.hosting.emea.com root 8943 0.0 0.1 63424 1132 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes viedvmw5.doam.hosting.emea.com root 2256 0.0 2.1 47664 17124 ? D Feb23 0:05 /usr/sbin/snmpd -s -l /dev/null -P /var/run/snmpd -a root 4747 0.0 0.8 9076 6652 ? D Feb23 0:00 ./racsvm -u root 5359 0.0 0.0 1668 520 ? D Feb23 0:00 /usr/sbin/updfstab root 5370 0.0 0.0 1688 524 ? D Feb23 0:00 /usr/sbin/updfstab root 5651 0.0 0.0 1672 516 ? D Feb23 0:00 /usr/sbin/updfstab root 5658 0.0 0.0 1688 524 ? D Feb23 0:00 /usr/sbin/updfstab root 5669 0.0 0.0 1684 520 ? D Feb23 0:00 /usr/sbin/updfstab root 5672 0.0 0.0 1688 516 ? D Feb23 0:00 /usr/sbin/updfstab root 5834 0.0 0.0 1680 524 ? D Feb23 0:00 /usr/sbin/updfstab root 5842 0.0 0.0 1684 520 ? D Feb23 0:00 /usr/sbin/updfstab root 5855 0.0 0.0 1688 520 ? D Feb23 0:00 /usr/sbin/updfstab root 15685 0.0 0.0 1684 520 ? D Feb26 0:00 /usr/sbin/updfstab root 15694 0.0 0.0 1680 520 ? D Feb26 0:00 /usr/sbin/updfstab root 15699 0.0 0.0 1680 520 ? D Feb26 0:00 /usr/sbin/updfstab root 6631 0.0 0.1 4248 1048 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes viedvmw6.doam.hosting.emea.com root 2256 0.0 2.3 48732 18480 ? D Feb23 0:05 /usr/sbin/snmpd -s -l /dev/null -P /var/run/snmpd -a root 6278 0.0 0.8 9056 6656 ? D Feb23 0:00 ./racsvm -u root 6624 0.0 0.0 1680 524 ? D Feb23 0:00 /usr/sbin/updfstab root 6629 0.0 0.0 1688 528 ? D Feb23 0:00 /usr/sbin/updfstab root 6630 0.0 0.0 1680 524 ? D Feb23 0:00 /usr/sbin/updfstab root 6682 0.0 0.0 1680 520 ? D Feb23 0:00 /usr/sbin/updfstab root 6693 0.0 0.0 1684 516 ? D Feb23 0:00 /usr/sbin/updfstab root 7084 0.0 0.0 1688 520 ? D Feb23 0:00 /usr/sbin/updfstab root 7095 0.0 0.0 1688 520 ? D Feb23 0:00 /usr/sbin/updfstab root 7106 0.0 0.0 1680 520 ? D Feb23 0:00 /usr/sbin/updfstab root 7111 0.0 0.0 1676 520 ? D Feb23 0:00 /usr/sbin/updfstab root 7262 0.0 0.0 1664 516 ? D Feb23 0:00 /usr/sbin/updfstab root 7281 0.0 0.0 1680 520 ? D Feb23 0:00 /usr/sbin/updfstab root 7294 0.0 0.0 1684 516 ? D Feb23 0:00 /usr/sbin/updfstab root 16220 0.0 0.0 1688 520 ? D Feb26 0:00 /usr/sbin/updfstab root 16223 0.0 0.0 1680 520 ? D Feb26 0:00 /usr/sbin/updfstab root 16233 0.0 0.0 1680 520 ? D Feb26 0:00 /usr/sbin/updfstab root 26429 0.0 0.1 4260 1048 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes viedvmw7.doam.hosting.emea.com root 872 0.0 0.1 4252 1056 ? S 11:20 0:00 /bin/sh -c ps auxwww | grep ' D ' root 874 0.0 0.0 3692 676 ? S 11:20 0:00 grep D

Checking for dead processes viedvmw8.doam.hosting.emea.com root 783 0.0 0.1 4252 1048 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes vieivmw1.oam.hosting.emea.com root 24940 0.0 0.1 61316 1044 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes vieivmw12.oam.hosting.emea.com root 27658 0.0 0.1 4248 1052 ? S 11:20 0:00 /bin/sh -c ps auxwww | grep ' D ' root 27660 0.0 0.0 3688 676 ? S 11:20 0:00 grep D

Checking for dead processes vieivmw13.oam.hosting.emea.com root 27720 0.0 0.1 4248 1056 ? S 11:20 0:00 /bin/sh -c ps auxwww | grep ' D ' root 27722 0.0 0.0 3692 676 ? S 11:20 0:00 grep D

Checking for dead processes vieivmw2.oam.hosting.emea.com root 26953 0.0 0.1 61328 1040 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes vieivmw3.oam.hosting.emea.com root 25185 0.0 0.1 61312 1048 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes vieivmw4.oam.hosting.emea.com root 2683 0.0 0.1 61312 1044 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes vieivmw5.oam.hosting.emea.com root 1363 0.0 0.1 63400 1132 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes vieivmw6.oam.hosting.emea.com root 12124 0.0 0.8 9056 6648 ? D Feb23 0:00 ./racsvm -u root 12327 0.0 0.0 1668 520 ? D Feb23 0:00 /usr/sbin/updfstab root 12329 0.0 0.0 1676 516 ? D Feb23 0:00 /usr/sbin/updfstab root 12666 0.0 0.0 1692 532 ? D Feb23 0:00 /usr/sbin/updfstab root 12677 0.0 0.0 1692 520 ? D Feb23 0:00 /usr/sbin/updfstab root 12688 0.0 0.0 1672 516 ? D Feb23 0:00 /usr/sbin/updfstab root 12689 0.0 0.0 1672 516 ? D Feb23 0:00 /usr/sbin/updfstab root 12860 0.0 0.0 1672 520 ? D Feb23 0:00 /usr/sbin/updfstab root 12884 0.0 0.0 1684 520 ? D Feb23 0:00 /usr/sbin/updfstab root 12892 0.0 0.0 1676 516 ? D Feb23 0:00 /usr/sbin/updfstab root 27181 0.0 0.8 9056 6648 ? D Feb23 0:00 ./racsvm -u root 27320 0.0 0.0 1688 520 ? D Feb23 0:00 /usr/sbin/updfstab root 27331 0.0 0.0 1696 516 ? D Feb23 0:00 /usr/sbin/updfstab root 27618 0.0 0.0 1668 528 ? D Feb23 0:00 /usr/sbin/updfstab root 27630 0.0 0.0 1680 520 ? D Feb23 0:00 /usr/sbin/updfstab root 27641 0.0 0.0 1696 516 ? D Feb23 0:00 /usr/sbin/updfstab root 27642 0.0 0.0 1688 516 ? D Feb23 0:00 /usr/sbin/updfstab root 27810 0.0 0.0 1676 520 ? D Feb23 0:00 /usr/sbin/updfstab root 27834 0.0 0.0 1672 520 ? D Feb23 0:00 /usr/sbin/updfstab root 27842 0.0 0.0 1680 516 ? D Feb23 0:00 /usr/sbin/updfstab root 26382 0.0 0.0 1684 520 ? D Feb26 0:00 /usr/sbin/updfstab root 26395 0.0 0.0 1688 520 ? D Feb26 0:00 /usr/sbin/updfstab root 26398 0.0 0.0 1684 516 ? D Feb26 0:00 /usr/sbin/updfstab root 30555 0.0 0.1 4244 1056 ? S 11:20 0:00 /bin/sh -c ps auxwww | grep ' D ' root 30557 0.0 0.0 3692 672 ? S 11:20 0:00 grep D

Checking for dead processes vieivmw7.oam.hosting.emea.com root 28918 0.0 0.1 4244 1056 ? S 11:20 0:00 /bin/sh -c ps auxwww | grep ' D ' root 28920 0.0 0.0 3680 672 ? S 11:20 0:00 grep D

Checking for dead processes vieivmw8.oam.hosting.emea.com root 11398 0.0 0.1 4248 1048 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

Checking for dead processes vieivmw9.oam.hosting.emea.com root 21171 0.0 0.1 4252 1048 ? R 11:20 0:00 /bin/sh -c ps auxwww | grep ' D '

osctrl08:home/hpekospa (root) cat nsn.txt viedvmw1.doam.hosting.emea.com viedvmw2.doam.hosting.emea.com viedvmw3.doam.hosting.emea.com viedvmw4.doam.hosting.emea.com viedvmw5.doam.hosting.emea.com viedvmw6.doam.hosting.emea.com viedvmw7.doam.hosting.emea.com viedvmw8.doam.hosting.emea.com vieivmw1.oam.hosting.emea.com vieivmw12.oam.hosting.emea.com vieivmw13.oam.hosting.emea.com vieivmw2.oam.hosting.emea.com vieivmw3.oam.hosting.emea.com vieivmw4.oam.hosting.emea.com vieivmw5.oam.hosting.emea.com vieivmw6.oam.hosting.emea.com vieivmw7.oam.hosting.emea.com vieivmw8.oam.hosting.emea.com vieivmw9.oam.hosting.emea.com osctrl08:home/hpekospa (root)

 Adecco:
 restarting SSHD , using OVO from adeovo01(OVO8)

 	/opt/OV/bin/ovdeploy -cmd "service nscd restart" -node Node_FQDN


	/opt/OV/bin/ovdeploy -cmd "service sshd restart" -node Node_FQDN

You have to restart BOTH nscd and ssh NOT just SSH - Adecco uses LDAP and ssh won't start, if there is no NSCD running !!!

Georgiev, Aleksandar (Unix TS) 22:11 dbovo01:/root (root) /opt/OV/bin/ovdeploy -cmd "id; uptime; uname -a; date;" -node w96468.nat.danskenet.net uid=0(root) gid=0(root) 21:11:03 up 28 days, 21:56, 1 user, load average: 0.14, 0.08, 0.08 Linux W96468 2.6.18-274.17.1.el5 #1 SMP Wed Jan 4 22:45:44 EST 2012 x86_64 x86_64 x86_64 GNU/Linux Fri Feb 15 21:11:03 CET 2013

dbovo01:/roooot)

separate /opt/OV/bin/ovdeploy -cmd "id; uptime; uname -a; date;" -node triton4-back.dnb.lv

Upload:

[root@alovom20 ~]# ovdeploy -upload -file /root/home/hpenedem/libcrypto.so.0.9.8e -td /tmp/ -host myseslbrmwsaps01.gemoney.net INFO: File successfully uploaded.

/opt/OV/bin/ovdeploy -cmd "/etc/init.d/sshd restart" -node sl04744.e-ssi.net