Friday, June 17, 2011

5120's & LDOMs

New Stuff from sun with T5120 T5140's


Power on via the ilom

start /SYS

then switch to the console

start /SP/console

Default ilom password
root , changeme

Changing the ilom default password
set /SP/users/root password

Enter new Password: *******

Reset machine back to factory defaults
with a running machine
ldm set-spconfig factory-default
or from the service processor
bootmode config="factory-default"



Enable/ Disable SSH
set /SP/services/ssh state=[enable|disable]

Display the active ILOM sessions
show /SP/sessions


Display information about commands
show /SP/cli/commands



Add a local user
create /SP/users/bob password=password role=administrator|operator


Delete a local user
delete /SP/users/fred




change the ip address to static

cd /SP/network
set pendingipdiscovery=static
set pendingipaddress=xxx.xxx.xxx.xxx
set pendingipnetmask=yyy.yyy.yyy.yyy
set pendingipgatwat=zzz.zzz.zzz.zzz

show to verify setings

set commitpending=true



Enable web interface
set /SP/services/http state=enabled


Reset Service Processor
reset /SP


Boot System
start /SYS
(reset /SYS - for a vulgar system reset)





How do i break into the obp on a 5120

set /HOST send_break_action=break
start /SP/console
r)eboot, o)k prompt, h)alt?
o


- why , every domain gets it's own obp

the older revision procedure is to send a break , select r to reboot , then send another break to get to the obp for that ldom


one other way
set /HOST/bootmode script="setenv auto-boot? false"
then init 0 or / reset /sys

Jumpstart from the control domain ldom

so you have build your jet/jumpstart server and your first ldom host , and try and boot net install

but all you get is arp timeouts

you even snoop your interfaces looking for packets ... but still nothing!

ok here is the fix

(my example uses two interfaces)


first you need to use other interfaces than your e1000g0 g1 etc

remove your vsw interfaces from primary control domain (if configured)

then examine the real mac addesses for your interfaces

-bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000

e1000g0: flags=1000843 mtu 1500 index 2
inet xx.yy.zz.40 netmask fffff000 broadcast x.x.x.x
ether xx:xx:xx:xx:be:7b

e1000g1: flags=1000843 mtu 1500 index 3
inet zz.yy.zz.40 netmask fffff000 broadcast x.x.x.x
ether xx:xx:xx:xx:be:7a



now unplumb both interfaces

-bash-3.00# ifconfig e1000g0 unplumb

-bash-3.00# ifconfig e1000g1 unplumb

now configure the virtual switch interfaces

-bash-3.00# ldm add-vsw mac-addr= net-dev=e1000g0 vsw0 primary
-bash-3.00# ldm add-vsw mac-addr= net-dev=e1000g0 vsw0 primary

now ifconfig vsw0 and vsw1

-bash-3.00# ifconfig vsw0 plumb

-bash-3.00# ifconfig vsw1 plumb

and ifconfig your interfaces as you add them before

-bash-3.00# ifconfig vsw0 up xx.yy.zz.40 netmask + broadcast +


-bash-3.00# ifconfig vsw1up zz.yy.zz.40 netmask + broadcast +

no ifconfig will show :


-bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000

vsw0: flags=1000843 mtu 1500 index 2
inet xx.yy.zz.40 netmask fffff000 broadcast x.x.x.x
ether xx:xx:xx:xx:be:7b

vsw1: flags=1000843 mtu 1500 index 3
inet zz.yy.zz.40 netmask fffff000 broadcast x.x.x.x
ether xx:xx:xx:xx:be:7a
copy your interface files from the old e1000g0 files so your changes survive a reboot


-bash-3.00# mv /etc/hostname.e1000g0 /etc/hostname.vsw0

-bash-3.00# mv /etc/hostname.e1000g1 /etc/hostname.vsw1



you should now be able to jumpstart from your control domain

these steps are not required if your jumpstart is remote from the ldom host




Set up the hardware mirror (t2000 & t5120)
think really carefully before you do this ... if you dont leave a slice for liveupgrade you will not be able to split the mirror for patching or os upgrade


(best done from cdrom/net boot as you get into the well known catch 22 problem)

/usr/sbin/raidctl [-f] -c cXtXdX cYtYdY


# raidctl
Controller: 1
Disk: 0.0.0
Disk: 0.1.0
# format < /dev/null Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c1t0d0
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
1. c1t1d0
/pci@0/pci@0/pci@2/scsi@0/sd@1,0
Specify disk (enter its number):
# raidctl -c c1t0d0 c1t1d0
Creating RAID volume will destroy all data on spare space of member disks, proceed (yes/no)? y
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
Physical disk 0 created.
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
Physical disk 1 created.
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
Volume 0 is |enabled||optimal|
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
Volume 0 is |enabled||optimal|
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
Volume 0 created.
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
Physical disk (target 1) is |out of sync||online|
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
Volume 0 is |enabled||degraded|
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
Volume 0 is |enabled||resyncing||degraded|
Volume c1t0d0 is created successfully!
# raidctl
Controller: 1
Volume:c1t0d0
Disk: 0.0.0
Disk: 0.1.0


List Ldoms
# ./ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-c- SP 32 8064M 0.0% 3m



Setup ldom default service

# /opt/SUNWldm/bin/ldm add-vds primary-vds0 primary
# /opt/SUNWldm/bin/ldm add-vcc port-range=5000-5100 primary-vcc0 primary
# /opt/SUNWldm/bin/ldm add-vsw net-dev=e1000g0 primary-vsw0 primary
# /opt/SUNWldm/bin/ldm set-mau 1 primary
# /opt/SUNWldm/bin/ldm set-vcpu 4 primary


List ldom services

# /opt/SUNWldm/bin/ldm list-services primary
VCC
NAME PORT-RANGE
primary-vcc0 5000-5100

VSW
NAME MAC NET-DEV DEVICE MODE
primary-vsw0 00:14:4f:fb:37:32 e1000g0 switch@0

VDS
NAME VOLUME OPTIONS DEVICE
primary-vds0


Listing ldom sp config

# /opt/SUNWldm/bin/ldm list-spconfig
factory-default [current]



Saving sp config
# /opt/SUNWldm/bin/ldm add-spconfig initial
# /opt/SUNWldm/bin/ldm list-spconfig
factory-default [current]
initial [next]

(remember to reboot if you have made changes)

List ldom service bindings
bash-3.00# ./ldm list-bindings primary
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv SP 4 1G 0.0% 3m

MAC
00:21:28:ZZ:YY:XX

VCPU
VID PID UTIL STRAND
0 0 0.0% 100%
1 1 0.0% 100%
2 2 0.0% 100%
3 3 0.0% 100%

MAU
ID CPUSET
0 (0, 1, 2, 3, 4, 5, 6, 7)

MEMORY
RA PA SIZE
0x8000000 0x8000000 1G

VARIABLES
auto-boot?=false
boot-device=/pci@0/pci@0/pci@2/scsi@0/disk@0,0:a disk net
keyboard-layout=US-English

IO
DEVICE PSEUDONYM OPTIONS
pci@0 pci
niu@80 niu

VCC
NAME PORT-RANGE
primary-vcc0 5000-5100

VSW
NAME MAC NET-DEV DEVICE MODE
primary-vsw0 00:14:4f:fb:37:32 e1000g0 switch@0

VDS
NAME VOLUME OPTIONS DEVICE
primary-vds0

VCONS
NAME SERVICE PORT
SP



Example simple Ldom
bash-3.00# ldm add-domain secondary
bash-3.00# ldm add-vcpu 12 secondary
bash-3.00# ldm add-memory 1G secondary
bash-3.00# ldm add-vnet vnet1 primary-vsw0 secondary
bash-3.00# ldm add-vdsdev /dev/zvol/rdsk/domains/secondary vol1@primary-vds0
bash-3.00# ldm add-vdisk vdisk1 vol1@primary-vds0 secondary
bash-3.00# ldm set-variable auto-boot\?=false secondary
bash-3.00# ldm set-variable boot-device=/virtual-devices@100/channel-devices@20
0/disk@0 secondary
bash-3.00# ldm bind-domain secondary
bash-3.00# ldm start-domain secondary

LDom secondary started
bash-3.00# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "secondary" in group "secondary" ....
Press ~? for control options ..

{0} ok banner

SPARC Enterprise T5120, No Keyboard
Copyright 2008 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.28.0, 1024 MB memory available, Serial #66796533.
Ethernet address 0:14:4f:fb:3b:f5, Host ID: 83fb3bf5.




##list physical cdrom / cdr drives
-bash-3.00# cdrw -l
Looking for CD devices...
Node Connected Device Device type
----------------------+--------------------------------+-----------------
/dev/rdsk/c0t0d0s2 | TSSTcorp CD/DVDW TS-T632A SR03 | CD Reader/Writer



Set up an iso image as a vdisk (to boot from)
-bash-3.00# ldm add-vdsdev /var/tmp/sol-10-u5-ga-sparc-dvd.iso iso@primary-vds0
-bash-3.00# ldm add-vdisk iso iso@primary-vds0 secondary


Yes it will run a sparc linux system:)
this was a quick test using the iso tip abobe and booting from the cdrom




~ # cat /proc/cpuinfo
cpu : UltraSparc T2 (Niagara2)
fpu : UltraSparc T2 integrated FPU
prom : OBP 4.28.0 2008/01/22 21:10
type : sun4v
ncpus probed : 12
ncpus active : 1
D$ parity tl1 : 0
I$ parity tl1 : 0
Cpu0ClkTck : 00000000457646c0
MMU Type : Hypervisor (sun4v)






ilom/Sp Is LINUX !

U-Boot 1.1.1 (Apr 3 2008 - 19:06:21)

CPU: MPC885ZPnn at 133 MHz: 8 kB I-Cache 8 kB D-Cache FEC present
Board: SPARC885
Watchdog enabled
I2C: ready
DRAM:
trying 128 MBytes
trying 64 MBytes
(64 MB SDRAM) 64 MB
Memory Tests: DA A1 A2 00 FF 55 AA T2 T3 T4
POST memory PASSED
FLASH: 32 MB
In: serial
Out: serial
Err: serial
Net: FEC ETHERNET
POST i2c c d 18 20 23 2a 2d 2e 30 40 43 46 51 53 54 56 68 69 6a 6b 70 71 PASSED
POST cpu PASSED
POST ethernet PASSED
Booting linux in 5 seconds...
## Booting image at fe080000 ...
Image Name: Linux-2.4.22
Image Type: PowerPC Linux Kernel Image (gzip compressed)
Data Size: 814987 Bytes = 795.9 kB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK

Solaris Patching (SPARC) - non liveupgrade

Prep work - Backup the entire system :
Flar , ufsdump , networker , etc

I find flar the fastest as it will give you an option for complete recovery in the event of a total abort

a good practice is to copy your /etc/system file somewhere save , I would also grab the metastat -p , metastat , metadb and eeprom , outputs and keep them off the machine

metastat -p |mailx -s "meta db for target x" me@company

repeat for any metasets !

and while we are at it .. grab the /etc/lvm directory too!




Verify Quorum flag in /etc/system
set md:mirrored_root_flag=1



backup metadb's

metadb
flags first blk block count
a m p luo 16 8192 /dev/dsk/c1t0d0s3
a p luo 8208 8192 /dev/dsk/c1t0d0s3
a p luo 16 8192 /dev/dsk/c1t0d0s4
a p luo 8208 8192 /dev/dsk/c1t0d0s4
a p luo 16 8192 /dev/dsk/c1t1d0s4
a p luo 8208 8192 /dev/dsk/c1t1d0s4


dd if=/dev/rdsk/c1t0d0s3 of=//metadb.c1t0d0s3 bs=2048k
dd if=/dev/rdsk/c1t0d0s4 of=//metadb.c1t0d0s4 bs=2048k
dd if=/dev/rdsk/c1t1d0s3 of=//metadb.c1t1d0s3 bs=2048k





DETACH ROOT MIRROR DISK

Check which is the root meta device: df –k /
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d10 66419890 8134395 57621297 13% /

# metastat -p d10

a grab the second copy


# metadetach d10 d12
d10: submirror d12 is detached

verify with a metastat command the detach successful

remove disk from metadb


# metadb -d /dev/dsk/c1t1d0s7

You are now ready to patch , i find localcopy to work best as Solaris 10 had some issues wth patching accross nfs!

Patching is Successful

resilver the submirror

# metattach d10 d12
Add disk back into metadb


# metadb -a -c3 /dev/dsk/c1t1d0s7

resilver will take a while , you can monitor in the usual way



DISASTER - Patching has Failed , total recal required

(you need to back out the patches patchrm them one by one)

Boot using corrupt/patched environment. If the patched environment is not bootable, boot using net , cdrom.
now mount the basic device file from the old mirror disk and replace the rootdev entry in the /etc/system file (remember to edit the correct etc/system file)



vi system
details are from your saved eeprom outputs!


The details of below can be seen in step 3 output (add whatever 100,blk comes after the disk id)
example :
rootdev:/pseudo/md@0:0,100,blk
change to:
rootdev:/ssm@0,0/pci@18,600000/scsi@2/disk@1,0:a,100,blk


revert your vfstab back to pre svm

Replace disk entries of / with the new disk (comment out the original and make a copy with the following)

/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no logging
change to
/dev/dsk/c1t1d0s0 /dev/rdsk/c1t1d0s0 / ufs 1 no logging

lockfs -fa;sync;sync;reboot


now boot from your mirror disk

if there are any remaning metadevices remove them



system will now be in a svm less pre patch state

rebuild the svm state from scratch using the now new boot disk as the primary mirror drive

Recover Metadb from backup

if all copies of the metadb have a problem, use dd to restore the dd backup.

Example:

dd if= /opt/sun/sdsbackup/metadb.c0t0d0s7.dd of=/dev/rdsk/c0t0d0s7 bs=2048k

Jumpstarting Zones

PREP

Zone host:
mkdir /usr/lib/jetzone/
ln -s /net//opt/SUNWjet/Products/zones/smf/jetzonefinish.xml /usr/lib/jetzone/jetzonefinish.xml
(or copy it)

add /etc/default/jetzone
JET_SERVER="i"
JET_CONFIG="/opt/SUNWjet"
JET_ZONEDIR="/zones"


cp /net//opt/SUNWjet/Products/zones/jetzone /usr/local/bin
chmod +x /usr/local/bin/jetzone



Configure Zone on JET server

Jumpstart server :
make_zone_template -f -T SUS_ZONE
make_client -f


BUILDING
jetzone -F bluecat
JumpStart Enterprise Toolkit Zone creation: bluecat
Cleaning out zone bluecat
Uninstall zone bluecat
zoneadm: zone 'bluecat': is already in state 'configured'.
Cleaning up zone area
Mounting JET configuration area on /tmp/jet.8791
Loading zone bluecat configuration data
Forcing permissions to 700 for /zones/bluecat
Building zone bluecat configuration
Add network (aggr1/192.168.1.38/255.255.255.0) to zone
Creating zone bluecat
Installing zone bluecat
Preparing to install zone .
Creating list of files to copy from the global zone.
Copying <3135> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <585> packages on the zone.
Initialized <585> packages on zone.
Zone is initialized.
Installation of <2> packages was skipped.
The file contains a log of the zone installation.
Populating zone bluecat sysidcfg
Disable NFSv4 prompt
Adding JET hook to zone bluecat
cp: cannot access /usr/lib/jetzone/jetzonefinish.xml
Booting zone bluecat
To monitor the installation of the zone, please run the command
zlogin -C bluecat

[Connected to zone 'bluecat' console]
142/142
Reading ZFS config: done.
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: aggr1.

rebooting system due to change(s) in /etc/default/init


[NOTICE: Zone rebooting]


SunOS Release 5.10 Version Generic_142909-17 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
Hostname: bluecat
Reading ZFS config: done.
Starting JumpStart Enterprise Toolkit helper script
Mounting JET filesystem
Starting JET finish script
Installation of bluecat at 12:28 on 07-Dec-2010
Loading JumpStart Server variables
JumpStart Enterprise Toolkit version 4.8
Loading Client configuration file
Creating directory: /var/opt/sun/jet/post_install
Creating directory: /var/opt/sun/jet/Utils
Dec 7 12:28:18 bluecat sendmail[22716]: My unqualified host name (bluecat) unknown; sleeping for retry
Creating directory: /var/opt/sun/jet/config
Creating directory: /var/opt/sun/jet/js_media/patch
Creating directory: /var/opt/sun/jet/js_media/pkg
Dec 7 12:28:18 bluecat sendmail[22755]: My unqualified host name (bluecat) unknown; sleeping for retry
Copying file Clients/bluecat/sysidcfg to //var/opt/sun/jet/config/sysidcfg
Copying file Clients/bluecat/host.config to //var/opt/sun/jet/config/host.config
Copying file Utils/solaris/releaseinfo to //var/opt/sun/jet/config/releaseinfo
Copying functions to //var/opt/sun/jet/Utils/lib
Copying file Clients/bluecat/module_hints to //var/opt/sun/jet/config/module_hints
NFS Mounting Media Directories
Mounting nfs://192.168.1.86/export/install/patches on //var/opt/sun/jet/js_media/patch
Mounting nfs://192.168.1.86/export/install/pkgs on //var/opt/sun/jet/js_media/pkg
BASE_CONFIG: Running base_config install script....
BASE_CONFIG: Product base_config started
BASE_CONFIG: Trying to use external matrix cmd /tmp/jet/Products/zones/empty-matrix
BASE_CONFIG: External matrix cmd returned non-zero status or no entries
BASE_CONFIG: Unable to locate package.matrix file... exiting.
BASE_CONFIG: No HW specific packages for platform SUNW,SPARC-Enterprise-T5220
BASE_CONFIG: Trying to use external matrix cmd /tmp/jet/Products/zones/empty-matrix
BASE_CONFIG: External matrix cmd returned non-zero status or no entries
BASE_CONFIG: Unable to locate patch.matrix file... exiting.
BASE_CONFIG: No HW specific patches for platform SUNW,SPARC-Enterprise-T5220
BASE_CONFIG: Set root password
BASE_CONFIG: Setting netmask for primary interface
BASE_CONFIG: Add netmask 192.168.1.0 / 255.255.255.0
BASE_CONFIG: Disabling power management
BASE_CONFIG: Creating directory: /var/opt/sun/jet/post_install/n-post
BASE_CONFIG: Register postinstall script 'setupdumpdevice' for boot n
BASE_CONFIG: Creating directory: /var/opt/sun/jet/post_install/n
BASE_CONFIG: Register postinstall script 'run_sshkeygen' for boot n
BASE_CONFIG: Setting system terminal type to vt100
BASE_CONFIG: Register postinstall script 'console' for boot n
BASE_CONFIG: Setting NFSv4 domain
BASE_CONFIG: Creating directory: /var/opt/sun/jet/system.add
BASE_CONFIG: Product base_config finished
BASE_CONFIG: Running base_config install script....
BASE_CONFIG: Product base_config started
BASE_CONFIG: Trying to use external matrix cmd /tmp/jet/Products/zones/empty-matrix
BASE_CONFIG: External matrix cmd returned non-zero status or no entries
BASE_CONFIG: Unable to locate package.matrix file... exiting.
BASE_CONFIG: No HW specific packages for platform SUNW,SPARC-Enterprise-T5220
BASE_CONFIG: Trying to use external matrix cmd /tmp/jet/Products/zones/empty-matrix
BASE_CONFIG: External matrix cmd returned non-zero status or no entries
BASE_CONFIG: Unable to locate patch.matrix file... exiting.
BASE_CONFIG: No HW specific patches for platform SUNW,SPARC-Enterprise-T5220
BASE_CONFIG: Set root password
BASE_CONFIG: Setting netmask for primary interface
BASE_CONFIG: Disabling power management
BASE_CONFIG: Register postinstall script 'setupdumpdevice' for boot n
BASE_CONFIG: Register postinstall script 'run_sshkeygen' for boot n
BASE_CONFIG: Setting system terminal type to vt100
BASE_CONFIG: Register postinstall script 'console' for boot n
BASE_CONFIG: Setting NFSv4 domain
BASE_CONFIG: Product base_config finished
CUSTOM: Running custom install script....
CUSTOM: Copying file Clients/bluecat/../common.files/ntp.conf to //etc/inet/ntp.conf
CUSTOM: Copying file Clients/bluecat/../common.files/etc_profile to //etc/profile
CUSTOM: Copying file Clients/bluecat/../common.files/ITC_jet_post to //post_install/ITC_jet_post
CUSTOM: Copying file Clients/bluecat/../common.files/deploy_foo_solaris to //post_install/deploy_altiris_solaris
CUSTOM: Copying file Clients/bluecat/../common.files/syslog.conf to //etc/syslog.conf
CUSTOM: Copying file Clients/bluecat/../common.files/jfdi to //post_install/jfdi
CUSTOM: Register postinstall script 'ITC_jet_post' for boot n
----------------------------------------------------------
Product modules processed, finish up installation tasks
Creating directory: /var/opt/sun/jet/system.add/updated
Copying file etc/jumpstart.conf to //var/opt/sun/jet/config/jumpstart.conf
Copying file Utils/smf/jetjump.xml to //var/svc/manifest/site/jetjump.xml
Copying file Utils/S99jumpstart to //var/opt/sun/jet/post_install/S99jumpstart
NFS Unmounting Media Directories
Unmounting /var/opt/sun/jet/js_media/pkg
Unmounting /var/opt/sun/jet/js_media/patch
Make a link to finish log...
Updating boot-archive
/: not a boot archive based Solaris instance
Disable & delete SMF tag svc:/site/jetzonefinish

bluecat console login: Dec 7 12:28:28 bluecat reboot: rebooted by LOGIN
Dec 7 12:28:28 bluecat syslogd: going down on signal 15
Dec 7 12:28:28 /usr/lib/snmp/snmpdx: received signal 15

[NOTICE: Zone rebooting]


SunOS Release 5.10 Version Generic_142909-17 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
Hostname: bluecat
Loading smf(5) service descriptions: 1/1
Reading ZFS config: done.
JumpStart (/var/opt/sun/jet/post_install/S99jumpstart) started @ Tue Dec 7 12:28:59 MET 2010
Loading JumpStart Server variables
Loading Client configuration file
No more reboots required
Running additional install files for reboot n
NFS Mounting Media Directories
Mounting nfs://192.168.1.86/export/install/patches on /var/opt/sun/jet/js_media/patch
Dec 7 12:28:59 bluecat sendmail[24360]: My unqualified host name (bluecat) unknown; sleeping for retry
Dec 7 12:28:59 bluecat sendmail[24364]: My unqualified host name (bluecat) unknown; sleeping for retry
Mounting nfs://192.168.1.86/export/install/pkgs on /var/opt/sun/jet/js_media/pkg
BASE_CONFIG: Running 001.base_config.001.run_sshkeygen
BASE_CONFIG: Running 001.base_config.002.console
BASE_CONFIG: Setting terminal type to vt100
BASE_CONFIG: Running 001.base_config.003.run_sshkeygen
BASE_CONFIG: Running 001.base_config.004.console
BASE_CONFIG: Setting terminal type to vt100
CUSTOM: Running 002.custom.001.ITC_jet_post
Dec 7 12:29:31 bluecat sendmail[25179]: My unqualified host name (bluecat) unknown; sleeping for retry
Dec 7 12:29:59 bluecat sendmail[24360]: unable to qualify my own domain name (bluecat) -- using short name
Dec 7 12:29:59 bluecat sendmail[24364]: unable to qualify my own domain name (bluecat) -- using short name
Dec 7 12:30:14 bluecat sshd[25284]: Failed keyboard-interactive for root from 10.128.101.58 port 56954 ssh2
Dec 7 12:30:28 bluecat login: ROOT LOGIN /dev/pts/4
Dec 7 12:30:31 bluecat sendmail[25179]: unable to qualify my own domain name (bluecat) -- using short name
Dec 7 12:31:12 bluecat ntpdate[25718]: can't find host ntp1.ch.sus.local
Dec 7 12:31:12 bluecat ntpdate[25718]: can't find host ntp2.ch.sus.local
Dec 7 12:31:12 bluecat ntpdate[25718]: can't find host ntp3.ch.sus.local
Dec 7 12:31:12 bluecat ntpdate[25718]: can't find host ntp4.ch.sus.local
Dec 7 12:31:12 bluecat ntpdate[25718]: no servers can be used, exiting
Dec 7 12:31:12 bluecat xntpd[25720]: sched_setscheduler(): Not owner
Dec 7 12:31:12 bluecat xntpd[25720]: loop_config: ntp_adjtime() failed: Not owner
Dec 7 12:31:12 bluecat last message repeated 1 time
Dec 7 12:31:14 bluecat xntpd[25721]: couldn't resolve `ntp1.ch.sus.local', giving up on it
Dec 7 12:31:14 bluecat xntpd[25721]: couldn't resolve `ntp2.ch.sus.local', giving up on it
Dec 7 12:31:14 bluecat xntpd[25721]: couldn't resolve `ntp3.ch.sus.local', giving up on it
Dec 7 12:31:14 bluecat xntpd[25721]: couldn't resolve `ntp4.SUS', giving up on it

BASE_CONFIG: Running 001.base_config.001.setupdumpdevice
BASE_CONFIG: Running 001.base_config.002.setupdumpdevice
NFS Unmounting Media Directories
Unmounting /var/opt/sun/jet/js_media/pkg
Unmounting /var/opt/sun/jet/js_media/patch
Disable & delete SMF tag svc:/site/jetjump

bluecat console login: JumpStart is complete @ Tue Dec 7 12:32:44 MET 2010

Solaris and SAN

Scanning for a new Lun
luxadm probe -p
cfgadm -al


Should be Enough:
emcpdiscover
powermt check




Force HBA to scan bus


cfgadm -al|grep fc-fabric
c1 fc-fabric connected configured unknown
c2 fc-fabric connected configured unknown


cfgadm -o force_update -c configure cX


cX taken from above




UNLIC paths
suddenly your paths become like this :
Pseudo name=emcpower28a
CLARiiON ID=CKxxxxxxxx [host]
Logical device ID=600601604506260086790600F2F3DF11 [DOCUMENTUM_zone_seg_2]
state=alive; policy=BasicFailover; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A Array failover mode: 1
==============================================================================
---------------- Host --------------- - Stor - -- I/O Path - -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
3076 pci@2,600000/SUNW,qlc@0/fp@0,0 c1t500601623B20211Ed20s0 SP A2 active alive 0 0
3076 pci@2,600000/SUNW,qlc@0/fp@0,0 c1t5006016B3B20211Ed20s0 SP B3 active alive 0 0
3074 pci@3,700000/SUNW,qlc@0,1/fp@0,0 c2t500601633B20211Ed20s0 SP A3 unlic alive 0 0
3074 pci@3,700000/SUNW,qlc@0,1/fp@0,0 c2t5006016A3B20211Ed20s0 SP B2 unlic alive 0 0




fix:
reset the policy to Clariion Optimization
powermt set policy=co dev=all
powermt save


Now :
Pseudo name=emcpower28a
CLARiiON ID=CKxxxxxxxxxx [host]
Logical device ID=600601604506260086790600F2F3DF11 [DOCUMENTUM_zone_seg_2]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A Array failover mode: 1
==============================================================================
---------------- Host --------------- - Stor - -- I/O Path - -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
3076 pci@2,600000/SUNW,qlc@0/fp@0,0 c1t500601623B20211Ed20s0 SP A2 active alive 0 0
3076 pci@2,600000/SUNW,qlc@0/fp@0,0 c1t5006016B3B20211Ed20s0 SP B3 active alive 0 0
3074 pci@3,700000/SUNW,qlc@0,1/fp@0,0 c2t500601633B20211Ed20s0 SP A3 active alive 0 0
3074 pci@3,700000/SUNW,qlc@0,1/fp@0,0 c2t5006016A3B20211Ed20s0 SP B2 active alive 0 0

Thursday, May 26, 2011

Non-global Solaris zone migration to another machine

Non-global Solaris zone migration to another machine

I had task to migrate Nagios with NagiosGrapher (running on Sparc zone) to another machine (SunFire X4200, yes i386 platform). Trus me, it was pain in the neck.

First I manually migrated "Nagios sparc zone" to test i386 platform and then it was simple doing migration from there to production platform (another X4200 with same Solaris release and additional patches).

So in this page I just explain how to migrate non-global zone between 2 totally identical systems.

System Serbia has zone France. France zone will be migrated to system Japan and will have new zone name India!

1. Configure whole root zone France to Serbia

{serbia}/zones> mkdir france


{serbia}/> zonecfg -z france

france: No such zone configured

Use 'create' to begin configuring a new zone.

zonecfg:france> create

zonecfg:france> set zonepath=/zones/france

zonecfg:france> add net

zonecfg:france:net> set physical=e1000g0

zonecfg:france:net> set address=192.168.20.222

zonecfg:france:net> end

zonecfg:france> verify

zonecfg:france> commit

zonecfg:france> exit

Since zone is whole root, remove inherited dirs

{serbia}/zones> zonecfg -z france
zonecfg:france> remove inherit-pkg-dir dir=/lib
zonecfg:france> remove inherit-pkg-dir dir=/platform
zonecfg:france> remove inherit-pkg-dir dir=/sbin
zonecfg:france> remove inherit-pkg-dir dir=/usr
zonecfg:france> verify
zonecfg:france> commit
zonecfg:france> exit

{serbia}/zones> zonecfg -z france info

zonename: france
zonepath: /zones/france
brand: native
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: shared
net:

address: 192.168.20.222

physical: e1000g0

defrouter not specified


2. Install the zone

{serbia}/zones> zoneadm -z france install

A ZFS file system has been created for this zone.

Preparing to install zone .

Creating list of files to copy from the global zone.

Copying <48022> files to the zone.

Initializing zone product registry.

Determining zone package initialization order.

Preparing to initialize <455> packages on the zone.

Initialized <455> packages on zone.

Zone is initialized.

The file contains a log of the zone installation.


The zone size is:

{serbia}/zones/france> du -sh .

680M .


{serbia}/zones/france> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- france installed /zones/france native shared


3. Boot the zone, but first place sysidcfg file in /zones/france/root/etc/


#The sysidcfg file:

#--------------------

system_locale=en_US

timezone=US/Pacific

timeserver=localhost


#name_service=DNS

# {domain_name=googlux.com

# name_server=xxx.xxx.xxx.xxx}



name_service=NONE


nfs4_domain=genesyslab.com

service_profile=limited_net

terminal=vt100

security_policy=NONE

root_password=NP

network_interface=primary { hostname=france

netmask=255.255.255.0

protocol_ipv6=no

ip_address=192.168.20.222

default_route=192.168.20.1}


{serbia}/> zoneadm -z france boot

{serbia}/> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

1 france running /zones/france native shared





4. Login to zone's console and make sure it works well.


{serbia}/> zlogin -C france

[Connected to zone 'france' console]

france console login:

or login to zone's terminal session


{serbia}/> zlogin france

[Connected to zone 'france' pts/2]

Sun Microsystems Inc. SunOS 5.10 Generic January 2005

# pwd

/

# hostname


*************************************************

Migration to the another machine

*************************************************


1. Now, when having zone, let's migrate it to another machine.

Halt and Detach the zone

{serbia}/> zoneadm -z france halt

{serbia}/> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- france installed /zones/france native shared


{serbia}/> zoneadm -z france detach

{serbia}/> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared


2. Migrate zone path to the new host

{serbia}/zones> rsync -avH france japan:/zones/


3. Rename {japan}/zones/france to /zones/india and adjust sysidcfg file as below

network_interface=primary { hostname=india

netmask=255.255.255.0

protocol_ipv6=no

ip_address=192.168.24.35

default_route=192.168.24.1}



4. Configure zone on the new system

{japan}/tmp> zonecfg -z india

india: No such zone configured

Use 'create' to begin configuring a new zone.

zonecfg:india> create -a /zones/india


zonecfg:india> info ---> see zone info

zonename: india

zonepath: /zones/india

brand: native

autoboot: false

bootargs:

pool:

limitpriv:

scheduling-class:

ip-type: shared

net:

address: 192.168.20.222 ----> change IP

physical: e1000g0

defrouter not specified



zonecfg:india> select net physical=e1000g0

zonecfg:india:net> set address=192.168.24.35

zonecfg:india:net> end

zonecfg:india> commit

zonecfg:india> exit

5. Attach the zone

{japan}/tmp> zoneadm -z india attach

{japan}/tmp> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- india installed /zones/india native shared

6. Boot the zone

{japan}/tmp> zoneadm -z india boot

{japan}/tmp> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

3 india running /zones/india native shared



{japan}/tmp> zlogin -C india

[Connected to zone 'india' console]

france console login: --------------> hostname has to be changed

7. Login to zone and change hostname to /etc/hosts and /etc/nodename. Done.

Tuesday, January 4, 2011

LDOM(1.0.2) quick guide

Terminalogy

Hypervisor

Hypervisor is a thin firmware layer that provides a stable virtualized machine architecture to which an operating system can be written.
It resides in the flas PROM of the motherboard and act as interface between operating system and the hardware.
It provides a set of support fuctions to the operating system, so that the OS does not need to know intimate details of how to
perform fuctions with the hardware.

Logical Domain

Logical domain is a discrete logical grouping with its own operating system, resources and identity within a single computer system.
Each logical domain can be created, destroyed, reconfigured, and rebooted independently, without requiring a power cycle of the
server. We can run a variety of applications software in different logical domains and keep them independent of performance
and security purposes.

Logical Domains Manager

The Logical Domains Manager is used to create and manage logical domains. There can be only
one Logical Domains Manager per serer. The Logical Domains Manager maps logical domains to physical resources.

Control Domain

Domain in which the Logical Domain Manager runs allowing you to create and manager other logical domains and allocate virtual resources
to other domains. There can be only one control domain per servers. The initial domain created when installing Logical Domains software is a
control domain and is named primary.

Service Domain

Domain that provides virtual device services to other domains, such as a virtual switch, a virtual console concentrator, and a
virtual disk server.

I/O Domain

Domain that has direct ownership of and direct access to physical I/O devices, such as a network card in a PCI express controller.
Shares the devices with other domains in the form of virtual devices wehn the I/O domain is also the control domain.

Guest Domain

Domain that is managed by the control domain and uses services from the I/O and service domains.


Patch Installation

Refer the Logical Domains 1.0.2 Release notes for the required patches/softwares for your OS release

System Firmware Upgrade

Refer the Logical Domains 1.0.2 Release notes for the required System Firmware patches for your OS release

Quick steps for upgrading Firmware

1) check the system firmware version using showhost

sc>showhost

2) Download the latest firmware based on your hardware model from http://sunsolve.sun.com



3) Use sysfwdownload utility comes with the downloaded firmware package to install the firmware

#./sysfwdownload [firmware].bin



4) Power off the server

#shutdown -i5 -g0 -y


5) Upgrade the firmware using flashupdate


sc>flashupdate -s 127.0.0.1



6) reset the system controller and poweron

sc>resetsc -y
sc>poweron
sc>boot

Logical Domain manager Installation


1) Download Logical DOmains Manager Package(LDoms_Manager-1_0_2.zip) from here http://www.sun.com/ldoms

2) unzip LDoms_Manager-1_0_2.zip file

3) Execute install-ldm script from the unzip folder



#./install-ldm
Welcome to the LDoms installer.

You are about to install the domain manager package that will enable
you to create, destroy and control other domains on your system. Given
the capabilities of the domain manager, you can now change the security
configuration of this Solaris instance using the Solaris Security
Toolkit.

Select a security profile from this list:

a) Hardened Solaris configuration for LDoms (recommended)
b) Standard Solaris configuration
c) Your custom-defined Solaris security configuration profile

Enter a, b, or c [a]:
The changes made by selecting this option can be undone through the
Solaris Security Toolkit's undo feature. This can be done with the
'/opt/SUNWjass/bin/jass-execute -u' command.
Installing LDoms and Solaris Security Toolkit packages.
pkgadd -n -d "/ldom/LDOM/LDoms_Manager-1_0_2/Product" -a pkg_admin SUNWldm.v
Copyright 2008 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.

Installation of (SUNWldm) was successful.
pkgadd -n -d "/ldom/LDOM/LDoms_Manager-1_0_2/Product" -a pkg_admin SUNWjass
Copyright 2005 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.

Installation of (SUNWjass) was successful.

Verifying that all packages are fully installed. OK.
Enabling services: svc:/ldoms/ldmd:default
Running Solaris Security Toolkit 4.2.0 driver ldm_control-secure.driver.
Please wait. . .
/opt/SUNWjass/bin/jass-execute -q -d ldm_control-secure.driver
Executing driver, ldm_control-secure.driver
Solaris Security Toolkit hardening executed successfully; log file
/var/opt/SUNWjass/run/20080617183524/jass-install-log.txt. It will not
take effect until the next reboot. Before rebooting, make sure SSH or
the serial line is setup for use after the reboot.

Default Service Creation

All LDOM related commands will be available under /opt/SUNWldm/bin/ldm. Add this path to your PATH variable(export PATH=$PATH:/opt/SUNWldm/bin/ldm)

Creating Virtual Disk server(vds)

Virtual disk server helps importing virtual disks into a logical domain from the control domain.

primary#ldm add-vds primary-vds0 primary

Creating Virtual Console concentrator Server(vcc)

Virtual Console concentrator server provides terminal service to logical domain consoles.

primary#ldm add-vcc port-range=5000-5100 primary-vcc0 primary


Creating Virtual Switch server(vsw)

Virtual Switch server enables networking between virtual network devices in logical domains.

primary#ldm add-vsw net-dev=e1000g0 primary-vsw0 primary

Listing the default service created

# ldm list-services primary
------------------------------------------------------------------------------
Notice: the LDom Manager is running in configuration mode. Configuration and
resource information is displayed for the configuration under construction;
not the current active configuration. The configuration being constructed
will only take effect after it is downloaded to the system controller and
the host is reset.
------------------------------------------------------------------------------
VDS
NAME VOLUME OPTIONS DEVICE
primary-vds0

VCC
NAME PORT-RANGE
primary-vcc0 5000-5100

VSW
NAME MAC NET-DEV DEVICE MODE
primary-vsw0 00:11:4a:19:ac:3c e1000g1 switch@0 prog,promisc

ether 0:2:ac:d9:11:12

Control Domain Creation

Creating the control domain with 4 cpu's and 1gb RAM.

#/opt/SUNWldm/bin/ldm set-mau 0 primary

#/opt/SUNWldm/bin/ldm set-vcpu 4 primary

#/opt/SUNWldm/bin/ldm set-memory 1024m primary

Make the modified configuration permanent using list-spconfig option

#ldm list-spconfig
factory-default [current]

#ldm add-spconfig initial

#ldm list-spconfig
factory-default [current]
initial [next]

Reboot the server to come up with initial configuration.

#shutdown -i6 -g0 -y


Enable Networking between domains

Networking between control,service and other domains is disabled by default. To enable this, the virtual switch device
should be configured as a network device.


Login to the server console and perform the following network configuration steps,

primary#ifconfig -a

Note down the name,ip configured for the primary interface.

Plumb the virtual switch(vsw0)

primary#ifconfig vsw0 plumb

Bring down the primary interface(use appropriate interface name in place of e1000g1)

primary#ifconfig e1000g1 down unplumb

Configure Virtual switch with the primary interface details (replace primary_interface_up,primary_interface_netmask appropriately)

primary#ifconfig vsw0 primary_interface_ip netmask primary_interface_netmask broadcast + up

Modify the hostname file to make this configuration permanent

primary#mv /etc/hostname.e1000g1 /etc/hostname.vsw0


Enable Virtual Network terminal server daemon

#svcadm enable vntsd

Logical Domain Creation

Create a logical domain with the following configuration

Domain name - gdomain
Number of CPU - 8
Memory - 1GB

# ldm add-domain gdomain
# ldm add-vcpu 8 gdomain
# ldm add-memory 1G gdomain
# ldm add-vnet vnet1 primary-vsw0 gdomain
# ldm add-vdsdev /dev/dsk/c0t2d0s2 vol1@primary-vds0
# ldm add-vdisk vdisk1 vol1@primary-vds0 gdomain
# ldm bind gdomain
# ldm set-var auto-boot\?=false gdomain
# ldm start-domain gdomain



#ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv SP 4 1G 0.2% 1h 3m
gdomain inactive ----- 8 1G


Jumpstart/Install OS in Logical Domain

Configure the guest domain as jumpstart client and initiate jumpstart from guest domain console. Login to guest domain console from control domain

#telnet localhost (port)
ok boot vnet1 - install

Solaris Firmware upgrade

System Firmware Upgrade

Quick steps for upgrading Firmware

1) check the system firmware version using showhost

sc>showhost

2) Download the latest firmware based on your hardware model from http://sunsolve.sun.com

3) Use sysfwdownload utility comes with the downloaded firmware package to install the firmware

#./sysfwdownload [firmware].bin

4) Power off the server

#shutdown -i5 -g0 -y

5) Upgrade the firmware using flashupdate

sc>flashupdate -s 127.0.0.1

6) reset the system controller and poweron

sc>resetsc -y
sc>poweron
sc>boot