Solaris and SAN

Scanning for a new Lun
luxadm probe -p
cfgadm -al


Should be Enough:
emcpdiscover
powermt check




Force HBA to scan bus


cfgadm -al|grep fc-fabric
c1 fc-fabric connected configured unknown
c2 fc-fabric connected configured unknown


cfgadm -o force_update -c configure cX


cX taken from above




UNLIC paths
suddenly your paths become like this :
Pseudo name=emcpower28a
CLARiiON ID=CKxxxxxxxx [host]
Logical device ID=600601604506260086790600F2F3DF11 [DOCUMENTUM_zone_seg_2]
state=alive; policy=BasicFailover; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A Array failover mode: 1
==============================================================================
---------------- Host --------------- - Stor - -- I/O Path - -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
3076 pci@2,600000/SUNW,qlc@0/fp@0,0 c1t500601623B20211Ed20s0 SP A2 active alive 0 0
3076 pci@2,600000/SUNW,qlc@0/fp@0,0 c1t5006016B3B20211Ed20s0 SP B3 active alive 0 0
3074 pci@3,700000/SUNW,qlc@0,1/fp@0,0 c2t500601633B20211Ed20s0 SP A3 unlic alive 0 0
3074 pci@3,700000/SUNW,qlc@0,1/fp@0,0 c2t5006016A3B20211Ed20s0 SP B2 unlic alive 0 0




fix:
reset the policy to Clariion Optimization
powermt set policy=co dev=all
powermt save


Now :
Pseudo name=emcpower28a
CLARiiON ID=CKxxxxxxxxxx [host]
Logical device ID=600601604506260086790600F2F3DF11 [DOCUMENTUM_zone_seg_2]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A Array failover mode: 1
==============================================================================
---------------- Host --------------- - Stor - -- I/O Path - -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
3076 pci@2,600000/SUNW,qlc@0/fp@0,0 c1t500601623B20211Ed20s0 SP A2 active alive 0 0
3076 pci@2,600000/SUNW,qlc@0/fp@0,0 c1t5006016B3B20211Ed20s0 SP B3 active alive 0 0
3074 pci@3,700000/SUNW,qlc@0,1/fp@0,0 c2t500601633B20211Ed20s0 SP A3 active alive 0 0
3074 pci@3,700000/SUNW,qlc@0,1/fp@0,0 c2t5006016A3B20211Ed20s0 SP B2 active alive 0 0

Non-global Solaris zone migration to another machine

Non-global Solaris zone migration to another machine

I had task to migrate Nagios with NagiosGrapher (running on Sparc zone) to another machine (SunFire X4200, yes i386 platform). Trus me, it was pain in the neck.

First I manually migrated "Nagios sparc zone" to test i386 platform and then it was simple doing migration from there to production platform (another X4200 with same Solaris release and additional patches).

So in this page I just explain how to migrate non-global zone between 2 totally identical systems.

System Serbia has zone France. France zone will be migrated to system Japan and will have new zone name India!

1. Configure whole root zone France to Serbia

{serbia}/zones> mkdir france


{serbia}/> zonecfg -z france

france: No such zone configured

Use 'create' to begin configuring a new zone.

zonecfg:france> create

zonecfg:france> set zonepath=/zones/france

zonecfg:france> add net

zonecfg:france:net> set physical=e1000g0

zonecfg:france:net> set address=192.168.20.222

zonecfg:france:net> end

zonecfg:france> verify

zonecfg:france> commit

zonecfg:france> exit

Since zone is whole root, remove inherited dirs

{serbia}/zones> zonecfg -z france
zonecfg:france> remove inherit-pkg-dir dir=/lib
zonecfg:france> remove inherit-pkg-dir dir=/platform
zonecfg:france> remove inherit-pkg-dir dir=/sbin
zonecfg:france> remove inherit-pkg-dir dir=/usr
zonecfg:france> verify
zonecfg:france> commit
zonecfg:france> exit

{serbia}/zones> zonecfg -z france info

zonename: france
zonepath: /zones/france
brand: native
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: shared
net:

address: 192.168.20.222

physical: e1000g0

defrouter not specified


2. Install the zone

{serbia}/zones> zoneadm -z france install

A ZFS file system has been created for this zone.

Preparing to install zone .

Creating list of files to copy from the global zone.

Copying <48022> files to the zone.

Initializing zone product registry.

Determining zone package initialization order.

Preparing to initialize <455> packages on the zone.

Initialized <455> packages on zone.

Zone is initialized.

The file contains a log of the zone installation.


The zone size is:

{serbia}/zones/france> du -sh .

680M .


{serbia}/zones/france> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- france installed /zones/france native shared


3. Boot the zone, but first place sysidcfg file in /zones/france/root/etc/


#The sysidcfg file:

#--------------------

system_locale=en_US

timezone=US/Pacific

timeserver=localhost


#name_service=DNS

# {domain_name=googlux.com

# name_server=xxx.xxx.xxx.xxx}



name_service=NONE


nfs4_domain=genesyslab.com

service_profile=limited_net

terminal=vt100

security_policy=NONE

root_password=NP

network_interface=primary { hostname=france

netmask=255.255.255.0

protocol_ipv6=no

ip_address=192.168.20.222

default_route=192.168.20.1}


{serbia}/> zoneadm -z france boot

{serbia}/> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

1 france running /zones/france native shared





4. Login to zone's console and make sure it works well.


{serbia}/> zlogin -C france

[Connected to zone 'france' console]

france console login:

or login to zone's terminal session


{serbia}/> zlogin france

[Connected to zone 'france' pts/2]

Sun Microsystems Inc. SunOS 5.10 Generic January 2005

# pwd

/

# hostname


*************************************************

Migration to the another machine

*************************************************


1. Now, when having zone, let's migrate it to another machine.

Halt and Detach the zone

{serbia}/> zoneadm -z france halt

{serbia}/> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- france installed /zones/france native shared


{serbia}/> zoneadm -z france detach

{serbia}/> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared


2. Migrate zone path to the new host

{serbia}/zones> rsync -avH france japan:/zones/


3. Rename {japan}/zones/france to /zones/india and adjust sysidcfg file as below

network_interface=primary { hostname=india

netmask=255.255.255.0

protocol_ipv6=no

ip_address=192.168.24.35

default_route=192.168.24.1}



4. Configure zone on the new system

{japan}/tmp> zonecfg -z india

india: No such zone configured

Use 'create' to begin configuring a new zone.

zonecfg:india> create -a /zones/india


zonecfg:india> info ---> see zone info

zonename: india

zonepath: /zones/india

brand: native

autoboot: false

bootargs:

pool:

limitpriv:

scheduling-class:

ip-type: shared

net:

address: 192.168.20.222 ----> change IP

physical: e1000g0

defrouter not specified



zonecfg:india> select net physical=e1000g0

zonecfg:india:net> set address=192.168.24.35

zonecfg:india:net> end

zonecfg:india> commit

zonecfg:india> exit

5. Attach the zone

{japan}/tmp> zoneadm -z india attach

{japan}/tmp> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- india installed /zones/india native shared

6. Boot the zone

{japan}/tmp> zoneadm -z india boot

{japan}/tmp> zoneadm list -vi

ID NAME STATUS PATH BRAND IP

0 global running / native shared

3 india running /zones/india native shared



{japan}/tmp> zlogin -C india

[Connected to zone 'india' console]

france console login: --------------> hostname has to be changed

7. Login to zone and change hostname to /etc/hosts and /etc/nodename. Done.

LDOM(1.0.2) quick guide

Terminalogy

Hypervisor

Hypervisor is a thin firmware layer that provides a stable virtualized machine architecture to which an operating system can be written.
It resides in the flas PROM of the motherboard and act as interface between operating system and the hardware.
It provides a set of support fuctions to the operating system, so that the OS does not need to know intimate details of how to
perform fuctions with the hardware.

Logical Domain

Logical domain is a discrete logical grouping with its own operating system, resources and identity within a single computer system.
Each logical domain can be created, destroyed, reconfigured, and rebooted independently, without requiring a power cycle of the
server. We can run a variety of applications software in different logical domains and keep them independent of performance
and security purposes.

Logical Domains Manager

The Logical Domains Manager is used to create and manage logical domains. There can be only
one Logical Domains Manager per serer. The Logical Domains Manager maps logical domains to physical resources.

Control Domain

Domain in which the Logical Domain Manager runs allowing you to create and manager other logical domains and allocate virtual resources
to other domains. There can be only one control domain per servers. The initial domain created when installing Logical Domains software is a
control domain and is named primary.

Service Domain

Domain that provides virtual device services to other domains, such as a virtual switch, a virtual console concentrator, and a
virtual disk server.

I/O Domain

Domain that has direct ownership of and direct access to physical I/O devices, such as a network card in a PCI express controller.
Shares the devices with other domains in the form of virtual devices wehn the I/O domain is also the control domain.

Guest Domain

Domain that is managed by the control domain and uses services from the I/O and service domains.


Patch Installation

Refer the Logical Domains 1.0.2 Release notes for the required patches/softwares for your OS release

System Firmware Upgrade

Refer the Logical Domains 1.0.2 Release notes for the required System Firmware patches for your OS release

Quick steps for upgrading Firmware

1) check the system firmware version using showhost

sc>showhost

2) Download the latest firmware based on your hardware model from http://sunsolve.sun.com



3) Use sysfwdownload utility comes with the downloaded firmware package to install the firmware

#./sysfwdownload [firmware].bin



4) Power off the server

#shutdown -i5 -g0 -y


5) Upgrade the firmware using flashupdate


sc>flashupdate -s 127.0.0.1



6) reset the system controller and poweron

sc>resetsc -y
sc>poweron
sc>boot

Logical Domain manager Installation


1) Download Logical DOmains Manager Package(LDoms_Manager-1_0_2.zip) from here http://www.sun.com/ldoms

2) unzip LDoms_Manager-1_0_2.zip file

3) Execute install-ldm script from the unzip folder



#./install-ldm
Welcome to the LDoms installer.

You are about to install the domain manager package that will enable
you to create, destroy and control other domains on your system. Given
the capabilities of the domain manager, you can now change the security
configuration of this Solaris instance using the Solaris Security
Toolkit.

Select a security profile from this list:

a) Hardened Solaris configuration for LDoms (recommended)
b) Standard Solaris configuration
c) Your custom-defined Solaris security configuration profile

Enter a, b, or c [a]:
The changes made by selecting this option can be undone through the
Solaris Security Toolkit's undo feature. This can be done with the
'/opt/SUNWjass/bin/jass-execute -u' command.
Installing LDoms and Solaris Security Toolkit packages.
pkgadd -n -d "/ldom/LDOM/LDoms_Manager-1_0_2/Product" -a pkg_admin SUNWldm.v
Copyright 2008 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.

Installation of (SUNWldm) was successful.
pkgadd -n -d "/ldom/LDOM/LDoms_Manager-1_0_2/Product" -a pkg_admin SUNWjass
Copyright 2005 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.

Installation of (SUNWjass) was successful.

Verifying that all packages are fully installed. OK.
Enabling services: svc:/ldoms/ldmd:default
Running Solaris Security Toolkit 4.2.0 driver ldm_control-secure.driver.
Please wait. . .
/opt/SUNWjass/bin/jass-execute -q -d ldm_control-secure.driver
Executing driver, ldm_control-secure.driver
Solaris Security Toolkit hardening executed successfully; log file
/var/opt/SUNWjass/run/20080617183524/jass-install-log.txt. It will not
take effect until the next reboot. Before rebooting, make sure SSH or
the serial line is setup for use after the reboot.

Default Service Creation

All LDOM related commands will be available under /opt/SUNWldm/bin/ldm. Add this path to your PATH variable(export PATH=$PATH:/opt/SUNWldm/bin/ldm)

Creating Virtual Disk server(vds)

Virtual disk server helps importing virtual disks into a logical domain from the control domain.

primary#ldm add-vds primary-vds0 primary

Creating Virtual Console concentrator Server(vcc)

Virtual Console concentrator server provides terminal service to logical domain consoles.

primary#ldm add-vcc port-range=5000-5100 primary-vcc0 primary


Creating Virtual Switch server(vsw)

Virtual Switch server enables networking between virtual network devices in logical domains.

primary#ldm add-vsw net-dev=e1000g0 primary-vsw0 primary

Listing the default service created

# ldm list-services primary
------------------------------------------------------------------------------
Notice: the LDom Manager is running in configuration mode. Configuration and
resource information is displayed for the configuration under construction;
not the current active configuration. The configuration being constructed
will only take effect after it is downloaded to the system controller and
the host is reset.
------------------------------------------------------------------------------
VDS
NAME VOLUME OPTIONS DEVICE
primary-vds0

VCC
NAME PORT-RANGE
primary-vcc0 5000-5100

VSW
NAME MAC NET-DEV DEVICE MODE
primary-vsw0 00:11:4a:19:ac:3c e1000g1 switch@0 prog,promisc

ether 0:2:ac:d9:11:12

Control Domain Creation

Creating the control domain with 4 cpu's and 1gb RAM.

#/opt/SUNWldm/bin/ldm set-mau 0 primary

#/opt/SUNWldm/bin/ldm set-vcpu 4 primary

#/opt/SUNWldm/bin/ldm set-memory 1024m primary

Make the modified configuration permanent using list-spconfig option

#ldm list-spconfig
factory-default [current]

#ldm add-spconfig initial

#ldm list-spconfig
factory-default [current]
initial [next]

Reboot the server to come up with initial configuration.

#shutdown -i6 -g0 -y


Enable Networking between domains

Networking between control,service and other domains is disabled by default. To enable this, the virtual switch device
should be configured as a network device.


Login to the server console and perform the following network configuration steps,

primary#ifconfig -a

Note down the name,ip configured for the primary interface.

Plumb the virtual switch(vsw0)

primary#ifconfig vsw0 plumb

Bring down the primary interface(use appropriate interface name in place of e1000g1)

primary#ifconfig e1000g1 down unplumb

Configure Virtual switch with the primary interface details (replace primary_interface_up,primary_interface_netmask appropriately)

primary#ifconfig vsw0 primary_interface_ip netmask primary_interface_netmask broadcast + up

Modify the hostname file to make this configuration permanent

primary#mv /etc/hostname.e1000g1 /etc/hostname.vsw0


Enable Virtual Network terminal server daemon

#svcadm enable vntsd

Logical Domain Creation

Create a logical domain with the following configuration

Domain name - gdomain
Number of CPU - 8
Memory - 1GB

# ldm add-domain gdomain
# ldm add-vcpu 8 gdomain
# ldm add-memory 1G gdomain
# ldm add-vnet vnet1 primary-vsw0 gdomain
# ldm add-vdsdev /dev/dsk/c0t2d0s2 vol1@primary-vds0
# ldm add-vdisk vdisk1 vol1@primary-vds0 gdomain
# ldm bind gdomain
# ldm set-var auto-boot\?=false gdomain
# ldm start-domain gdomain



#ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv SP 4 1G 0.2% 1h 3m
gdomain inactive ----- 8 1G


Jumpstart/Install OS in Logical Domain

Configure the guest domain as jumpstart client and initiate jumpstart from guest domain console. Login to guest domain console from control domain

#telnet localhost (port)
ok boot vnet1 - install

Solaris Firmware upgrade

System Firmware Upgrade

Quick steps for upgrading Firmware

1) check the system firmware version using showhost

sc>showhost

2) Download the latest firmware based on your hardware model from http://sunsolve.sun.com

3) Use sysfwdownload utility comes with the downloaded firmware package to install the firmware

#./sysfwdownload [firmware].bin

4) Power off the server

#shutdown -i5 -g0 -y

5) Upgrade the firmware using flashupdate

sc>flashupdate -s 127.0.0.1

6) reset the system controller and poweron

sc>resetsc -y
sc>poweron
sc>boot

Sun clutser

Sun Cluster


Introduction

Sun Cluster 3.2 has the following features and limitations:

• Support for 2-16 nodes.

• Global device capability--devices can be shared across the cluster.

• Global file system --allows a file system to be accessed simultaneously by all cluster nodes.

• Tight implementation with Solaris--The cluster framework services have been implemented in the kernel.

• Application agent support.

• Tight integration with zones.

• Each node must run the same revision and update of the Solaris OS.

• Two node clusters must have at least one quorum device.

• Each cluster needs at least two separate private networks. (Supported hardware, such as ce and bge may use tagged VLANs to run private and public networks on the same physical connection.)

• Each node's boot disk should include a 500M partition mounted at /globaldevices

• Attached storage must be multiply connected to the nodes.

• ZFS is a supported file system and volume manager. Veritas Volume Manager (VxVM) and Solaris Volume Manager (SVM) are also supported volume managers.

• Veritas multipathing (vxdmp) is not supported. Since vxdmp must be enabled for current VxVM versions, it must be used in conjunction with mpxio or another similar solution like EMC's Powerpath.

• SMF services can be integrated into the cluster, and all framework daemons are defined as SMF services

• PCI and SBus based systems cannot be mixed in the same cluster.

• Boot devices cannot be on a disk that is shared with other cluster nodes. Doing this may lead to a locked-up cluster due to data fencing.

The overall health of the cluster may be monitored using the cluster status or scstat -v commands. Other useful options include:

• scstat -g: Resource group status

• scstat -D: Device group status

• scstat -W: Heartbeat status

• scstat -i: IPMP status

• scstat -n: Node status

Failover applications (also known as "cluster-unaware" applications in the Sun Cluster documentation) are controlled by rgmd (the resource group manager daemon). Each application has a data service agent, which is the way that the cluster controls application startups, shutdowns, and monitoring. Each application is typically paired with an IP address, which will follow the application to the new node when a failover occurs.

"Scalable" applications are able to run on several nodes concurrently. The clustering software provides load balancing and makes a single service IP address available for outside entities to query the application.

"Cluster aware" applications take this one step further, and have cluster awareness programmed into the application. Oracle RAC is a good example of such an application.

All the nodes in the cluster may be shut down with cluster shutdown -y -g0. To boot a node outside of the cluster (for troubleshooting or recovery operations, run boot -x

clsetup is a menu-based utility that can be used to perform a broad variety of configuration tasks, including configuration of resources and resource groups.

Cluster Configuration

The cluster's configuration information is stored in global files known as the "cluster configuration repository" (CCR). The cluster framework files in /etc/cluster/ccr should not be edited manually; they should be managed via the administrative commands.

The cluster show command displays the cluster configuration in a nicely-formatted report.

The CCR contains:

• Names of the cluster and the nodes.

• The configuration of the cluster transport.

• Device group configuration.

• Nodes that can master each device group.

• NAS device information (if relevant).

• Data service parameter values and callback method paths.

• Disk ID (DID) configuration.

• Cluster status.

Some commands to directly maintain the CCR are:

• ccradm: Allows (among other things) a checksum re-configuration of files in /etc/cluster/ccr after manual edits. (Do NOT edit these files manually unless there is no other option. Even then, back up the original files.) ccradm -i /etc/cluster/ccr/filename -o

• scgdefs: Brings new devices under cluster control after they have been discovered by devfsadm.

The scinstall and clsetup commands may

We have observed that the installation process may disrupt a previously installed NTP configuration (even though the installation notes promise that this will not happen). It may be worth using ntpq to verify that NTP is still working properly after a cluster installation.

Resource Groups

Resource groups are collections of resources, including data services. Examples of resources include disk sets, virtual IP addresses, or server processes like httpd.

Resource groups may either be failover or scalable resource groups. Failover resource groups allow groups of services to be started on a node together if the active node fails. Scalable resource groups run on several nodes at once.

The rgmd is the Resource Group Management Daemon. It is responsible for monitoring, stopping, and starting the resources within the different resource groups.

Some common resource types are:

• SUNW.LogicalHostname: Logical IP address associated with a failover service.

• SUNW.SharedAddress: Logical IP address shared between nodes running a scalable resource group.

• SUNW.HAStoragePlus: Manages global raw devices, global file systems, non-ZFS failover file systems, and failover ZFS zpools.

Resource groups also handle resource and resource group dependencies. Sun Cluster allows services to start or stop in a particular order. Dependencies are a particular type of resource property. The r_properties man page contains a list of resource properties and their meanings. The rg_properties man page has similar information for resource groups. In particular, the Resource_dependencies property specifies something on which the resource is dependent.

Some resource group cluster commands are:

• clrt register resource-type: Register a resource type.

• clrt register -n node1name,node2name resource-type: Register a resource type to specific nodes.

• clrt unregister resource-type: Unregister a resource type.

• clrt list -v: List all resource types and their associated node lists.

• clrt show resource-type: Display all information for a resource type.

• clrg create -n node1name,node2name rgname: Create a resource group.

• clrg delete rgname: Delete a resource group.

• clrg set -p property-name rgname: Set a property.

• clrg show -v rgname: Show resource group information.

• clrs create -t HAStoragePlus -g rgname -p AffinityOn=true -p FilesystemMountPoints=/mountpoint resource-name

• clrg online -M rgname

• clrg switch -M -n nodename rgname

• clrg offline rgname: Offline the resource, but leave it in a managed state.

• clrg restart rgname

• clrs disable resource-name: Disable a resource and its fault monitor.

• clrs enable resource-name: Re-enable a resource and its fault monitor.

• clrs clear -n nodename -f STOP_FAILED resource-name

• clrs unmonitor resource-name: Disable the fault monitor, but leave resource running.

• clrs monitor resource-name: Re-enable the fault monitor for a resource that is currently enabled.

• clrg suspend rgname: Preserves online status of group, but does not continue monitoring.

• clrg resume rgname: Resumes monitoring of a suspended group

• clrg status: List status of resource groups.

• clrs status -g rgname

Data Services

A data service agent is a set of components that allow a data service to be monitored and fail over within the cluster. The agent includes methods for starting, stopping, monitoring, or failing the data service. It also includes a registration information file allowing the CCR to store the information about these methods in the CCR. This information is encapsulated as a resource type.

The fault monitors for a data sevice place the daemons under the control of the process monitoring facility (rpc.pmfd), and the service, using client commands.

Public Network

The public network uses pnmd (Public Network Management Daemon) and the IPMP in.mpathd daemon to monitor and control the public network addresses.

IPMP should be used to provide failovers for the public network paths. The health of the IPMP elements can be monitored with scstat -i

The clrslh and clrssa commands are used to configure logical and shared hostnames, respectively.

• clrslh create -g rgname logical-hostname

Private Network

The "private," or "cluster transport" network is used to provide a heartbeat between the nodes so that they can determine which nodes are available. The cluster transport network is also used for traffic related to global devices.

While a 2-node cluster may use crossover cables to construct a private network, switches should be used for anything more than two nodes. (Ideally, separate switching equipment should be used for each path so that there is no single point of failure.)

The default base IP address is 172.16.0.0, and private networks are assigned subnets based on the results of the cluster setup.

Available network interfaces can be identified by using a combination of dladm show-dev and ifconfig.

Private networks should be installed and configured using the scinstall command during cluster configuration. Make sure that the interfaces in question are connected, but down and unplumbed before configuration. The clsetup command also has menu options to guide you through the private network setup process.

Alternatively, something like the following command string can be used to establish a private network:

• clintr add nodename1:ifname

• clintr add nodename2:ifname2

• clintr add switchname

• clintr add nodename1:ifname1,switchname

• clintr add nodename2:ifname2,switchname

• clintr status

The health of the heartbeat networks can be checked with the scstat -W command. The physical paths may be checked with clintr status or cluster status -t intr.

Quorum

Sun Cluster uses a quorum voting system to prevent split-brain and cluster amnesia. The Sun Cluster documentation refers to "failure fencing" as the mechanism to prevent split-brain (where two nodes run the same service at the same time, leading to potential data corruption).

"Amnesia" occurs when a change is made to the cluster while a node is down, then that node attempts to bring up the cluster. This can result in the changes being forgotten, hence the use of the word "amnesia."

One result of this is that the last node to leave a cluster when it is shut down must be the first node to re-enter the cluster. Later in this section, we will discuss ways of circumventing this protection.

Quorum voting is defined by allowing each device one vote. A quorum device may be a cluster node, a specified external server running quorum software, or a disk or NAS device. A majority of all defined quorum votes is required in order to form a cluster. At least half of the quorum votes must be present in order for cluster services to remain in operation. (If a node cannot contact at least half of the quorum votes, it will panic. During the reboot, if a majority cannot be contacted, the boot process will be frozen. Nodes that are removed from the cluster due to a quorum problem also lose access to any shared file systems. This is called "data fencing" in the Sun Cluster documentation.)

• Quorum devices must be available to at least two nodes in the cluster.

• Disk quorum devices may also contain user data. (Note that if a ZFS disk is used as a quorum device, it should be brought into the zpool before being specified as a quorum device.)

• Sun recommends configuring n-1 quorum devices (the number of nodes minus 1). Two node clusters must contain at least one quorum device.

• Disk quorum devices must be specified using the DID names.

• Quorum disk devices should be at least as available as the storage underlying the cluster resource groups.

Quorum status and configuration may be investigating using:

• scstat -q

• clq status

These commands report on the configured quorum votes, whether they are present, and how many are required for a majority.

Quorum devices can be manipulated through the following commands:

• clq add did-device-name

• clq remove did-device-name: (Only removes the device from the quorum configuration. No data on the device is affected.)

• clq enable did-device-name

• clq disable did-device-name: (Removes the quorum device from the total list of available quorum votes. This might be valuable if the device is down for maintenance.)

• clq reset: (Resets the configuration to the default.)

By default, doubly-connected disk quorum devices use SCSI-2 locking. Devices connected to more than two nodes use SCSI-3 locking. SCSI-3 offers persistent reservations, but SCSI-2 requires the use of emulation software. The emulation software uses a 64-bit reservation key written to a private area on the disk.

In either case, the cluster node that wins a race to the quorum device attempts to remove the keys of any node that it is unable to contact, which cuts that node off from the quorum device. As noted before, any group of nodes that cannot communicate with at least half of the quorum devices will panic, which prevents a cluster partition (split-brain).

In order to add nodes to a 2-node cluster, it may be necessary to change the default fencing with scdidadm -G prefer3 or cluster set -p global_fencing=prefer3, create a SCSI-3 quorum device with clq add, then remove the SCSI-2 quorum device with clq remove.

NetApp filers and systems running the scqsd daemon may also be selected as quorum devices. NetApp filers use SCSI-3 locking over the iSCSI protocol to perform their quorum functions.

The claccess deny-all command may be used to deny all other nodes access to the cluster. claccess allow nodename re-enables access for a node.

Purging Quorum Keys

CAUTION: Purging the keys from a quorum device may result in amnesia. It should only be done after careful diagnostics have been done to verify why the cluster is not coming up. This should never be done as long as the cluster is able to come up. It may need to be done if the last node to leave the cluster is unable to boot, leaving everyone else fenced out. In that case, boot one of the other nodes to single-user mode, identify the quorum device, and:

For SCSI 2 disk reservations, the relevant command is pgre, which is located in /usr/cluster/lib/sc:

• pgre -c pgre_inkeys -d /dev/did/rdks/d#s2 (List the keys in the quorum device.)

• pgre -c pgre_scrub -d /dev/did/rdks/d#s2 (Remove the keys from the quorum device.)

Similarly, for SCSI 3 disk reservations, the relevant command is scsi:

• scsi -c inkeys -d /dev/did/rdks/d#s2 (List the keys in the quorum device.)

• scsi -c scrub -d /dev/did/rdks/d#s2 (Remove the keys from the quorum device.)

Global Storage

Sun Cluster provides a unique global device name for every disk, CD, and tape drive in the cluster. The format of these global device names is /dev/did/device-type. (eg /dev/did/dsk/d2s3) (Note that the DIDs are a global naming system, which is separate from the global device or global file system functionality.)

DIDs are componentsof SVM volumes, though VxVM does not recognize DID device names as components of VxVM volumes.

DID disk devices, CD-ROM drives, tape drives, SVM volumes, and VxVM volumes may be used as global devices. A global device is physically accessed by just one node at a time, but all other nodes may access the device by communicating across the global transport network.

The file systems in /global/.devices store the device files for global devices on each node. These are mounted on mount points of the form /global/.devices/node@nodeid, where nodeid is the identification number assigned to the node. These are visible on all nodes. Symbolic links may be set up to the contents of these file systems, if they are desired. Sun Cluster sets up some such links in the /dev/global directory.

Global file systems may be ufs, VxFS, or hsfs. To mount a file system as a global file system, add a "global" mount option to the file system's vfstab entry and remount. Alternatively, run a mount -o global... command.

(Note that all nodes in the cluster should have the same vfstab entry for all cluster file systems. This is true for both global and failover file systems, though ZFS file systems do not use the vfstab at all.)

In the Sun Cluster documentation, global file systems are also known as "cluster file systems" or "proxy file systems."

Note that global file systems are different from failover file systems. The former are accessible from all nodes; the latter are only accessible from the active node.

Maintaining Devices

New devices need to be read into the cluster configuration as well as the OS. As usual, we should run something like devfsadm or drvconfig; disks to create the /device and /dev links across the cluster. Then we use the scgdevs or scdidadm command to add more disk devices to the cluster configuration.

Some useful options for scdidadm are:

• scdidadm -l: Show local DIDs

• scdidadm -L: Show all cluster DIDs

• scdidadm -r: Rebuild DIDs

We should also clean up unused links from time to time with devfsadm -C and scdidadm -C

The status of device groups can be checked with scstat -D. Devices may be listed with cldev list -v. They can be switched to a different node via a cldg switch -n target-node dgname command.

Monitoring for devices can be enabled and disabled by using commands like:

• cldev monitor all

• cldev unmonitor d#

• cldev unmonitor -n nodename d#

• cldev status -s Unmonitored

Parameters may be set on device groups using the cldg set command, for example:

• cldg set -p failback=false dgname

A device group can be taken offline or placed online with:

• cldg offline dgname

• cldg online dgname

VxVM-Specific Issues

Since vxdmp cannot be disabled, we need to make sure that VxVM can only see one path to each disk. This is usually done by implementing mpxio or a third party product like Powerpath. The order of installation for such an environment would be:

1. Install Solaris and patches.

2. Install and configure multipathing software.

3. Install and configure Sun Cluster.

4. Install and configure VxVM

If VxVM disk groups are used by the cluster, all nodes attached to the shared storage must have VxVM installed. Each vxio number in /etc/name_to_major must also be the same on each node. This can be checked (and fixed, if necessary) with the clvxvm initialize command. (A reboot may be necessary if the /etc/name_to_major file is changed.)

The clvxvm encapsulate command should be used if the boot drive is encapsulated (and mirrored) by VxVM. That way the /global/.devices information is set up properly.

The clsetup "Device Groups" menu contains items to register a VxVM disk group, unregister a device group, or synchronize volume information for a disk group. We can also re-synchronize with the cldg sync dgname command.

Solaris Volume Manager-Specific Issues

Sun Cluster allows us to add metadb or partition information in the /dev/did format or in the usual format. In general:

• Use local format for boot drive mirroring in case we need to boot outside the cluster framework.

• Use cluster format for shared disksets because otherwise we will need to assume the same controller numbers on each node.

Configuration information is kept in the metadatabase replicas. At least three local replicas are required to boot a node; these should be put on their own partitions on the local disks. They should be spread across controllers and disks to the degree possible. Multiple replicas may be placed on each partition; they should be spread out so that if any one disk fails, there will still be at least three replicas left over, constituting at least half of the total local replicas.

When disks are added to a shared diskset, database replicas are automatically added. These will always be added to slice 7, where they need to remain. If a disk containing replicas is removed, the replicas must be removed using metadb.

If fewer than 50% of the replicas in a diskset are available, the diskset ceases to operate. If exactly 50% of the replicas are available, the diskset will continue to operate, but will not be able to be enabled or switched on another node.

A mediator can be assigned to a shared diskset. The mediator data is contained within a Solaris process on each node and counts for two votes in the diskset quorum voting.

Standard c#t#d#s# naming should be used when creating local metadb replicas, since it will make recovery easier if we need to boot the node outside of a cluster context. On the other hand, /dev/did/rdsk/d#s# naming should be used for shared disksets, since otherwise the paths will need to be identical on all nodes.

Creating a new shared diskset involves the following steps:

(Create an empty diskset.)

metaset -s set-name -a -h node1-name node2-name

(Create a mediator.)

metaset -s set-name -a -m node1-name node2-name

(Add disks to the diskset.)

metaset -s set-name -a /dev/did/rdsk/d# /dev/did/rdsk/d#

(Check that the diskset is present in the cluster configuration.)

cldev list -v

cldg status

cldg show set-name

ZFS-Specific Issues

ZFS is only available as a Sun Cluster failover file system, not as a global file system. No vfstab entries are required, since that information is contained in the zpools. No synchronization commands are required like in VxVM; Sun Cluster takes care of the synchronization automatically.

Zones

Non-global zones may be treated as virtual nodes. Keep in mind that some services, such as NFS, will not run in non-global zones.

Services can be failed over between zones, even zones on the same server. Where possible, it is best to use full rather than sparse zones. Certain types of failures within the non-global zone can cause a crash in the global zone.

Configuration of cluster resources and resource groups must be performed in the global zone. The rgmd runs in the global zone.

To specify a non-global zone as a node, use the form

nodename:zonename

or specify

-n nodename -z zonename

Cluster Quick Command Refernces

Shut down a resource group:

scswitch -F -g [RESOURCE_GROUP_NAME]

Bring up a resource group:
scswitch -Z -g [RESOURCE_GROUP_NAME]

Move failover resource group to node_name:
cswitch -z -g [RESOURCE_GROUP_NAME] -h [NODE_NAME]

Restart resource group:
scswitch -R -h [NODE_NAME] -g [RESOURCE_GROUP_NAME]

Evacuate all resources from node_name:
scswitch -S -h [NODE_NAME]

Disable resource:
scswitch -n -j [RESOURCE]

Enable resource:
scswitch -e -j [RESOURCE]

Clear STOP_FAILED on resource:
scswitch -c -j [RESOURCE] -h [NODE_NAME] -f STOP_FAILED

Disable resource’s fault monitor:
scswitch -n -M -j [RESOURCE]

Enable resource’s fault monitor:
scswitch -e -M -j [RESOURCE]

Enable resource’s fault monitor:
scswitch -e -M -j [RESOURCE]

Lists currently configured DID’s:
scdidadm -L

Put a new device under cluster control:
scgdevs

Displays status of the cluster, resources, resource groups, etc.:
scstat

Display useful setup info about cluster nodes, cluster transport, disksets, etc.:
scconf -p -v

Solaris NIC speed and duplex settings

The following is a Bourne shell script I wrote to determine the speed and duplex settings for all active network interfaces on a Solaris system.

#!/bin/sh # $Id: speed_duplex,v 1.4 2007/07/10 16:04:42 hutch Exp $ PATH=/bin:/usr/bin:/usr/sbin # Print column header information echo "Interface\tSpeed\t\tDuplex" echo "---------\t-----\t\t------" # Determine the speed and duplex for each live NIC on the system for INTERFACE in `netstat -i | egrep -v "^Name|^lo0" \ | awk '{ print $1 }' | cut -d: -f1 | sort | uniq` do # Only gather information for active interfaces # Note: "ce" interfaces can be "UP" in "ifconfig" but have link down ifconfig $INTERFACE | grep "^$INTERFACE:.* /dev/null 2>&1 || continue # Skip "cip" ATM interfaces echo $INTERFACE | grep "^cip" > /dev/null 2>&1 && continue # "ce" interfaces if [ "`echo $INTERFACE | awk '/^ce[0-9]+/ { print }'`" ] ; then kstat > /dev/null 2>&1 if [ $? -ne 0 ] ; then echo "The \"kstat\" command failed for interface $INTERFACE." continue fi # Determine the ce interface number INSTANCE=`echo $INTERFACE | cut -c 3-` DUPLEX=`kstat ce:$INSTANCE | grep link_duplex | awk '{ print $2 }'` case "$DUPLEX" in 0) DUPLEX="link down" ;; 1) DUPLEX="half" ;; 2) DUPLEX="full" ;; esac SPEED=`kstat ce:$INSTANCE | grep link_speed | awk '{ print $2 }'` case "$SPEED" in 0) SPEED="link down" ;; 10) SPEED="10 Mbit/s" ;; 100) SPEED="100 Mbit/s" ;; 1000) SPEED="1 Gbit/s" ;; esac # "dmfe" interfaces elif [ "`echo $INTERFACE | awk '/^dmfe[0-9]+/ { print }'`" ] ; then # Only the root user should run "ndd" if [ "`id | cut -c1-5`" != "uid=0" ] ; then echo "You must be the root user to determine \ ${INTERFACE_TYPE}${INSTANCE} speed and duplex information." continue fi DUPLEX=`ndd /dev/${INTERFACE} link_mode` case "$DUPLEX" in 0) DUPLEX="half" ;; 1) DUPLEX="full" ;; esac SPEED=`ndd /dev/${INTERFACE} link_speed` case "$SPEED" in 10) SPEED="10 Mbit/s" ;; 100) SPEED="100 Mbit/s" ;; 1000) SPEED="1 Gbit/s" ;; esac # "bge" and "iprb" interfaces elif [ "`echo $INTERFACE | awk '/^iprb[0-9]+|bge[0-9]+/ { print }'`" ] ; then kstat > /dev/null 2>&1 if [ $? -ne 0 ] ; then DUPLEX="The \"kstat\" command failed for interface $INTERFACE." continue fi # Determine the bge|iprb interface number INSTANCE=`echo $INTERFACE | tr -d '[a-z]'` INTERFACE=`echo $INTERFACE | tr -d '[0-9]'` DUPLEX=`kstat $INTERFACE:$INSTANCE | grep duplex | awk '{ print $2 }'` SPEED=`kstat $INTERFACE:$INSTANCE | grep ifspeed | awk '{ print $2 }'` case "$SPEED" in 10000000) SPEED="10 Mbit/s" ;; 100000000) SPEED="100 Mbit/s" ;; 1000000000) SPEED="1 Gbit/s" ;; esac elif [ "`echo $INTERFACE | awk '/^e1000g[0-9]+/ { print }'`" ] ; then INSTANCE=`echo $INTERFACE | cut -c7-` # The duplex for e1000g devices can only be found with "dladm" DUPLEX=`dladm show-dev $INTERFACE | awk '{ print $NF }'` SPEED=`kstat e1000g:$INSTANCE | grep ifspeed | awk '{ print $2 }'` case "$SPEED" in 10000000) SPEED="10 Mbit/s" ;; 100000000) SPEED="100 Mbit/s" ;; 1000000000) SPEED="1 Gbit/s" ;; esac # le interfaces are always 10 Mbit half-duplex elif [ "`echo $INTERFACE | awk '/^le[0-9]+/ { print }'`" ] ; then DUPLEX="half" SPEED="10 Mbit/s" # All other interfaces else INTERFACE_TYPE=`echo $INTERFACE | sed -e "s/[0-9]*$//"` INSTANCE=`echo $INTERFACE | sed -e "s/^[a-z]*//"` # Only the root user should run "ndd" if [ "`id | cut -c1-5`" != "uid=0" ] ; then echo "You must be the root user to determine \ ${INTERFACE_TYPE}${INSTANCE} speed and duplex information." continue fi ndd -set /dev/$INTERFACE_TYPE instance $INSTANCE SPEED=`ndd -get /dev/$INTERFACE_TYPE link_speed` case "$SPEED" in 0) SPEED="10 Mbit/s" ;; 1) SPEED="100 Mbit/s" ;; 1000) SPEED="1 Gbit/s" ;; esac DUPLEX=`ndd -get /dev/$INTERFACE_TYPE link_mode` case "$DUPLEX" in 0) DUPLEX="half" ;; 1) DUPLEX="full" ;; *) DUPLEX="" ;; esac fi echo "$INTERFACE\t\t$SPEED\t$DUPLEX" done

Example output:

Interface Speed Duplex --------- ----- ------ hme0 100 Mbit/s full hme1 100 Mbit/s full hme2 100 Mbit/s full

Setting NIC speed and duplex
Solaris is often unable to correctly auto-negotiate duplex settings with a link partner (e.g. switch), especially when the switch is set to 100Mbit full-duplex. You can force the NIC into 100Mbit full-duplex by disabling auto-negotiation and 100Mbit half-duplex capability.

Example with hme0:

1. Make the changes to the running system.

# ndd -set /dev/hme adv_100hdx_cap 0 # ndd -set /dev/hme adv_100fdx_cap 1 # ndd -set /dev/hme adv_autoneg_cap 0

2. Make kernel parameter changes to preserve the speed and duplex settings after a reboot.
# vi /etc/system

Add:

# set hme:hme_adv_autoneg_cap=0 # set hme:hme_adv_100hdx_cap=0 # set hme:hme_adv_100fdx_cap=1

Note: the /etc/system change affects all hme interfaces if multiple NICs are present (e.g. hme0, hme1).



The /etc/system settings listed above are not supported for configuring ce Ethernet adapters during system startup; you may either use ndd commands in an /etc/rc?.d script or create a /platform/sun4u/kernel/drv/ce.conf file with appropriate settings.

Example: /etc/init.d/nddconfig

#!/bin/sh ndd -set /dev/ce instance 0 ndd -set /dev/ce adv_1000fdx_cap 0 ndd -set /dev/ce adv_1000hdx_cap 0 ndd -set /dev/ce adv_100fdx_cap 1 ndd -set /dev/ce adv_100hdx_cap 0 ndd -set /dev/ce adv_10fdx_cap 0 ndd -set /dev/ce adv_10hdx_cap 0 ndd -set /dev/ce adv_autoneg_cap 0 # ln -s /etc/init.d/nddconfig /etc/rc2.d/S31nddconfig

$ dmesg | grep ce0
Jan 20 11:05:01 crmmdb22 genunix: [ID 611667 kern.info] NOTICE: ce0: xcvr addr:0x01 - link up 100 Mbps half duplex
Jan 20 11:05:15 crmmdb22 genunix: [ID 408822 kern.info] NOTICE: ce0: no fault external to device; service available
Jan 20 11:05:15 crmmdb22 genunix: [ID 611667 kern.info] NOTICE: ce0: xcvr addr:0x01 - link up 100 Mbps full duplex


Manually determining NIC speed and duplex

If you have ce or bge interfaces, use kstat ce and kstat bge, respectively, to return NIC settings. All other interfaces may use ndd to determine NIC settings.

ndd example with hme0, assuming "instance" is 0:

# ndd -get /dev/hme link_mode
Interpretation:
0 -- half-duplex
1 -- full-duplex

# ndd -get /dev/hme link_speed
Interpretation:
0 -- 10 Mbit
1 -- 100 Mbit
1000 -- 1 Gbit

To query a different NIC, such as hme1, set the "instance" to 1, and then perform the link_mode and link_speed queries above.

# ndd -set /dev/hme instance 1

Note: the ndd commands above must be run as root. Otherwise, you will receive errors such as "couldn't push module 'hme0', No such device or address."


ce Ethernet adapters

Older versions of the Sun GigaSwift Ethernet 1.0 driver do not support the ndd link_mode and link_speed parameters. You may either install the latest Sun GigaSwift Ethernet adapter patch (111883) or you may use kstat ce ce_device to get speed and duplex information for ce Ethernet adapters.



$ netstat -k ce | egrep 'link_speed|link_status|link_duplex'

Interpretation:
link_up - 0 down, 1 up
link_speed - speed in Mbit/s
link_duplex - 1 half duplex, 2 full duplex, 0 down


Host/link partner mismatch example
A large number of output errors or collisions may indicate a host and link partner mismatch. The following is netstat -in output from a system configured for 100 half-duplex while the switch was configured for 100 full-duplex.

Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
ce0 1500 192.168.1.0 192.168.1.1 2707133478 25 2895422910 142310052 182856975 0

In this example, the switch was configured for 100 half-duplex and the system was configured for 100 full-duplex. Note the percentage of Ierrs/Ipkts (~ 1.7%).

Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
qfe1 1500 10.0.0.0 10.0.0.3 1430247 24663 1779341 0 0 0


CHANGELOG
2007/07/10 -- Added e1000g interfaces
2006/11/29 -- Added dmfe interfaces, fixed bge and ce interfaces, many other modifications and fixes.
2006/04/27 -- Solaris 7 and earlier include subinterface information with netstat -i. Patch by Royce Williams to only query physical NIC interfaces.
2006/03/09 -- Fixed bge interfaces, added support for iprb interfaces, added "support" for le interfaces, modified script to only require root access if using /usr/sbin/ndd to determine NIC speed and duplex settings
2005/05/25 -- Fixed ce interface handling.
2004/09/15 -- Added support for ce and bge interfaces.