Restarting VCS in Veritas SF for Oracle RAC environment

The following procedures to restart VCS and bring the service groups on the domain online.

To restart LLT, GAB, VCS, and DBE/AC processes

1. Restart LLT.
   # /etc/rc2.d/S70llt start
2. Restart GAB.
   # /etc/rc2.d/S92gab start
3. Restart the LMX driver.
   # /etc/rc2.d/S71lmx start
4. Restart the VCSMM driver.
   # /etc/rc2.d/S98vcsmm start
5. Restart the VXFEN driver
   # /etc/rc2.d/S97vxfen start
6. Restart the ODM driver.
   # mount /dev/odm
7. Start VCS.
   # hastart
8. Verify that the CVM service group is online.
   # hagrp -state cvm
9. Verify the GAB memberships required for DBE/AC for Oracle9i RAC are configured.
   # /sbin/gabconfig -a
   

   GAB Port Memberships
   ============================================================
   Port a gen 4a1c0001 membership 012
   Port b gen g8ty0002 membership 012
   Port d gen 40100001 membership 012
   Port f gen f1990002 membership 012
   Port h gen g8ty0002 membership 012
   Port o gen f1100002 membership 012
   Port q gen 28d10002 membership 012
   Port v gen 1fc60002 membership 012
   Port w gen 15ba0002 membership 012

10. Bring the service groups that had been take offline in step 3 on page 11 online.
    # hagrp -online service_grp_name -sys wildcat

Stopping VCS in Veritas SF for Oracle RAC environment

If you must stop VCS on a domain where Veritas SF for Oracle RAC is running, the Oracle RAC application on the domain being reconfigured must be brought offline. In addition, the GAB, LLT, LMX, and VXFEN modules must be unconfigured. Performing these steps ensures that other instances do not attempt communication with the stopped instance. This could cause the application to hang, when the instance does not respond.

To stop VCS in a Veritas SF for Oracle RAC environment

1. Log in as administrator to the domain being reconfigured (wildcat, for example).
2. List the configured VCS service groups and see which are online in the domain:
   # hagrp -list
3. Based on the output of step 2, bring each service group that is online to offline in the domain wildcat. Use the following command:
   # hagrp -offline service_grp_name -sys wildcat
4. Stop VCS.
   # hastop -local
   In addition to port h, this command stops the CVM drivers using ports v and w.
5. If any CFS file systems outside of VCS control are mounted, unmount them.
6. Stop and unconfigure the drivers required by DBE/AC:
   # cd /opt/VRTSvcs/rac
   # ./uload_drv
   

   Unloading qlog
   Unloading odm
   Unloading fdd
   Unloading vxportal
   Unloading vxfs

7. Unconfigure the VCSMM and I/O fencing drivers, which use ports b and o, respectively:
   # /sbin/vxfenconfig -U
   # /sbin/vcsmmconfig -U
8. Unconfigure the LMX driver:
   # /sbin/lmxconfig -U
9. Verify that the drivers h, v, w, f, q, d, b, and o are stopped. They should not show memberships when you use the gabconfig -a command:
   # gabconfig -a
   

   GAB Port Memberships
   ============================================================
   Port a gen 4a1c0001 membership 01

10. Unload the VCSMM, I/O fencing, and LMX modules.
    Determine the module IDs for VCSMM, I/O fencing, and LMX:
    # modinfo | egrep "lmx|vxfen|vcsmm"
    

    237 783e4000 25497 237 1 vcsmm (VERITAS Membership
    Manager)
    238 78440000 263df 238 1 vxfen (VERITAS I/O Fencing)
    239 7845a000 12b1e 239 1 lmx (LLT Mux 3.5B2)

    Unload the VCSMM, I/O fencing, and LMX modules based on their module IDs:
    # modunload -i 237
    # modunload -i 238
    # modunload -i 239
11. Unconfigure GAB
    # /sbin/gabconfig -U
12. Unconfigure LLT
    # /sbin/lltconfig -U
13. Remove the GAB and LLT modules from the kernel.
    Determine the IDs of the GAB and LLT modules:
    # modinfo | egrep "gab|llt"
    

    305 78531900 30e 305 1 gab
    292 78493850 30e 292 1 llt

    Unload the GAB and LLT modules based on their module IDs:
    # modunload -i 305
    # modunload -i 292
14. You can begin performing dynamic reconfiguration.

Restarting VCS in a standard environment

If you are ready to restart VCS in the domain where you are performing dynamic reconfiguration, use the following procedure. If you are running Veritas SF for Oracle RAC, and are ready to restart VCS.

To restart LLT, GAB, and VCS

1. Restart LLT.
   # /etc/rc2.d/S70llt start
2. Restart GAB.
   # /etc/rc2.d/S92gab start
3. Start VCS.
   # hastart
4. Verify GAB and VCS are started.
   # /sbin/gabconfig -a
   

   GAB Port Memberships
   ================================================
   Port a gen 4a1c0001 membership 012
   Port h gen g8ty0002 membership 012

To bring service groups online

1. Determine which service groups are frozen.
   # hagrp -display | grep Frozen
2. Make the configuration writable.
   # haconf -makerw
3. Unfreeze the frozen service groups.
   # hagrp -unfreeze service_grp_name -persistent
4. Make the configuration read-only.
   # haconf -dump -makero

Stopping VCS in a standard environment

Stopping VCS in a standard environment

When you dynamically reconfigure CPU/Memory boards and I/O boards, it may be necessary, in some circumstances, to stop VCS in the domain.
Applications running on clusters of three or more domains remain highly available on two or more domains if VCS operation must be stopped on one domain. In a cluster of two domains, the applications running during reconfiguration are not highly available when VCS must be stopped on one of the domains.

To stop VCS in a standard environment

1. Log in as administrator to the domain (wildcat, for example) you are reconfiguring.
2. List the VCS service groups to determine which are online on the domain.
   # hagrp -list
3. If you can switch the service groups running on the domain to another domain (cheetah, for example), switch the service groups.
   # hagrp -switch service_grp_name -to cheetah
   Verify the service groups are offline on wildcat.
   # hastatus
   Stop VCS on wildcat.
   # hastop -local
4. If you cannot switch the online service groups to another system, freeze each of them for the duration of dynamic reconfiguration.
   Make the VCS configuration writable.
   # haconf -makerw
   Freeze each of the service groups persistently.
   # hagrp -freeze service_grp_name -persistent
   Verify the groups are frozen.
   # hagrp display | grep Frozen
   Make the configuration read-only.
   # haconf -dump -makero
   Stop VCS.
   # hastop -local -force
5. Unconfigure GAB.
   # /sbin/gabconfig -U
6. Unconfigure LLT.
   # /sbin/lltconfig -U
   Answer "y" to confirm that you want to stop LLT.
7. Remove the GAB and LLT modules from the kernel.
   Determine the IDs of the GAB and LLT modules:
   # modinfo | egrep "gab|llt"
   

   305 78531900 30e 305 1 gab
   292 78493850 30e 292 1 llt

   Unload the GAB and LLT modules based on their module IDs:
   # modunload -i 305
   # modunload -i 292
8. You can begin performing dynamic reconfiguration.

Detaching I/O system boards with DMP enabled

(Performing dynamic reconfiguration on SunEnterprise 10K)

Make sure the kernel_cage_enable variable is set.
To attach an I/O board with DMP enabled

1. Freeze the VCS service groups running on the domain where you intend to perform dynamic reconfiguration operations. Freezing the service groups prevents them from being taken offline or failed over. Repeat the following command for each service group:
   # hagrp -freeze ser_grp_name
2. Connect to the SSP server and log in to the domain whose system board requires Dynamic Reconfiguration.
   ssp:D1% echo $SUNW_HOSTNAME
3. Enter the dr(1M) shell:
   ssp:D1% dr
4. To verify the board is an I/O board, enter:
   dr> drshow sb# IO
   If the display lists the disks connected to the controller, the system board is an I/O board.
5. If the system board is an I/O board, open another window and log in as root to the domain you are currently reconfiguring.
6. Disable the controller on the I/O system board:
   # vxdmpadm disable ctlr=ctlr#
7. In the window where you are running dynamic reconfiguration, start detaching the I/O board by entering:
   dr> drain sb#
8. Monitor the progress of the drain operation by entering:
   dr> drshow sb# drain
9. When you see the message:
   

   Percent Complete= 100% (0 KBytes remaining)

   complete the detach operation:
   dr> complete_detach sb#
10. To verify that the board is no longer configured, type the following command:
    dr> drshow sb#
    The detached board should not appear in the detailed listing.
11. Exit the dr shell:
    dr> exit
12. If the board is not to be immediately replaced, unfreeze any frozen service groups:
    # hagrp -unfreeze ser_grp_name
    Repeat for each service group.

Attaching I/O system boards with DMP enabled

(Performing dynamic reconfiguration on SunEnterprise 10K)

You can attach a system I/O board using the following procedure:

To attach I/O system boards with DMP enabled

1. Freeze the VCS service groups running on the domain where you intend to attach a system board. Repeat the following command for each service group:
   # hagrp -freeze ser_grp_name
2. After physically replacing a previously removed I/O board, make sure it is connected to the shared storage.
3. From the SSP server, enter the dr(1M) shell:
   ssp:D1% dr
4. Follow the Sun procedure to attach the system board, described here briefly:
   dr> init_attach sb#
   Complete the attach operation:
   dr> complete_attach sb#
5. Verify that the dynamic reconfiguration attach operation has succeeded. Type:
   dr> drshow #sb
   The new system board should show in the list of configured boards.
6. Exit the dr shell.
   dr> exit
7. Log in as root to the domain where you are adding the system board. Enable the controller by entering:
   # vxdmpadm enable ctlr=ctlr#
8. When you have successfully attached and enabled the system I/O board, unfreeze any frozen service groups:
   # hagrp -unfreeze ser_grp_name
   Repeat for each service group.
9. Verify that VCS is still up and running.

Detaching CPU/memory boards

Detaching CPU/memory boards (Performing dynamic reconfiguration on SunEnterprise 10K)
Use the following procedure if no I/O devices on the system board are used.
Make sure the kernel_cage_enable variable is set.

To detach CPU/memory boards

1.Freeze the VCS service groups running on the domain where you intend to detach a CPU/Memory board. Freezing the service groups prevents them from being taken offline or failed over. Repeat the following command for each service group:
# hagrp -freeze ser_grp_name

2.Connect to the SSP server and log in to the domain whose system board requires Dynamic Reconfiguration.
ssp:D1% echo $SUNW_HOSTNAME

3.Enter the dr(1M) shell:
ssp:D1% dr

4.In the window where you are running dynamic reconfiguration, start detaching the I/O board by entering:
dr> drain sb#

5.Monitor the progress of the drain operation by entering:
dr> drshow sb# drain

6.When you see the message
Percent Complete= 100% (0 KBytes remaining)complete the detach operation:
dr> complete_detach sb#

7.To verify that the board is no longer configured, type the following command:
dr> drshow sb#

The detached board should not appear in the detailed listing.

8.Exit the dr shell:
dr > exit

9.If the board is not to be immediately replaced, unfreeze any frozen service groups:
# hagrp -unfreeze ser_grp_name

10.Repeat for each service group.