Gathering fibre channel info using fcinfo

I came across fcinfo command reading some man page. So, I decided to see what kind of useful fibre channel information it could give me. It turns out fcinfo knows quite a bit.

We can list all local fibre channel ports:

bash-3.00# fcinfo hba-port
HBA Port WWN: 21000003ba16dbd2
OS Device Name: /dev/cfg/c1
Manufacturer: QLogic Corp.
Model: 2200
Firmware Version: 2.1.144
FCode/BIOS Version: ISP2200 FC-AL Host Adapter Driver: 1.12 01/01/16
Type: L-port
State: online
Supported Speeds: 1Gb
Current Speed: 1Gb
Node WWN: 20000003ba16dbd2
HBA Port WWN: 2100001b320e3853
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: QLA2462
Firmware Version: 4.0.27
FCode/BIOS Version: QLA2462 Host Adapter Driver(SPARC): 1.24 11/15/06
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 2Gb
Node WWN: 2000001b320e3853
HBA Port WWN: 2101001b322e3853
OS Device Name: /dev/cfg/c4
Manufacturer: QLogic Corp.
Model: QLA2462
Firmware Version: 4.0.27
FCode/BIOS Version: QLA2462 Host Adapter Driver(SPARC): 1.24 11/15/06
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 2Gb
Node WWN: 2001001b322e3853

We can take a look what remote ports are seen by particular local fibre channel port, in this case 2100001b320e3853:

bash-3.00# fcinfo remote-port -p 2100001b320e3853
Remote Port WWN: 100000e00216aef3
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 100000e01124b88f
Remote Port WWN: 50001fe15003b384
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50001fe15037e759
Remote Port WWN: 100000e0022744f3
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 100000e0020744f3
Remote Port WWN: 50001fe150216de9
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50001fe11025bb53
Remote Port WWN: 100000e00228f492
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 100000e00208f492
Remote Port WWN: 50001fe15076b59b
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50001fe15037e759
Remote Port WWN: 50001fe150216ded
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50001fe11025bb53

Here, we get link statistics for remote fibre channel device whose WWN is 100000e0020744f3:

bash-3.00# fcinfo remote-port -l -p 2100001b320e3853 100000e0020744f3
Remote Port WWN: 100000e0022744f3
Active FC4 Types: SCSI
SCSI Target: unknown
Node WWN: 100000e0020744f3
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0

We can also get link statistics and SCSI target information for all remote fibre channel devices see on local port whose WWN is 2100001b320e3853:

bash-3.00# fcinfo remote-port -sl -p 2100001b320e3853
Remote Port WWN: 100000e00216aef3
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 100000e01124b88f
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: HP
Product: MSL6000 Series
OS Device Name: /devices/pci@8,600000/SUNW,qlc@1/fp@0,0/sgen@w100000e00216aef3,0
LUN: 1
Vendor: HP
Product: Ultrium 2-SCSI
OS Device Name: /dev/rmt/2n
Remote Port WWN: 50001fe15003b384
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50001fe15037e759
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: COMPAQ
Product: HSV111 (C)COMPAQ
OS Device Name: Unknown
LUN: 1
Vendor: COMPAQ
Product: HSV111 (C)COMPAQ
OS Device Name: /dev/rdsk/c5t600508B4001031250000900000540000d0s2
Remote Port WWN: 100000e0022744f3
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 100000e0020744f3
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: HP
Product: MSL6000 Series
OS Device Name: /devices/pci@8,600000/SUNW,qlc@1/fp@0,0/sgen@w100000e0022744f3,0
LUN: 1
Vendor: HP
Product: Ultrium 2-SCSI
OS Device Name: /dev/rmt/1n
LUN: 2
Vendor: HP
Product: Ultrium 2-SCSI
OS Device Name: /dev/rmt/0n
Remote Port WWN: 50001fe150216de9
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50001fe11025bb53
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: COMPAQ
Product: HSV111 (C)COMPAQ
OS Device Name: Unknown
LUN: 1
Vendor: COMPAQ
Product: HSV111 (C)COMPAQ
OS Device Name: /dev/rdsk/c5t600508B400011C370000C00003210000d0s2
Remote Port WWN: 100000e00228f492
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 100000e00208f492
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: HP
Product: Ultrium 2-SCSI
OS Device Name: /dev/rmt/4n
LUN: 1
Vendor: HP
Product: Ultrium 2-SCSI
OS Device Name: /dev/rmt/3n
Remote Port WWN: 50001fe15076b59b
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50001fe15037e759
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: COMPAQ
Product: HSV111 (C)COMPAQ
OS Device Name: Unknown
LUN: 1
Vendor: COMPAQ
Product: HSV111 (C)COMPAQ
OS Device Name: /dev/rdsk/c5t600508B4001031250000900000540000d0s2
Remote Port WWN: 50001fe150216ded
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50001fe11025bb53
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: COMPAQ
Product: HSV111 (C)COMPAQ
OS Device Name: Unknown
LUN: 1
Vendor: COMPAQ
Product: HSV111 (C)COMPAQ
OS Device Name: /dev/rdsk/c5t600508B400011C370000C00003210000d0s2

This command is quite handy when troubleshooting fibre channel. Very cool…

Forcing network speed and duplex in Open Boot prompt

I had a Sun box attached to a switch I had not control over and I needed to jumpstart it. Unfortunately, I was getting nowhere. After crosschecking everything in my setup the only thing I did not try was to force speed and duplex on the Sun box to see if it would start booting.

I wondered if it would be possible to do so in Open Boot prompt. So I fired up the most logical man page in the given situation - the boot man page. Sure enough it is possible to force speed and duplex from ok prompt:

{1} ok boot net:speed=100,duplex=full - install

And off it went happily booting…

Managing core files

If you have grown tired of having core files laying around all over the place you can manage them using coreadm command. You can set up system to save core files to a specific location.

First, here are the default core settings on a Solaris box:

bash-3.00# coreadm
global core file pattern:
global core file content: default
init core file pattern: core
init core file content: default
global core dumps: disabled
per-process core dumps: enabled
global setid core dumps: disabled
per-process setid core dumps: disabled
global core dump logging: disabled

Now, let’s configure the system to save init core files in /var/crash/cores in format EUID.execName.core:

bash-3.00# coreadm -i /var/crash/cores/%u.%f.core
bash-3.00# coreadm
global core file pattern:
global core file content: default
init core file pattern: /var/crash/cores/%u.%f.core
init core file content: default
global core dumps: disabled
per-process core dumps: enabled
global setid core dumps: disabled
per-process setid core dumps: disabled
global core dump logging: disabled

To test our setup, we can use gcore command.

bash-3.00# gcore -p 741
gcore: /var/crash/cores/0.bash.core dumped
bash-3.00# ls /var/crash/cores
0.bash.core hsperfdata_noaccess hsperfdata_root
bash-3.00#

Above, we generated core file for bash process with PID 741 run under effective UID 0. This should make the system tidier as far as core files are concerned.

Mirroring root disk using SVM

There are about 487359 documents on the Internet about how to mirror root disk in Solaris. So, here is 487360th.

The assumptions are following: the first disk has Solaris already installed, root slice is slice 1, and the disks are identical with the same size and geometry. If they have different cylinder, head, sector count or different size you will have to fiddle with sizing slices more.

The first step is to recreate the same slice arrangement on the second disk:

bash-3.00# prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2
fmthard: New volume table of contents now in place.

You can check both disks have the same VTOC using prtvtoc command

bash-3.00# prtvtoc /dev/rdsk/c1t0d0s2
* /dev/rdsk/c1t0d0s2 partition map
*
* Dimensions:
*     512 bytes/sector
*     424 sectors/track
*      24 tracks/cylinder
*   10176 sectors/cylinder
*   14089 cylinders
*   14087 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
0      3    01          0   4100928   4100927
1      2    00    4100928  20484288  24585215
2      5    00          0 143349312 143349311
3      8    00   24585216 118672512 143257727
7      0    00  143257728     91584 143349311

Now we have to create state database replicas on slice 7. We will be adding two replicas to each slice:

bash-3.00# metadb -a -f -c2 /dev/dsk/c1t0d0s7
bash-3.00# metadb -a -f -c2 /dev/dsk/c1t1d0s7

Database replicas are crucial part of SVM. Here is some important information about how they work, how many you need, etc.

Since the database replicas are in place we can start creating metadevices. The following commands will create metadevice d31 from slice c1t0d0s3, and metadevice d32 from slice c1t1d0s3. Then we create mirror d30 with d31 attached as a submirror. Finally we will attach submirror d32 to mirror d30. Once d32 is attached, the mirror d30 will automatically start syncing.

bash-3.00# metainit -f d31 1 1 c1t0d0s3
d31: Concat/Stripe is setup
bash-3.00# metainit -f d32 1 1 c1t1d0s3
d32: Concat/Stripe is setup
bash-3.00# metainit d30 -m d31
d30: Mirror is setup
bash-3.00# metattach d30 d32
d30: submirror d32 is attached

The procedure is the same for all other mirrors you might want to create. Root filesystem is slightly different. First you will have to create your submirrors. Then you will have to attach submirror with existing root filesystem, in this case d11, to the new mirror metadevice d10. Then you will have to run metaroot command. It will alter / entry in /etc/vfstab. Finally, you flush the filesystem using lockfs command and reboot.

bash-3.00# metainit -f d11 1 1 c1t0d0s1
d31: Concat/Stripe is setup
bash-3.00# metainit -f d12 1 1 c1t1d0s1
d32: Concat/Stripe is setup
bash-3.00# metainit d10 -m d11
d30: Mirror is setup
bash-3.00# metaroot d10
bash-3.00# lockfs -fa
bash-3.00# reboot

When the system reboots, you can attach the second submirror to d10 as follows:

bash-3.00# metattach d10 d12

You can check the sync progress using metastat command. Once all mirrors are synced up the next step is to configure the new swap metadevice, in my case d0, to be crash dump device. This is done using dumpadm command:

bash-3.00# dumpadm
Dump content: kernel pages
Dump device: /dev/dsk/c1t0d0s0 (dedicated)
Savecore directory: /var/crash/ultra
Savecore enabled: yes
bash-3.00# dumpadm -d /dev/md/dsk/d0

The final step is to modify PROM. First we need to find out which two physical devices c1t0d0 and c1t1d0 refer to:

bash-3.00# ls -l /dev/dsk/c1t0d0s1
lrwxrwxrwx 1 root root 43 Mar 4 14:38 /dev/dsk/c1t0d0s1 -> ../../devices/pci@1c,600000/scsi@2/sd@0,0:b
bash-3.00# ls -l /dev/dsk/c1t1d0s1
lrwxrwxrwx 1 root root 43 Mar 4 14:38 /dev/dsk/c1t1d0s1 -> ../../devices/pci@1c,600000/scsi@2/sd@1,0:b

The physical device path is everything starting from /pci…. Please make a note of sd towards the end of the device string. When creating device aliases below, sd will have to be changed to disk.

Now we create two device aliases called root and backup_root. Then we set boot-device to be root and backup_root. The :b refers to slice 1(root) on that particular disk.

bash-3.00# eeprom “use-nvramrc?=true”
bash-3.00# eeprom “nvramrc=devalias root /pci@1c,600000/scsi@2/disk@0,0 devalias backup_root /pci@1c,600000/scsi@2/disk@1,0″
bash-3.00# eeprom “boot-device=root:b backup_root:b net”

Now we can test that the system boot from both root and backup_root devices.

There is one more optional step. Adding two kernel tunables to /etc/system file. The first one is md_mirror:md_resync_bufsz which will speed up mirror resync. The second one is md:mirrored_root_flag. When this flag is enabled the system will boot even if less than majority of database replicas is available. Personally, I do not use the second tunable. More on these can be found in Solaris Tunable Parameters Reference Guide.

Moving boot disk to a different target or controller

I had a need for a system that can boot two different versions of Solaris from two different disks. Both disks were on target 0 when Solaris was installed on them. But when I moved one of the disks to be target 1, obviously there was going to be a problem with booting from that disk.

There is a simple way to get the disk booting from the new target. Here are the steps to take:

1. Boot the system off a DVD or CD or jumpstart server in single user mode
2. Mount the root filesystem on the disk in question and edit vfstab to reflect the new controller/target setup

The last step is to regenerate /etc/path_to_inst and device links in /dev. Searching the internet for some unrelated info I found out devfsadm has an undocumented -p switch that recreates path_to_inst file. The -r switch specifies location of root filesystem.

bash-3.00# devfsadm -r /mnt -p /mnt/etc/path_to_inst

Now you can reboot and the drive should be bootable again.