AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20080410141006.14563067@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<john.keiffer@onstor.com>,<tim.gardner@onstor.com>,<sandrine.boulanger@onstor.com>,<vikas.saini@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E0959684A@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 10 Apr 2008 14:14:51 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "John Keiffer" <john.keiffer@onstor.com>
Cc: "Tim Gardner" <tim.gardner@onstor.com>, "Sandrine Boulanger"
 <sandrine.boulanger@onstor.com>, "Vikas Saini" <vikas.saini@onstor.com>
Subject: Re: g5r10, badbeef, oops, etc
Message-ID: <20080410141451.414cb2af@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0959684A@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E0959684A@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

The oops, a deficiency in the mgmt_bus driver, I have attempted to
address in sub#17.  This filer sounds like a real problem child, but
what do I know.

You can do the change to your filer yourself now if you want, with the
following command:

# for d in /etc/rc?.d/K*onstor ; do
mv $d /etc/$(basename $(dirname $d))/K21onstor &&
echo renamed $d to /etc/$(basename $(dirname $d))/K21onstor
done

On Thu, 10 Apr 2008 13:45:59 -0700 "John Keiffer"
<john.keiffer@onstor.com> wrote:

> Tim,
> 
>  
> 
> My filer is g5r10 10.2.10.5. The SSC terminal is 10.2.10.236 port
> 9010, and I have closed it. I believe this is the 8h or 9th time this
> has happened since this morning.
> 
>  
> 
> There are no core files (maybe due to the following?), and besides, my
> mgmt volume is not accessible due to Input/output errors!
> 
>  
> 
> g5r10:~# cat /var/log/messages | grep crash
> 
> Apr 10 10:39:09 g5r10 kernel: fp0: Invalid crashdump signature
> 
> Apr 10 10:44:25 g5r10 kernel: fp0: Invalid crashdump signature
> 
> Apr 10 10:51:31 g5r10 kernel: fp0: Invalid crashdump signature
> 
> Apr 10 10:55:58 g5r10 kernel: fp0: Invalid crashdump signature
> 
> Apr 10 11:32:52 g5r10 kernel: fp0: Invalid crashdump signature
> 
> Apr 10 12:32:21 g5r10 kernel: fp0: Invalid crashdump signature
> 
> Apr 10 12:44:54 g5r10 kernel: fp0: Invalid crashdump signature
> 
> Apr 10 13:22:55 g5r10 kernel: fp0: Invalid crashdump signature
> 
> Apr 10 13:35:34 g5r10 kernel: fp0: Invalid crashdump signature
> 
> g5r10:~#
> 
>  
> 
> Here is an example of the kernel oops, and at bottom are the beginning
> of the badbeef messages after rebooting:
> 
>  
> 
> Apr 10 13:33:10 g5r10 : 0:0:vsd:ERROR: vsd_disableAllVsProc : Timeout
> disabling VS (unknown vs)
> 
> Apr 10 13:33:10 g5r10 : 0:0:vsd:ERROR: vsd_disableAllVsProc : disable
> of one or more VS failed
> 
> Apr 10 13:33:10 g5r10 : 0:0:nfxsh:DEBUG: cmd[0]: stats txrx kpi
> get_summary 8 : status[0]
> 
> Apr 10 13:33:10 g5r10 : 0:0:cluster2:ERROR: Node going down for
> reboot! (cluster_server: invalidating clusDb).
> 
> Apr 10 13:33:11 g5r10 : 0:0:eventd:DEBUG: > ems_logEvent()
> 
> Apr 10 13:33:11 g5r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
> 'local', State Down, Msg 'Node going down for reboot! (cluster_server:
> invalidating clusDb).'
> 
> INIT: Sending processes the TEApr 10 13:33:11 g5r10 :
> 0:0:eventd:DEBUG: < ems_logEvent()
> 
> Apr 10 13:33:11 g5r10 : 0:0:vtm:DEBUG:
> vtm_get_filer_config_and_start_vsvr_trans: start collecting failover
> vsvr
> 
> Apr 10 13:33:11 g5r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost PCC for
> 
> Apr 10 13:33:12 g5r10 : 0:0:asd:ERROR: Rcvd post request APP: unknown
> EVENT: Node
> 
> Stopping deferred execution scheduler: atd.
> 
> Stopping periodic command scheduler: crond.
> 
> Stopping MTA: exim4_listener.
> 
> Stopping internet superserver: inetd.
> 
> Stopping OpenBSD Secure Shell server: sshd.
> 
> Stopping automounter: done.
> 
> Stopping NTP server: ntpdApr 10 13:33:13 g5r10 : 0:0:cluster2:ERROR:
> ClusterContrl_GetRecordWhole: ubik_call failed, recType 5(group), code
> 5376, rc 30
> 
> .
> 
> Saving the system clock..
> 
> Stopping NFS common utilities: statd.
> 
> Stopping kernel log daemon: klogd.
> 
> Stopping system log daemon: syslogd.
> 
> Stopping ONStor services:DBE physical address: 0041000000
> 
> Data bus error, epc == ffffffff8218afe4, ra == ffffffff8204f614
> 
> Oops[#1]:
> 
> Cpu 0
> 
> $ 0   : 0000000000000000 0000000010001fe1 ffffffffffffffff
> 9000000041000000
> 
> $ 4   : 0000000000000038 a8000000049b6000 0100000000000000
> 0000000000000000
> 
> $ 8   : a8000000049b6000 9000000000000000 ffffffff822f0198
> 0000000100000000
> 
> $12   : 0000000010001fe0 000000001000001f 0000000000000000
> 0000000000000000
> 
> $16   : a80000000490e620 0000000000000000 0000000000000038
> 0000000000000000
> 
> $20   : 0000000000000001 900000f81a7084a0 0000000000000500
> a80000008eca88e0
> 
> $24   : 0000000000000010 ffffffff82195ec0
> 
> $28   : a80000008e740000 a80000008e7438c0 0000000000000000
> ffffffff8204f614
> 
> Hi    : 0000000000000000
> 
> Lo    : 0000000000001340
> 
> epc   : ffffffff8218afe4 yenta_interrupt+0x14/0x118     Not tainted
> 
> ra    : ffffffff8204f614 handle_IRQ_event+0x6c/0xe8
> 
> Status: 10001fe3    KX SX UX KERNEL EXL IE
> 
> Cause : 0080841c
> 
> PrId  : 00040103
> 
> Modules linked in: autofs4
> 
> Process eventd (pid: 997, threadinfo=a80000008e740000,
> task=a8000000049440c8)
> 
> Stack : ffffffff8204f614 0000000000000000 ffffffff822b6820
> 0000000000000038
> 
>         a80000000490e620 fffffffffffffbff ffffffff82310000
> ffffffff8204f764
> 
>         0100000000000000 a80000008e21c010 a80000008e1ea1a0
> 900000008f0012e0
> 
>         900000008f008c00 ffffffff820011a4 0000000000000000
> ffffffff82001840
> 
>         0000000000000000 00000000004ab4d0 900000f81a4bfd00
> 00000000004bfd00
> 
>         900000f81a4bfe40 a80000008e21c152 00000000000003d0
> ffffff0000000000
> 
>         0000000000000000 0000000000000000 0000000000000000
> 0000000100000000
> 
>         0000000000000010 a80000008e743de8 0000000000000000
> 0000000000000000
> 
>         0000000000000510 a80000008e21c010 a80000008e1ea1a0
> 900000008f0012e0
> 
>         900000008f008c00 900000f81a7084a0 0000000000000500
> a80000008eca88e0
> 
>         ...
> 
> Call Trace:
> 
> [<ffffffff8218afe4>] yenta_interrupt+0x14/0x118
> 
> [<ffffffff8204f614>] handle_IRQ_event+0x6c/0xe8
> 
> [<ffffffff8204f764>] __do_IRQ+0xd4/0x160
> 
> [<ffffffff820011a4>] plat_irq_dispatch+0x1e4/0x1f0
> 
> [<ffffffff82001840>] ret_from_irq+0x0/0x4
> 
> [<ffffffff8212e3e4>] src_unaligned_dst_aligned+0xc/0x50
> 
> [<ffffffff82195fd8>] mgmtbus_hard_start_xmit+0x118/0x178
> 
> [<ffffffff821a9ea4>] dev_queue_xmit+0x30c/0x458
> 
> [<ffffffff82220938>] eee_dgram_sendmsg+0x2b8/0x440
> 
> [<ffffffff82199540>] sock_sendmsg+0x98/0xe8
> 
> [<ffffffff821997d8>] sys_sendmsg+0x248/0x320
> 
> [<ffffffff8200fec8>] handle_sys+0x108/0x124
> 
>  
> 
>  
> 
> Code: ffbf0000  dca30010  8c620000 <0040202d> ac620000  dca60010
> 8cc30000  90c20804  1480001f
> 
> Kernel panic - not syncing: Fatal exception in interrupt
> 
> Rebooting in 5 seconds..<2>SiByte Watchdog in danger of initiating
> system reset in 7.1 seconds
> 
>  
> 
>  
> 
>  
> 
> PowerOn Self Test........OK
> 
>  
> 
> Initializing System......please wait
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
> PMON [SSC,EL,FP,64]
> 
> ONStor Inc. PROM_SIBYTE_CG : Cougar-prom-1.0.5 : Tue Mar 18 11:08:26
> 2008
> 
> CPU type SB1125.  Rev 35  600 MHz
> 
> module: SSC, Slot 0, CPU 0
> 
> Memory size 512 MB.
> 
> Icache size  32 KB, 32/line (4 way)
> 
> Dcache size  32 KB, 32/line (4 way)
> 
> Scache size 256 KB, 32/line (4 way)
> 
> debug IP addr = 10.2.10.5
> 
> debug IP mask = 255.255.0.0
> 
>  
> 
>  
> 
> Initializing Autoloader, hit control-E to bypass
> 
> ........................................................................
> ........
> 
>  
> 
> Type ctrl-e to stop autoload.
> 
> Waiting for SSC to enter autoload init state...done.
> 
>  ext2_load_file /dev/sda1/boot/vmlinux.bin at location
> ffffffff82000000
> 
> disk model: CF 1GB
> 
> disk geometry: cylinders=2044 heads=16 sectors=63
> 
> Type ctrl-e to stop autoload.
> 
> Waiting for TXRX to enter autoload init state...done.
> 
>  ext2_load_file /dev/sda1/boot/txrx_cg.bin at location 42000000
> 
> disk model: CF 1GB
> 
> disk geometry: cylinders=2044 heads=16 sectors=63
> 
> Type ctrl-e to stop autoload.
> 
> Waiting for FP to enter autoload init state...done.
> 
>  ext2_load_file /dev/sda1/boot/fp_cg.bin at location 44000000
> 
> disk model: CF 1GB
> 
> disk geometry: cylinders=2044 heads=16 sectors=63
> 
>  do_bsd_launch argc = 5 argv[5] = NULL
> 
>  
> 
> env[0] = 0xffffffff80b8c1e8:.cpuclock=4894967296.
> 
> env[1] = 0xffffffff80b8c238:.memsize=512.
> 
> env[2] = 0xffffffff80b8c288:.osloadoptions=mAt.
> 
> env[3] = 0xffffffff80b8c2d8:.boot=cold.
> 
> env[4] = 0xffffffff80b8c328:.busclock=600.
> 
> env[5] = 0xffffffff80b8c378:.ipaddr=10.2.10.5.
> 
> env[6] = 0xffffffff80b8c3c8:.netmask=255.255.0.0.
> 
> env[7] = 0xffffffff80b8c418:.macaddr0=.00:07:34:07:57:80.
> 
> env[8] = 0xffffffff80b8c468:.macaddr1=.00:07:34:07:57:81.
> 
> env[9] = 0xffffffff80b8c4b8:.bootdev=/dev/sda1.
> 
>  Load options and params for [g]
> 
>   Address 0xffffffff82000000 argc = 5
> 
>    argv [0] = g
> 
>    argv [1] = root=/dev/sda1
> 
>    argv [2] = ip=none
> 
>    argv [3] = rootdelay=1
> 
>    argv [4] = NULL
> 
>  pointer to Prom Util routines = 0x0
> 
>  Command should be  (addr)(argc, argv, env_strings,
> ptr_prom_util_routines)
> 
>  
> 
>  
> 
> Linux version 2.6.22-cg (build@k3.onstor.lab) (gcc version 4.1.2
> 20061115 (prerelease) (Debian 4.1.1-21)) #1 Wed Apr 9 17:40:39 PDT
> 2008
> 
> Booting Linux kernel...Mips64 Cougar
> 
> cougar_pmon_init: argc=5, arg=ffffffff80c04c30, env=ffffffff80b8c168
> 
> prom_init: env[0] = 'cpuclock=4894967296'
> 
> prom_init: env[1] = 'memsize=512'
> 
> prom_init: env[2] = 'osloadoptions=mAt'
> 
> prom_init: env[3] = 'boot=cold'
> 
> prom_init: env[4] = 'busclock=600'
> 
> prom_init: env[5] = 'ipaddr=10.2.10.5'
> 
> prom_init: env[6] = 'netmask=255.255.0.0'
> 
> prom_init: env[7] = 'macaddr0=00:07:34:07:57:80'
> 
> prom_init: env[8] = 'macaddr1=00:07:34:07:57:81'
> 
> prom_init: env[9] = 'bootdev=/dev/sda1'
> 
> CPU revision is: 00040103
> 
> FPU revision is: 000f0103
> 
> Broadcom SiByte BCM1125H A4 @ 600 MHz (SB1 rev 3)
> 
> Board type: ONStor Cougar
> 
> This kernel optimized for ONStor Cougar board without CFE
> 
> Determined physical RAM map:
> 
>  memory: 0000000002000000 @ 0000000000000000 (ROM data)
> 
>  memory: 000000000e000000 @ 0000000002000000 (usable)
> 
>  memory: 000000000f000000 @ 0000000080000000 (usable)
> 
>  memory: 0000000001000000 @ 000000008f000000 (reserved)
> 
> Wasting 458752 bytes for tracking 8192 unused pages
> 
> Built 1 zonelists.  Total pages: 577720
> 
> Kernel command line: console=duart0,57600n8 root=/dev/sda1
> root=/dev/sda1 ip=none rootdelay=1 NULL
> 
> Primary instruction cache 32kB, 4-way, linesize 32 bytes.
> 
> Primary data cache 32kB, 4-way, linesize 32 bytes.
> 
> Secondary cache 256kB, 4-way, linesize 32 bytes.
> 
> Synthesized TLB refill handler (38 instructions).
> 
> Synthesized TLB load handler fastpath (49 instructions).
> 
> Synthesized TLB store handler fastpath (49 instructions).
> 
> Synthesized TLB modify handler fastpath (48 instructions).
> 
> PID hash table entries: 4096 (order: 12, 32768 bytes)
> 
> Using 1.000 MHz high precision timer.
> 
> Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
> 
> Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
> 
> Memory: 433408k/475136k available (2202k kernel code, 41556k reserved,
> 690k data, 112k init, 0k highmem)
> 
> Mount-cache hash table entries: 256
> 
> Checking for the multiply/shift bug... no.
> 
> Checking for the daddi bug... no.
> 
> Checking for the daddiu bug... no.
> 
> NET: Registered protocol family 16
> 
> SCSI subsystem initialized
> 
> PCI: Bus 1, cardbus bridge: 0000:00:07.0
> 
>   IO window: 00008000-000080ff
> 
>   IO window: 00008400-000084ff
> 
>   PREFETCH window: 44000000-47ffffff
> 
>   MEM window: 48000000-4bffffff
> 
> PCI: Bus 5, cardbus bridge: 0000:00:07.1
> 
>   IO window: 00008800-000088ff
> 
>   IO window: 00008c00-00008cff
> 
>   PREFETCH window: 4c000000-4fffffff
> 
>   MEM window: 50000000-53ffffff
> 
> Time: MIPS clocksource has been installed.
> 
> NET: Registered protocol family 2
> 
> IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
> 
> TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
> 
> TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
> 
> TCP: Hash tables configured (established 131072 bind 65536)
> 
> TCP reno registered
> 
> io scheduler noop registered
> 
> io scheduler anticipatory registered
> 
> io scheduler deadline registered
> 
> io scheduler cfq registered (default)
> 
> SiByte Watchdog: timeout is 8.3 secs
> 
> eth0: enabling TCP rcv checksum
> 
> eth0: SiByte Ethernet at 0x10064000, address: 00:07:34:07:57:80
> 
> eth1: enabling TCP rcv checksum
> 
> eth1: SiByte Ethernet at 0x10065000, address: 00:07:34:07:57:81
> 
> Yenta: CardBus bridge found at 0000:00:07.0 [0000:0000]
> 
> Yenta: Enabling burst memory read transactions
> 
> Yenta: Using INTVAL to route CSC interrupts to PCI
> 
> Yenta: Routing CardBus interrupts to PCI
> 
> Yenta TI: socket 0000:00:07.0, mfunc 0x00000022, devctl 0x60
> 
> Yenta: ISA IRQ mask 0x0000, PCI irq 56
> 
> Socket status: 30000059
> 
> pcmcia: parent PCI bridge I/O window: 0x0 - 0x1ffffff
> 
> cs: IO port probe 0x0-0x1ffffff: clean.
> 
> pcmcia: parent PCI bridge Memory window: 0x41000000 - 0x5fffffff
> 
> Yenta: CardBus bridge found at 0000:00:07.1 [0000:0000]
> 
> Yenta: Using INTVAL to route CSC interrupts to PCI
> 
> Yenta: Routing CardBus interrupts to PCI
> 
> Yenta TI: socket 0000:00:07.1, mfunc 0x00000022, devctl 0x60
> 
> Yenta: ISA IRQ mask 0x0000, PCI irq 57
> 
> Socket status: 30000059
> 
> pcmcia: parent PCI bridge I/O window: 0x0 - 0x1ffffff
> 
> cs: IO port probe 0x0-0x1ffffff: clean.
> 
> pcmcia: parent PCI bridge Memory window: 0x41000000 - 0x5fffffff
> 
> pccard: PCMCIA card inserted into slot 0
> 
> cs: memory probe 0x41000000-0x5fffffff: excluding
> 0x41000000-0x417fffff 0x44000000-0x53ffffff
> 
> pcmcia: registering new device pcmcia0.0
> 
> scsi0 : pata_pcmcia
> 
> ata1: PATA max PIO0 cmd 0x90000000dc000000 ctl 0x90000000dc00000e
> bmdma 0x0000000000000000 irq 56
> 
> ds1511 ds1511: rtc core: registered ds1511 as rtc0
> 
> MGMTBUS: using irq 28
> 
> MGMTBUS: registered net device mgmtbus @ 900000008f000000
> 
> rcon device will be in the hizzy @ 900000008f0000e0
> 
> TCP cubic registered
> 
> NET: Registered protocol family 1
> 
> NET: Registered protocol family 17
> 
> NET: Registered protocol family 30
> 
> ds1511 ds1511: setting the system clock to 2008-04-10 20:35:07
> (1207859707)
> 
> Waiting 1sec before mounting root device...
> 
> ata1.00: CFA: CF 1GB, 06/07/29, max PIO2
> 
> ata1.00: 2060352 sectors, multi 0: LBA
> 
> ata1.00: configured for PIO0
> 
> scsi 0:0:0:0: Direct-Access     ATA      CF 1GB           06/0 PQ: 0
> ANSI: 5
> 
> sd 0:0:0:0: [sda] 2060352 512-byte hardware sectors (1055 MB)
> 
> sd 0:0:0:0: [sda] Write Protect is off
> 
> sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't
> support DPO or FUA
> 
> sd 0:0:0:0: [sda] 2060352 512-byte hardware sectors (1055 MB)
> 
> sd 0:0:0:0: [sda] Write Protect is off
> 
> sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't
> support DPO or FUA
> 
>  sda: sda1 sda2 sda3 sda4
> 
> sd 0:0:0:0: [sda] Attached SCSI removable disk
> 
> pccard: PCMCIA card inserted into slot 1
> 
> cs: memory probe 0x41000000-0x5fffffff: excluding
> 0x41000000-0x41ffffff 0x44000000-0x53ffffff
> 
> pcmcia: registering new device pcmcia1.0
> 
> scsi1 : pata_pcmcia
> 
> ata2: PATA max PIO0 cmd 0x90000000dc000010 ctl 0x90000000dc00001e
> bmdma 0x0000000000000000 irq 57
> 
> ata2.00: CFA: CF 1GB, 06/07/29, max PIO2
> 
> ata2.00: 2060352 sectors, multi 0: LBA
> 
> ata2.00: configured for PIO0
> 
> scsi 1:0:0:0: Direct-Access     ATA      CF 1GB           06/0 PQ: 0
> ANSI: 5
> 
> sd 1:0:0:0: [sdb] 2060352 512-byte hardware sectors (1055 MB)
> 
> sd 1:0:0:0: [sdb] Write Protect is off
> 
> sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't
> support DPO or FUA
> 
> sd 1:0:0:0: [sdb] 2060352 512-byte hardware sectors (1055 MB)
> 
> sd 1:0:0:0: [sdb] Write Protect is off
> 
> sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't
> support DPO or FUA
> 
>  sdb: sdb1 sdb2 sdb3 sdb4
> 
> sd 1:0:0:0: [sdb] Attached SCSI removable disk
> 
> EXT3-fs: INFO: recovery required on readonly filesystem.
> 
> EXT3-fs: write access will be enabled during recovery.
> 
> kjournald starting.  Commit interval 5 seconds
> 
> EXT3-fs: recovery complete.
> 
> EXT3-fs: mounted filesystem with ordered data mode.
> 
> VFS: Mounted root (ext3 filesystem) readonly.
> 
> Freeing unused kernel memory: 112k freed
> 
> INIT: version 2.86 booting
> 
> Starting the hotplug events dispatcher: udevd.
> 
> Synthesizing the initial hotplug events...done.
> 
> Waiting for /dev to be fully populated...done.
> 
> Setting parameters of disc: (none).
> 
> Setting kernel variables...done.
> 
> Activating swap...done.
> 
> Checking root file system...fsck 1.40-WIP (14-Nov-2006)
> 
> /dev/sda1: clean, 9619/89088 files, 72156/178156 blocks
> 
> done.
> 
> Setting the system clock..
> 
> Cleaning up ifupdown....
> 
> Loading kernel modules...done.
> 
> Loading device-mapper support.
> 
> Checking file systems...fsck 1.40-WIP (14-Nov-2006)
> 
> /dev/sda3: recovering journal
> 
> /dev/sda3: clean, 1036/43648 files, 25081/174384 blocks
> 
> /dev/sda4: recovering journal
> 
> /dev/sda4: clean, 21/28224 files, 11026/112896 blocks
> 
> done.
> 
> Mounting local filesystems...done.
> 
> Activating swapfile swap...done.
> 
> Starting ONStor initial configuration utility:.
> 
> Setting up networking....
> 
> Configuring network interfaces...SIOCADDRT: Network is unreachable
> 
> SIOCADDRT: Network is unreachable
> 
> run-parts: /etc/network/if-up.d/addroutes exited with return code 7
> 
> SIOCADDRT: Network is unreachable
> 
> Don't seem to be have all the variables for eth1/inet.
> 
> Failed to bring up eth1.
> 
> done.
> 
> Starting portmap daemon....
> 
> INIT: Entering runlevel: 2
> 
> Starting system log daemon: syslogd.
> 
> Starting kernel log daemon: klogd.
> 
> Starting portmap daemon...Already running..
> 
> Starting MTA: exim4.
> 
> * Not starting internet superserver: no services enabled.
> 
> Starting OpenBSD Secure Shell server: sshd.
> 
> Starting NFS common utilities: statd.
> 
> Starting automounter: loading autofs4 kernel module, no automount maps
> defined.
> 
> Starting NTP server: ntpd.
> 
> Starting deferred execution scheduler: atd.
> 
> Starting periodic command scheduler: crond.
> 
> Starting ONStor services: mgmtbus emrscron /onstor/bin/pm.
> 
> Apr 10 13:35:44 g5r10 pm: /onstor/bin/elog: finished initialization.
> 
> Apr 10 13:35:45 g5r10 : 0:0:pm:INFO: /onstor/bin/ncmd: finished
> initialization.
> 
> Apr 10 13:35:45 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_add_filers: added
> filer(g6r10) ip(ncm@10.2.10.6)
> 
> Apr 10 13:35:46 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_add_filers: added
> filer(g5r10) ip(ncm@10.2.10.5)
> 
> Apr 10 13:35:46 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_get_cluster_conf:
> my_filer(g5r10)
> 
> Apr 10 13:35:46 g5r10 : 0:0:ncm:DEBUG: ncmd : main: ncm(3) initialized
> and starting
> 
> Apr 10 13:35:46 g5r10 : 0:0:pm:INFO: /onstor/bin/eventd: finished
> initialization.
> 
> Apr 10 13:35:46 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_open_filer: opening
> filer(g6r10) app(ncm@10.2.10.6)
> 
> Apr 10 13:35:46 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_open_filer: open
> filer(g6r10) msg[0x489028] sock[8]
> 
> Apr 10 13:35:46 g5r10 : 0:0:pm:INFO: /onstor/bin/timekeeper: finished
> initialization.
> 
> Apr 10 13:35:47 g5r10 : 0:0:pm:INFO: /onstor/bin/chassisd: finished
> initialization.
> 
> Apr 10 13:35:47 g5r10 : 0:0:cm:NOTICE: CHASSISD: Chassis Manager:
> Started
> 
> Apr 10 13:35:47 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_filer_msg_done
> filer(g6r10) open - setting timeouts
> 
>  
> 
> Apr 10 13:35:47 g5r10 : 0:0:cluster2:INFO: ClusterCtrl_InitUbik:
> connection opened 60a020a
> 
> OnStor GNU/Linux 4.0 g5r10 duart0
> 
>  
> 
> g5r10 login: Apr 10 13:35:48 g5r10 : 0:0:cluster2:INFO:
> ClusterCtrl_InitUbik: connection opened 50a020a
> 
> Apr 10 13:35:48 g5r10 : 0:0:cluster2:INFO: ClusterCtrl_InitUbik:
> connection opened 60a020a
> 
> Apr 10 13:35:48 g5r10 : 0:0:cluster2:INFO: ClusterCtrl_InitUbik:
> connection opened 50a020a
> 
> Apr 10 13:35:48 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_add_filers: existing
> filer(g6r10) ip(ncm@10.2.10.6)
> 
> Apr 10 13:35:48 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_add_filers: existing
> filer(g5r10) ip(ncm@10.2.10.5)
> 
> Apr 10 13:35:48 g5r10 : 0:0:ncm:DEBUG: ncmd : ncm_get_cluster_conf:
> my_filer(g5r10)
> 
> Apr 10 13:35:48 g5r10 : 0:0:cluster2:INFO: ClusterCtrl_RecvPingMsg:
> connection up from host g6r10, ip 10.2.10.6
> 
> Apr 10 13:35:51 g5r10 : 0:0:cluster2:NOTICE: Using 10.2.10.5 as my
> primary address
> 
> Apr 10 13:35:51 g5r10 : 0:0:cluster2:NOTICE: Ubik: SDISK_SendFile:
> Synchronize database with server 10.2.10.6, version 0x47fe5b71:18f
> 
> Apr 10 13:35:53 g5r10 : 0:0:cluster2:NOTICE: Ubik: SDISK_SendFile:
> Synchronize database with server 10.2.10.6 completed
> 
> Apr 10 13:35:54 g5r10 : 0:0:cluster2:INFO: ClusterServ_UpdateState:
> database synchronized with the new cluster
> 
> Apr 10 13:35:54 g5r10 : 0:0:cluster2:INFO: ClusterCtrl_iUpdateState:
> post PCC up pccname g6r10
> 
> Apr 10 13:35:54 g5r10 : 0:0:cluster2:NOTICE: ubik init with buff size
> 64
> 
> Apr 10 13:35:54 g5r10 : 0:0:eventd:DEBUG: > ems_logEvent()
> 
> Apr 10 13:35:54 g5r10 : 0:0:eventd:WARNING: Process-EVENT 0.0.0.0:
> Mgmt Port 0.0.0.0 PCC, State Up
> 
> Apr 10 13:35:54 g5r10 : 0:0:eventd:DEBUG: < ems_logEvent()
> 
> Apr 10 13:35:55 g5r10 : 0:0:cluster2:INFO:
> ClusterCtrl_readyRestoreFiler: data base ready to restore
> 
> Apr 10 13:35:55 g5r10 : 0:0:pm:INFO: /onstor/bin/cluster_server:
> finished initialization.
> 
> Apr 10 13:35:55 g5r10 : 0:0:sdm:DEBUG: sdm_openRmcToFC : RMC Open
> session sent for port SP2.0 successful, tag[badbeef]
> 
> Apr 10 13:35:55 g5r10 : 0:0:pm:INFO: /onstor/bin/sdm_cfgd: finished
> initialization.
> 
> Apr 10 13:35:55 g5r10 : 0:0:sdm:INFO: ONStor Storage Device Manager
> (c)2003: Started
> 
> Apr 10 13:35:56 g5r10 : 0:0:evm:DEBUG: EVM daemon is (re)starting...
> 
> Apr 10 13:35:56 g5r10 : 0:0:evm:DEBUG: EVM myNfx: cluster[g6r10]
> node[g5r10] chassisId[67415]
> 
> Apr 10 13:35:56 g5r10 : 0:0:pm:INFO: /onstor/bin/evm_cfgd: finished
> initialization.
> 
> Apr 10 13:35:56 g5r10 : 0:0:pm:INFO: /onstor/bin/crashsaved: finished
> initialization.
> 
> Apr 10 13:35:56 g5r10 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> 
> Apr 10 13:35:56 g5r10 : 0:0:pm:INFO: /onstor/bin/ea: finished
> initialization.
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:INFO: evm_rcvRmcMsg : NCM session open
> failed
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:INFO: evm_rcvRmcMsg : SDM session open
> failed
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:INFO: evm_closeSess : NCM session
> closed
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:INFO: evm_closeSess : SDM session
> closed
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:DEBUG: evm_openSessAsync : NCM session
> is being opened
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:DEBUG: evm_ncm_send : No session to
> NCM
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:DEBUG: evm_sendScanReqProc: Unable to
> send sync msg to SPM, rc[-5624]
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:INFO: evm_rcvRmcMsg : NCM session open
> success
> 
> Apr 10 13:35:57 g5r10 : 0:0:evm:INFO: evm_rcvRmcMsg : SDM session open
> success
> 
> Apr 10 13:35:57 g5r10 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM session open
> failed
> 
> Apr 10 13:35:58 g5r10 : 0:0:pm:INFO: /onstor/bin/spm: finished
> initialization.
> 
> Apr 10 13:35:58 g5r10 : 0:0:pm:INFO: /onstor/bin/ipmd: finished
> initialization.
> 
> Apr 10 13:35:58 g5r10 : 0:0:pm:INFO: /onstor/bin/tape-driver: finished
> initialization.
> 
> Apr 10 13:36:00 g5r10 : 0:0:ssc_ndmp:INFO: ndmp_GetFPState:  fp card
> is DOWN
> 
> Apr 10 13:36:00 g5r10 : 0:0:pm:INFO: /onstor/bin/ndmp_cfgd: finished
> initialization.
> 
> Apr 10 13:36:01 g5r10 : 0:0:auth_agent:NOTICE: main: auth-agent
> started
> 
> Apr 10 13:36:02 g5r10 : 0:0:pm:INFO: /onstor/bin/auth-agent: finished
> initialization.
> 
> Apr 10 13:36:02 g5r10 : 0:0:ea:NOTICE: ea_closeSess : NCM session
> closed
> 
> Apr 10 13:36:03 g5r10 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM session open
> success
> 
> Apr 10 13:36:03 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=5, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=5, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=5, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=14, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=14, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=14, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=10, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=10, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=10, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=2, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=2, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=2, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=12, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=12, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=12, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=6, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=6, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=6, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=3, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=3, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=3, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=4, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=4, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=4, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=11, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=11, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=11, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=7, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=7, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=7, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=8, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=8, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=8, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=9, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=9, file=72, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=9, file=74, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=16, file=71, #masterRec=0
> 
> Apr 10 13:36:04 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=16, file=72, #masterRec=0
> 
> Apr 10 13:36:05 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=16, file=74, #masterRec=0
> 
> Apr 10 13:36:05 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=13, file=71, #masterRec=0
> 
> Apr 10 13:36:05 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=13, file=72, #masterRec=0
> 
> Apr 10 13:36:05 g5r10 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1230] : vs=13, file=74, #masterRec=0
> 
> Apr 10 13:36:05 g5r10 : 0:0:nfxsh:DEBUG: cmd[0]: autosupport emrs show
> config : status[0]
> 
> Apr 10 13:36:05 g5r10 : 0:0:pm:INFO: /onstor/bin/vsd: finished
> initialization.
> 
> Apr 10 13:36:05 g5r10 : 0:0:cluster2:INFO: ClusterCtrl_ReleaseFiler:
> called by vtm
> 
> Apr 10 13:36:05 g5r10 : 0:0:vtm:DEBUG:
> vtm_get_filer_config_and_start_vsvr_trans: start collecting failover
> vsvr
> 
> Apr 10 13:36:05 g5r10 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card DOWN to g6r10
> 
> Apr 10 13:36:06 g5r10 : 0:0:pm:INFO: /onstor/bin/vtmd: finished
> initialization.
> 
> Apr 10 13:36:08 g5r10 : 0:0:pm:INFO: /onstor/bin/sanmd: finished
> initialization.
> 
> Apr 10 13:36:08 g5r10 : 0:0:pm:INFO: /onstor/bin/cluster_relay:
> finished initialization.
> 
> Apr 10 13:36:08 g5r10 : 0:0:sanm:NOTICE: SANM: ONStor Data Mirror
> (c)2006: Started
> 
> Apr 10 13:36:08 g5r10 : 0:0:pm:INFO: /onstor/bin/snmpd: finished
> initialization.
> 
> Apr 10 13:36:09 g5r10 : 0:0:nfxsh:DEBUG: cmd[0]: autosupport emrs show
> config : status[0]
> 
> Apr 10 13:36:09 g5r10 : 0:0:nfxsh:DEBUG: cmd[0]: system show chassis :
> status[0]
> 
> Apr 10 13:36:10 g5r10 : 0:0:snmpd:DEBUG: Local config already synced
> with PCC
> 
> Apr 10 13:36:10 g5r10 : 0:0:snmpd:INFO: UCD-SNMP version 4.2.2
> 
> Apr 10 13:36:15 g5r10 : 0:0:asd:INFO: asd_main: auto support conf is
> TO john.keiffer@onstor.com, NOTETO john.keiffer@onstor.com, FROM
> G6R10@onstor.com, server 0.0.0.0, enable 1
> 
> Apr 10 13:36:15 g5r10 : 0:0:pm:INFO: /onstor/bin/asd: finished
> initialization.
> 
> Apr 10 13:36:16 g5r10 : 0:0:asd:INFO: asd_register_events: Registered
> for all events
> 
> Apr 10 13:36:16 g5r10 : 0:0:pm:INFO: /onstor/bin/sscccc: finished
> initialization.
> 
> Apr 10 13:36:16 g5r10 : 0:0:sscccc:DEBUG: listen_rmc: Starting RMC
> Listener
> 
> Apr 10 13:36:16 g5r10 : 0:0:sscccc:DEBUG: listen_rmc: listening as
> 'sscccc'
> 
> Apr 10 13:36:16 g5r10 : 0:0:sscccc:DEBUG: listen_rmc: Returning 0
> 
> Apr 10 13:36:16 g5r10 : 0:0:sscccc:INFO: no
> /onstor/etc/sscccc_hosts_deny file
> 
> Apr 10 13:36:21 g5r10 : 0:0:eventd:DEBUG: > ems_logEvent()
> 
> Apr 10 13:36:21 g5r10 : 0:0:eventd:NOTICE: Process-EVENT Node: Name
> 'g5r10', State Up, Msg ''
> 
> Apr 10 13:36:22 g5r10 : 0:0:eventd:DEBUG: < ems_logEvent()
> 
> Apr 10 13:36:22 g5r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: NOT PCC
> message for g5r10
> 
> Apr 10 13:36:22 g5r10 : 0:0:vtm:DEBUG:
> vtm_get_filer_config_and_start_vsvr_trans: start collecting failover
> vsvr
> 
> Apr 10 13:36:22 g5r10 : 0:0:asd:DEBUG: Rcvd post request APP: cluster
> EVENT: Node
> 
> Apr 10 13:36:26 g5r10 : 0:0:sdm:INFO: Success reading configuration
> record from cluster db.
> 
> Apr 10 13:36:26 g5r10 : 0:0:sdm:INFO: Start SCSI Level device
> discovery.
> 
> Apr 10 13:36:26 g5r10 : 0:0:sdm:DEBUG: sdm_rcvRmcMsg:RMC open not
> successful, type [1] {sdm:Agile1075}->{:scsi2.2.0} msg flags[1]
> msg_id[0] msg_seq[0] status[-19] rc[-20]
> 
> Apr 10 13:36:26 g5r10 : 0:0:sdm:DEBUG: sdm_openRmcToFC : RMC Open
> session sent for port SP2.0 successful, tag[badbeef]
> 
> Apr 10 13:36:27 g5r10 : 0:0:sdm:DEBUG: sdm_rcvRmcMsg:RMC open not
> successful, type [1] {sdm:Agile1258}->{:scsi2.2.0} msg flags[1]
> msg_id[0] msg_seq[0] status[-19] rc[-20]
> 
> Apr 10 13:36:27 g5r10 : 0:0:sdm:DEBUG: sdm_openRmcToFC : RMC Open
> session sent for port SP2.0 successful, tag[badbeef]
> 
> Apr 10 13:36:27 g5r10 : 0:0:sdm:DEBUG: sdm_rcvRmcMsg:RMC open not
> successful, type [1] {sdm:Agile1261}->{:scsi2.2.0} msg flags[1]
> msg_id[0] msg_seq[0] status[-19] rc[-20]
> 
> Apr 10 13:36:27 g5r10 : 0:0:sdm:DEBUG: sdm_openRmcToFC : RMC Open
> session sent for port SP2.0 successful, tag[badbeef]
> 
>  
> 
> Thank you,
> 
> John Keiffer
> 
>  
> 
>  
> 
