AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20090331160313.0b9e6d22@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:mail.onstor.net
NSV:
SSH:
R:<maxim.kozlovsky@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@exch1.onstor.net/INBOX	0	2779531E7C760D4491C96305019FEEB52AC8EA6374@exch1.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Tue, 31 Mar 2009 16:03:16 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: Maxim Kozlovsky <maxim.kozlovsky@onstor.com>
Subject: Re: linux crash loop
Message-ID: <20090331160316.69b12c75@ripper.onstor.net>
In-Reply-To: <2779531E7C760D4491C96305019FEEB52AC8EA6374@exch1.onstor.net>
References: <2779531E7C760D4491C96305019FEEB52AC8EA6374@exch1.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

You're getting a machine check exception.  Hardware problem.  Call
hardware tech support </Brian_Stark>.  It happened while reading the
PCI bus.  Not good.

Ouch, looks like something is badly on the fritz.  Machine dying?  Or
hardware not properly initialized?


On Tue, 31 Mar 2009 15:32:02 -0700 Maxim Kozlovsky
<maxim.kozlovsky@onstor.com> wrote:

> I've got that Linux crash loop again:
> 
>  -- Do Linux defaults launch
> 
> env[0] = 0xffffffff80b92778:.cpuclock=4894967296.
> env[1] = 0xffffffff80b927c8:.memsize=512.
> env[2] = 0xffffffff80b92818:.osloadoptions=mAt.
> env[3] = 0xffffffff80b92868:.boot=cold.
> env[4] = 0xffffffff80b928b8:.busclock=600.
> env[5] = 0xffffffff80b92908:.ipaddr=10.2.203.11.
> env[6] = 0xffffffff80b92958:.netmask=255.255.0.0.
> env[7] = 0xffffffff80b929a8:.macaddr0=.00:07:34:07:69:00.
> env[8] = 0xffffffff80b929f8:.macaddr1=.00:07:34:07:69:01.
> env[9] = 0xffffffff80b92a48:.bootdev=/dev/sdb1.
>  Load options and params for [g]
>   Address 0xffffffff82000000 argc = 5
>    argv [0] = g
>    argv [1] = root=/dev/sdb1
>    argv [2] = ip=none
>    argv [3] = rootdelay=1
>    argv [4] = onstor_model=ONS-SYS-6700
> Linux version 2.6.22-cg-g931f5229-dirty (andys@ripper) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #39 Thu Feb 19 17:46:09 PST 2009
> Booting Linux kernel...Mips64 Cougar
> cougar_pmon_init: argc=5, arg=ffffffff80c0b1e0, env=ffffffff80b926f8
> prom_init: env[0] = 'cpuclock=4894967296'
> prom_init: env[1] = 'memsize=512'
> prom_init: env[2] = 'osloadoptions=mAt'
> prom_init: env[3] = 'boot=cold'
> prom_init: env[4] = 'busclock=600'
> prom_init: env[5] = 'ipaddr=10.2.203.11'
> prom_init: env[6] = 'netmask=255.255.0.0'
> prom_init: env[7] = 'macaddr0=00:07:34:07:69:00'
> prom_init: env[8] = 'macaddr1=00:07:34:07:69:01'
> prom_init: env[9] = 'bootdev=/dev/sdb1'
> CPU revision is: 00040103
> FPU revision is: 000f0103
> Broadcom SiByte BCM1125H A4 @ 600 MHz (SB1 rev 3)
> Board type: ONStor Cougar
> This kernel optimized for ONStor Cougar board without CFE
> Determined physical RAM map:
>  memory: 0000000010000000 @ 0000000000000000 (usable)
>  memory: 000000000f000000 @ 0000000080000000 (usable)
> Built 1 zonelists.  Total pages: 577720
> Kernel command line: console=duart0,57600n8 root=/dev/nfs nfsroot=10.0.0.42:/var/nfsroot/cougar,v3,tcp ip=10.2.10.8:10.0.0.42:10.2.0.1:255.255.0.0:coolcat:eth0:none root=/dev/sdb1 ip=none rootdelay=1 onstor_model=ONS-SYS-6700
> Primary instruction cache 32kB, 4-way, linesize 32 bytes.
> Primary data cache 32kB, 4-way, linesize 32 bytes.
> Secondary cache 256kB, 4-way, linesize 32 bytes.
> Synthesized TLB refill handler (38 instructions).
> Synthesized TLB load handler fastpath (49 instructions).
> Synthesized TLB store handler fastpath (49 instructions).
> Synthesized TLB modify handler fastpath (48 instructions).
> PID hash table entries: 4096 (order: 12, 32768 bytes)
> Using 1.000 MHz high precision timer.
> Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
> Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
> Memory: 466176k/507904k available (2203k kernel code, 41588k reserved, 720k data, 112k init, 0k highmem)
> Mount-cache hash table entries: 256
> Checking for the multiply/shift bug... no.
> Checking for the daddi bug... no.
> Checking for the daddiu bug... no.
> NET: Registered protocol family 16
> SCSI subsystem initialized
> Got mcheck at ffffffff82197c38
> Cpu 0
> $ 0   : 0000000000000000 0000000014001fe0 0000000000000002 0000000014001fe2
> $ 4   : 000000000012166d 0000000000000005 0000000000000001 0000000000000001
> $ 8   : a8000000048c7e20 ffffffff822ba890 ffffffffffff0a70 ffffffff82310000
> $12   : ffffffff82320000 ffffffff82310000 fffffffffffffffd ffffffff8223c568
> $16   : 0000000000000004 0000000000000000 a80000000495a800 0000000000000028
> $20   : a8000000048c7e20 ffffffff82300000 ffffffff822f0000 ffffffff822f0000
> $24   : 0000000000000000 0000000000000030
> $28   : a8000000048c4000 a8000000048c75a0 0000000000000000 ffffffff82197c18
> Hi    : 0000000000000000
> Lo    : 0000000000000118
> epc   : ffffffff82197c38 sb1250_pcibios_read+0x70/0x180     Not tainted
> ra    : ffffffff82197c18 sb1250_pcibios_read+0x50/0x180
> Status: 14001fe2    KX SX UX KERNEL EXL
> Cause : 00809060
> PrId  : 00040103
> 
> Code: 24020002  12020037  0000182d <ae840000> dfbf0028  dfb40020  dfb30018  dfb20010  0060102d <0>Kernel panic - not syncing: Caught Machine Check exception - not caused by multiple matching entries in the TLB.
> Rebooting in 5 seconds..<1>CPU 0 Unable to handle kernel paging request at virtual address 000000000000000c, epc == ffffffff8218eebc, ra == ffffffff8218efe4
> Oops[#1]:
> Cpu 0
> $ 0   : 0000000000000000 0000000014001fe0 ffffffff82320000 0000000000000000
> $ 4   : 0000000000000000 000000000000000c 0000000000000000 0000000000000000
> $ 8   : ffffffff822c0000 ffffffff822ba890 ffffffffffff0f1a ffffffff82310000
> $12   : ffffffff82320000 ffffffff82310000 fffffffffffffffd ffffffff8223c568
> $16   : cccccccccccccccd 0000000000000003 000000000000012c ffffffff82300000
> $20   : ffffffff82300000 00000000000003e8 ffffffff822f0000 ffffffff822f0000
> $24   : 0000000000000000 ffffffff8212da88
> $28   : a8000000048c4000 a8000000048c7390 0000000000000000 ffffffff8218efe4
> Hi    : 0000000000000000
> Lo    : 0000000000000000
> epc   : ffffffff8218eebc rtc_write+0x1c/0x28     Not tainted
> ra    : ffffffff8218efe4 ds1511_wdog_set+0xcc/0x148
> Status: 14001fe2    KX SX UX KERNEL EXL
> Cause : 1080900c
> BadVA : 000000000000000c
> PrId  : 00040103
> Modules linked in:
> Process swapper (pid: 1, threadinfo=a8000000048c4000, task=a8000000048c1768)
> Stack : 0000000000001388 fffffffffffffd98 ffffffff82300000 ffffffff8212da98
>         ffffffff820244c4 ffffffff82310000 a8000000048c7418 ffffffff82300000
>         0000000000000000 a8000000048c7470 a80000000495a800 0000000000000028
>         a8000000048c7e20 ffffffff82300000 ffffffff82006fa8 ffffffff822ba890
>         ffffffffffff0e81 ffffffff8226edd8 0000000000000001 0000000000000000
>         ffffffff822c0000 ffffffff822ba890 ffffffffffff0e8b ffffffff82310000
>         0000000000000004 0000000000000000 ffffffff82001820 ffffffff82310000
>         0000000000000000 0000000014001fe0 0000000000000002 0000000014001fe2
>         000000000012166d 0000000000000005 0000000000000001 0000000000000001
>         a8000000048c7e20 ffffffff822ba890 ffffffffffff0a70 ffffffff82310000
>         ...
> Call Trace:
> [<ffffffff8218eebc>] rtc_write+0x1c/0x28
> [<ffffffff8218efe4>] ds1511_wdog_set+0xcc/0x148
> [<ffffffff8212da98>] prom_linux_restart+0x10/0x18
> [<ffffffff820244c4>] panic+0x134/0x1a0
> [<ffffffff82006fa8>] do_mcheck+0xb8/0xd0
> [<ffffffff82001820>] ret_from_exception+0x0/0x20
> [<ffffffff82197c38>] sb1250_pcibios_read+0x70/0x180
> 
> 
> Code: 308400ff  dc437a20  00a3282d <a0a40000> 03e00008  00000000  3c028232  0004203c  0004203e
> Kernel panic - not syncing: Attempted to kill init!
> Rebooting in 5 seconds..<1>CPU 0 Unable to handle kernel paging request at virtual address 000000000000000c, epc == ffffffff8218eebc, ra == ffffffff8218efe4
> Oops[#2]:
> Cpu 0
> $ 0   : 0000000000000000 0000000014001fe0 ffffffff82320000 0000000000000000
> $ 4   : 0000000000000000 000000000000000c 0000000000000000 0000000000000000
> $ 8   : ffffffff822c0000 ffffffff822ba890 ffffffffffff18f8 ffffffff82310000
> $12   : ffffffff82320000 ffffffff82310000 fffffffffffffffd ffffffff8223c568
> $16   : cccccccccccccccd 0000000000000003 000000000000012c ffffffff82300000
> $20   : ffffffff82300000 00000000000003e8 0000000000000060 0000000000030000
> $24   : 0000000000000000 ffffffff8212da88
> $28   : a8000000048c4000 a8000000048c7040 0000000000000000 ffffffff8218efe4
> Hi    : 0000000000000000
> Lo    : 0000000000000000
> epc   : ffffffff8218eebc rtc_write+0x1c/0x28     Not tainted
> ra    : ffffffff8218efe4 ds1511_wdog_set+0xcc/0x148
> Status: 14001fe3    KX SX UX KERNEL EXL IE
> Cause : 1080800c
> BadVA : 000000000000000c
> PrId  : 00040103
> Modules linked in:
> Process swapper (pid: 1, threadinfo=a8000000048c4000, task=a8000000048c1768)
> Stack : 0000000000001388 fffffffffffffd98 ffffffff82300000 ffffffff8212da98
>         ffffffff820244c4 ffffffff82024498 a8000000048c70c8 00000000000018a6
>         ffffffff8226f070 a8000000048c1768 a8000000048c1768 000000000000000b
>         ffffffff822c0000 0000000000000001 ffffffff82028874 ffffffff8223c568
>         ffffffff8226f070 00000000000018a6 ffffffffffffffff 00000000000018a6
>         ffffffff822c0000 ffffffff822ba890 ffffffffffff18a6 ffffffff82310000
>         0000000000000000 0000000000000030 a8000000048c7138 ffffffff822ba890
>         ffffffff8226f070 a8000000048c7260 a8000000048c1768 0000000000000000
>         a8000000048c7260 0000000000000001 0000000000000060 0000000000030000
>         0000000000000000 ffffffff82007130 000000000000000c 0000000000000003
>         ...
> Call Trace:
> [<ffffffff8218eebc>] rtc_write+0x1c/0x28
> [<ffffffff8218efe4>] ds1511_wdog_set+0xcc/0x148
> [<ffffffff8212da98>] prom_linux_restart+0x10/0x18
> [<ffffffff820244c4>] panic+0x134/0x1a0
> [<ffffffff82028874>] do_exit+0x884/0x920
> [<ffffffff82007130>] nmi_exception_handler+0x0/0x38
> 
> 
> Code: 308400ff  dc437a20  00a3282d <a0a40000> 03e00008  00000000  3c028232  0004203c  0004203e
> Kernel panic - not syncing: Attempted to kill init!
> Rebooting in 5 seconds..<1>CPU 0 Unable to handle kernel paging request at virtual address 000000000000000c, epc == ffffffff8218eebc, ra == ffffffff8218efe4
> Oops[#3]:
> Cpu 0
> $ 0   : 0000000000000000 0000000014001fe0 ffffffff82320000 0000000000000000
> $ 4   : 0000000000000000 000000000000000c 0000000000000000 0000000000000000
> $ 8   : ffffffff822c0000 ffffffff822ba890 ffffffffffff22a5 ffffffff82310000
> $12   : ffffffff82320000 ffffffff82310000 fffffffffffffffd ffffffff8223c568
> $16   : cccccccccccccccd 0000000000000003 000000000000012c ffffffff82300000
> $20   : ffffffff82300000 00000000000003e8 0000000000000060 0000000000030000
> $24   : 0000000000000000 ffffffff8212da88
> $28   : a8000000048c4000 a8000000048c6cf0 0000000000000000 ffffffff8218efe4
> Hi    : 0000000000000000
> Lo    : 0000000000000000
> epc   : ffffffff8218eebc rtc_write+0x1c/0x28     Not tainted
> ra    : ffffffff8218efe4 ds1511_wdog_set+0xcc/0x148
> Status: 14001fe3    KX SX UX KERNEL EXL IE
> Cause : 1080800c
> BadVA : 000000000000000c
> PrId  : 00040103
> Modules linked in:
> Process swapper (pid: 1, threadinfo=a8000000048c4000, task=a8000000048c1768)
> Stack : 0000000000001388 fffffffffffffd98 ffffffff82300000 ffffffff8212da98
>         ffffffff820244c4 ffffffff82024498 a8000000048c6d78 0000000000002253
>         ffffffff8226f070 a8000000048c1768 a8000000048c1768 000000000000000b
>         ffffffff822c0000 0000000000000001 ffffffff82028874 ffffffff8223c568
>         ffffffff8226f070 0000000000002253 ffffffffffffffff 0000000000002253
>         ffffffff822c0000 ffffffff822ba890 ffffffffffff2253 ffffffff82310000
>         0000000000000000 0000000000000030 a8000000048c6de8 ffffffff822ba890
>         ffffffff8226f070 a8000000048c6f10 a8000000048c1768 0000000000000000
>         a8000000048c6f10 0000000000000001 0000000000000060 0000000000030000
>         0000000000000000 ffffffff82007130 000000000000000c 0000000000000003
>         ...
> Call Trace:
> [<ffffffff8218eebc>] rtc_write+0x1c/0x28
> [<ffffffff8218efe4>] ds1511_wdog_set+0xcc/0x148
> [<ffffffff8212da98>] prom_linux_restart+0x10/0x18
> [<ffffffff820244c4>] panic+0x134/0x1a0
> [<ffffffff82028874>] do_exit+0x884/0x920
> [<ffffffff82007130>] nmi_exception_handler+0x0/0x38
> 
> 
> Code: 308400ff  dc437a20  00a3282d <a0a40000> 03e00008  00000000  3c028232  0004203c  0004203e
> Kernel panic - not syncing: Attempted to kill init!
> Rebooting in 5 seconds..<1>CPU 0 Unable to handle kernel paging request at virtual address 000000000000000c, epc == ffffffff8218eebc, ra == ffffffff8218efe4
> Oops[#4]:
> Cpu 0
> $ 0   : 0000000000000000 0000000014001fe0 ffffffff82320000 0000000000000000
> $ 4   : 0000000000000000 000000000000000c 0000000000000000 0000000000000000
> $ 8   : ffffffff822c0000 ffffffff822ba890 ffffffffffff2c52 ffffffff82310000
> $12   : ffffffff82320000 ffffffff82310000 fffffffffffffffd ffffffff8223c568
> $16   : cccccccccccccccd 0000000000000003 000000000000012c ffffffff82300000
> $20   : ffffffff82300000 00000000000003e8 0000000000000060 0000000000030000
> $24   : 0000000000000000 ffffffff8212da88
> $28   : a8000000048c4000 a8000000048c69a0 0000000000000000 ffffffff8218efe4
> Hi    : 0000000000000000
> Lo    : 0000000000000000
> epc   : ffffffff8218eebc rtc_write+0x1c/0x28     Not tainted
> ra    : ffffffff8218efe4 ds1511_wdog_set+0xcc/0x148
> Status: 14001fe3    KX SX UX KERNEL EXL I
> Cause : 1080800c
> BadVA : 000000000000000c
> PrId  : 00040103<1>CPU 0 Unable to handle kernel paging request at virtual address 000000000000000c, epc == ffffffff8218eebc, ra == ffffffff8218efe4
> Oops[#5]:
> Cpu 0
> $ 0   : 0000000000000000 0000000014001fe0 ffffffff82320000 0000000000000000
> $ 4   : 0000000000000000 000000000000000c 0000000000000000 0000000000000000
> $ 8   : ffffffff822c0000 ffffffff822ba890 ffffffffffff35ff ffffffff82310000
> $12   : ffffffff82320000 ffffffff82310000 fffffffffffffffd ffffffff8223c568
> $16   : cccccccccccccccd 0000000000000003 000000000000012c ffffffff82300000
> $20   : ffffffff82300000 00000000000003e8 0000000000000060 0000000000030000
> $24   : 0000000000000000 ffffffff8212da88
> $28   : a8000000048c4000 a8000000048c6650 0000000000000000 ffffffff8218efe4
> Hi    : 0000000000000000
> Lo    : 0000000000000000
> epc   : ffffffff8218eebc rtc_write+0x1c/0x28     Not tainted
> ra    : ffffffff8218efe4 ds1511_wdog_set+0xcc/0x148
> Status: 14001fe3    KX SX UX KERNEL EXL IE
> Cause : 1080800c
> BadVA : 000000000000000c
> PrId  : 00040103
> Modules linked in:
> Process swapper (pid: 1, threadinfo=a8000000048c4000, task=a8000000048c1768)
> Stack : 0000000000001388 fffffffffffffd98 ffffffff82300000 ffffffff8212da98
>         ffffffff820244c4 ffffffff82024498 a8000000048c66d8 00000000000035ad
>         ffffffff8226f070 a8000000048c1768 a8000000048c1768 000000000000000b
>         ffffffff822c0000 0000000000000001 ffffffff82028874 ffffffff8223c568
>         ffffffff8226f070 00000000000035ad ffffffffffffffff 00000000000035ad
>         ffffffff822c0000 ffffffff822ba890 ffffffffffff35ad ffffffff82310000
>         0000000000000000 0000000000000030 a8000000048c6748 ffffffff822ba890
>         ffffffff8226f070 a8000000048c6870 a8000000048c1768 0000000000000000
>         a8000000048c6870 0000000000000001 0000000000000060 0000000000030000
>         0000000000000000 ffffffff82007130 000000000000000c 0000000000000003
>         ...
> Call Trace:
> [<ffffffff8218eebc>] rtc_write+0x1c/0x28
> [<ffffffff8218efe4>] ds1511_wdog_set+0xcc/0x148
> [<ffffffff8212da98>] prom_linux_restart+0x10/0x18
> [<ffffffff820244c4>] panic+0x134/0x1a0
> [<ffffffff82028874>] do_exit+0x884/0x920
> [<ffffffff82007130>] nmi_exception_handler+0x0/0x38
> 
> 
> Code: 308400ff  dc437a20  00a3282d <a0a40000> 03e00008  00000000  3c028232  0004203c  0004203e
> Kernel panic - not syncing: Attempted to kill init!
> Rebooting in 5 seconds..<1>CPU 0 Unable to handle kernel paging request at virtual address 000000000000000c, epc == ffffffff8218eebc, ra == ffffffff8218efe4
> Oops[#6]:
> Cpu 0
> $ 0   : 0000000000000000 0000000014001fe0 ffffffff82320000 0000000000000000
> $ 4   : 0000000000000000 000000000000000c 0000000000000000 0000000000000000
> $ 8   : ffffffff822c0000 ffffffff822ba890 ffffffffffff3fac ffffffff82310000
> $12   : ffffffff82320000 ffffffff82310000 fffffffffffffffd ffffffff8223c568
> $16   : cccccccccccccccd 0000000000000003 000000000000012c ffffffff82300000
> $20   : ffffffff82300000 00000000000003e8 0000000000000060 0000000000030000
> $24   : 0000000000000000 ffffffff8212da88
> $28   : a8000000048c4000 a8000000048c6300 0000000000000000 ffffffff8218efe4
> Hi    : 0000000000000000
> Lo    : 0000000000000000
> epc   : ffffffff8218eebc rtc_write+0x1c/0x28     Not tainted
> ra    : ffffffff8218efe4 ds1511_wdog_set+0xcc/0x148
> Status: 14001fe3    KX SX UX KERNEL EXL IE
> Cause : 1080800c
> BadVA : 000000000000000c
> PrId  : 00040103
> Modules linked in:
> Process swapper (pid: 1, threadinfo=a8000000048c4000, task=a8000000048c1768)
> Stack : 0000000000001388 fffffffffffffd98 ffffffff82300000 ffffffff8212da98
>         ffffffff820244c4 ffffffff82024498 a8000000048c6388 0000000000003f5a
>         ffffffff8226f070 a8000000048c1768 a8000000048c1768 000000000000000b
>         ffffffff822c0000 0000000000000001 ffffffff82028874 ffffffff8223c568
>         ffffffff8226f070 0000000000003f5a ffffffffffffffff 0000000000003f5a
>         ffffffff822c0000 ffffffff822ba890 ffffffffffff3f5a ffffffff82310000
>         0000000000000000 0000000000000030 a8000000048c63f8 ffffffff822ba890
>         ffffffff8226f070 a8000000048c6520 a8000000048c1768 0000000000000000
>         a8000000048c6520 0000000000000001 0000000000000060 0000000000030000
>         0000000000000000 ffffffff82007130 000000000000000c 0000000000000003
>         ...
> Call Trace:
> [<ffffffff8218eebc>] rtc_write+0x1c/0x28
> [<ffffffff8218efe4>] ds1511_wdog_set+0xcc/0x148
> [<ffffffff8212da98>] prom_linux_restart+0x10/0x18
> [<ffffffff820244c4>] panic+0x134/0x1a0
> [<ffffffff82028874>] do_exit+0x884/0x920
> [<ffffffff82007130>] nmi_exception_handler+0x0/0x38
> 
> 
> Code: 308400ff  dc437a20  00a3282d <a0a40000> 03e00008  00000000  3c028232  0004203c  0004203e
> Kernel panic - not syncing: Attempted to kill init!
> Rebooting in 5 seconds..
> 
