AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20080605132348.30c73525@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<jobi.ariyamannil@onstor.com>,<jan.seidel@onstor.com>,<john.rogers@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E0A501342@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 5 Jun 2008 16:03:14 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>
Cc: "Jan Seidel" <jan.seidel@onstor.com>, "John Rogers"
 <john.rogers@onstor.com>
Subject: Re: Linux client 10.2.27.18 crashes
Message-ID: <20080605160314.3a29238c@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0A501342@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E083217C7@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E0A501342@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Doesn't look good.  I would want to gather the following data:

What is the swap device and is it configured and working.

How much memory on the afflicted clients.

Is this the latest kernel for this release of Redhat, or do they have a
more recent kernel?


On Thu, 5 Jun 2008 11:22:47 -0700 "Jobi Ariyamannil"
<jobi.ariyamannil@onstor.com> wrote:

> Hi Jan,
> 
>  
> 
> Please open an ELAB request to address this issue.
> 
>  
> 
> Andy, do you know why these linux systems are crashing?
> 
>  
> 
> Regards,
> 
> Jobi
> 
>  
> 
> ________________________________
> 
> From: Jan Seidel 
> Sent: Thursday, June 05, 2008 11:00 AM
> To: John Rogers
> Cc: Jobi Ariyamannil
> Subject: Linux client 10.2.27.18 crashes
> 
>  
> 
> Hi John, Hi Jobi
> 
>  
> 
> During the stress test the client I am using (10.2.27.18) sometimes
> crashes. This is not related to one specific test and happens at
> different times.
> 
>  
> 
> Does anyone of you have an idea what can be the reason for that and
> what I can do about it? Last week Durai told me that he also
> experienced crashes related to nfs with this and some other clients.
> Unfortunately I can't test a different client, because all of them
> are either reserved or not working with the test harness. 
> 
>  
> 
> Below you can find the part of /var/log/messages that shows the
> crash. I also included the log for a crash that happened last week.
> It looks very similar but a different process produced the crash
> there.
> 
>  
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: Unable to handle kernel paging
> request at virtual address 7c0b1140
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  printing eip:
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: 7c0b1140
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: *pde = 08509001
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: Oops: 0010 [#1]
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: SMP
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core
> sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state ip_conntrack
> iptable_filter ip_tables button battery ac uhci_hcd ehci_hcd e1000
> floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod mptscsih
> mptbase sd_mod scsi_mod
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: CPU:    1
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: EIP:    0060:[<7c0b1140>]    Not
> tainted VLI
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: EFLAGS: 00010202
> (2.6.9-5.ELsmp)
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: EIP is at 0x7c0b1140
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: eax: 121b020a   ebx: f7fc420c
> ecx: 7c0b1140   edx: 00000002
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: esi: c12f1f20   edi: eb2b8c24
> ebp: f3d5cdb4   esp: f3d5cc64
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: Process fsstress_operat (pid:
> 5956, threadinfo=f3d5c000 task=ccc401b0)
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: Stack: c01445c3 eb2b8c24 c01446ad
> c12f1d60 c12f1d80 c12f1da0 c12f1dc0 c12f1de0
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:        c12f1e00 c12f1e20 c12f1e40
> 00000246 00000000 f3d5cd28 c12f1f00 eb2b8c24
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:        c12f1f20 eb2b8c24 c014492c
> 00000001 00000000 0000001d 00000000 f3d5cd28
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: Call Trace:
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01445c3>]
> may_write_to_queue+0x32/0x5b
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01446ad>] pageout+0x55/0xcc
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c014492c>]
> shrink_list+0x208/0x3ee
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0144cef>]
> shrink_cache+0x1dd/0x34d
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01453ad>]
> shrink_zone+0xa7/0xb6
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0145408>]
> shrink_caches+0x4c/0x57
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01454ce>]
> try_to_free_pages+0xbb/0x187
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c013f0e9>]
> __alloc_pages+0x1b5/0x298
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c014159f>]
> do_page_cache_readahead+0xe7/0x158
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c013c6b5>]
> filemap_nopage+0x158/0x2f3
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0148d0f>]
> do_no_page+0xc5/0x2d3
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0149094>]
> handle_mm_fault+0xbd/0x175
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0119851>]
> do_page_fault+0x1ae/0x5b6
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c014104f>]
> vma_prio_tree_insert+0x17/0x2a
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0149e16>] vma_link+0xb4/0xbc
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c014a998>]
> do_mmap_pgoff+0x568/0x662
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c010b3fd>] sys_mmap2+0x80/0xb4
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01196a3>]
> do_page_fault+0x0/0x5b6
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c02c6d7f>]
> error_code+0x2f/0x38
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c02c007b>]
> unix_detach_fds+0x2e/0x31
> 
> Jun  4 18:08:26 c18r27-rhel4 kernel: Code:  Bad EIP value.
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel:  <1>Unable to handle kernel
> paging request at virtual address a1000394
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel:  printing eip:
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: c016b9d4
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: *pde = 00000000
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: Oops: 0000 [#2]
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: SMP
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core
> sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state ip_conntrack
> iptable_filter ip_tables button battery ac uhci_hcd ehci_hcd e1000
> floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod mptscsih
> mptbase sd_mod scsi_mod
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: CPU:    2
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: EIP:    0060:[<c016b9d4>]    Not
> tainted VLI
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: EFLAGS: 00010282
> (2.6.9-5.ELsmp)
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: EIP is at iput+0x25/0x61
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: eax: a1000380   ebx: e733edfc
> ecx: 00000016   edx: c0319e00
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: esi: 00000000   edi: e733edfc
> ebp: fffffff7   esp: f7d89ddc
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: Process kswapd0 (pid: 62,
> threadinfo=f7d89000 task=f7db38b0)
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: Stack: 01c80971 f8bdede5 00000010
> 00000001 00001000 00001c80 00000000 f7d89e14
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel:        c1294c80 e733eeac c1294c80
> e733eeac f7d89f58 c01446e5 00000000 00000000
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel:        00000000 00000020 00000000
> 00000000 00000000 00000000 00000000 00000009
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel: Call Trace:
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel:  [<f8bdede5>]
> nfs_writepage+0x1d4/0x1de [nfs]
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel:  [<c01446e5>] pageout+0x8d/0xcc
> 
> Jun  4 18:08:28 c18r27-rhel4 kernel:  [<c014492c>]
> shrink_list+0x208/0x3ee
> 
> Jun  5 10:37:03 c18r27-rhel4 syslogd 1.4.1: restart.
> 
> Jun  5 10:37:03 c18r27-rhel4 syslog: syslogd startup succeeded
> 
>  
> 
>  
> 
>  
> 
> Crash from last week:
> 
>  
> 
> May 28 16:46:40 c18r27-rhel4 kernel: VFS: Busy inodes after unmount.
> Self-destruct in 5 seconds.  Have a nice day...
> 
> May 28 16:56:56 c18r27-rhel4 kernel: VFS: Busy inodes after unmount.
> Self-destruct in 5 seconds.  Have a nice day...
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Unable to handle kernel paging
> request at virtual address 3538314
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  printing eip:
> 
> May 28 17:03:01 c18r27-rhel4 kernel: c016b9d4
> 
> May 28 17:03:01 c18r27-rhel4 kernel: *pde = 26071001
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Oops: 0000 [#1]
> 
> May 28 17:03:01 c18r27-rhel4 kernel: SMP
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core
> sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state ip_conntrack
> iptable_filter ip_tables button battery ac uhci_hcd ehci_hcd e1000
> floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod mptscsih
> mptbase sd_mod scsi_mod
> 
> May 28 17:03:01 c18r27-rhel4 kernel: CPU:    
> 
> May 28 17:03:01 c18r27-rhel4 kernel: EIP:    0060:[<c016b9d4>]    Not
> tainted VLI
> 
> May 28 17:03:01 c18r27-rhel4 kernel: EFLAGS: 00010202
> (2.6.9-5.ELsmp)
> 
> May 28 17:03:01 c18r27-rhel4 kernel: EIP is at iput+0x25/0x61
> 
> May 28 17:03:01 c18r27-rhel4 kernel: eax: 35383134   ebx: e4861154
> ecx: 00000016   edx: c0319e00
> 
> May 28 17:03:01 c18r27-rhel4 kernel: esi: 00000000   edi: e4861154
> ebp: fffffff7   esp: f7d89ddc
> 
> May 28 17:03:01 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Process kswapd0 (pid: 62,
> threadinfo=f7d89000 task=f7db38b0)
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Stack: 046a9d1a f8bdede5 00000010
> 00000001 00001000 000046a9 00000000 f7d89e14
> 
> May 28 17:03:01 c18r27-rhel4 kernel:        c1175a00 e4861204 c1175a00
> e4861204 f7d89f58 c01446e5 00000000 00000000
> 
> May 28 17:03:01 c18r27-rhel4 kernel:        00000000 00000020 00000000
> 00000000 00000000 00000000 00000000 00000009
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Call Trace:
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<f8bdede5>]
> nfs_writepage+0x1d4/0x1de [nfs]
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c01446e5>] pageout+0x8d/0xcc
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c014492c>]
> shrink_list+0x208/0x3ee
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c0144cef>]
> shrink_cache+0x1dd/0x34d
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c01453ad>]
> shrink_zone+0xa7/0xb6
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c0145750>]
> balance_pgdat+0x1b6/0x2f8
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c014595c>] kswapd+0xca/0xcc
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c011e912>]
> autoremove_wake_function+0x0/0x2d
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c02c61c2>]
> ret_from_fork+0x6/0x14
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c011e912>]
> autoremove_wake_function+0x0/0x2d
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c0145892>] kswapd+0x0/0xcc
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  [<c01041f1>]
> kernel_thread_helper+0x5/0xb
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Code: ff e9 fa fe ff ff 53 85 c0
> 89 c3 74 58 83 bb 3c 01 00 00 20 8b 80 a4 00 00 00 8b 40 24 75 08 0f
> 0b 4c 04 79 d4 2d c0 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2
> 8d 43 1c ba f0 d8 31 c0 e8 98
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  <1>Unable to handle kernel
> paging request at virtual address 6e616834
> 
> May 28 17:03:01 c18r27-rhel4 kernel:  printing eip:
> 
> May 28 17:03:01 c18r27-rhel4 kernel: c016b9d4
> 
> May 28 17:03:01 c18r27-rhel4 kernel: *pde = 00000000
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Oops: 0000 [#2]
> 
> May 28 17:03:01 c18r27-rhel4 kernel: SMP
> 
> May 28 17:03:01 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core
> sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state ip_conntrack
> iptable_filter ip_tables button battery ac uhci_hcd ehci_hcd e1000
> floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod mptscsih
> mptbase sd_mod scsi_mod
> 
> May 28 17:03:01 c18r27-rhel4 kernel: CPU:    0
> 
> May 28 17:03:01 c18r27-rhel4 kernel: EIP:    0060:[<c016b9d4>]    Not
> tainted VLI
> 
> May 28 17:03:01 c18r27-rhel4 kernel: EFLAGS: 00010202
> (2.6.9-5.ELsmp)
> 
> May 28 17:03:01 c18r27-rhel4 kernel: EIP is at iput+0x25/0x61
> 
> May 28 17:03:01 c18r27-rhel4 kernel: eax: 6e616820   ebx: e4861154
> ecx: 00000016   edx: c0319e00
> 
> May 28 17:03:01 c18r27-rhel4 kernel: esi: 00000000   edi: e4861154
> ebp: fffffff7   esp: ee8fabd0
> 
> May 28 17:03:01 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Process bonnie++ (pid: 8383,
> threadinfo=ee8fa000 task=f5ae19b0)
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Stack: 046a9d1a f8bdede5 00000010
> 00000001 00001000 000046a9 00000000 ee8fac08
> 
> May 28 17:03:02 c18r27-rhel4 kernel:        c1175d60 e4861204 c1175d60
> e4861204 ee8fad4c c01446e5 00000000 00000000
> 
> May 28 17:03:02 c18r27-rhel4 kernel:        00000000 00000020 00000000
> 00000000 00000000 00000000 00000000 00000009
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Call Trace:
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<f8bdede5>]
> nfs_writepage+0x1d4/0x1de [nfs]
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c01446e5>] pageout+0x8d/0xcc
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c014492c>]
> shrink_list+0x208/0x3ee
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c0144cef>]
> shrink_cache+0x1dd/0x34d
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c013fcc8>]
> get_writeback_state+0x30/0x35
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c013fcea>]
> get_dirty_limits+0x1d/0xc1
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c013ff1e>]
> throttle_vm_writeout+0x18/0x53
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c01453ad>]
> shrink_zone+0xa7/0xb6
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c0145408>]
> shrink_caches+0x4c/0x57
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c01454ce>]
> try_to_free_pages+0xbb/0x187
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c013f0e9>]
> __alloc_pages+0x1b5/0x298
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c013d181>]
> generic_file_buffered_write+0x10d/0x47c
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c011ce7c>]
> scheduler_tick+0x3ce/0x3e5
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c013d872>]
> generic_file_aio_write_nolock+0x382/0x3b0
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c013d9ab>]
> generic_file_aio_write+0x72/0xc6
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<f8bd76d4>]
> nfs_file_write+0xa3/0xbe [nfs]
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c0155604>]
> do_sync_write+0x97/0xc9
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c010b0f3>]
> timer_interrupt+0xd6/0xde
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c011e912>]
> autoremove_wake_function+0x0/0x2d
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c02c454d>]
> schedule+0x83d/0x869
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c01556ec>] vfs_write+0xb6/0xe2
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c01557b6>] sys_write+0x3c/0x62
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c02c62a3>]
> syscall_call+0x7/0xb
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c02c007b>]
> unix_detach_fds+0x2e/0x31
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Code: ff e9 fa fe ff ff 53 85 c0
> 89 c3 74 58 83 bb 3c 01 00 00 20 8b 80 a4 00 00 00 8b 40 24 75 08 0f
> 0b 4c 04 79 d4 2d c0 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2
> 8d 43 1c ba f0 d8 31 c0 e8 98
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  <1>Unable to handle kernel
> paging request at virtual address 6e616834
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  printing eip:
> 
> May 28 17:03:02 c18r27-rhel4 kernel: c016b9d4
> 
> May 28 17:03:02 c18r27-rhel4 kernel: *pde = 00000000
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Oops: 0000 [#3]
> 
> May 28 17:03:02 c18r27-rhel4 kernel: SMP
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core
> sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state ip_conntrack
> iptable_filter ip_tables button battery ac uhci_hcd ehci_hcd e1000
> floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod mptscsih
> mptbase sd_mod scsi_mod
> 
> May 28 17:03:02 c18r27-rhel4 kernel: CPU:    1
> 
> May 28 17:03:02 c18r27-rhel4 kernel: EIP:    0060:[<c016b9d4>]    Not
> tainted VLI
> 
> May 28 17:03:02 c18r27-rhel4 kernel: EFLAGS: 00010202
> (2.6.9-5.ELsmp)
> 
> May 28 17:03:02 c18r27-rhel4 kernel: EIP is at iput+0x25/0x61
> 
> May 28 17:03:02 c18r27-rhel4 kernel: eax: 6e616820   ebx: e4861154
> ecx: 00000016   edx: c0319e00
> 
> May 28 17:03:02 c18r27-rhel4 kernel: esi: 00000000   edi: e4861154
> ebp: fffffff7   esp: f5e07afc
> 
> May 28 17:03:02 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Process rpciod (pid: 3058,
> threadinfo=f5e07000 task=f6789130)
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Stack: 046a9d1a f8bdede5 00000010
> 00000001 00001000 000046a9 00000000 f5e07b34
> 
> May 28 17:03:02 c18r27-rhel4 kernel:        c10da8a0 e4861204 c10da8a0
> e4861204 f5e07c78 c01446e5 00000000 00000000
> 
> May 28 17:03:02 c18r27-rhel4 kernel:        00000000 00000020 00000000
> 00000000 00000000 00000000 00000000 00000009
> 
> May 28 17:03:02 c18r27-rhel4 kernel: Call Trace:
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<f8bdede5>]
> nfs_writepage+0x1d4/0x1de [nfs]
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c01446e5>] pageout+0x8d/0xcc
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c014492c>]
> shrink_list+0x208/0x3ee
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c0144cef>]
> shrink_cache+0x1dd/0x34d
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c011cf19>] __wake_up+0x29/0x3c
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c01453ad>]
> shrink_zone+0xa7/0xb6
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c0145408>]
> shrink_caches+0x4c/0x57
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c01454ce>]
> try_to_free_pages+0xbb/0x187
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c013f0e9>]
> __alloc_pages+0x1b5/0x298
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c0290e2a>]
> tcp_sendmsg+0x3ac/0xb5b
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c02acad1>]
> inet_sendmsg+0x38/0x42
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c026aa05>]
> sock_sendmsg+0xdb/0xf7
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c02acad1>]
> inet_sendmsg+0x38/0x42
> 
> May 28 17:03:02 c18r27-rhel4 kernel:  [<c026aa05>]
> sock_sendmsg+0xdb/0xf7
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c010781b>] do_IRQ+0x125/0x130
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c01461f0>]
> kmap_high+0x18e/0x198
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c011e912>]
> autoremove_wake_function+0x0/0x2d
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c011b506>]
> recalc_task_prio+0x128/0x133
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c026aa47>]
> kernel_sendmsg+0x26/0x2c
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c026d937>]
> sock_no_sendpage+0x4d/0x60
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c026d8ea>]
> sock_no_sendpage+0x0/0x60
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b392c5>]
> xdr_sendpages+0x171/0x200 [sunrpc]
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<f8be289c>]
> nfs3_xdr_writeargs+0x104/0x10e [nfs]
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b318a0>]
> xprt_transmit+0x12f/0x419 [sunrpc]
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b2fbda>]
> call_transmit+0x5f/0x7f [sunrpc]
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b32d4f>]
> __rpc_execute+0xa6/0x327 [sunrpc]
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c02c4566>]
> schedule+0x856/0x869
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b3306e>]
> __rpc_schedule+0x3f/0x93 [sunrpc]
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b337b1>] rpciod+0xf7/0x251
> [sunrpc]
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c011e912>]
> autoremove_wake_function+0x0/0x2d
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c02c61c2>]
> ret_from_fork+0x6/0x14
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c011e912>]
> autoremove_wake_function+0x0/0x2d
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b336ba>] rpciod+0x0/0x251
> [sunrpc]
> 
> May 28 17:03:03 c18r27-rhel4 kernel:  [<c01041f1>]
> kernel_thread_helper+0x5/0xb
> 
> May 28 17:03:03 c18r27-rhel4 kernel: Code: ff e9 fa fe ff ff 53 85 c0
> 89 c3 74 58 83 bb 3c 01 00 00 20 8b 80 a4 00 00 00 8b 40 24 75 08 0f
> 0b 4c 04 79 d4 2d c0 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2
> 8d 43 1c ba f0 d8 31 c0 e8 98
> 
