AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20080605221853.7abc7c9d@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<john.rogers@onstor.com>,<jan.seidel@onstor.com>,<jobi.ariyamannil@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E09C4250A@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 5 Jun 2008 22:19:28 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "John Rogers" <john.rogers@onstor.com>
Cc: "Jan Seidel" <jan.seidel@onstor.com>, "Jobi Ariyamannil"
 <jobi.ariyamannil@onstor.com>
Subject: Re: Linux client 10.2.27.18 crashes
Message-ID: <20080605221928.7337d450@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E09C4250A@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E083217C9@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E09C4250A@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

John,

If we upgrade to that kernel (dude, version 5 v. version 67, does
that tell you anything?) and the problem goes away, afaic, we're all
done.

Jan, that would be the first thing I would do regardless.

Cheers,

a

On Thu, 5 Jun 2008 18:04:29 -0700 "John Rogers"
<john.rogers@onstor.com> wrote:

> There is a more recent kernel, but if this is an interaction with our
> nfs server, we should probably track it down.
> 
> -----Original Message-----
> From: Jan Seidel 
> Sent: Thursday, June 05, 2008 4:41 PM
> To: Andy Sharp; Jobi Ariyamannil
> Cc: John Rogers
> Subject: RE: Linux client 10.2.27.18 crashes
> 
> Hi Andy,
> 
> Thank you for looking at this.
> 
> For the swap device it shows me:
> [root@c18r27-rhel4 test]# swapon -s
> Filename                                Type            Size    Used
> Priority
> /dev/mapper/VolGroup00-LogVol01         partition       2031608 452
> -1
> 
> The client has 2 GB of memory.
> 
> It's a rhel4 client with kernel version 2.6.9-5.ELsmp which is the
> kernel of the first rhel4 release. The update 6 uses kernel 2.6.9-67
> according to http://kbase.redhat.com/faq/FAQ_85_5859.shtm
> 
> Regards,
> Jan
> 
> -----Original Message-----
> From: Andy Sharp 
> Sent: Thursday, June 05, 2008 4:03 PM
> To: Jobi Ariyamannil
> Cc: Jan Seidel; John Rogers
> Subject: Re: Linux client 10.2.27.18 crashes
> 
> Doesn't look good.  I would want to gather the following data:
> 
> What is the swap device and is it configured and working.
> 
> How much memory on the afflicted clients.
> 
> Is this the latest kernel for this release of Redhat, or do they have
> a more recent kernel?
> 
> 
> On Thu, 5 Jun 2008 11:22:47 -0700 "Jobi Ariyamannil"
> <jobi.ariyamannil@onstor.com> wrote:
> 
> > Hi Jan,
> > 
> >  
> > 
> > Please open an ELAB request to address this issue.
> > 
> >  
> > 
> > Andy, do you know why these linux systems are crashing?
> > 
> >  
> > 
> > Regards,
> > 
> > Jobi
> > 
> >  
> > 
> > ________________________________
> > 
> > From: Jan Seidel 
> > Sent: Thursday, June 05, 2008 11:00 AM
> > To: John Rogers
> > Cc: Jobi Ariyamannil
> > Subject: Linux client 10.2.27.18 crashes
> > 
> >  
> > 
> > Hi John, Hi Jobi
> > 
> >  
> > 
> > During the stress test the client I am using (10.2.27.18) sometimes
> > crashes. This is not related to one specific test and happens at
> > different times.
> > 
> >  
> > 
> > Does anyone of you have an idea what can be the reason for that and
> > what I can do about it? Last week Durai told me that he also
> > experienced crashes related to nfs with this and some other clients.
> > Unfortunately I can't test a different client, because all of them
> > are either reserved or not working with the test harness. 
> > 
> >  
> > 
> > Below you can find the part of /var/log/messages that shows the
> > crash. I also included the log for a crash that happened last week.
> > It looks very similar but a different process produced the crash
> > there.
> > 
> >  
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: Unable to handle kernel paging
> > request at virtual address 7c0b1140
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  printing eip:
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: 7c0b1140
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: *pde = 08509001
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: Oops: 0010 [#1]
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: SMP
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> > exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev
> > i2c_core sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state
> > ip_conntrack iptable_filter ip_tables button battery ac uhci_hcd
> > ehci_hcd e1000 floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd
> > dm_mod mptscsih mptbase sd_mod scsi_mod
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: CPU:    1
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: EIP:    0060:[<7c0b1140>]
> > Not tainted VLI
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: EFLAGS: 00010202
> > (2.6.9-5.ELsmp)
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: EIP is at 0x7c0b1140
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: eax: 121b020a   ebx: f7fc420c
> > ecx: 7c0b1140   edx: 00000002
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: esi: c12f1f20   edi: eb2b8c24
> > ebp: f3d5cdb4   esp: f3d5cc64
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: Process fsstress_operat (pid:
> > 5956, threadinfo=f3d5c000 task=ccc401b0)
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: Stack: c01445c3 eb2b8c24
> > c01446ad c12f1d60 c12f1d80 c12f1da0 c12f1dc0 c12f1de0
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:        c12f1e00 c12f1e20
> > c12f1e40 00000246 00000000 f3d5cd28 c12f1f00 eb2b8c24
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:        c12f1f20 eb2b8c24
> > c014492c 00000001 00000000 0000001d 00000000 f3d5cd28
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: Call Trace:
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01445c3>]
> > may_write_to_queue+0x32/0x5b
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01446ad>] pageout+0x55/0xcc
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c014492c>]
> > shrink_list+0x208/0x3ee
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0144cef>]
> > shrink_cache+0x1dd/0x34d
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01453ad>]
> > shrink_zone+0xa7/0xb6
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0145408>]
> > shrink_caches+0x4c/0x57
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01454ce>]
> > try_to_free_pages+0xbb/0x187
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c013f0e9>]
> > __alloc_pages+0x1b5/0x298
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c014159f>]
> > do_page_cache_readahead+0xe7/0x158
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c013c6b5>]
> > filemap_nopage+0x158/0x2f3
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0148d0f>]
> > do_no_page+0xc5/0x2d3
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0149094>]
> > handle_mm_fault+0xbd/0x175
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0119851>]
> > do_page_fault+0x1ae/0x5b6
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c014104f>]
> > vma_prio_tree_insert+0x17/0x2a
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c0149e16>]
> > vma_link+0xb4/0xbc
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c014a998>]
> > do_mmap_pgoff+0x568/0x662
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c010b3fd>]
> > sys_mmap2+0x80/0xb4
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c01196a3>]
> > do_page_fault+0x0/0x5b6
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c02c6d7f>]
> > error_code+0x2f/0x38
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel:  [<c02c007b>]
> > unix_detach_fds+0x2e/0x31
> > 
> > Jun  4 18:08:26 c18r27-rhel4 kernel: Code:  Bad EIP value.
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel:  <1>Unable to handle kernel
> > paging request at virtual address a1000394
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel:  printing eip:
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: c016b9d4
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: *pde = 00000000
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: Oops: 0000 [#2]
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: SMP
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> > exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev
> > i2c_core sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state
> > ip_conntrack iptable_filter ip_tables button battery ac uhci_hcd
> > ehci_hcd e1000 floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd
> > dm_mod mptscsih mptbase sd_mod scsi_mod
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: CPU:    2
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: EIP:    0060:[<c016b9d4>]
> > Not tainted VLI
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: EFLAGS: 00010282
> > (2.6.9-5.ELsmp)
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: EIP is at iput+0x25/0x61
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: eax: a1000380   ebx: e733edfc
> > ecx: 00000016   edx: c0319e00
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: esi: 00000000   edi: e733edfc
> > ebp: fffffff7   esp: f7d89ddc
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: Process kswapd0 (pid: 62,
> > threadinfo=f7d89000 task=f7db38b0)
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: Stack: 01c80971 f8bdede5
> > 00000010 00000001 00001000 00001c80 00000000 f7d89e14
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel:        c1294c80 e733eeac
> > c1294c80 e733eeac f7d89f58 c01446e5 00000000 00000000
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel:        00000000 00000020
> > 00000000 00000000 00000000 00000000 00000000 00000009
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel: Call Trace:
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel:  [<f8bdede5>]
> > nfs_writepage+0x1d4/0x1de [nfs]
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel:  [<c01446e5>] pageout+0x8d/0xcc
> > 
> > Jun  4 18:08:28 c18r27-rhel4 kernel:  [<c014492c>]
> > shrink_list+0x208/0x3ee
> > 
> > Jun  5 10:37:03 c18r27-rhel4 syslogd 1.4.1: restart.
> > 
> > Jun  5 10:37:03 c18r27-rhel4 syslog: syslogd startup succeeded
> > 
> >  
> > 
> >  
> > 
> >  
> > 
> > Crash from last week:
> > 
> >  
> > 
> > May 28 16:46:40 c18r27-rhel4 kernel: VFS: Busy inodes after unmount.
> > Self-destruct in 5 seconds.  Have a nice day...
> > 
> > May 28 16:56:56 c18r27-rhel4 kernel: VFS: Busy inodes after unmount.
> > Self-destruct in 5 seconds.  Have a nice day...
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Unable to handle kernel paging
> > request at virtual address 3538314
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  printing eip:
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: c016b9d4
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: *pde = 26071001
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Oops: 0000 [#1]
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: SMP
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> > exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev
> > i2c_core sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state
> > ip_conntrack iptable_filter ip_tables button battery ac uhci_hcd
> > ehci_hcd e1000 floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd
> > dm_mod mptscsih mptbase sd_mod scsi_mod
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: CPU:    
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: EIP:    0060:[<c016b9d4>]
> > Not tainted VLI
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: EFLAGS: 00010202
> > (2.6.9-5.ELsmp)
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: EIP is at iput+0x25/0x61
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: eax: 35383134   ebx: e4861154
> > ecx: 00000016   edx: c0319e00
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: esi: 00000000   edi: e4861154
> > ebp: fffffff7   esp: f7d89ddc
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Process kswapd0 (pid: 62,
> > threadinfo=f7d89000 task=f7db38b0)
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Stack: 046a9d1a f8bdede5
> > 00000010 00000001 00001000 000046a9 00000000 f7d89e14
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:        c1175a00 e4861204
> > c1175a00 e4861204 f7d89f58 c01446e5 00000000 00000000
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:        00000000 00000020
> > 00000000 00000000 00000000 00000000 00000000 00000009
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Call Trace:
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<f8bdede5>]
> > nfs_writepage+0x1d4/0x1de [nfs]
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c01446e5>] pageout+0x8d/0xcc
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c014492c>]
> > shrink_list+0x208/0x3ee
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c0144cef>]
> > shrink_cache+0x1dd/0x34d
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c01453ad>]
> > shrink_zone+0xa7/0xb6
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c0145750>]
> > balance_pgdat+0x1b6/0x2f8
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c014595c>] kswapd+0xca/0xcc
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c011e912>]
> > autoremove_wake_function+0x0/0x2d
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c02c61c2>]
> > ret_from_fork+0x6/0x14
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c011e912>]
> > autoremove_wake_function+0x0/0x2d
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c0145892>] kswapd+0x0/0xcc
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  [<c01041f1>]
> > kernel_thread_helper+0x5/0xb
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Code: ff e9 fa fe ff ff 53 85
> > c0 89 c3 74 58 83 bb 3c 01 00 00 20 8b 80 a4 00 00 00 8b 40 24 75
> > 08 0f 0b 4c 04 79 d4 2d c0 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8
> > ff d2 8d 43 1c ba f0 d8 31 c0 e8 98
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  <1>Unable to handle kernel
> > paging request at virtual address 6e616834
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel:  printing eip:
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: c016b9d4
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: *pde = 00000000
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Oops: 0000 [#2]
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: SMP
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> > exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev
> > i2c_core sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state
> > ip_conntrack iptable_filter ip_tables button battery ac uhci_hcd
> > ehci_hcd e1000 floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd
> > dm_mod mptscsih mptbase sd_mod scsi_mod
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: CPU:    0
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: EIP:    0060:[<c016b9d4>]
> > Not tainted VLI
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: EFLAGS: 00010202
> > (2.6.9-5.ELsmp)
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: EIP is at iput+0x25/0x61
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: eax: 6e616820   ebx: e4861154
> > ecx: 00000016   edx: c0319e00
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: esi: 00000000   edi: e4861154
> > ebp: fffffff7   esp: ee8fabd0
> > 
> > May 28 17:03:01 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Process bonnie++ (pid: 8383,
> > threadinfo=ee8fa000 task=f5ae19b0)
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Stack: 046a9d1a f8bdede5
> > 00000010 00000001 00001000 000046a9 00000000 ee8fac08
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:        c1175d60 e4861204
> > c1175d60 e4861204 ee8fad4c c01446e5 00000000 00000000
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:        00000000 00000020
> > 00000000 00000000 00000000 00000000 00000000 00000009
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Call Trace:
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<f8bdede5>]
> > nfs_writepage+0x1d4/0x1de [nfs]
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c01446e5>] pageout+0x8d/0xcc
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c014492c>]
> > shrink_list+0x208/0x3ee
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c0144cef>]
> > shrink_cache+0x1dd/0x34d
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c013fcc8>]
> > get_writeback_state+0x30/0x35
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c013fcea>]
> > get_dirty_limits+0x1d/0xc1
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c013ff1e>]
> > throttle_vm_writeout+0x18/0x53
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c01453ad>]
> > shrink_zone+0xa7/0xb6
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c0145408>]
> > shrink_caches+0x4c/0x57
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c01454ce>]
> > try_to_free_pages+0xbb/0x187
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c013f0e9>]
> > __alloc_pages+0x1b5/0x298
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c013d181>]
> > generic_file_buffered_write+0x10d/0x47c
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c011ce7c>]
> > scheduler_tick+0x3ce/0x3e5
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c013d872>]
> > generic_file_aio_write_nolock+0x382/0x3b0
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c013d9ab>]
> > generic_file_aio_write+0x72/0xc6
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<f8bd76d4>]
> > nfs_file_write+0xa3/0xbe [nfs]
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c0155604>]
> > do_sync_write+0x97/0xc9
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c010b0f3>]
> > timer_interrupt+0xd6/0xde
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c011e912>]
> > autoremove_wake_function+0x0/0x2d
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c02c454d>]
> > schedule+0x83d/0x869
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c01556ec>]
> > vfs_write+0xb6/0xe2
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c01557b6>]
> > sys_write+0x3c/0x62
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c02c62a3>]
> > syscall_call+0x7/0xb
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c02c007b>]
> > unix_detach_fds+0x2e/0x31
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Code: ff e9 fa fe ff ff 53 85
> > c0 89 c3 74 58 83 bb 3c 01 00 00 20 8b 80 a4 00 00 00 8b 40 24 75
> > 08 0f 0b 4c 04 79 d4 2d c0 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8
> > ff d2 8d 43 1c ba f0 d8 31 c0 e8 98
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  <1>Unable to handle kernel
> > paging request at virtual address 6e616834
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  printing eip:
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: c016b9d4
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: *pde = 00000000
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Oops: 0000 [#3]
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: SMP
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Modules linked in: nfs nfsd
> > exportfs lockd md5 ipv6 parport_pc lp parport autofs4 i2c_dev
> > i2c_core sunrpc ipt_MASQUERADE iptable_nat ipt_REJECT ipt_state
> > ip_conntrack iptable_filter ip_tables button battery ac uhci_hcd
> > ehci_hcd e1000 floppy sg dm_snapshot dm_zero dm_mirror ext3 jbd
> > dm_mod mptscsih mptbase sd_mod scsi_mod
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: CPU:    1
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: EIP:    0060:[<c016b9d4>]
> > Not tainted VLI
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: EFLAGS: 00010202
> > (2.6.9-5.ELsmp)
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: EIP is at iput+0x25/0x61
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: eax: 6e616820   ebx: e4861154
> > ecx: 00000016   edx: c0319e00
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: esi: 00000000   edi: e4861154
> > ebp: fffffff7   esp: f5e07afc
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: ds: 007b   es: 007b   ss: 0068
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Process rpciod (pid: 3058,
> > threadinfo=f5e07000 task=f6789130)
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Stack: 046a9d1a f8bdede5
> > 00000010 00000001 00001000 000046a9 00000000 f5e07b34
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:        c10da8a0 e4861204
> > c10da8a0 e4861204 f5e07c78 c01446e5 00000000 00000000
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:        00000000 00000020
> > 00000000 00000000 00000000 00000000 00000000 00000009
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel: Call Trace:
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<f8bdede5>]
> > nfs_writepage+0x1d4/0x1de [nfs]
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c01446e5>] pageout+0x8d/0xcc
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c014492c>]
> > shrink_list+0x208/0x3ee
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c0144cef>]
> > shrink_cache+0x1dd/0x34d
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c011cf19>]
> > __wake_up+0x29/0x3c
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c01453ad>]
> > shrink_zone+0xa7/0xb6
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c0145408>]
> > shrink_caches+0x4c/0x57
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c01454ce>]
> > try_to_free_pages+0xbb/0x187
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c013f0e9>]
> > __alloc_pages+0x1b5/0x298
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c0290e2a>]
> > tcp_sendmsg+0x3ac/0xb5b
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c02acad1>]
> > inet_sendmsg+0x38/0x42
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c026aa05>]
> > sock_sendmsg+0xdb/0xf7
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c02acad1>]
> > inet_sendmsg+0x38/0x42
> > 
> > May 28 17:03:02 c18r27-rhel4 kernel:  [<c026aa05>]
> > sock_sendmsg+0xdb/0xf7
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c010781b>]
> > do_IRQ+0x125/0x130
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c01461f0>]
> > kmap_high+0x18e/0x198
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c011e912>]
> > autoremove_wake_function+0x0/0x2d
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c011b506>]
> > recalc_task_prio+0x128/0x133
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c026aa47>]
> > kernel_sendmsg+0x26/0x2c
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c026d937>]
> > sock_no_sendpage+0x4d/0x60
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c026d8ea>]
> > sock_no_sendpage+0x0/0x60
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b392c5>]
> > xdr_sendpages+0x171/0x200 [sunrpc]
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<f8be289c>]
> > nfs3_xdr_writeargs+0x104/0x10e [nfs]
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b318a0>]
> > xprt_transmit+0x12f/0x419 [sunrpc]
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b2fbda>]
> > call_transmit+0x5f/0x7f [sunrpc]
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b32d4f>]
> > __rpc_execute+0xa6/0x327 [sunrpc]
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c02c4566>]
> > schedule+0x856/0x869
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b3306e>]
> > __rpc_schedule+0x3f/0x93 [sunrpc]
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b337b1>] rpciod+0xf7/0x251
> > [sunrpc]
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c011e912>]
> > autoremove_wake_function+0x0/0x2d
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c02c61c2>]
> > ret_from_fork+0x6/0x14
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c011e912>]
> > autoremove_wake_function+0x0/0x2d
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<f8b336ba>] rpciod+0x0/0x251
> > [sunrpc]
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel:  [<c01041f1>]
> > kernel_thread_helper+0x5/0xb
> > 
> > May 28 17:03:03 c18r27-rhel4 kernel: Code: ff e9 fa fe ff ff 53 85
> > c0 89 c3 74 58 83 bb 3c 01 00 00 20 8b 80 a4 00 00 00 8b 40 24 75
> > 08 0f 0b 4c 04 79 d4 2d c0 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8
> > ff d2 8d 43 1c ba f0 d8 31 c0 e8 98
> > 
