AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:
CFG:
PT:0
S:andy.sharp@lsi.com
RQ:
SSV:mhbs.lsil.com
NSV:
SSH:
R:<Jobi.Ariyamannil@lsi.com>,<Maxim.Kozlovsky@lsi.com>,<Brian.Stark@lsi.com>,<Bill.Fisher@lsi.com>,<Rendell.Fong@lsi.com>
MAID:2
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/LSI/INBOX	0	5D3F3304755C724285AA5789BA15992004158B5E@cosmail03.lsi.com
X-Sylpheed-End-Special-Headers: 1
Date: Mon, 26 Oct 2009 11:31:41 -0700
From: Andrew Sharp <andy.sharp@lsi.com>
To: "Ariyamannil, Jobi" <Jobi.Ariyamannil@lsi.com>
Cc: "Kozlovsky, Maxim" <Maxim.Kozlovsky@lsi.com>, "Stark, Brian"
 <Brian.Stark@lsi.com>, "Fisher, Bill" <Bill.Fisher@lsi.com>, "Fong,
 Rendell" <Rendell.Fong@lsi.com>
Subject: Re: FS threads design overview
Message-ID: <20091026113141.5a2823d2@ripper.onstor.net>
In-Reply-To: <5D3F3304755C724285AA5789BA15992004158B5E@cosmail03.lsi.com>
References: <861DA0537719934884B3D30A2666FECC94373C7A@cosmail02.lsi.com>
	<4014E6EE2F9ED44299897AD701ED1C5101515BF917@cosmail03.lsi.com>
	<861DA0537719934884B3D30A2666FECC94373CA0@cosmail02.lsi.com>
	<4014E6EE2F9ED44299897AD701ED1C5101515BF91E@cosmail03.lsi.com>
	<20090930142009.5b66d4ad@ripper.onstor.net>
	<5D3F3304755C724285AA5789BA15992004158B5E@cosmail03.lsi.com>
Organization: LSI
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Mon, 26 Oct 2009 10:56:32 -0600 "Ariyamannil, Jobi"
<Jobi.Ariyamannil@lsi.com> wrote:

> One thing we have not discussed before.  EEE could terminate threads
> without dropping resources like locks, memory etc during a volume
> exception. This may be tricky with a general purpose OS.  I am not
> sure whether threads can exit while holding spin locks though. We
> have hooks in the yield code path to take care of special cases
> during volume exception processing, which we may have to preserve
> with the linux port as well (so FS threads cannot simply use standard
> kernel APIs to yield CPU).

It sounds to me like the "special case" handling code is the poor man's
equivalent of an OS doing the right thing and dropping a threads
resources when it ceases to exist.  Hence no preservation should be
needed.

> Regards,
> Jobi
> 
> -----Original Message-----
> From: Andrew Sharp [mailto:andy.sharp@lsi.com] 
> Sent: Wednesday, September 30, 2009 2:20 PM
> To: Ariyamannil, Jobi
> Cc: Kozlovsky, Maxim; Stark, Brian; Fisher, Bill; Fong, Rendell
> Subject: Re: FS threads design overview
> 
> I've put Max's design overview on the Tuxstor wiki where it can be
> maintained.  Included at the end is some of the questions and answers.
> 
> 
> On Tue, 29 Sep 2009 22:30:31 -0600 "Ariyamannil, Jobi"
> <Jobi.Ariyamannil@lsi.com> wrote:
> 
> > Thanks.  In typical environments, the kernel can wait for memory to
> > become available.  The design notes say that the system will be
> > rebooted if memory does not become available in 4 seconds.  So I was
> > wondering whether we needed some way to ask the kernel to free up
> > memory earlier and faster.
> > 
> > When the FS threads are interrupted, will the interrupt processing
> > use the same stack?  This can become messy with stack sizes.
> 
> I seriously believe you're over-thinking things.  It's not like these
> situations don't happen every microsecond on every "generic" Linux
> system.  If we are really blowing up a stack, rarely an occurance
> these days, we can just increase the stack size.  Also remember that
> on X86 and even MIPS these days, stacks are protected, so there's no
> silent stack corruption and you can't execute stack pages.
> 
> > On a side note - typical file systems process the IO done event in
> > the interrupt context, but that is not the case with our
> > implementation.
> 
> It's been a long time since this was done at interrupt time.  These
> days that sort of thing is done in a completion thread, sometimes
> referred to as a "soft irq" in Linux for reasons that I can't fathom,
> but more properly called a "delayed procedure call" in NT.
> 
> From "ps ax":
> 
> .
> .
> .
>     4 ?        S<     0:01 [ksoftirqd/0]
>     7 ?        S<     0:05 [ksoftirqd/1]
>    10 ?        S<     0:02 [ksoftirqd/2]
>    13 ?        S<     0:00 [ksoftirqd/3]
>    15 ?        S<     0:00 [events/0]
>    16 ?        S<     0:00 [events/1]
>    17 ?        S<     0:00 [events/2]
>    18 ?        S<     0:00 [events/3]
> .
> .
> .
> 
> 
> 
> 
> > Regards,
> > Jobi
> > 
> > _____________________________________________
> > From: Kozlovsky, Maxim
> > Sent: Tuesday, September 29, 2009 8:56 PM
> > To: Ariyamannil, Jobi; Stark, Brian; Fisher, Bill; Sharp, Andy;
> > Fong, Rendell Subject: RE: FS threads design overview
> > 
> > 
> > 
> > 
> > _____________________________________________
> > From: Ariyamannil, Jobi
> > Sent: Tuesday, September 29, 2009 8:23 PM
> > To: Kozlovsky, Maxim; Stark, Brian; Fisher, Bill; Sharp, Andy; Fong,
> > Rendell Subject: RE: FS threads design overview
> > 
> > Hi Max,
> > 
> > Thanks for writing this up.  A few questions:
> > 
> > 1.      The current implementation has a limit on the number of file
> > system threads.  Are we going to restrict the number of FS threads
> > or will it be as many the system could support?  I think some of the
> > threshold/tuning conditions in the file system code are derived from
> > the maximum number of concurrent FS threads. [MK] Yes we'll have the
> > limit on the number of the threads, I don't think it has to be any
> > different from the current FS limits. 2.      Are we going to
> > pre-allocate the stacks for all the FS threads?  Will there be any
> > predefined limit on the stack space for a given thread? [MK]
> > Pre-allocating the stacks seems like a good idea. I think we'll
> > raise the stack limit a bit just in case. 3.      Can an interrupt
> > pre-empt a FS thread? Can a thread be interrupted while holding a
> > spin lock? There can be different priorities for the threads, have
> > you thought the priority for the FS threads? [MK] We'll be using
> > the version of spinlock that does not allow interrupts, other than
> > spin lock protected regions the interrupt can preempt the thread
> > but it should not do any harm. 4.      When the system runs low on
> > memory, how does the FS/ACPU code ask the kernel to free up more
> > memory? [MK] I don't know the exact mechanism in the linux kernel,
> > but typically the OS will not wait for the memory to completely run
> > out before it starts freeing things and will be doing the cleaning
> > process in the background in attempt to keep some amount of memory
> > free. We'll need to set that threshold high enough to satisfy
> > possible FS/ACPU allocations. If this is not how it works in linux,
> > please let me know. 5.      Will there be any mechanism for the
> > kernel to ask FS/ACPU threads to free up memory from the caches
> > like buffer cache, dcache etc?  Currently, we don't do anything to
> > free up a complete slab and without doing that, the memory cannot
> > be really freed with slab allocators. [MK] Not in the first
> > release, but eventually we should do it. Could be first release if
> > somebody ends up with a bunch of free time on their hands.
> > 
> > Regards,
> > Jobi
> > 
> > _____________________________________________
> > From: Kozlovsky, Maxim
> > Sent: Tuesday, September 29, 2009 5:18 PM
> > To: Stark, Brian; Fisher, Bill; Sharp, Andy; Ariyamannil, Jobi;
> > Fong, Rendell Subject: FS threads design overview
> > 
> > 
> > 
> >  << File: FS-threads.txt >>
> > 
