AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:
CFG:
PT:0
S:andy.sharp@lsi.com
RQ:
SSV:mhbs.lsil.com
NSV:
SSH:
R:<Jobi.Ariyamannil@lsi.com>,<Maxim.Kozlovsky@lsi.com>,<Brian.Stark@lsi.com>,<Bill.Fisher@lsi.com>,<Rendell.Fong@lsi.com>
MAID:2
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/LSI/INBOX	0	4014E6EE2F9ED44299897AD701ED1C5101515BF91E@cosmail03.lsi.com
X-Sylpheed-End-Special-Headers: 1
Date: Wed, 30 Sep 2009 14:20:09 -0700
From: Andrew Sharp <andy.sharp@lsi.com>
To: "Ariyamannil, Jobi" <Jobi.Ariyamannil@lsi.com>
Cc: "Kozlovsky, Maxim" <Maxim.Kozlovsky@lsi.com>, "Stark, Brian"
 <Brian.Stark@lsi.com>, "Fisher, Bill" <Bill.Fisher@lsi.com>, "Fong,
 Rendell" <Rendell.Fong@lsi.com>
Subject: Re: FS threads design overview
Message-ID: <20090930142009.5b66d4ad@ripper.onstor.net>
In-Reply-To: <4014E6EE2F9ED44299897AD701ED1C5101515BF91E@cosmail03.lsi.com>
References: <861DA0537719934884B3D30A2666FECC94373C7A@cosmail02.lsi.com>
	<4014E6EE2F9ED44299897AD701ED1C5101515BF917@cosmail03.lsi.com>
	<861DA0537719934884B3D30A2666FECC94373CA0@cosmail02.lsi.com>
	<4014E6EE2F9ED44299897AD701ED1C5101515BF91E@cosmail03.lsi.com>
Organization: LSI
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

I've put Max's design overview on the Tuxstor wiki where it can be
maintained.  Included at the end is some of the questions and answers.


On Tue, 29 Sep 2009 22:30:31 -0600 "Ariyamannil, Jobi"
<Jobi.Ariyamannil@lsi.com> wrote:

> Thanks.  In typical environments, the kernel can wait for memory to
> become available.  The design notes say that the system will be
> rebooted if memory does not become available in 4 seconds.  So I was
> wondering whether we needed some way to ask the kernel to free up
> memory earlier and faster.
>=20
> When the FS threads are interrupted, will the interrupt processing
> use the same stack?  This can become messy with stack sizes.

I seriously believe you're over-thinking things.  It's not like these
situations don't happen every microsecond on every "generic" Linux
system.  If we are really blowing up a stack, rarely an occurance these
days, we can just increase the stack size.  Also remember that on X86
and even MIPS these days, stacks are protected, so there's no silent
stack corruption and you can't execute stack pages.

> On a side note - typical file systems process the IO done event in
> the interrupt context, but that is not the case with our
> implementation.

It's been a long time since this was done at interrupt time.  These days
that sort of thing is done in a completion thread, sometimes referred
to as a "soft irq" in Linux for reasons that I can't fathom, but more
properly called a "delayed procedure call" in NT.

=46rom "ps ax":

.
.
.
    4 ?        S<     0:01 [ksoftirqd/0]
    7 ?        S<     0:05 [ksoftirqd/1]
   10 ?        S<     0:02 [ksoftirqd/2]
   13 ?        S<     0:00 [ksoftirqd/3]
   15 ?        S<     0:00 [events/0]
   16 ?        S<     0:00 [events/1]
   17 ?        S<     0:00 [events/2]
   18 ?        S<     0:00 [events/3]
.
.
.




> Regards,
> Jobi
>=20
> _____________________________________________
> From: Kozlovsky, Maxim
> Sent: Tuesday, September 29, 2009 8:56 PM
> To: Ariyamannil, Jobi; Stark, Brian; Fisher, Bill; Sharp, Andy; Fong,
> Rendell Subject: RE: FS threads design overview
>=20
>=20
>=20
>=20
> _____________________________________________
> From: Ariyamannil, Jobi
> Sent: Tuesday, September 29, 2009 8:23 PM
> To: Kozlovsky, Maxim; Stark, Brian; Fisher, Bill; Sharp, Andy; Fong,
> Rendell Subject: RE: FS threads design overview
>=20
> Hi Max,
>=20
> Thanks for writing this up.  A few questions:
>=20
> 1.      The current implementation has a limit on the number of file
> system threads.  Are we going to restrict the number of FS threads or
> will it be as many the system could support?  I think some of the
> threshold/tuning conditions in the file system code are derived from
> the maximum number of concurrent FS threads. [MK] Yes we'll have the
> limit on the number of the threads, I don't think it has to be any
> different from the current FS limits. 2.      Are we going to
> pre-allocate the stacks for all the FS threads?  Will there be any
> predefined limit on the stack space for a given thread? [MK]
> Pre-allocating the stacks seems like a good idea. I think we'll raise
> the stack limit a bit just in case. 3.      Can an interrupt pre-empt
> a FS thread? Can a thread be interrupted while holding a spin lock?
> There can be different priorities for the threads, have you thought
> the priority for the FS threads? [MK] We'll be using the version of
> spinlock that does not allow interrupts, other than spin lock
> protected regions the interrupt can preempt the thread but it should
> not do any harm. 4.      When the system runs low on memory, how does
> the FS/ACPU code ask the kernel to free up more memory? [MK] I don't
> know the exact mechanism in the linux kernel, but typically the OS
> will not wait for the memory to completely run out before it starts
> freeing things and will be doing the cleaning process in the
> background in attempt to keep some amount of memory free. We'll need
> to set that threshold high enough to satisfy possible FS/ACPU
> allocations. If this is not how it works in linux, please let me
> know. 5.      Will there be any mechanism for the kernel to ask
> FS/ACPU threads to free up memory from the caches like buffer cache,
> dcache etc?  Currently, we don't do anything to free up a complete
> slab and without doing that, the memory cannot be really freed with
> slab allocators. [MK] Not in the first release, but eventually we
> should do it. Could be first release if somebody ends up with a bunch
> of free time on their hands.
>=20
> Regards,
> Jobi
>=20
> _____________________________________________
> From: Kozlovsky, Maxim
> Sent: Tuesday, September 29, 2009 5:18 PM
> To: Stark, Brian; Fisher, Bill; Sharp, Andy; Ariyamannil, Jobi; Fong,
> Rendell Subject: FS threads design overview
>=20
>=20
>=20
>  << File: FS-threads.txt >>
>=20
