X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7CB27.B934F5FD@onstor-exch02.onstor.net>; Fri, 20 Jul 2007 15:43:07 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7CB27.B934F5FD"
Content-class: urn:content-classes:message
Subject: RE: SMP FC locking
Date: Fri, 20 Jul 2007 15:43:07 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E018A69C9@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E049DE7FA@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: SMP FC locking
Thread-Index: AcfLI2WbSODYorRwRPa2C6RVMJ0hMQAA2evwAAAZjjA=
From: "Jeff Miller" <jeff.miller@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>,
	"dl-Cougar" <dl-Cougar@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7CB27.B934F5FD
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Getting the device lock lets you queue an I/O even while another CPU
owns the port.  You don't have to wait for the port lock, which may be
unavailable for a while.  The device lock is only ever kept for a very
short time.

_____________________________________________
From: Maxim Kozlovsky=20
Sent: Friday, July 20, 2007 4:39 PM
To: Jeff Miller; dl-Cougar
Subject: RE: SMP FC locking

What exactly does the device lock buy you? Wouldn't you already have the
port lock anyway by the time you have to get the device lock in most of
the cases?=20

_____________________________________________
From: Jeff Miller=20
Sent: Friday, July 20, 2007 4:12 PM
To: dl-Cougar
Subject: SMP FC locking

This is what I am thinking for a SMP FC.  Any comments?
Thanks,
Jeff

MP locking strategy
Lock port during port control structure or hardware access.  A FC port
is not permanantly tied to a particular CPU as in Cheetah, but is owned
by a CPU only for the period time that it needs attention from a CPU. =20

Lock device only to protect device queues while adding or removing scsi
descriptors.  All other device fields are protected by the port lock.

A CPU may have to wait (in spin lock) for another CPU, but only for a
very short time while a scsi descriptor is being linked or unlinked.
The lock for the port is held for a much longer time, but CPUs do not
wait at all for the port lock. =20


Some assumptions:
A device is associated with only one port.


Changes
Add port spin lock to ispfc_cb_t structure.
do NFX_SPIN_TRYLOCK() at beginning of polling functions and before
issuing command to hardware in receive function, do nothing (don't wait)
if lock not acquired.
Release lock at function returns.

Add device spin lock to scsi_dev_t structure.  Add lock/unlock only
around code that manipulates the device queues.  Do lock and unlock
within loops that handle multiple scsi descriptors.

Polling functions that need to lock port:
ispfc_ct_desc_start_handler()
ispfc_completion_handler()
ispfc_link_management_handler()
ispfc_device_management_handler()
scsi_test_daemon() ?

Port also needs to be locked for calls to scsi_sendNextCmd().

Add device lock only to protect device queues. All other device fields
are protected by the port locks.

These functions manipulate the device queues and need to use the device
lock:
scsi_devO_procWriteGateTimer()
scsi_sendNextCmd()
scsi_flushReqQ()
scsi_addToReqQ()

------_=_NextPart_001_01C7CB27.B934F5FD
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: SMP FC locking</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Getting the device lock lets you queue an I/O even while =
another CPU owns the port.&nbsp; You don</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">t have to wait for the port lock, which may be =
unavailable for a while.&nbsp; The device lock is only ever kept for a =
very short time.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Maxim Kozlovsky<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, July 20, 2007 =
4:39 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jeff Miller; =
dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: SMP FC locking</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">What exactly does the device lock buy you? Wouldn&#8217;t =
you already have the port lock anyway by the time you have to get the =
device lock in most of the cases? </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jeff Miller<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, July 20, 2007 =
4:12 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> SMP FC locking</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">This =
is what I am thinking for a SMP FC.&nbsp; Any =
comments?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Jeff</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">MP =
locking strategy</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Lock =
port during port control structure or hardware access.&nbsp; A FC port =
is not permanantly tied to a particular CPU as in Cheetah, but is owned =
by a CPU only for the period time that it needs attention from a =
CPU.&nbsp; </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Lock =
device only to protect device queues while adding or removing scsi =
descriptors.&nbsp; All other device fields are protected by the port =
lock.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">A CPU =
may have to wait (in spin lock) for another CPU, but only for a very =
short time while a scsi descriptor is being linked or unlinked.&nbsp; =
The lock for the port is held for a much longer time, but CPUs do not =
wait at all for the port lock.&nbsp; </FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Some =
assumptions:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">A =
device is associated with only one port.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Changes</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Add =
port spin lock to ispfc_cb_t structure.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">do =
NFX_SPIN_TRYLOCK() at beginning of polling functions and before issuing =
command to hardware in receive function, do nothing (don't wait) if lock =
not acquired.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Release lock at function returns.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Add =
device spin lock to scsi_dev_t structure.&nbsp; Add lock/unlock only =
around code that manipulates the device queues.&nbsp; Do lock and unlock =
within loops that handle multiple scsi descriptors.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Polling functions that need to lock =
port:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">ispfc_ct_desc_start_handler()</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">ispfc_completion_handler()</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">ispfc_link_management_handler()</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">ispfc_device_management_handler()</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">scsi_test_daemon() ?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Port =
also needs to be locked for calls to =
scsi_sendNextCmd().</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Add =
device lock only to protect device queues. All other device fields are =
protected by the port locks.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">These =
functions manipulate the device queues and need to use the device =
lock:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">scsi_devO_procWriteGateTimer()</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">scsi_sendNextCmd()</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">scsi_flushReqQ()</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">scsi_addToReqQ()</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C7CB27.B934F5FD--
