X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C717C7.213543DF@onstor-exch02.onstor.net>; Mon, 4 Dec 2006 09:10:42 -0800
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Latest Clio status - 12/03/06
Date: Mon, 4 Dec 2006 09:10:40 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E0193E060@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E01335FD0@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Latest Clio status - 12/03/06
thread-index: AccXKE67m7VwrlBDRCqxwKSb3N63GwAGm3OFAAMbXpAABvSaNQAAOOgWABZvAsA=
From: "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>
To: "Ken Renshaw" <ken.renshaw@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>,
	"Vikas Saini" <vikas.saini@onstor.com>,
	"dl-Clio" <dl-Clio@onstor.com>
Cc: "Jerry Lopatin" <jerry.lopatin@onstor.com>

IMO, we have the best stable filesystem product ever in 2.1.  We have
stabilized snapshots and fixed some corruption issues in the allocator.
Also a lot of improvements have been made to EEK.  Any customer using
snapshots should immediately upgrade to 2.1 when that is available.

It does not mean that there are no more corruption issues.  We still
have not looked at the log replay code and mirroring and many more
areas.

Its not possible to fix rest of the corruption issues in the given
timeframe of Clio.  Trying to fix corruption problems in short timeframe
is not advisable as we may be introducing more issues than fixing.  A
complete investigation of log replay related issues are currently
planned for Delorean and any issues found in Clio tests related to that
can be addressed only in Delorean.  I prioritized investigation of
corruption issues in Clio based on the nature of escalations and
provided the best possible in Clio.
A lot of burning issues at customer sites have been addressed in 2.1.

Regards,
Jobi

-----Original Message-----
From: Ken Renshaw=20
Sent: Sunday, December 03, 2006 10:23 PM
To: Paul Hammer; Tim Gardner; Vikas Saini; dl-Clio
Cc: Jerry Lopatin
Subject: RE: Latest Clio status - 12/03/06

I also thought that we were shipping with zero known corruption bugs,
and log replay is a vital underpinning to failover and other workflows
we need to protect. Can't we get a little more focus from Dev this week
on Clio to finish it off properly rather than shift over too soon to
Delorean? You still have to do the same work, and it's only a greater
task now with the added deferred defects. I've noticed that the
percentage of checkins has heavily swayed towards the Delorean branch
the last week or two, and am worried that we might be sacrificing
quality in the current Clio release when we shouldn't be. I think we
should all try and get as much quality in the 2.1 release as we can
before we shift our total attention to the next project.=20

Thanks, just my two cents :)

-Ken


-----Original Message-----
From: Paul Hammer
Sent: Sun 12/3/2006 10:11 PM
To: Tim Gardner; Vikas Saini; dl-Clio
Cc: Jerry Lopatin
Subject: RE: Latest Clio status - 12/03/06
=20
Mostly sounds good, Vikas have we reduced the testing to 8 mirrors (I
did not hear that this was agreed to)?
=20
Are (16545 16550) the fs corruption defects that Raj ran into? If so I
think those have to remain MF for 2.1, can't push our fs corruption bugs
to 2.2?=20
=20
Thoughts?
=20
-Paul
=20

________________________________

From: Tim Gardner
Sent: Sun 12/3/2006 7:00 PM
To: Paul Hammer; Vikas Saini; dl-Clio
Cc: Jerry Lopatin
Subject: RE: Latest Clio status - 12/03/06



The only difficult MF left for clio that we are planning to fix is
16559. This is the ultra buffer leak during remote restore.

No eta for this. Still trying to find the problem. There is also the
txrx small buffer leak but that is currently marked

can't reproduce.

=20

Narayan was ok with shipping clio with a limit of 8 concurrent mirror
sessions. This means we will push the 5

16 concurrent mirror defects to lambo (16538 16539 16595 16622 15896).

=20

We are not planning on addressing log replay corruption defects until
delorean. This pushes 2 defects to delorean.

(16545 16550)

=20

There is one web-ui defect. That will be fixed in a day or so. (16609)

=20

The remaining two are WADs (16421 16630).

=20

We started daily meetings last week. Suggest we continue this week and
use the 10am time slot

that we have been using.

=20

Tim

=20

=20

________________________________

From: Paul Hammer=20
Sent: Sunday, December 03, 2006 5:23 PM
To: Vikas Saini; dl-Clio
Cc: Jerry Lopatin
Subject: RE: Latest Clio status - 12/03/06

=20

Thanks Vikas, what is the current ETA for test complete? Any idea why
the Bobcat ops are so bad? Do we know what happend on the Cheetah soak?
Eta for the defect backlog in QA being complete?

=20

Tim, do we have an eta for having all the MF's fixed?

=20

Assume we return to daily Clio meeting starting tomorrow, anyone
agree/disagree?

=20

Thanks,

=20

-Paul

=20

________________________________

From: Vikas Saini
Sent: Sun 12/3/2006 2:13 PM
To: dl-Clio
Cc: Jerry Lopatin
Subject: Latest Clio status - 12/03/06

On defects side.

MF in Dev for Clio      11

MF in QA for Clio       30

All in QA for Clio      38

All in QA               98

QE Testing Status

Testing completed so far in addition to previous list.

Switch related tests                    DONE

8k tcp connections                      DONE

Domain trust and forest                 DONE

FP/SP crash/recovery                    DONE

Robocopy                                DONE

Pcap                                    DONE

Ping                                    DONE

Local mirror                            DONE

Upgrade Tests                           DONE

Multi protocol                          DONE

File System                             DONE

Audit                                   DONE

=20

Testing in progress.

DMIP                    117 out of 160 executed, 43 left.

Share import export     212 out of 232 executed, 20 left.

Port checker            48 out of 65 executed, 3 left.

Backup/Restore  Veritas: 91.78% 12 to do, 134 done

NTLMv2:                 3 to do, 36 done

OpenLDAP:               11 to do, 77 done

AD LDAP:                19 to do, 85 done=20

Netbios-less:           6 to do, 43 done

NIS Remote:             4 to do, 19 done

NIS Localmap:           12 to do, 2 done=20

Admin Privileges:       41 to do, 46 done=20

Exec Privileges:        4 to do, 26 done=20

NCM                     90% done

NGM                     90% done

NCM tree view           95% done

16 concurrent mirrors   under test

VSVR                    TBD

Rsync                   TBD

Soak Status

=3D=3D=3D=3D=3D=3D=3D=3D=3D

Super Soak is still in setup stages. Elab is working on it.

Bobcat Soak

-------------------

nfxsh# nfxsh -t

Welcome to the ONStor NAS Gateway.

13:34:58 eng51 diag> vsvr stats agg -i 1

------------------------------------------------------------------------
--------------------------

| VS             | Speed (Ops/sec)                   | Thruput
(Bytes/sec)                       |

------------------------------------------------------------------------
--------------------------

|                |    NFS |   CIFS |    NFS |   CIFS |      NFS |
CIFS |      NFS |     CIFS |

|                |   (IN) |   (IN) |  (OUT) |  (OUT) |     (IN) |
(IN) |    (OUT) |    (OUT) |

------------------------------------------------------------------------
--------------------------

| -N/A-          |   3453 |   2213 |   3446 |   2214 |   21.44M |
314.84K |  636.16K |  121.81K |

| -N/A-          |   1124 |   2263 |   1124 |   2261 |   12.60M |
331.62K |  199.87K |  127.36K |

| -N/A-          |   1492 |   1157 |   1494 |   1157 |   22.65M |
168.21K |  270.38K |   62.55K |

| -N/A-          |   1694 |   2120 |   1690 |   2117 |   22.03M |
306.15K |  343.58K |  115.81K |

| -N/A-          |   1495 |   1018 |   1497 |   1022 |   30.84M |
145.64K |  284.64K |   56.11K |

| -N/A-          |    865 |    852 |    867 |    851 |    8.59M |
122.79K |  181.89K |   46.87K |

| -N/A-          |   1348 |    433 |   1347 |    433 |   11.25M |
61.50K |  259.52K |   22.62K |

| -N/A-          |   2194 |    788 |   2193 |    788 |   40.34M |
113.39K |  420.50K |   44.46K |

| -N/A-          |    753 |    298 |    770 |    298 |    5.64M |
43.13K |  165.37K |   16.26K |

| -N/A-          |   2429 |   2609 |   2432 |   2612 |   34.76M |
387.99K |  405.78K |  145.78K |

| -N/A-          |   1815 |   5121 |   1800 |   5121 |   31.26M |
762.50K |  361.22K |  275.62K |

=20

Cheetah Soak

----------------------

Just looked at Cheetah Soak. Cheetah Soak is down. No updates available
about the crash.

AREAS OF CONCERN

16 concurrent mirror support is not there in Clio. We have seen problems
while testing 16 concurrent mirror sessions.=20

There are known File System corruption issues (in the log replay area)
which are not fixed.=20

Below are few charts from Clio.

RAW Find rate for Clio

=20

MF Find rate for Clio

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

MF Close Rate for Clio

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

=20

Cummulative Trends for Clio

=20

Thanks

Vikas

=20

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D





Open LDAP                               DONE

NFS Share and different export options  DONE    (may need to repeat it
again because of latest changes in that area.)

Snapshot                                DONE

File System (log Replay)                DONE

File System (eek)                       DONE

Symlinks                                DONE

DFS                                     DONE

Widelinks                               DONE

ABE                                     DONE

Cifs servers                            DONE

Computer mgmt                           DONE

AD LDAP                         DONE

Full first time install                 DONE

Upgrade from 1.3.x to 2.1               DONE

Lun discovery                           DONE

System compare                  DONE

=20

Features which should have been completed but not completed yet.

Volume Shadow copy                      20%

Lun label,unlabel,writeback label               90%

Port checker                            60%     (waiting on Elab for
firewall)

Virus Scan                              50%     (can't do offshore as
Symantec License is not working over there)

CIFS                                    80%

Quota LDAP                              80%     (can't do offshore as
Sepaton is not working)

Quota NIS                               80%     (can't do offshore as
Sepaton is not working)

Robocopy                                30%

NDMP                           =20

Lports

Lab setup(domain controller)

Jumbo Frames                                    (waiting on Elab )

Direct/Fabric attached storage

NTLMV2

NIS(Remote)

NGM

DMIP                                    10% (Should have been 40% by
now, lost a week because of merge issues)

Super Soak      Elab setup is going on.As of now ETA is 11/22.

_____________________________________________
From: Vikas Saini
Sent: Tuesday, November 21, 2006 2:23 PM
To: Vikas Saini; dl-Clio
Cc: Jerry Lopatin
Subject: Clio Testing Update 11/20/06

Adding to the previous list.

Testing completed so far in addition to previous list.

Open LDAP                               DONE

NFS Share and different export options  DONE    (may need to repeat it
again because of latest changes in that area.)

Snapshot                                DONE

File System (log Replay)                DONE

File System (eek)                       DONE

Symlinks                                DONE

DFS                                     DONE

Widelinks                               DONE

ABE                                     DONE

Cifs servers                            DONE

Computer mgmt                           DONE

AD LDAP                         DONE

Full first time install                 DONE

Upgrade from 1.3.x to 2.1               DONE

Lun discovery                           DONE

System compare                  DONE

=20

Features which should have been completed but not completed yet.

Volume Shadow copy                      20%

Lun label,unlabel,writeback label               90%

Port checker                            60%     (waiting on Elab for
firewall)

Virus Scan                              50%     (can't do offshore as
Symantec License is not working over there)

CIFS                                    80%

Quota LDAP                              80%     (can't do offshore as
Sepaton is not working)

Quota NIS                               80%     (can't do offshore as
Sepaton is not working)

Robocopy                                30%

NDMP                           =20

Lports

Lab setup(domain controller)

Jumbo Frames                                    (waiting on Elab )

Direct/Fabric attached storage

NTLMV2

NIS(Remote)

NGM

DMIP                                    10% (Should have been 40% by
now, lost a week because of merge issues)

Super Soak      Elab setup is going on.As of now ETA is 11/22.

=20

Also any blocker issues please let me know immediately.

=20

Also on defects side.

MF in Dev for Clio      24

MF in QA for Clio       54

All in QA for Clio      70

All in QA               130

So please verify all the defects in your court. First preference should
be given to MI, WAD, can't reproduce types of bugs so that we have a
closure on those. We need to be 0 by month END

Areas of Concern

QE is behind schedule by one week, we need to cover that ASAP.=20

Still a lot of MF in DEV, we need to resolve them ASAP.

Thanks

Vikas

_____________________________________________
From: Vikas Saini
Sent: Tuesday, November 14, 2006 6:37 PM
To: dl-Clio
Cc: Jerry Lopatin
Subject: Clio Testing Update 11/14/06
Importance: High

I thought I should share with everyone where we are wrt to Clio testing.
Here is the testing update so far. We are little behind schedule but
will catch up with schedule.

Testing completed so far.

NFS v2 udp/tcp          DONE            Mary

NFS v3 udp/tcp          DONE            Mary

NLM/NSM         DONE            Mary

Symlinks                DONE            Mary

DFS                     DONE            Mary

CIFS                    80%             Sahoo

Volume          80%             Selva

Volume Import           DONE            Selva

Quotas(NIS)             80%             Sangeetha       (Backup/Restore
related test cases could not be executed because of sepaton issue.)

Quota(LDAP)             80%             Sangeetha       (Backup/Restore
related test cases could not be executed because of sepaton issue.)

Mirror                  5%              Selva

DMIP                    5%              May             (DMIP regression
is having problems. We need to address this ASAP)=20

Port Scanner            60%             May             (Waiting for Lab
setup for firewall to execute the reset of test cases)

Snapshots               Done            Prashant

User accounts           DONE            Sandrine

Vlan tagging            82% core        Sandrine

Ssh keys                DONE            Sandrine

Backup/Restore          5%              Sandrine/Durai

Lun Discovery           DONE            Sangeetha

NGM                     15%             John K

NCM (web-ui)            70% core        John K

NCM(web-ui)             30% Full        John K

NCM Tree view           80%             John K

Netbios-less            Core DONE       Erik P

Privs - Exec            Core DONE       Erik P

Priv - Admin            Core DONE       Erik P

Audit Commands/log      DONE            Erik P

Open LDAP               DONE            Erik P

AD LDAP         50%             Erik P

DNS                     Done            Durai

Elog                    Done            Durai

NFS Share

and diff export options 95%             Durai

Autosupport             DONE            Prashant

EMRS                    DONE            Prashant

File System             80% DONE        Raj

Virus Scan              50%             Sahoo (waiting for Symantec
licenses from Mary)

Arp+Route+port          DONE            Prashant

Interface               DONE            Prashant

First time Install              DONE            Prashant

Mgmt volume/vsvr        DONE            Prashant

=20

Also any blocker issues please let me know immediately.

Also on defects side.

MF in Dev for Clio      27

MF in QA for Clio       34

All in QA for Clio      51     =20

All in QA               87

So we should start working on that too before these numbers increase. So
please verify all the defects in your court. First preference should be
given to MI, WAD, can't reproduce types of bugs so that we have a
closure on those.

Areas of Concern

DMIP is still not working properly. Having problems starting DMIP
regression.

File System code is very fragile. Getting a lot of File System crashes.=20

We need to resolve Clio MF ASAP. There are 27 MF in Clio right now and
if they don't get resolved by Nov 23rd, we will be in bad shape.

Thanks

Vikas







