SCSI - Connecting two PCs with FC cards

Hi all,

I’d like to transfer data between two NT workstations using QLogic 2100
Fibre Channel card.

I have read SCSI-2 ANSI specification and especially the part about
‘Communication devices’ and SEND MESSAGE & GET MESSAGE commands.

Is it possible to make this kind of communication using SPTI (SCSI pass
through interface )?

Thanks in advance,
Kristian


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Doubts.
IIRC SCSI miniports do not support incoming messages and target mode.

Max

----- Original Message -----
From:
To: “NT Developers Interest List”
Sent: Sunday, July 22, 2001 12:00 AM
Subject: [ntdev] SCSI - Connecting two PCs with FC cards

> Hi all,
>
> I’d like to transfer data between two NT workstations using QLogic 2100
> Fibre Channel card.
>
> I have read SCSI-2 ANSI specification and especially the part about
> ‘Communication devices’ and SEND MESSAGE & GET MESSAGE commands.
>
> Is it possible to make this kind of communication using SPTI (SCSI pass
> through interface )?
>
> Thanks in advance,
> Kristian
>
> —
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com
>


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Kristian

You can easily do card to card communication with the Qlogic ISP2100. My
company produces a fibre channel adapter that uses the Qlogic ISP-2100. For
this card I provide a peer to peer driver and a SCSI miniport. The peer to
peer driver allows block sizes up to 4meg and can transfer data at about 102
megabytes per second. I do not know if you can achieve that with SPTI. If
you are limited to the SCSI block sizing, your maximum block will be 128K,
and that maxs out at about 80 meg/sec in peer to peer.

Without being to commercial, if you are interested in a functional adapter
and driver, and do not have to abide by SPTI, contact me at the address
below, and I will provide you more information.

Gary
Broadband Storage, Inc
xxxxx@broadstor.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@atia.com
Sent: Sunday, July 22, 2001 12:00 AM
To: NT Developers Interest List
Subject: [ntdev] SCSI - Connecting two PCs with FC cards

Hi all,

I’d like to transfer data between two NT workstations using QLogic 2100
Fibre Channel card.

I have read SCSI-2 ANSI specification and especially the part about
‘Communication devices’ and SEND MESSAGE & GET MESSAGE commands.

Is it possible to make this kind of communication using SPTI (SCSI pass
through interface )?

Thanks in advance,
Kristian


You are currently subscribed to ntdev as: xxxxx@inland.net
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Max,

SCSI-miniports are initiators only. An initiator can post a request to a
target to receive or send, within the SCSI protocol. If SPTI allows that, it
would probably depend on how each end does it. The best route is to go peer
to peer between the 2 cards, and use the communication line like it is a
gigabit serial port.

Gary

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Maxim S. Shatskih
Sent: Sunday, July 22, 2001 2:43 AM
To: NT Developers Interest List
Subject: [ntdev] Re: SCSI - Connecting two PCs with FC cards

Doubts.
IIRC SCSI miniports do not support incoming messages and target mode.

Max

----- Original Message -----
From:
To: “NT Developers Interest List”
Sent: Sunday, July 22, 2001 12:00 AM
Subject: [ntdev] SCSI - Connecting two PCs with FC cards

> Hi all,
>
> I’d like to transfer data between two NT workstations using QLogic 2100
> Fibre Channel card.
>
> I have read SCSI-2 ANSI specification and especially the part about
> ‘Communication devices’ and SEND MESSAGE & GET MESSAGE commands.
>
> Is it possible to make this kind of communication using SPTI (SCSI pass
> through interface )?
>
> Thanks in advance,
> Kristian
>
> —
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com
>


You are currently subscribed to ntdev as: xxxxx@inland.net
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

> Doubts.

IIRC SCSI miniports do not support incoming messages and target mode.

Max

Max,
I am new in SCSI and quite confused, but as I read in QLogic’s
specification about QLA2100 Fibre Channel Adapter, there is a key feature
“SCSI Initiator & Target Operation”. What this mean?
Is Fibre Channel a “communication device” part for these kind of SCSI cards
or I am wrong?
When I connected two stations with FC cards, I saw that they didn’t find
each other. May be this is because thay both are initiators, as you
mention.

Max,

SCSI-miniports are initiators only. An initiator can post a request to a
target to receive or send, within the SCSI protocol. If SPTI allows that, it
would probably depend on how each end does it. The best route is to go peer
to peer between the 2 cards, and use the communication line like it is a
gigabit serial port.

Gary

Gary,
what do you mean with “peer to peer”?
Do you send SCSI command to your cards or you make direct communication
with fiber channel hardware, bypassing SCSI?
It sounds great using FC like “gigabit serial port” but I have no idea
how?
I don’t insist using SPTI, just I made some tests with spti sample from DDK

  • I have tried direct read/writes from the connected Storage Array and
    everything was fine.

Thanks,
Kristian


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

> > IIRC SCSI miniports do not support incoming messages and target mode.

>
> Max

Max,
I am new in SCSI and quite confused, but as I read in QLogic’s
specification about QLA2100 Fibre Channel Adapter, there is a key feature
“SCSI Initiator & Target Operation”. What this mean?

I mean that the SCSIPORT’s programming model does not support target
operations.

Max


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Not quite sure what you are trying to do here, but if you are simply trying
to communicate between two nodes over Fibre Channel, there are established
methods.

a) IP over Fibre Channel
b) VI over Fibre Channel
c) Winsock Direct.

I’m not sure of the 2100’s features, but I think it supports IP over
FC. The simplest method is to use Winsock Direct, if supported by your
flavour of Windows and the card vendor.

Caveat: b) and c) are user mode methods. a) can be both, depending upon
the usefulness of the vendor SDK and documentation.

Mark.

At 05:06 AM 7/23/2001 +0000, xxxxx@atia.com wrote:

> Doubts.
> IIRC SCSI miniports do not support incoming messages and target mode.
>
> Max

Max,
I am new in SCSI and quite confused, but as I read in QLogic’s
specification about QLA2100 Fibre Channel Adapter, there is a key feature
“SCSI Initiator & Target Operation”. What this mean?
Is Fibre Channel a “communication device” part for these kind of SCSI cards
or I am wrong?
When I connected two stations with FC cards, I saw that they didn’t find
each other. May be this is because thay both are initiators, as you
mention.

> Max,
>
> SCSI-miniports are initiators only. An initiator can post a request to a
> target to receive or send, within the SCSI protocol. If SPTI allows
that, it
> would probably depend on how each end does it. The best route is to go peer
> to peer between the 2 cards, and use the communication line like it is a
> gigabit serial port.
>
> Gary

Gary,
what do you mean with “peer to peer”?
Do you send SCSI command to your cards or you make direct communication
with fiber channel hardware, bypassing SCSI?
It sounds great using FC like “gigabit serial port” but I have no idea
how?
I don’t insist using SPTI, just I made some tests with spti sample from DDK

  • I have tried direct read/writes from the connected Storage Array and
    everything was fine.

Thanks,
Kristian


You are currently subscribed to ntdev as: xxxxx@muttsnuts.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

The QLogic ISP2x00 family of chips supports the Fibre Channel protocol. This
protocol requires an initiator and a target, with the initiator controlling
the loop. In other words the target never does anything unless the initiator
tells the target to do something. The NT/Windows drivers that are provided
by QLogic provide a SCSI mini–port connected to SCSI targets such as
CD-ROMs, HDDs, or JBODs. QLogic also provides an IP driver, but that is for
the ISP 2200.

Once you start thinking outside the box, the ISP 2x00 chip is nothing more
than a very fast COM port. Peer to peer is simply our designation for using
two ISP adapters in a high-speed communication link. The driver interface I
provide allows an application writer to open a handle to our driver, and
then do data transfer by building a fibre channel request block and passing
it to the driver using standard Read/WriteFile commands.

The driver also provides RDMA as well as buffering incoming requests from
the initiator.

Gary G. Little
Staff Engineer
Broadband Storage, Inc.
xxxxx@broadstor.com

-----Original Message-----
From: xxxxx@atia.com [mailto:xxxxx@atia.com]
Sent: Sunday, July 22, 2001 5:00 PM
To: NT Developers Interest List
Subject: [ntdev] Re: SCSI - Connecting two PCs with FC cards

Doubts.
IIRC SCSI miniports do not support incoming messages and target mode.

Max

Max,
I am new in SCSI and quite confused, but as I read in QLogic’s
specification about QLA2100 Fibre Channel Adapter, there is a key feature
“SCSI Initiator & Target Operation”. What this mean?
Is Fibre Channel a “communication device” part for these kind of SCSI cards
or I am wrong?
When I connected two stations with FC cards, I saw that they didn’t find
each other. May be this is because thay both are initiators, as you
mention.

Max,

SCSI-miniports are initiators only. An initiator can post a request to a
target to receive or send, within the SCSI protocol. If SPTI allows that,
it
would probably depend on how each end does it. The best route is to go
peer
to peer between the 2 cards, and use the communication line like it is a
gigabit serial port.

Gary

Gary,
what do you mean with “peer to peer”?
Do you send SCSI command to your cards or you make direct communication
with fiber channel hardware, bypassing SCSI?
It sounds great using FC like “gigabit serial port” but I have no idea
how?
I don’t insist using SPTI, just I made some tests with spti sample from DDK

  • I have tried direct read/writes from the connected Storage Array and
    everything was fine.

Thanks,
Kristian


You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Not quite so easy without a device driver. The problem is that both
cards will define themselves as initiators. Your software will need to
reprgram the card on the second machine to act as a target rather than
the default initiator. It must be given an ID that is defferent than the
ID of the card in the first machine.

It can be done, but I suspect it will require a special device driver to
drive the card on the second machine as a target. I doubt a miniport
driver will work. You will probably have to write a monolithic SCSI port
driver for your card.

Just my thoughts…

Jamey

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@atia.com
Sent: Sunday, July 22, 2001 12:00 AM
To: NT Developers Interest List
Subject: [ntdev] SCSI - Connecting two PCs with FC cards

Hi all,

I’d like to transfer data between two NT workstations using
QLogic 2100
Fibre Channel card.

I have read SCSI-2 ANSI specification and especially the part about
‘Communication devices’ and SEND MESSAGE & GET MESSAGE commands.

Is it possible to make this kind of communication using SPTI
(SCSI pass
through interface )?

Thanks in advance,
Kristian


You are currently subscribed to ntdev as:
xxxxx@storagecraft.com To unsubscribe send a blank email to
leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

I’ve changed two cards with QLA2200 / with ISP2200 chip /.

I’ve tried QLogic’s IP driver and it seems to work fine, but unfortunately
the performance is to low.
I have not found any usefull information neither for VI over Fibre Channel
or Winsock Direct both mentioned by Mark.

In my case I need simply to transfer big data chunks between two nodes
using high performance fibre channel hardware.
May be Gary’s solution is just mine, but I would like to make my own
driver.

Now, as Gary said “ISP 2x00 chip is nothing more than a very fast COM
port”.

But in this case if I undertood properly there are next important things
/ tell me if I miss something, please /

  1. I have to reprogram QLA2200 card, as Jamey mentioned.
    I believe that this is true, because the QLogic’s QL2x00IP NDIS driver
    needs “Driver Load RISC Code” to be Enabled. They use enhanced version of
    their QL2200 SCSI miniport driver and it is not MS sertified.

  2. Fibre Channel protocol specification

  3. To transfer data chunks I need to build Fibre Channel Request Blocks and
    pass them directly to the card.
    / This sounds great but does someone know how can I obtain some programming
    specification about these cards?
    I didn’t see anything useful info at www.qlogic.com

Thanks,
Kristian


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

>I’ve tried QLogic’s IP driver and it seems to work fine, but unfortunately

the performance is to low.
I have not found any usefull information neither for VI over Fibre Channel
or Winsock Direct both mentioned by Mark.

That might be because it tries to use ethernet size transfers (about 1500
bytes). You need BIG transfers to get FC to go really fast. You also need
the destination buffer ready to pour data into when you send it.

If you sequentially transfer a block at 100 MBytes/sec at the source bus,
and 100 MBytes/sec across the FC, and 100 MBytes/sec at the destination
bus, that’s only 33 MBytes/sec. Everything must run in PARALLEL (i.e.
smaller blocks pipelined) at 100 MBytes/sec to get good performance. The
little latencies turn into big performance hits.

If you have a client/server interaction architecture, think about your
server response time too. Synchronously requesting a 64 kbyte transfer,
with a 1 millisecond response time will give you < 64 MBytes/sec (it would
be something like 1 / (request processing time + data transfer time) /
block size)… Realistically, the transport time for the request will take
time, even if it’s only few bytes. Most hardware does not have a linear
curve of processing time vs. transfer size. Good questions are: what is
your transfer block size (hopefully megabytes), what is the command
processing time of the FC hardware, and can you pipeline requests. I’d
strongly suggest you draw a little timeline for your data flow, and add up
how much time it takes each step. The hardware folks tend to only talk
about raw maximum transfer rates, and tend to ignore how the
hardware/software will interact with other subsystems. Also, heavily
loaded, multiple streams are very different than a single stream/client.
Understand YOUR problem, and ask the vendors for data that fit’s what YOU
need to do.

My understanding was the Winsock Direct stuff was supposed to be a low
latency peer-to-peer channel for clusters and such, and bypassed the TCP/IP
stack.

In a previous message, Gary said his company made cards+drivers that did
100+ MBytes/sec under some unspecified conditions. Is there some reason you
want to do large amounts of development?

Depending on the FC hardware design, you might need to know little to a lot
about FC. The command interface presented by the firmware may have little
to do with FC protocol formats (the Emulex card was like this), or the
hardware may just be a thinner layer between the FC wire and the PCI bus
interface (the HP Tachyon as I remember). Either way, a fast, high quality
driver is a LOT of work.

Perhaps you can explain more details about what you need to do?

  • Jan

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

At 03:52 AM 7/24/2001 -0700, Jan Bottorff wrote:

My understanding was the Winsock Direct stuff was supposed to be a low
latency peer-to-peer channel for clusters and such, and bypassed the
TCP/IP stack.

WSD is implemented as a Winsock2 service provider. Vendors with VI
interfaces to their SAN cards, including FC, TCP, InfiniBand and
proprietary protocols such as CLAN provide the WSD module to map sockets
over the top of VI. Because the WSD provider activates automatically
existing sockets based applications can use the low-latency high-speed path
without realising it or needing to be re-written.

Microsoft’s figures for a null RPC service are quite, there is almost a
100% improvement in throughput from the same binary when WSD is used.

Your comments about properly designing the system to be truly asynchronous,
to have parallel operations in progress and to always have receive buffers
queued should be in 30 point bold and should be branded on the forehead of
any comms programmer who forgets them :slight_smile:

Cheers,

Mark.


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Hi,

That can be done easily. When you need to post a message during data out
phase you program your card as initiator, when the message will be sent you
program the card back to the target mode so it can accept messages from
other initators on the shared bus. If someone will post your card a message
when it’s in initiator mode it will just fail selection so you’ll need to
retry after timeout. That’s all. About port driver. I do not think the
described architecture will fit into Windows storage driver stack model.
Windows is not OpenVMS AFAIR.

Regards,
Anton Kolomyeytsev

CoolDev.Com - Toolkits for Network & Storage Kernel Software Developers
“KoolSockets” & “KoolStorage” - TDI Client, Kernel Sockets, iSCSI port
www.CoolDev.Com xxxxx@CoolDev.Com xxxxx@CoolDev.Com

On 07/23/01, ““Jamey Kirby” ” wrote:
> Not quite so easy without a device driver. The problem is that both
> cards will define themselves as initiators. Your software will need to
> reprgram the card on the second machine to act as a target rather than
> the default initiator. It must be given an ID that is defferent than the
> ID of the card in the first machine.
>
> It can be done, but I suspect it will require a special device driver to
> drive the card on the second machine as a target. I doubt a miniport
> driver will work. You will probably have to write a monolithic SCSI port
> driver for your card.
>
> Just my thoughts…
>
> Jamey
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@atia.com
> > Sent: Sunday, July 22, 2001 12:00 AM
> > To: NT Developers Interest List
> > Subject: [ntdev] SCSI - Connecting two PCs with FC cards
> >
> >
> > Hi all,
> >
> > I’d like to transfer data between two NT workstations using
> > QLogic 2100
> > Fibre Channel card.
> >
> > I have read SCSI-2 ANSI specification and especially the part about
> > ‘Communication devices’ and SEND MESSAGE & GET MESSAGE commands.
> >
> > Is it possible to make this kind of communication using SPTI
> > (SCSI pass
> > through interface )?
> >
> > Thanks in advance,
> > Kristian
> >
> > —
> > You are currently subscribed to ntdev as:
> > xxxxx@storagecraft.com To unsubscribe send a blank email to
> > leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com
> >
>
>
> —
> You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

They do not, indeed, but we do. We also are adding FC SCSI Target support
soon. Sorry for the plug but I had to get the information out somehow :).

Bill Casey

== SCSI Adapters & VirtualSCSI Target Mode Libs ==
Advanced Storage Concepts, Inc. (409) 744-2129
2720 Terminal Drive xxxxx@virtualscsi.com
Galveston, TX 77554 USA www.virtualscsi.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Maxim S. Shatskih
Sent: Sunday, July 22, 2001 4:43 AM
To: NT Developers Interest List
Subject: [ntdev] Re: SCSI - Connecting two PCs with FC cards

Doubts.
IIRC SCSI miniports do not support incoming messages and target mode.

Max

----- Original Message -----
From:
> To: “NT Developers Interest List”
> Sent: Sunday, July 22, 2001 12:00 AM
> Subject: [ntdev] SCSI - Connecting two PCs with FC cards
>
>
> > Hi all,
> >
> > I’d like to transfer data between two NT workstations using QLogic 2100
> > Fibre Channel card.
> >
> > I have read SCSI-2 ANSI specification and especially the part about
> > ‘Communication devices’ and SEND MESSAGE & GET MESSAGE commands.
> >
> > Is it possible to make this kind of communication using SPTI (SCSI pass
> > through interface )?
> >
> > Thanks in advance,
> > Kristian
> >
> > —
> > You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> > To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com
> >
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@advstor.com
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Jan mentions “under unspecified conditions” … :slight_smile:

In both the QLogic and Broadband cards using the ISP family of chips, the
driver is responsible for loading the firmware into the cards NVRAM. To
write a driver for these cards you have to have that firmware, since it
vaporizes with power down. Broadband has an NDA with QLogic and we develop a
lot of our own firmware to do what we do. The NT/2000/XP drivers that I
provide wrap a proprietary fibre channel API that includes that firmware.

As to the pipelining that Jan talked about, the ISP 2x00 family can handle
multiple simultaneous transfer commands. Our proprietary API and firmware
allows the ISP chip to handle up to 64 commands, each with it’s own
scatter/gather list of up, to but no more than, 4096 descriptors… I’m not
sure what QLogic allows, but remember, their solution is directed more at
SCSI and IP where as we target customers that want large blocks of data
transmitted quickly without the overhead of ScsiPort.

Kristian, you will need to provide that firmware, as well as the low level
IO code that controls the board. To do this yourself, you really really need
to answer Jan’s question: “. Is there some reason you want to do large
amounts of development?” You can acquire an API from either QLogic or from
Broadband. Broadband provides either completed drivers for NT/2000/XP, or a
platform independent API. There may be another source for such an API for
the ISP chips, but working for Broadband, I only know of our product.

What Jan called “under unspecified conditions” turns out to be almost 6
man-years of blood sweat and tears. :slight_smile: Which anyone in this industry, doing
this kind of work, is intimately familiar.

Gary G. Little
Staff Engineer
Broadband Storage, Inc.
xxxxx@broadstor.com

-----Original Message-----
From: Jan Bottorff [mailto:xxxxx@pmatrix.com]
Sent: Tuesday, July 24, 2001 3:53 AM
To: NT Developers Interest List
Cc: xxxxx@atia.com
Subject: [ntdev] Re: SCSI - Connecting two PCs with FC cards

I’ve tried QLogic’s IP driver and it seems to work fine, but unfortunately
the performance is to low.
I have not found any usefull information neither for VI over Fibre Channel
or Winsock Direct both mentioned by Mark.

That might be because it tries to use ethernet size transfers (about 1500
bytes). You need BIG transfers to get FC to go really fast. You also need
the destination buffer ready to pour data into when you send it.

If you sequentially transfer a block at 100 MBytes/sec at the source bus,
and 100 MBytes/sec across the FC, and 100 MBytes/sec at the destination
bus, that’s only 33 MBytes/sec. Everything must run in PARALLEL (i.e.
smaller blocks pipelined) at 100 MBytes/sec to get good performance. The
little latencies turn into big performance hits.

If you have a client/server interaction architecture, think about your
server response time too. Synchronously requesting a 64 kbyte transfer,
with a 1 millisecond response time will give you < 64 MBytes/sec (it would
be something like 1 / (request processing time + data transfer time) /
block size)… Realistically, the transport time for the request will take
time, even if it’s only few bytes. Most hardware does not have a linear
curve of processing time vs. transfer size. Good questions are: what is
your transfer block size (hopefully megabytes), what is the command
processing time of the FC hardware, and can you pipeline requests. I’d
strongly suggest you draw a little timeline for your data flow, and add up
how much time it takes each step. The hardware folks tend to only talk
about raw maximum transfer rates, and tend to ignore how the
hardware/software will interact with other subsystems. Also, heavily
loaded, multiple streams are very different than a single stream/client.
Understand YOUR problem, and ask the vendors for data that fit’s what YOU
need to do.

My understanding was the Winsock Direct stuff was supposed to be a low
latency peer-to-peer channel for clusters and such, and bypassed the TCP/IP
stack.

In a previous message, Gary said his company made cards+drivers that did
100+ MBytes/sec under some unspecified conditions. Is there some reason you
want to do large amounts of development?

Depending on the FC hardware design, you might need to know little to a lot
about FC. The command interface presented by the firmware may have little
to do with FC protocol formats (the Emulex card was like this), or the
hardware may just be a thinner layer between the FC wire and the PCI bus
interface (the HP Tachyon as I remember). Either way, a fast, high quality
driver is a LOT of work.

Perhaps you can explain more details about what you need to do?

  • Jan

You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

>What Jan called “under unspecified conditions” turns out to be almost 6

man-years of blood sweat and tears. :slight_smile: Which anyone in this industry, doing
this kind of work, is intimately familiar.

My reference to “under unspecified conditions” was referring to the 100+
MBytes/sec performance number you mentioned. I didn’t mean to discount 6
man/years of work, which certainly classes as a “lot of work”.

I do like to remind people that every performance number is under some
specific environment, and a specific application’s environment may or may
not match. Is there a technical white paper that describes how applications
can be designed to get that 100 MBytes/sec?

  • Jan

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Jan,

To my knowledge we have not published a white paper, at least I was not
privy to it if we did. The numbers were generated using a rather brain-dead
app that does nothing but send or receive data and compute data rates, with
nothing else going on in the system. In order to achieve a 100+ mBps, the
app was doing 8 concurrent IO’s. In other words, the command processor in
the ISP was always working on 8 transfers of 2 megs apiece. The other end
was a mirror of the sender side but was doing target receives of 2 meg, so
there was little “getting ready” delay on the target. Performance drops off
as the block size decreases, as the bus speed changes, and as the number of
IO’s in progress decreases. Using non-concurrent IO, performance drops to
about 80+ mBps.

So you are right … “brag numbers” need to be qualified.

The main point for Kristian, is that what he wants to do has already been
done.

Gary G. Little
Staff Engineer
Broadband Storage, Inc.
xxxxx@broadstor.com

-----Original Message-----
From: Jan Bottorff [mailto:xxxxx@pmatrix.com]
Sent: Tuesday, July 24, 2001 12:13 PM
To: NT Developers Interest List
Subject: [ntdev] Re: SCSI - Connecting two PCs with FC cards

What Jan called “under unspecified conditions” turns out to be almost 6
man-years of blood sweat and tears. :slight_smile: Which anyone in this industry, doing
this kind of work, is intimately familiar.

My reference to “under unspecified conditions” was referring to the 100+
MBytes/sec performance number you mentioned. I didn’t mean to discount 6
man/years of work, which certainly classes as a “lot of work”.

I do like to remind people that every performance number is under some
specific environment, and a specific application’s environment may or may
not match. Is there a technical white paper that describes how applications
can be designed to get that 100 MBytes/sec?

  • Jan

You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

FC is 100MB/sec nominally, but its full duplex. With two nodes,
you can run point-to-point. In a loop configuration you have a
repeater on each node, which degrades your performance.

So you’d like to believe that you can get 200MB/sec across the wire.
But first its a nominal 100MB/sec, the real number I think closer to 90MB
for SCSI. I’d guess a packet driver could get better theoretical performance,
but many of the FC chips are designed for SCSI.

With the right load you *could* get full duplex performance at 2X one
direction, but which SCSI you need buffer credits, and convoys form
pretty easily (you can’t sent your data because your buffer credit is behind
a big transfer in the other direction).

The FC interface chip may not work as much in parallel as you’d like, also
degrading the performance.

So I think there are a large number of driver/chipset issues that limit performance.

At the “application” level, the key is to have a decent number of decent sized blocks
queued for transfer to the FC chip. Clearly the biggest issue is block size. If you
limit the transfer size to an Ethernet MTU, you’ll never get close to the maximum
bandwidth.

-DH

----- Original Message -----
From: “Jan Bottorff”
To: “NT Developers Interest List”
Sent: Tuesday, July 24, 2001 3:12 PM
Subject: [ntdev] Re: SCSI - Connecting two PCs with FC cards

>
> >What Jan called “under unspecified conditions” turns out to be almost 6
> >man-years of blood sweat and tears. :slight_smile: Which anyone in this industry, doing
> >this kind of work, is intimately familiar.
>
> My reference to “under unspecified conditions” was referring to the 100+
> MBytes/sec performance number you mentioned. I didn’t mean to discount 6
> man/years of work, which certainly classes as a “lot of work”.
>
> I do like to remind people that every performance number is under some
> specific environment, and a specific application’s environment may or may
> not match. Is there a technical white paper that describes how applications
> can be designed to get that 100 MBytes/sec?
>
> - Jan
>
>
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@syssoftsol.com
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com