Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTDEV

Before Posting...

Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


NVMeOF TCP Initiator for Windows

sandilyabhagisandilyabhagi Member Posts: 9

Hi,

Is there anything available for nvme like MS-iSCSI initiator ?
I have looked around and couldn't find any.
Intent is to discover/connect nvme target via TCP from Windows machine. Similar to Linux's nvme, nvme-tcp and nvme tool.
I have searched but couldn't get much on nvme.

Hence, looking to implement NVMeOF (over TCP) initiator, similar to MS-iSCSI initiator(May be), unless Microsoft is expected to release for nvme.
I have searched for MS-iSCSI initiator sample code for reference but couldn't get.
Any help in this regard would be very helpful.

Thanks,
Sandilya

Comments

  • MBond2MBond2 Member Posts: 466

    NVMeOF is a protocol that is much different than iSCSI. I do not believe that there is anything from Microsoft, and I would be surprised if they have plans for anything anytime soon. The major storage vendors have too much invested in their proprietary versions

    Usually, NMVEOF is used with Fiber Channel. It can be used with Ethernet too, but it is not TCP

  • sandilyabhagisandilyabhagi Member Posts: 9

    @MBond2 said:
    NVMeOF is a protocol that is much different than iSCSI. I do not believe that there is anything from Microsoft, and I would be surprised if they have plans for anything anytime soon. The major storage vendors have too much invested in their proprietary versions

    Usually, NMVEOF is used with Fiber Channel. It can be used with Ethernet too, but it is not TCP

    Thanks you for your inputs. However in my Google search came across, It says it could work over TCP.
    https://www.starwindsoftware.com/starwind-nvme-of-initiator

    I want to develop a similar one probably tightly coupled with our NIC. Any references would be helpful.

  • MBond2MBond2 Member Posts: 466

    I know nothing about this product, or what you want to do. With respect, if you are asking advice on how to approach this from an Internet forum, you have a long way to go on this project.

  • sandilyabhagisandilyabhagi Member Posts: 9

    @MBond2 said:
    I know nothing about this product, or what you want to do. With respect, if you are asking advice on how to approach this from an Internet forum, you have a long way to go on this project.

    I apologize if I have sounded so, I am not asking on how to approach but, I have 2 questions first, Is MSFT has plans to provide for which I have received information in your first reply. My second questions is to know if there is any MS-iSCSI initiator sample code available only to understand the TCP part. I have learned that WSK is one way to communicate via sockets from kernel. Just want evaluate any other options if available for socket communication in kernel code.
    Thank you again.

  • MBond2MBond2 Member Posts: 466

    well, I guess you are right

    https://nvmexpress.org/wp-content/uploads/NVMe-over-Fabrics-1.1a-2021.07.12-Ratified.pdf

    the specification does include explicit TCP support. Typically this is deployed using lossless Ethernet or Fiber Channel, but I suppose TCP would qualify as a similar lossless transport. The performance would be abysmal, and it would be much more fragile if you intend to rely on the Windows TCP stack. I think that you would need a TCP stack on your hardware if you intend to support bootable volumes, but I have not looked into the details enough to know.

    Anyways, I would expect that you would plan to be enumerated as an additional device on your NIC, and that your upper edge would be storage controller, and your lower edge would be your NIC - the NVME part being implemented entirely between your driver and firmware. But that's just a guess

  • sandilyabhagisandilyabhagi Member Posts: 9

    @MBond2 said:
    well, I guess you are right

    https://nvmexpress.org/wp-content/uploads/NVMe-over-Fabrics-1.1a-2021.07.12-Ratified.pdf

    the specification does include explicit TCP support. Typically this is deployed using lossless Ethernet or Fiber Channel, but I suppose TCP would qualify as a similar lossless transport. The performance would be abysmal, and it would be much more fragile if you intend to rely on the Windows TCP stack. I think that you would need a TCP stack on your hardware if you intend to support bootable volumes, but I have not looked into the details enough to know.

    Anyways, I would expect that you would plan to be enumerated as an additional device on your NIC, and that your upper edge would be storage controller, and your lower edge would be your NIC - the NVME part being implemented entirely between your driver and firmware. But that's just a guess

    Apologies again for delay in response. I am not keeping well recently.
    Yes, that's correct. One alternative is It would be enumerated as additional device on NIC with storage as upper edge and lower edge would be NIC and private implementation in the firmware/driver.

    But, not all vendors/models does have required support to implement things in the driver/firmware and hence this alternative.
    It would be enumerated as root device like MS-iSCSI initiator with storage as upper edge and lower edge would be any NIC (It could be extended to RDMA capable NIC).
    So for this, looking for the right interface to communicate with the NICs in the driver.

  • MBond2MBond2 Member Posts: 466

    if you are not in control of the NIC hardware, then your task is immensely harder and is probably not worth the effort.

    Also, for this project, RDMA is an irrelevant feature. RDMA is a technology to accelerate the transfer of data from the NIC into RAM. But you are primarily concerned with lossless transfer over the fabric - or emulated lossless transfer over TCP over the fabric. And it is likely that whatever mechanism that the NIC driver uses, your throughput will be much lower - and if it relies on TCP, and the hardware does not actually implement lossless ethernet, then it is likely to be much lower

    your question is whether you can or should use the in box TCP stack. If you have specific hardware that can implement the stack on your device, that would be the standard approach. If not, then whatever you implement will suck and there isn't much you can do about it

  • sandilyabhagisandilyabhagi Member Posts: 9

    Completely agree with all your points. But, the problem at hand is having something working (acknowledging lower throughput) vs not having at anything all. Especially when Linux having a standard tool which does seamlessly for any NIC.

    I have understood that it is possible to implement using WSK and I'll summarize my study. I have noted that it is not easy and immensely harder.
    Thank you very much again for your insight.
    My opinion this should ideally come from MSFT if it is worth having it like MS-iSCSI Initiator but I am talking without complete knowledge here.

  • Yatindra_VaishnavYatindra_Vaishnav Member Posts: 11
    edited July 30

    I implemented this NVMe over TCP couple of years back. It is working fantastically similar to Linux and in some workload better then Linux.

  • Yatindra_VaishnavYatindra_Vaishnav Member Posts: 11

    One more point I've written the library for WinSock Kernel to be used in the kernel applications. Which made whole effort much easier.

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,996

    @Yatindra_Vaishnav Awesome... are you able to share any of your work with the community? That would, I'm sure, be most appreciated.

    Peter Viscarola
    OSR
    @OSRDrivers

  • Yatindra_VaishnavYatindra_Vaishnav Member Posts: 11
    edited July 31

    @Peter_Viscarola_(OSR) I would love to and trying to get something for community, which will not conflict with the organization.

  • Yatindra_VaishnavYatindra_Vaishnav Member Posts: 11

    Meanwhile I would help if someone had issue to get this piece of puzzle working.

  • sandilyabhagisandilyabhagi Member Posts: 9

    @Yatindra_Vaishnav Great to hear about your product. However, I am about to start on it before been pulled off to another priority project. However, I would start soon. I was planned to use WSK for the underlying TCP communication and looks like it is the choice. I understand if you cannot share your work but any pointers which helps if can be shareable would be of great help.

  • Yatindra_VaishnavYatindra_Vaishnav Member Posts: 11

    Start looking at https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wsk/ns-wsk-_wsk_provider_connection_dispatch. This is the first thing you will have to have for doing anything to with Windows Kernel Sockets.

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. Sign in or register to get started.

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Writing WDF Drivers 12 September 2022 Live, Online
Internals & Software Drivers 23 October 2022 Live, Online
Kernel Debugging 14 November 2022 Live, Online
Developing Minifilters 5 December 2022 Live, Online