WinUSB can not read RAW data from Generic USB Composite Cap Card device

Alright, so I have extracted the data and reading it in a hex editor, for a 5 second capture. I see that there is 3,422 0xFFD8 JPEG Start of Image (SOI) and 3,820 0xFFD9 JPEG End of Image (EOI).

There is a difference of 402 0xFFD8 JPEG SOI missing, is this normal for MJPEG streams of 1080P @ 60FPS.

What are some clues I can tell which is the start of new JPEG frame and end of it? Are UVC headers included in the capture that is placed in the buffer which I've extracted?

Anyhow I will leave my code here for others may find it helpful to get their WinUSB working:

/*

1 sec = 1,000 ms
camera is 60 FPS.

1,000ms/60FPS = 16.666 ms / camera frame input stream.

*/

#include <Windows.h>
#include <tchar.h>
#include <strsafe.h>
#include <Winusb.h>
#include <Usb.h>
#include <cfgmgr32.h>
#include <stdio.h>  // For printf
#include <iostream>
#include <string>
#include <initguid.h>

#define ISOCH_DATA_SIZE_MS 1               // How long to capture? (Do not use this as any midflight errors will throw off the entire previous data stream and need to start over streaming again).
#define ISOCH_TRANSFER_COUNT 5000              // How many instances of requested capture time to be repeated.

DEFINE_GUID(GUID_DEVINTERFACE_USBApplication1, 0xD3CACD43, 0xD2AF, 0x4589, 0x8A, 0x20, 0x7B, 0x1B, 0xB2, 0x08, 0xC4, 0x75); // Device Interface GUID.

typedef struct _DEVICE_DATA {
    BOOL                    HandlesOpen;
    WINUSB_INTERFACE_HANDLE WinusbHandle;
    WINUSB_INTERFACE_HANDLE AssociatedInterfaceHandle;
    HANDLE                  DeviceHandle;
    TCHAR                   DevicePath[MAX_PATH];
    UCHAR                   IsochInPipe;
    ULONG                   IsochInTransferSize;
    ULONG                   IsochInPacketCount;
} DEVICE_DATA, * PDEVICE_DATA;

HRESULT GetIsochPipes(_Inout_ PDEVICE_DATA DeviceData)
{
    BOOL result;
    USB_INTERFACE_DESCRIPTOR usbInterface;
    WINUSB_PIPE_INFORMATION_EX pipe;
    HRESULT hr = S_OK;
    UCHAR i;

    printf("Initiating: `WinUsb_GetAssociatedInterface`\n");

    result = WinUsb_GetAssociatedInterface(DeviceData->WinusbHandle, 0, &DeviceData->AssociatedInterfaceHandle);

    if (result == FALSE)
    {
        hr = HRESULT_FROM_WIN32(GetLastError());
        std::cerr << "WinUsb_GetAssociatedInterface failed to get handle for interface 0." << std::endl;
        CloseHandle(DeviceData->WinusbHandle);
        return hr;
    }
    else if (result)
    {
        printf("`WinUsb_GetAssociatedInterface` has been Initiated.\n\n");
    }

    printf("Initiating: `WinUsb_QueryInterfaceSettings`\n");

    result = WinUsb_QueryInterfaceSettings(DeviceData->AssociatedInterfaceHandle, 1, &usbInterface);

    if (result == FALSE)
    {
        hr = HRESULT_FROM_WIN32(GetLastError());
        std::cerr << "WinUsb_QueryInterfaceSettings failed to get USB interface for Alternate Setting 1." << std::endl;
        CloseHandle(DeviceData->AssociatedInterfaceHandle);
        return hr;
    }
    else if (result)
    {
        printf("`WinUsb_QueryInterfaceSettings` has been Initiated.\n\n");
    }


    printf("Initiating: `WinUsb_SetCurrentAlternateSetting`\n");

    // Select the alternate setting for the video stream interface
    result = WinUsb_SetCurrentAlternateSetting(DeviceData->AssociatedInterfaceHandle, 1);

    if (result == FALSE)
    {
        printf("`WinUsb_SetCurrentAlternateSetting`: Failed to set alternate setting.\n");
        //WinUsb_Free(&DeviceData->AssociatedInterfaceHandle);
        CloseHandle(&DeviceData->AssociatedInterfaceHandle);
        return hr;
    }
    else if (result)
    {
        printf("`WinUsb_SetCurrentAlternateSetting` has been Initiated.\n\n");
    }


    for (i = 0; i < usbInterface.bNumEndpoints; i++) {
        result = WinUsb_QueryPipeEx(
            DeviceData->AssociatedInterfaceHandle,
            1,
            (UCHAR)i,
            &pipe);

        if (result == FALSE) {
            hr = HRESULT_FROM_WIN32(GetLastError());
            printf("WinUsb_QueryPipeEx failed to get USB pipe.\n");
            CloseHandle(DeviceData->DeviceHandle);
            return hr;
        }


        if (pipe.PipeType == UsbdPipeTypeIsochronous)
        {
            // Check if this is an IN endpoint (bit 7 set)
            if ((pipe.PipeId & 0x80) != 0)
            {
                DeviceData->IsochInPipe = pipe.PipeId;
                wprintf(L"Isochronous `IN` Pipe ID: [ %d ]\n", DeviceData->IsochInPipe);
            }
            else
            {
                wprintf(L"Isochronous OUT pipe found (ID: %d), skipping.\n", pipe.PipeId);
                return hr;
            }

            if (pipe.MaximumBytesPerInterval == 0 || (pipe.Interval == 0)) {
                hr = E_INVALIDARG;
                wprintf(L"Isoch Out: MaximumBytesPerInterval or Interval value is 0.\n");
                CloseHandle(DeviceData->DeviceHandle);
                return hr;
            }

            else
            {
                std::wcout << "Isochronous `IN` MaximumBytesPerInterval Size: " << pipe.MaximumBytesPerInterval << std::endl;
                
                // 1ms = 1 frame.
                // (ISOCH_DATA_SIZE_MS = per ms, pipe.MaximumBytesPerInterval = 3072 bytes per microframe, x 8 microframes) = total size of data in terms of total size of Frames.
                // Calculating, how large of data in terms of total amount of time (frames) requested:
                DeviceData->IsochInTransferSize = ISOCH_DATA_SIZE_MS * pipe.MaximumBytesPerInterval * (8 / pipe.Interval);
                std::wcout << "Isochronous `IN` Pipe Total Transfer Size (with requested capture time): " << DeviceData->IsochInTransferSize << std::endl;

                // 1 frame = 8 microframes or (8 x 125 microseconds).
                // Calculating how many microframes in total size of data transfer of time (frames) requested:
                DeviceData->IsochInPacketCount = DeviceData->IsochInTransferSize / pipe.MaximumBytesPerInterval;
                std::wcout << "Isochronous `IN` Pipe Total Packet Count (Total Microframes): " << DeviceData->IsochInPacketCount << std::endl;

                std::wcout << "Isochronous Requested Capture Time: " << ISOCH_DATA_SIZE_MS << "ms." << std::endl;
                std::wcout << "Isochronous Requested repeated amount: " << ISOCH_TRANSFER_COUNT << std::endl;

            }
        }
    }
    return hr;
}



VOID SendIsochInTransfer(_Inout_ PDEVICE_DATA DeviceData)
{
    DWORD lastError = 0;
    LPOVERLAPPED overlapped;
    overlapped = NULL;
    PUCHAR readBuffer = NULL;
    BOOL result = FALSE;
    ULONG numBytes;
    ULONG i;
    ULONG j;
    //ULONG length = DeviceData->IsochInTransferSize;
    WINUSB_ISOCH_BUFFER_HANDLE isochReadBufferHandle;
    PUSBD_ISO_PACKET_DESCRIPTOR isochPackets;
    BOOL ContinueStream = FALSE;
    ULONG totalTransferSize;


    printf("\nPerforming to Read transfer:\n");

    // Calculate total transfer size of how many instances should the requested capture time should be repeated:
    totalTransferSize = DeviceData->IsochInTransferSize * ISOCH_TRANSFER_COUNT;
    printf("Total Transfer Size WITH requested repeat amount: %lu\n\n", totalTransferSize);

    // Allocate memory for the read buffer
    readBuffer = new UCHAR[totalTransferSize];
    if (readBuffer == NULL) 
    {
        wprintf(L"Unable to allocate memory.\n");
        goto Error;
    }

    ZeroMemory(readBuffer, totalTransferSize);

    // Allocate memory for the isoch packets
    isochPackets = new USBD_ISO_PACKET_DESCRIPTOR[DeviceData->IsochInPacketCount * ISOCH_TRANSFER_COUNT];
    ZeroMemory(isochPackets, DeviceData->IsochInPacketCount * ISOCH_TRANSFER_COUNT);

    // Allocate memory for overlapped
    overlapped = new OVERLAPPED[ISOCH_TRANSFER_COUNT];
    ZeroMemory(overlapped, (sizeof(OVERLAPPED) * ISOCH_TRANSFER_COUNT));

    for (i = 0; i < ISOCH_TRANSFER_COUNT; i++)
    {
        overlapped[i].hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);

        if (overlapped[i].hEvent == NULL)
        {
            printf("Unable to set event for overlapped operation.\n");
            goto Error;
        }
    }

    printf("Initiating: `WinUsb_RegisterIsochBuffer`\n");

    // Register the isoch buffer
    result = WinUsb_RegisterIsochBuffer(
        DeviceData->AssociatedInterfaceHandle,
        DeviceData->IsochInPipe,
        readBuffer,
        DeviceData->IsochInTransferSize * ISOCH_TRANSFER_COUNT,
        &isochReadBufferHandle);

    if (!result)
    {
        lastError = GetLastError();
        wprintf(L"Isoch buffer registration failed. Error: %x\n", lastError);
        goto Error;
    }

    else if (result)
    {
        printf("`WinUsb_SetCurrentAlternateSetting` has been Initiated.\n\n");
    }


    // Validate parameters before calling WinUsb_ReadIsochPipeAsap:

    // Check if the isochReadBufferHandle is valid
    if (isochReadBufferHandle == INVALID_HANDLE_VALUE) {
        wprintf(L"Invalid isochReadBufferHandle.\n");
        goto Error;
    }

    if (DeviceData->IsochInPacketCount == 0) {
        wprintf(L"Invalid number of packets: %lu\n", DeviceData->IsochInPacketCount);
        goto Error;
    }

    if (isochPackets == NULL) {
        wprintf(L"Invalid isochPackets array.\n");
        goto Error;
    }

    // Start reading from the isoch pipe
	i = 0; // Reset the counter, each count is 1ms of data.

	while (i < ISOCH_TRANSFER_COUNT)   // While loop is to test the 'ContinueStream' is continuous without any midflight errors:
    {
        printf("Value of `ContinueStream` is: %d\n", ContinueStream);
        printf("Initiating: `WinUsb_ReadIsochPipeAsap` and `i` is: %d\n\n", i);

            // Perform the isochronous read, the parameters reads data based on the total capture time requested:
            result = WinUsb_ReadIsochPipeAsap(
                isochReadBufferHandle,
                DeviceData->IsochInTransferSize * i, // offest
                DeviceData->IsochInTransferSize,
                ContinueStream,
                DeviceData->IsochInPacketCount,
                &isochPackets[i * DeviceData->IsochInPacketCount], &overlapped[i]);

            lastError = GetLastError();
            
            if (!result && lastError != ERROR_IO_PENDING)       // If results fail AND there is no pending I/O, then there is an error.
            {
                printf("`WinUsb_ReadIsochPipeAsap` failed. Error: %x\n", lastError);

                // Log detailed information about the parameters
                printf("Parameters:\n");
                printf("IsochReadBufferHandle: %p\n", isochReadBufferHandle);
                printf("Offset: %lu\n", DeviceData->IsochInTransferSize * i);
                printf("ContinueStream: %d\n", ContinueStream);
                printf("IsochInPacketCount: %lu\n", DeviceData->IsochInPacketCount);
                printf("IsochPackets: %p\n", &isochPackets[i * DeviceData->IsochInPacketCount]);
                goto Error;
            }

            if (!result && lastError != ERROR_IO_PENDING && ContinueStream)  // If the stream failed and the `ContinueStream` is true, need to start a new continue stream again since we need a continous stream with no errors.
            {
                ContinueStream = FALSE;
                i = 0;
                DWORD lastError = GetLastError();
                printf("!result && lastError != ERROR_IO_PENDING && `ContinueStream`. Error: %x\n", lastError);
                continue;
            }
            
            printf("`WinUsb_ReadIsochPipeAsap` has been Initiated. `i` is: %d\n", i);
            printf("Value of `ContinueStream` is: %d\n\n", ContinueStream);
            ContinueStream = TRUE;
            i++;

            if (i == ISOCH_TRANSFER_COUNT) // Note: `WinUsb_ReadIsochPipeAsap` uses 0 based-indexing while `ISOCH_TRANSFER_COUNT` is 1 based-indexing.
            {
                break;
            }
    }

    i = 0; // Reset the counter, each count is 1ms of data.

    //    Wait for transfers to complete
    for (i = 0; i < ISOCH_TRANSFER_COUNT; i++)
    {
        result = WinUsb_GetOverlappedResult(
            DeviceData->AssociatedInterfaceHandle,
            &overlapped[i],
            &numBytes,
            TRUE);

        if (!result)
        {
            lastError = GetLastError();
            printf("Failed to read with error %x\n", lastError);
        }
        else
        {
            numBytes = 0;
            for (j = 0; j < DeviceData->IsochInPacketCount; j++)
            {
                numBytes += isochPackets[j].Length;
            }

            printf("Requested %d bytes in %d packets per transfer.\n", DeviceData->IsochInTransferSize, DeviceData->IsochInPacketCount);
        }

        printf("Transfer %d completed. Read %d bytes. \n\n", i + 1, numBytes);
    }



    // Dump the data to a file
    FILE* file = fopen("isoch_data.bin", "wb");
    if (file != NULL)
    {
        fwrite(readBuffer, 1, totalTransferSize, file);
        fclose(file);
        printf("Data successfully written to isoch_data.bin\n");
    }
    else
    {
        printf("Failed to open file for writing.\n");
    }




Error:
    // Cleanup resources
    if (isochReadBufferHandle != INVALID_HANDLE_VALUE && isochReadBufferHandle != NULL) {
        result = WinUsb_UnregisterIsochBuffer(isochReadBufferHandle);
        if (!result) {
            DWORD lastError = GetLastError();
            wprintf(L"Failed to unregister isoch read buffer. Error: %x\n", lastError);
        }
    }

    if (readBuffer != NULL) {
        delete[] readBuffer;
    }

    if (isochPackets != NULL) {
        delete[] isochPackets;
    }

    return;
}







HRESULT OpenDevice(
    _Out_ PDEVICE_DATA DeviceData,
    _Out_opt_ PBOOL FailureDeviceNotFound);

VOID CloseDevice(
    _Inout_ PDEVICE_DATA DeviceData);

HRESULT RetrieveDevicePath(
    _Out_bytecap_(BufLen) LPTSTR DevicePath,
    _In_                  ULONG  BufLen,
    _Out_opt_             PBOOL  FailureDeviceNotFound);

HRESULT OpenDevice(
    _Out_     PDEVICE_DATA DeviceData,
    _Out_opt_ PBOOL        FailureDeviceNotFound)

{
    HRESULT hr = S_OK;
    BOOL    bResult;

    DeviceData->HandlesOpen = FALSE;

    hr = RetrieveDevicePath(DeviceData->DevicePath,
        sizeof(DeviceData->DevicePath),
        FailureDeviceNotFound);

    if (FAILED(hr)) {
        return hr;
    }

    DeviceData->DeviceHandle = CreateFile(DeviceData->DevicePath,
        GENERIC_WRITE | GENERIC_READ,
        FILE_SHARE_WRITE | FILE_SHARE_READ,
        NULL,
        OPEN_EXISTING,
        FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED,
        NULL);

    if (INVALID_HANDLE_VALUE == DeviceData->DeviceHandle) {
        hr = HRESULT_FROM_WIN32(GetLastError());
        return hr;
    }

    bResult = WinUsb_Initialize(DeviceData->DeviceHandle,
        &DeviceData->WinusbHandle);

    if (FALSE == bResult) {
        hr = HRESULT_FROM_WIN32(GetLastError());
        CloseHandle(DeviceData->DeviceHandle);
        return hr;
    }

    DeviceData->HandlesOpen = TRUE;
    return hr;
}

VOID CloseDevice(
    _Inout_ PDEVICE_DATA DeviceData)

{
    if (FALSE == DeviceData->HandlesOpen) {
        return;
    }

    WinUsb_Free(DeviceData->WinusbHandle);
    CloseHandle(DeviceData->DeviceHandle);
    DeviceData->HandlesOpen = FALSE;

    return;
}

HRESULT RetrieveDevicePath(
    _Out_bytecap_(BufLen) LPTSTR DevicePath,
    _In_                  ULONG  BufLen,
    _Out_opt_             PBOOL  FailureDeviceNotFound)

{
    CONFIGRET cr = CR_SUCCESS;
    HRESULT   hr = S_OK;
    PTSTR     DeviceInterfaceList = NULL;
    ULONG     DeviceInterfaceListLength = 0;

    if (NULL != FailureDeviceNotFound) {
        *FailureDeviceNotFound = FALSE;
    }

    do {
        cr = CM_Get_Device_Interface_List_Size(&DeviceInterfaceListLength,
            (LPGUID)&GUID_DEVINTERFACE_USBApplication1,
            NULL,
            CM_GET_DEVICE_INTERFACE_LIST_PRESENT);

        if (cr != CR_SUCCESS) {
            hr = HRESULT_FROM_WIN32(CM_MapCrToWin32Err(cr, ERROR_INVALID_DATA));
            break;
        }

        DeviceInterfaceList = (PTSTR)HeapAlloc(GetProcessHeap(),
            HEAP_ZERO_MEMORY,
            DeviceInterfaceListLength * sizeof(TCHAR));

        if (DeviceInterfaceList == NULL) {
            hr = E_OUTOFMEMORY;
            break;
        }

        cr = CM_Get_Device_Interface_List((LPGUID)&GUID_DEVINTERFACE_USBApplication1,
            NULL,
            DeviceInterfaceList,
            DeviceInterfaceListLength,
            CM_GET_DEVICE_INTERFACE_LIST_PRESENT);

        if (cr != CR_SUCCESS) {
            HeapFree(GetProcessHeap(), 0, DeviceInterfaceList);

            if (cr != CR_BUFFER_SMALL) {
                hr = HRESULT_FROM_WIN32(CM_MapCrToWin32Err(cr, ERROR_INVALID_DATA));
            }
        }
    } while (cr == CR_BUFFER_SMALL);

    if (FAILED(hr)) {
        return hr;
    }

    if (*DeviceInterfaceList == TEXT('\0')) {
        if (NULL != FailureDeviceNotFound) {
            *FailureDeviceNotFound = TRUE;
        }

        hr = HRESULT_FROM_WIN32(ERROR_NOT_FOUND);
        HeapFree(GetProcessHeap(), 0, DeviceInterfaceList);
        return hr;
    }

    hr = StringCbCopy(DevicePath,
        BufLen,
        DeviceInterfaceList);

    HeapFree(GetProcessHeap(), 0, DeviceInterfaceList);

    return hr;
}

int __cdecl wmain(
    int     Argc,
    wchar_t* Argv[]
)
{
    DEVICE_DATA           deviceData;
    HRESULT               hr;
    USB_DEVICE_DESCRIPTOR deviceDesc;
    BOOL                  bResult;
    BOOL                  noDevice;
    ULONG                 lengthReceived;

    UNREFERENCED_PARAMETER(Argc);
    UNREFERENCED_PARAMETER(Argv);

    hr = OpenDevice(&deviceData, &noDevice);

    if (FAILED(hr)) {
        if (noDevice) {
            wprintf(L"Device not connected or driver not installed\n");
        }
        else {
            wprintf(L"Failed looking for device, HRESULT 0x%x\n", hr);
        }

        return 0;
    }

    bResult = WinUsb_GetDescriptor(deviceData.WinusbHandle,
        USB_DEVICE_DESCRIPTOR_TYPE,
        0,
        0,
        (PBYTE)&deviceDesc,
        sizeof(deviceDesc),
        &lengthReceived);

    if (FALSE == bResult || lengthReceived != sizeof(deviceDesc)) {
        wprintf(L"Error among LastError %d or lengthReceived %d\n",
            FALSE == bResult ? GetLastError() : 0,
            lengthReceived);
        CloseDevice(&deviceData);
        return 0;
    }

    printf("\nDevice found: VID_%04X&PID_%04X; bcdUsb %04X\n\n",
        deviceDesc.idVendor,
        deviceDesc.idProduct,
        deviceDesc.bcdUSB);

    hr = GetIsochPipes(&deviceData);

    if (SUCCEEDED(hr)) {
        SendIsochInTransfer(&deviceData);
    }
    else {
        wprintf(L"GetIsochPipes failed with HRESULT: 0x%08X\n", hr);
    }

    CloseDevice(&deviceData);
    return 0;
}

There are two other weirdnesses we haven't talked about that make your goal unattainable.

First, a UVC frame always has a header. The MJPEG header is between 2 and 12 bytes, depending on which timestamps they include. For a bulk pipe, you get one header per frame. For an isochronous pipe, you get one header per PACKET. That header includes bits that tell you whether the timestamps are included, and whether this packet represents the beginning, middle, or end of a frame.

Second, isochronous buffers are not contiguous. Your 24k buffer is divided into 8 3k packets. Each packet corresponds to one microframe. If your device sends nothing or the packet is dropped, there will be a gap in the buffer.

Let's say your images are 100kB. That occupies 4 full transfers and part of a fifth, which will have "3k 1k 0 0 0 0 0 0". Now we have 11ms of idle time until the next frame is available. So, you'll get 11 transfers that are completely empty. If the timing really is 16.6ms, then the next frame starts during the 6th microframe of the next transfer (625us in), so the packet contents will be "0 0 0 0 0 3k 3k 3k"; the image data starts 18k bytes into the buffer.

Because of that, there will always be a need to reconstruct the frame from the isochronous transfer packets. This is why I still strongly believe you should let uvcvideo.sys do that for you. You aren't gaining anything., and there's plenty of idle time in your stream to take care of the overhead.

What are some clues I can tell which is the start of new JPEG frame and end of it?

The frame header in each packet will tell you that. If you haven't downloaded the USB Video Class spec and the UVC Spec for MJPEG Payloads, you should do that. It describes the fields in the header.

1 Like

Hello @Tim_Roberts, thanks for the reply.

Excellent explanation, you should write a book on the subject.

Just curious, can FFMPEG able to resolve the gaps and perform reconstruct? I'm running this command on FFMPEG:
ffmpeg -f mjpeg -i "isoch_data.bin" -c:v copy output.mp4
ffplay output.mp4

I get this, seems like the data extracted is legitimate, but needs reconstruction, this is due to the gaps most likely right? LibUVC doesn't have a library of somesort which has an algorithm which can do the reconstruction?:

For some reason I get better results using WinUsb_ReadIsochPipe.

What would be the best way to directly talk to uvcvideo.sys and make 1 copy from buffer to GPU?

Thanks.

You are still not comprehending the layers of responsibility here, which is a key concept in the Windows driver stack. The isochronous gap nonsense and the UVC header stuff is strictly between uvcvideo.sys and the USB host controller. Once you get above uvcvideo, all of that is gone. The stream consists of whole, reconstructed frames.

What would be the best way to directly talk to uvcvideo.sys and make 1 copy from buffer to GPU?

The best way to talk to uvcvideo.sys is by using DirectShow. DirectShow and Kernel Streaming were designed in parallel. The DirectShow camera interfaces map directly to Kernel Streaming conventions.

You will NEVER be able to implement the one-buffer-no-copy thing. While you are busy handling frame N, where do you think the framework is going to be storing the first pieces of frame N+1? There is a continuous stream of data here. It's not BLINK here is frame 1, then nothing until BLINK here is frame 2. It's like a stream of water filling up a bucket. When you remove that bucket, you had better have another bucket ready to fill.

1 Like

Thanks for the reply.

I believe I understood your great explanation correctly in your previous post. uvcvideo.sys takes care of reconstructing the idle gaps caused by the USB host controller. However, I am using winusb.sys, which I assume does not resolve any of the idle gaps. Thus, I was just curious if FFMPEG is able to detect the gaps and omit them during replay, for situations where the driver stack did not take care of the gaps. The same goes for LibUVC. I was hoping I could quickly grab the algorithm implemented in detecting the gaps.

I will start working and researching on DirectShow and Kernel Streaming. Do you have any recommendations on which API C/C++ calls I should use, any templates or examples I should look for?

Also, there is a special filter diagnostic tool called grapheditplus and graphedit. I tested its rendering for live webcam streams and was surprised that it is practically zero latency. I assume both grapheditplus and graphedit use DirectShow and Kernel Streaming? Maybe I can grab grapheditplus's algorithm on streaming and use that as a template to work with?

You mentioned earlier that a custom allocator could be made. I originally had the same plan to implement: create multiple buffer buckets. When one is filled up, the GPU will quickly grab it and do decoding on it, while the next buffer will be filled up and the cycle repeats.

The question is, how many buffer buckets are ideal? Two at least, or do I need more?

Thanks.

No. How could it possibly know about that? The only way the driver knows about it is because it gets the filled-in USB_ISO_PACKET_DESCRIPTOR array back as part of the URB, which tells it how many good bytes are in each packet.

FFMPEG is generic. It can't assume isochronous, nor indeed can it even assume USB. The data source might be a file, an IP camera, or even punched cards. Many USB 2 cameras did their video endpoints as bulk pipes, because that doubles the practical bandwidth.

1 Like

Hello @Tim_Roberts, thanks for the reply.

In WinUSB.sys, the PUSBD_ISO_PACKET_DESCRIPTOR stores variables for each packet transfer, such as: offset, length, and status.

One transfer will have 8 packets. After the first packet transfer, the offset will increase by 3,072 Bytes consecutively for every following packet.

What would the offset be if it was 2 transfers? Let's say the 1st transfer completed, buffer filled 8 packets, now it's time for the 2nd transfer.
Will the 1st packet in the 2nd transfer offset continue where it left off from the last 8th packet offset from the 1st transfer?

Right now I am seeing all the offsets in the PUSBD_ISO_PACKET_DESCRIPTOR are all duplicates for a two count transfer example:

Transfer 0, Packet 0:
  Offset: 0
  Length: 3072
  Status: 0
Transfer 0, Packet 1:
  Offset: 3072
  Length: 3072
  Status: 0
Transfer 0, Packet 2:
  Offset: 6144
  Length: 120
  Status: 0
Transfer 0, Packet 3:
  Offset: 9216
  Length: 96
  Status: 0
Transfer 0, Packet 4:
  Offset: 12288
  Length: 3072
  Status: 0
Transfer 0, Packet 5:
  Offset: 15360
  Length: 3072
  Status: 0
Transfer 0, Packet 6:
  Offset: 18432
  Length: 3072
  Status: 0
Transfer 0, Packet 7:
  Offset: 21504
  Length: 3072
  Status: 0

Transfer 1, Packet 0:
  Offset: 0
  Length: 0
  Status: 0
Transfer 1, Packet 1:
  Offset: 3072
  Length: 0
  Status: 0
Transfer 1, Packet 2:
  Offset: 6144
  Length: 3072
  Status: 0
Transfer 1, Packet 3:
  Offset: 9216
  Length: 0
  Status: 0
Transfer 1, Packet 4:
  Offset: 12288
  Length: 3072
  Status: 0
Transfer 1, Packet 5:
  Offset: 15360
  Length: 0
  Status: 0
Transfer 1, Packet 6:
  Offset: 18432
  Length: 0
  Status: 0
Transfer 1, Packet 7:
  Offset: 21504
  Length: 3072
  Status: 0

Also, the offsets stored in each packet in the PUSBD_ISO_PACKET_DESCRIPTOR must match how it is stored in the buffer too, right?

I find that on each transfer of the 5th packet's offset, labeled in the PUSBD_ISO_PACKET_DESCRIPTOR does not match precisely how it is stored in the buffer's offset.

The two offsets have a mismatch. I am assuming this is no good.

Thanks.

Each transfer is independent. The offsets do not carry over, because they are separate buffers. It is common, for example, to submit one transfer spanning many frames, so you might have 32 packets of 3072 spanning 4 frames.

In your first buffer, there will be 6,264 bytes of data at the beginning, then a gap of 6,024 bytes containing garbage, then 96 bytes of data, then 9,150 bytes of garbage, then 12,288 bytes of useful data.

In your second buffer, you will have 6,144 bytes of garbage at the beginning, then 3,072 bytes of data, then 3,072 byes of garbage, then 3,072 bytes of data, then 6,144 bytes of garbage, then 3,072 bytes of data to finish things out.

The offsets look fine to me. Where do you see a mismatch? In each case, packet[n].Offset should equal n*3072. WinUSB will set that up, and I BELIEVE (although I may be misremembering) that the host controller driver will even fix those on an input request if you forget.

The camera never sees any of this. All it knows is that, once every 125us, it gets an "IN" token giving it an opportunity to transmit. It can transmit any number of bytes up to the max packet size. The host controller hardware does the DMA operations based on the schedule set up by the HCD.

(It's actually slightly more complicated than that for a high-bandwidth isoch pipe. The max packet size is actually 1024, so the device gets an IN, and if it sends all 1024 bytes, it gets another IN, and if it sends another 1024 bytes, it gets a third IN. The hardware combines those three transactions into one here.)

1 Like

Hello @Tim_Roberts, thank you for your reply.

Right now, for testing, I am using 1 buffer, and I request 2 transfers where each transfer is 1 frame (8 packets).

The 1st transfer's packet offsets are perfectly aligned with the buffer as stated in the USBD_ISO_PACKET_DESCRIPTOR offsets for each packet.

The 2nd transfer's packets do not align as stated in the USBD_ISO_PACKET_DESCRIPTOR offsets to the buffer. So if I want to access the 1st packet of the 2nd transfer using 3072[8], I'll get an arbitrary offset which would never perfectly align to 0x6000. I find this to be strange but it makes sense with your explanation, every transfer needs its own buffer and can not be shared.

However, everything is great when I request 1 transfer with about 6,800 frames. I can precisely access the packet's offset in the buffer using 3072[6800]. Requesting anything more than 6,800 frames and that's when the offsets start to be misaligned. Not much concern for me since I'll be using only 1 transfer with about 20 frames or so. I would like to receive enough data to stitch up only 1 complete MJPEG frame and quickly decode it in GPU and repeat this infinitely.

I have created an algorithm which can parse the UVC header and MJPEG tags to stitch them up perfectly. It can do this up to 1/5th of the 1080P picture frame.

That is really interesting. I've learned immensely from this wonderful and complicated topic with your help. So it seems like I know how to use WinUSB for receiving data and I think I will stick with it. If Windows 11 natively supports WinUSB, that would also be a huge benefit as well. I like to use arbitrary cameras and will make my own camera hardware someday, and being able to see or customize the raw data gives more freedom and opportunity. I will keep an update how the progress goes.

Thanks.

Hello @Tim_Roberts, hope all is well.

I have created a working MJPEG parser, and it is able to completely reconstruct an MJPEG picture frame. I use the PUSBD_ISO_PACKET_DESCRIPTOR to reconstruct.

However, I found that some reconstructions look like this. Why?

The above picture was created by requesting one transfer of 16 frames of input. I used the PUSBD_ISO_PACKET_DESCRIPTOR to help reconstruct it by using the following packet order: 0, 1, 2, 3, 5, 15, 25, 36, 46, 56, and 61, which contained legitimate data.

Basically, the algorithm starts from the first transfer and checks if it has any data that starts with FFD8. It then combines the body data, and once it finds FFD9, it outputs a .JPEG frame.

I know you have said that the data is spread all over the buffer, but is the spanning in chronological order, or is it literally mixed? If it is in mixed order, how would I know which order is correct for reconstruction?

Some frames along the way have gibberish data; those are not included during reconstruction.

Thanks.

No, they are certainly in chronological order. Are you using the Length field from the USB_ISO_PACKET_DESCRIPTOR to know how many bytes to copy? Are you stripping the packet header that tells you whether it is "first of frame" and "last of frame"?

1 Like

Hello Tim, thanks for the reply and confirmation that the packet order is meant to be in chronological order.

Yes, I am using the Length field in the USB_ISO_PACKET_DESCRIPTOR. Packets are always 3072 bytes, and the packet which has the correct FFD9 tag is always less than 3072 bytes.

Below is an example of the packet header I receive. 0x0C tag is the packet header, the next byte is the bit field, then is the PTS, then SCR. I cannot figure out what the last two bytes are. As of now, I'm not too sure which is the start or last frame based on the UVC tags:

Transfer 0, Packet 0:
0C 0C 00 00 00 00 00 00 00 00 4E 06

Transfer 0, Packet 1:
0C 0C 00 00 00 00 00 00 00 00 63 06

Transfer 0, Packet 114:
0C 0E 00 00 00 00 00 00 00 00 73 06

I can only tell which is the start or last based on the MJPEG tags such as FFD8 and FFD9, but I realized that the USB camera device is giving packets which have a length field in the USB_ISO_PACKET_DESCRIPTOR and the data seems to be non-chronological order. Because when I stitch them together in chronological order, I get a MJPEG frame which looks skewed, like in my previous post.

It's like a hit or miss; sometimes I get a proper complete JPEG picture, sometimes I get a skewed one. I'm assuming this might be an issue of not implementing some sort of synchronization for every 16ms or something in that nature.

I've also checked the MJPEG data in the packets to see if there are any tags or markers which can be used to differentiate if the MJPEG data is part of an ongoing sequence/chronological order, but couldn't find anything meaningful, everything are 99% identical other than the compressed JPEG data.

Thanks.

Alright, so I guess I have perfected in capturing, with 1 transfer of 18 usb frames, seems to do the trick. I now get 99% legitimate hits for proper JPEG complete pictures.

However, some explanation about the UVC header tags as mentioned on my previous post would be great.

However the algo I am using, uses only one buffer and it loops infinitely and parses and then extracts JPEGs. Its super slow (randomly, every 1 to 4 secs would get a JPEG pic) because algo only looks for about total of 33 packets which must contain FFD8 and FFD9, if these conditions are true, the program extracts the JPEG and loops again and requests for another transfer and repeats... I also have tons and tons of printfs, I think if I eliminate these, it would reduce latency. But I think the main culprit of getting slow hits, could be due to no proper synchronization of some sort.

Possible to advice of a proper two buffer system for WinUSB? I would need to create two separate WinUsb_ReadIsochPipe and WinUsb_RegisterIsochBuffer instances?

Also how is the frame rate implemented? Do I like first get a complete JPEG and then Sleep(16) for 16ms and request again for another transfer?

Thanks.

That's not actually true. Note that your three packets have as their second byte 0x0C and 0x0E. That says those three all belong to the same frame, but the 0x0E says "this packet contains the end of the frame". The next packet you get should have the FID bit toggled, so that byte will be 0x0D, and the end of that frame will be 0x0F. You should be able to capture an arbitrary stream of packets and reconstruct the frames, triggering when the EOF bit gets set.

The SCR is 6 bytes long, little-endian. The high-order 2 bytes are supposed to contain the frame number (1kHz) when that packet got started. Your three packets have 1614, 1635 and 1651. That seems like a long time.

Possible to advice of a proper two buffer system for WinUSB?

Like I said, you need to use overlapped I/O to submit both buffers using OVERLAPPED structures, then use WaitForMultipleObjects to wait for them to complete. You process that one, then send it right back down.

Also how is the frame rate implemented?

The camera handles that. It's a "push" model. It will only gather images at the rate settled on during the probe/commit sequence. Assuming you are doing continuous reading, the first packet of a frame should arrive every 33ms or 16.6ms, depending on the rate you picked. Because you're attempting synchronous operation, the camera is probably throwing away frames (or packets) when you don't have a buffer ready.

1 Like

Hello @Tim_Roberts, thank you for your reply; much obliged.

I was trying to figure out how the FID and SCR bit toggling precisely works. I read the USB documentation, but it gave little insight into how it clearly works until your explanation made perfect sense to me. I will change the algorithm to deal with the FID and the SCR bytes directly.

I will read and research more on this.

I also suspect the camera is throwing away some packets due to synchronous implementation. So, just curious, does the camera's hardware always push frames out even if I am not requesting them? Let's say I stopped requesting to read the IN-Endpoint; does the camera still somehow capture and deliver the frames at the hardware level, as long as the camera is turned on? Or is this something irrelevant?

Thank you.

Well, a device can't ever send to USB on its own. A device is only allowed to send when the host controller gives it permission.

But you need to think about how the camera hardware works. You have a sensor, you have a microprocessor controlling the sensor and driving the JPEG compression, and you have a USB engine. Those things are all separate.

The microprocessor reads the sensor 30 or 60 times a second, whether you're listening or not. (Well, they probably stop scanning if its set to Alternate Setting 0.) It feeds that frame into the JPEG compressor. It shoves the encoded frame into the USB engine, which has a FIFO. If the USB gets a signal to transmit, it empties that FIFO out to the USB wire.

If the decoder produces another frame while the FIFO is full, the action depends on the hardware design. They probably have a way to replace the pending frame with the new one if it hasn't started to transmit.

1 Like