Hi,
We have created a custom UVC camera board with a Sony IMX586 48MP sensor module. The main chip is the Cypress CX3 and the solution is “working” - that is we get video, stills, and can control the focus etc. through the standard UVC methods on Windows 10.
The issue is that we did not include any image processing on the board to convert the sensor’s RAW8 format to YUV or MJPEG - something supported directly by UVC. So we pack the RAW8 data as YUV and then our custom app can convert it for proper display.
So we need our camera to work as any UVC webcam would. Before we respin our board and add an ISP or FPGA I wanted to explore writing a custom driver or filter to convert the RAW8 to YUV on the fly and automatically so that the camera will work in any windows app.
Can you driver experts comment on the best approach and if its even a good idea? or even possible?
UVC will supports any arbitrary fourcc you can think of. UYVY and NV12 are in the “uncompressed” formats, but if you use the “frame-based” format, you can embed any fourcc that your computer can handle.
By “RAW8”, do you mean 8-bit monochrome data? That’s usually described as “Y8” or “Y800”. There are many commonly available codecs that can handle that format.
Hi Tim,
Thanks for the response! Are you saying fourcc may already have a codec or filter to perform the conversion? My first time to hear of this so will learn what I can. Is it free?
By RAW8 I mean 1 byte per pixel in Bayer arrangement. This format is listed on fourcc.com as BA81 or possibly BYR1.
Not sure how to make use of fourcc… (?)
To be clear - what do you mean by “any fourcc that your computer can handle” ? Where do I find and/or augment the list that my computer can handle?
Thanks,
KenEE
what do you mean by “any fourcc that your computer can handle” ?
Most computers have accrued a rather wide variety of codecs through the installation of various video products. It used to be we all were keenly aware of what video we could handle and what video we couldn’t. We commonly used to refer people to the “FFDShow Codec Pack” or various other codec packs. OpenCV supports a large set of codecs.
My real point was that many people read the “Video Payload – Uncompressed” document in the UVC spec and conclude that YUY2 and NV12 are the only possible uncompressed formats. That’s not so. The “Video Payload – Frame Based” document describes ANY rectangular arrangement of pixels. The format descriptor is slightly different, so be careful. You just include the GUID of your format, and any DirectShow fourcc can be encoded as XXXXXXXX-0000-0010-8000-00AA00389B71. I have a couple of clients with 8-bit monochrome cameras; we advertise Y800 and most computers can decode it.
As for BYR1, this codec claims to support it: https://www.norpix.com/products/codecs/directshowcodec.php . I know nothing about that; my friend Google offered it to me.
Hi Tim, I am in contact with Norpix to see if their codec will work for us. I will report back any results when available.
But reading Microsoft’s documentation makes me think one might/should be able to get DirectX/Show/3D to do the conversion using fourcc/D3DFORMAT definition. But I don’t see any clear examples yet. Seems BayerToRGB/YUV would be super common since many/most Camera sensors output only RAW bayer encoded data. (at least all of the Sony, OnSemi, and OmniVision sensors we have)
But reading Microsoft’s documentation makes me think one might/should be able to get DirectX/Show/3D to do the conversion
using fourcc/D3DFORMAT definition.
Well, DirectShow doesn’t actually do any conversions. It just provides the mechanism and sets up the framework into which arbitrary converters can be loaded. The built-in codecs can convert between the most common YUV formats (YUY2, UYVY, YV12, I420/IYUV, a few others) and the common RGB formats (8, 15, 16, 24, 32). A DirectShow transform filter is not very hard to write; even if you can’t find one, you could write your own.
Seems BayerToRGB/YUV would be super common since many/most Camera sensors output only RAW bayer encoded data
But because Windows does not support it natively, every camera vendor I’ve worked with has had their own FPGA/CPLD to convert the Bayer codes to YUV or RGB, or to do MJPG compression.