How did you learn NT system development ?

NTDEV
1

I’m a .NET developer, and I’m moving to lower level, curiously I find it more interesting that all the metro development !

The documentation on kernel programming is sparse… It always seems a black art with a strange voodoo community behind it. So I want to ask to this voodoo community, how did it master the black art ?

My question is simple : Where are you reading about windows system development ?

I read a lot of books, and some of them told me wrong information about windows internals. (Windows via C/C++, even if I enjoyed reading it, it was great for app developers)

I read windows internals 1 and 2 from Mark Russinovich, great book, but alas, not developer oriented.
I read Windows Driver Foundation from Penny Orwick, but alas, only for seasoned WDM developers.
So I searched WDM book, but alas, only found a book that cost 160$ new on amazon…
I read good books on Winbg, but alas, it is only good for debugging, and don’t make me start to kernel development.

How did you started ? where can I master this black art ?
Should I climb Mt Himalaya bare footed or do other some strange ritual to find the knowledge source ? :frowning:

2

Well if you can take an OSR class, they are excellent. Other than that
start small, take one of the sample KMDF drivers that is pretty simple
and walk through it with the debugger and API documentation next to you
till you have a basic understanding of what it does. The big answer is
experience and that take time.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“xxxxx@gmail.com” wrote in message
news:xxxxx@ntdev:

> I’m a .NET developer, and I’m moving to lower level, curiously I find it more interesting that all the metro development !
>
> The documentation on kernel programming is sparse… It always seems a black art with a strange voodoo community behind it. So I want to ask to this voodoo community, how did it master the black art ?
>
> My question is simple : Where are you reading about windows system development ?
>
> I read a lot of books, and some of them told me wrong information about windows internals. (Windows via C/C++, even if I enjoyed reading it, it was great for app developers)
>
> I read windows internals 1 and 2 from Mark Russinovich, great book, but alas, not developer oriented.
> I read Windows Driver Foundation from Penny Orwick, but alas, only for seasoned WDM developers.
> So I searched WDM book, but alas, only found a book that cost 160$ new on amazon…
> I read good books on Winbg, but alas, it is only good for debugging, and don’t make me start to kernel development.
>
> How did you started ? where can I master this black art ?
> Should I climb Mt Himalaya bare footed or do other some strange ritual to find the knowledge source ? :frowning:

3

xxxxx@gmail.com wrote:

I’m a .NET developer, and I’m moving to lower level, curiously I find it more interesting that all the metro development !

The documentation on kernel programming is sparse… It always seems a black art with a strange voodoo community behind it. So I want to ask to this voodoo community, how did it master the black art ?

Well, we may be cranky, but there’s no voodoo here.

Have you done C++ programming without .NET? You need to have an
appreciation for how much CLR is doing for you, because there is no CLR
in kernel code. Doing non-CLR C++ programming gives you helpful insight
to that.

There really are a relatively small number of fundamental concepts that
lead to successful kernel programming. One of the most key concepts is
multiprocessing. Many things happen in drivers at the same time, and
usually at the worst possible time. Requests come in while interrupts
are being handled. You ALWAYS need to be aware of what sections of your
data need to be protected from simultaneous access, and you need to be
comfortable with the sync and lock primitives to ensure that doesn’t happen.

Another key concept is “who do you trust”. In an application, if you
dereference a bad pointer, your app dies and you debug it. In a kernel
driver, the system gets a blue screen, you reboot, and your history is
lost. You must validate everything that was not handed to you from a
trusted source.

Another key concept is the driver stack. Each driver acts as a
translator. It accepts some form of request from above, and it passes
some form of request down below. Its job is to translate from the
“above” language to the “below” language. Reading about PnP will start
to give you a glimmer in how these stacks are built and how they operate.

I learn best by example. For me, there’s nothing as valuable as
building a sample and single-stepping through it in a debugger, to see
what comes in and what goes out.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

4

It’s a multi-step process:

FIRST you need to learn general operating system concepts. The basics of addressing, memory management, how you get in and out of the operating system, scheduling, and device concepts (interrupts, registers, etc). Read a good book on OS theory.

NEXT you need to learn how these general concepts are dealt with in Windows. So, this is where Windows Internals comes in.

FINALLY, you’re ready to learn about kernel-mode programming in Windows. The WDF book by Orwick and Smith is pretty basic, and should be approachable once you have the right background.

As Don said, you can short-circuit a lot of this by taking a seminar (http://www.osr.com/seminars)… but you’ll need to ARRIVE at that seminar with a decent background in OS concepts or you’ll be frustrated.

Peter
OSR

5

I used to tell my students, “Once you get past the silly errors (if-test
backwards, attempt to use bad pointer, off-by-1 errors) the only thing
that remains are concurrency errors. The are nearly impossible to
reproduce, and are therefore hard to find, and harder still to fix. You
simply cannot ‘debug to correctness’ a concurrency problem; you have to
eliminate them at the design level.” As already pointed out, you need to
understand the concurrency issues. For example, consider the top-level
dispatch routine:
do domething, maybe as simple as getting the device extension
enqueue request();
do somthing else…

You have to recognize that *before* enqueue request() returns, the IRP
could already be completed and the thread that issued the I/O can already
be running.

The other important fact you must internalize is that the purpose of a
driver is the care and feeding of IRPs. If, along the way, your driver
talks to hardware, either directly (as a driver for a PCI device) or
indirectly (a USB driver, filter driver, etc.) that’s cool, but as far as
the system is concerned, if you obey the protocols for IRP management, the
system will be happy. Of course, if you do talk to hardware, you had
better satisfy *its* protocol requirements.

If you look at a lot of the questions in these newsgroups, they fall into
several categories

  1. “How do I do ?”
    2. “I did this and it , what did I do wrong?”
    3. “I am building a driver, and I want it to work like drivers did in
    . After building something that is so
    convoluted I can’t even explain it, it doesn’t work, what did I do
    wrong?”
    4. “How do I do ?”

    3 has only one answer: “This is Windows. You need to build a Windows
    driver, not something that worked in some embedded OS you once used. Your
    design is trash; start over and do it right.”
    4 resembles 1, except that 1 is a legitimate question, and 4 is some
    bizarre piece of behavior that usually runs counter to the entire
    philosophy of how a Windows driver works; often so far from standard
    practice that we can’t even figure out what problem it is trying to solve.

    Some hints
    *Don’t try to replicate the behavior of some weird embedded OS you once
    used. There is no future in this.
    If you have crashes you don’t understand, you will get more help if you
    say something about being a newbie, and include the !analyze -v output.
    Be prepared to show us the code
    _try/_except are never the right solution except in some
    carefully-documented situations
    *Don’t trust anyone. Validate values that you are unsure of
    *ASSERT is your friend
    *If you have ASSERT(condition), it should be followed by an if(!condition)
    { graceful recovery}
    *The Driver Verifier is your New Best Friend. Turn it on when you start
    debugging your driver. If you have to wait for someone in this group to
    ask if you used DV, it is already too late.
    *Find a prototype driver that appears functionally close to what you need
    to do. After you have confidence you understand nearly every line, you
    are ready to start coding
    *Using event objects shared with the application is always the wrong
    approach, until you hit one of the rare situations where it might be the
    correct solution
    *Understand “inverted call” and use it wisely
    Do not try to allocate memory in the kernel and share it with user space.
    This is always the wrong solution until you have enough experience to
    understand when you hit the once-in-a-lifetime driver where that is the
    only possible solution.
    “Graceful recovery” is an important concept. For example, if there’s
    something wrong in your driver, no matter what (bad buffer, whatever that
    means to you, device-has-caught-fire bit is set in the status register),
    don’t forget to complete the IRP. If it was a queued IRP, make sure the
    next IRP will be dequeued.
    *Trust no one. For example, if you have a switch statement decoding
    IOCTLs, it always has a default statement that does appropriate graceful
    recovery for the context you are in.
    *Do not reflect the defects of the hardware to the app. Do not
    unpleasantly surprise the user.
    For example, if the user does
    WriteFile(…);
    WriteFile(…);
    DeviceIoControl(…);

    make sure the driver does these sequentially. For example, in the
    serial port driver, the “change baud rate” request is done immediately,
    not queued up behind the WriteFiles. So the baud rate changes when the
    IOCTL is seen, often in the middle of sending a character. Maintain
    sensible semantics. Operations which change the state of your device
    should execute in the order the application issued them.

    *Understand the user I/O API fully. I’ve seen bizarre designs (even
    recently in this NG) that would have been trivial had the driver writer
    understood asynchronous I/O.
    *Write the programmer’s manual FIRST. If it takes a paragraph to explain
    some odd feature of the design, redesign.
    *ReadFile, WriteFile and DeviceIoControl are the low-level interface to
    your driver. If appropriate, you should design a DLL to interface to the
    user. Consider, as an example, SetDCB (the failure is that it makes
    immediate changes, even if there are queued operations; see previous note)

    That’s my off-the-top-of-my-head list; I might come up with other ideas
    and, if so, I’ll post them.
    joe

    > I’m a .NET developer, and I’m moving to lower level, curiously I find it
    > more interesting that all the metro development !
    >
    > The documentation on kernel programming is sparse… It always seems a
    > black art with a strange voodoo community behind it. So I want to ask to
    > this voodoo community, how did it master the black art ?
    >
    > My question is simple : Where are you reading about windows system
    > development ?
    >
    > I read a lot of books, and some of them told me wrong information about
    > windows internals. (Windows via C/C++, even if I enjoyed reading it, it
    > was great for app developers)
    >
    > I read windows internals 1 and 2 from Mark Russinovich, great book, but
    > alas, not developer oriented.
    > I read Windows Driver Foundation from Penny Orwick, but alas, only for
    > seasoned WDM developers.
    > So I searched WDM book, but alas, only found a book that cost 160$ new on
    > amazon…
    > I read good books on Winbg, but alas, it is only good for debugging, and
    > don’t make me start to kernel development.
    >
    > How did you started ? where can I master this black art ?
    > Should I climb Mt Himalaya bare footed or do other some strange ritual to
    > find the knowledge source ? :frowning:
    >
    > —
    > NTDEV is sponsored by OSR
    >
    > For our schedule of WDF, WDM, debugging and other seminars visit:
    > http://www.osr.com/seminars
    >
    > To unsubscribe, visit the List Server section of OSR Online at
    > http://www.osronline.com/page.cfm?name=ListServer
    >
6

windbg is your Himalayas
hello world driver is where your base camp lies (obviously you need
to be altitude trained to survive on rarified air at base camp ) if
you don’t have an altitude to train use you can use gamow bags
breaking say a orioned encrypted deeply hidden stuxnet virus driver is
akin to boasting that you conquered annapurna :slight_smile:
sometimes perseverance by climbing barefooted might be the only way to
reach the peak :slight_smile:

On 12/28/12, xxxxx@flounder.com wrote:
> I used to tell my students, “Once you get past the silly errors (if-test
> backwards, attempt to use bad pointer, off-by-1 errors) the only thing
> that remains are concurrency errors. The are nearly impossible to
> reproduce, and are therefore hard to find, and harder still to fix. You
> simply cannot ‘debug to correctness’ a concurrency problem; you have to
> eliminate them at the design level.” As already pointed out, you need to
> understand the concurrency issues. For example, consider the top-level
> dispatch routine:
> do domething, maybe as simple as getting the device extension
> enqueue request();
> do somthing else…
>
> You have to recognize that before enqueue request() returns, the IRP
> could already be completed and the thread that issued the I/O can already
> be running.
>
> The other important fact you must internalize is that the purpose of a
> driver is the care and feeding of IRPs. If, along the way, your driver
> talks to hardware, either directly (as a driver for a PCI device) or
> indirectly (a USB driver, filter driver, etc.) that’s cool, but as far as
> the system is concerned, if you obey the protocols for IRP management, the
> system will be happy. Of course, if you do talk to hardware, you had
> better satisfy its protocol requirements.
>
> If you look at a lot of the questions in these newsgroups, they fall into
> several categories
> 1. “How do I do ?”
> 2. “I did this and it , what did I do wrong?”
> 3. “I am building a driver, and I want it to work like drivers did in
> . After building something that is so
> convoluted I can’t even explain it, it doesn’t work, what did I do
> wrong?”
> 4. “How do I do ?”
>
> 3 has only one answer: “This is Windows. You need to build a Windows
> driver, not something that worked in some embedded OS you once used. Your
> design is trash; start over and do it right.”
> 4 resembles 1, except that 1 is a legitimate question, and 4 is some
> bizarre piece of behavior that usually runs counter to the entire
> philosophy of how a Windows driver works; often so far from standard
> practice that we can’t even figure out what problem it is trying to solve.
>
> Some hints
> *Don’t try to replicate the behavior of some weird embedded OS you once
> used. There is no future in this.
> *If you have crashes you don’t understand, you will get more help if you
> say something about being a newbie, and include the !analyze -v output.
> Be prepared to show us the code
> *_try/_except are never the right solution except in some
> carefully-documented situations
> *Don’t trust anyone. Validate values that you are unsure of
> *ASSERT is your friend
> *If you have ASSERT(condition), it should be followed by an if(!condition)
> { graceful recovery}
> *The Driver Verifier is your New Best Friend. Turn it on when you start
> debugging your driver. If you have to wait for someone in this group to
> ask if you used DV, it is already too late.
> *Find a prototype driver that appears functionally close to what you need
> to do. After you have confidence you understand nearly every line, you
> are ready to start coding
> *Using event objects shared with the application is always the wrong
> approach, until you hit one of the rare situations where it might be the
> correct solution
> *Understand “inverted call” and use it wisely
> *Do not try to allocate memory in the kernel and share it with user space.
> This is always the wrong solution until you have enough experience to
> understand when you hit the once-in-a-lifetime driver where that is the
> only possible solution.
> *“Graceful recovery” is an important concept. For example, if there’s
> something wrong in your driver, no matter what (bad buffer, whatever that
> means to you, device-has-caught-fire bit is set in the status register),
> don’t forget to complete the IRP. If it was a queued IRP, make sure the
> next IRP will be dequeued.
> *Trust no one. For example, if you have a switch statement decoding
> IOCTLs, it always has a default statement that does appropriate graceful
> recovery for the context you are in.
> *Do not reflect the defects of the hardware to the app. Do not
> unpleasantly surprise the user.
> For example, if the user does
> WriteFile(…);
> WriteFile(…);
> DeviceIoControl(…);
>
> make sure the driver does these sequentially. For example, in the
> serial port driver, the “change baud rate” request is done immediately,
> not queued up behind the WriteFiles. So the baud rate changes when the
> IOCTL is seen, often in the middle of sending a character. Maintain
> sensible semantics. Operations which change the state of your device
> should execute in the order the application issued them.
>
> *Understand the user I/O API fully. I’ve seen bizarre designs (even
> recently in this NG) that would have been trivial had the driver writer
> understood asynchronous I/O.
> *Write the programmer’s manual FIRST. If it takes a paragraph to explain
> some odd feature of the design, redesign.
> *ReadFile, WriteFile and DeviceIoControl are the low-level interface to
> your driver. If appropriate, you should design a DLL to interface to the
> user. Consider, as an example, SetDCB (the failure is that it makes
> immediate changes, even if there are queued operations; see previous note)
>
> That’s my off-the-top-of-my-head list; I might come up with other ideas
> and, if so, I’ll post them.
> joe
>
>
>
>> I’m a .NET developer, and I’m moving to lower level, curiously I find it
>> more interesting that all the metro development !
>>
>> The documentation on kernel programming is sparse… It always seems a
>> black art with a strange voodoo community behind it. So I want to ask to
>> this voodoo community, how did it master the black art ?
>>
>> My question is simple : Where are you reading about windows system
>> development ?
>>
>> I read a lot of books, and some of them told me wrong information about
>> windows internals. (Windows via C/C++, even if I enjoyed reading it, it
>> was great for app developers)
>>
>> I read windows internals 1 and 2 from Mark Russinovich, great book, but
>> alas, not developer oriented.
>> I read Windows Driver Foundation from Penny Orwick, but alas, only for
>> seasoned WDM developers.
>> So I searched WDM book, but alas, only found a book that cost 160$ new on
>> amazon…
>> I read good books on Winbg, but alas, it is only good for debugging, and
>> don’t make me start to kernel development.
>>
>> How did you started ? where can I master this black art ?
>> Should I climb Mt Himalaya bare footed or do other some strange ritual to
>> find the knowledge source ? :frowning:
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

7

> My question is simple : Where are you reading about windows system
> development ?

I can’t recommend a good book on kernel-mode driver dev. The only option
for you is to read the main DDK/WDK docs, but good.

Historical third-party kernel-mode books I’d consider looking at would
be: Nebbett’s Windows NT/2000 Native API Reference, and Peter/Tony’s
Windows NT Device Driver Development, and Nagar’s Windows NT File System
Internals.

For historical books on user-mode NT, I’d consider: Hart’s Windows
Systems Programming (4th ed) is a decent intro to Win32 APIs, esp if
you’re coming from a *nix background. Newcomer/Rector’s Win32
Programming is also useful for Win32 API stuff, esp. for historic
transition from Win16.

Helen Custer’s Inside Windows NT, aka Windows Internals 1st edition, is
useful to understand original NT arch. Not kernel-driver centric, but
useful historical background. Long out of print, maybe on MSDN archives?

As for hardcopy books on debugging, Hewardt/Pravat’s Advanced Windows
Debugging and Soulami’s Inside Windows Debugging aren’t bad. Both target
user-mode, hardly anything on kernel-mode.

I can’t recommend a good book on kernel debugging. The main option is
the Windbg help file, which is very good, at least as of a few years
ago. :slight_smile:

There are video archives of some key PDCs and WinHECs and IFS Kit
conferences, that are useful to see the original architects talking
about their designs, and current architects talking about new/updated
designs.

Learn to use KD/Windbg as well as NTSD/Windbg, and all of their
extensions. Learn how to turn enable different kinds of tracing and
diagnostics in the system.

Run a checked build with a kernel debugger with valid symbols, all the
time, not just when you’re trying to fix something. As well as a retail
build, to understand how differently the system and your code behaves.

Read all the SDK/WDK headers and samples. For years, the only docs for
IFS kit drivers was ntifs.h and a few samples. :slight_smile:

Learn to run/debug/trace all of the sample drivers. Especially learning
the various ways data moves from user-mode apps to kernel-mode drivers,
and related app/driver interactions.

Reading the NT sources is the best way to learn, of course. I’ve heard
some levels of being a Windows MVP will get you some readonly access to
the NT sources.

You could also look at the sources to ReactOS, a GPL-licensed NT clone.
But the internals of that project aren’t the same, so only somewhat
useful, but can be educational. Be careful, that GPL code might taint
you if you ever want to work in Redmond. :slight_smile:

Besides this NTdev list, OSR hosts a kernel-mode Windbg list, as well as
NTIFS list, if you’re working on file sytem [filter] drivers. So lots of
archives to read. :slight_smile:

That’s a start. HTH.

8

I am by no means an expert (as you can see by some of my posts), but I would recommend reading Mr. Newcomer’s book: Developing Windows NT Device Drivers and also the Walter Oney book: Microsoft Windows Driver Model. The WDM book will give you an appreciation of how good the WDF model is.

I have hard copies of these books, but they may be available as eBooks on OReilly for much less than the hard copy price.

9

Thanks so much for taking the time to give me so detailed and useful answer with great advices, even from book authors themselves ! :slight_smile:
There is so much passion here, it makes me even more envious to master these skills.

Here a resumee of your suggestions :

-Developing Windows NT Device Drivers of Mr. Newcomer’s
-WDF book by Orwick and Smith
-Microsoft Windows Driver Model of Walter Oney
-Windows NT/2000 Native API Reference of Nebbett
-Windows NT Device Driver Development of Peter/Tony’s
-Windows NT File System Internals of Nagarr
-Windows Systems Programming (4th ed) from Hart
-Inside Windows NT from Helen Custer
-Advanced Windows Debugging from Hewardt/Pravat’s
-Inside Windows Debugging from Soulami
-DDK/WDK docs (and header files :))
-PDCs and WinHECs and IFS Kit conferences
-OSR Courses (Canada and US only it seems)
-Perseverance by climbing barefooted to reach the peak :slight_smile:

For responses of previous posts :

Have you done C++ programming without .NET? You need to have an
appreciation for how much CLR is doing for you, because there is no CLR
in kernel code. Doing non-CLR C++ programming gives you helpful insight
to that.

I started with C++ long time ago, I always use it here and there with C++/CLI (despite all of its gotcha) so I can easily use the winapi in .NET. I have no problem with this language and idioms, but I don’t want to do high level stuff with it. I’m used to analyse stuff with OllyDbg/IDA pro, I’m a .NET developer familiar with the low level.

FINALLY, you’re ready to learn about kernel-mode programming in Windows. The
WDF book by Orwick and Smith is pretty basic, and should be approachable once
you have the right background.

I first read the WDF book when I was not familiar on the OS internals, so thanks to your advice, I’ll give it another immediately, I’ve improved a lot since my last read.

10

> -PDCs and WinHECs and IFS Kit conferences

Actually, WinHEC was a typo of mine. Instead of WinHEC, I meant the
original DDK Conference, dry-run in Bellevue hotel, done later at
Disneyland Hotel. One of those was videotaped. That and the File System
event at the Seattle hotel, those would be a good foundation for initial
NT arch. They were originally to attendees, as a huge set of VHS tapes.
I think I saw a DVD on some later MSDN distros with some of them.

But that’s all old stuff. Others here would be better suited to point
out current conference archives.

11

> *If you have crashes you don’t understand, you will get more help if you
> say something about being a newbie, and include the !analyze -v output.

Also, I’d suggest some current pentest books, focusing on Windows
shellcode. Like Grey Hat Hacking 3rd edition, or some older classics,
like Shellcodes Handbook. They look at NT drivers from a different
perspective, and it’s nice to see how attackers look at your code. The
recent msg with a pointer to the last MS driver talk at DefCon is a good
supplement to those docs.

12





Hmmmm… Is this the one that was done PRIOR to the release of NT… done some time in 1992 or MAYBE early 1993? Because, damn it, I missed that conference (I was still at Digital at the time). There are a couple of videos from this of presentations by Cutler on the NT Kernel. They’re quite rare… I’d love to get my hands on a copy, so if anyone has one… please get in touch. To the best of my knowledge, that was the last time Cutler spoke in public about the NT architecture.

The early driver development conference that was done AFTER the release of NT was at Disneyland… it was co-located and in parallel with the PDC. But it wasn’t videoed, IIRC (I was there, this conference took place AFTER we started OSR).



Actually, this event was at the Shilshole Bay Beach Club (I was at this one, too). It’s a LARGE set of tapes, and damn interesting archaeology. Dry. But fun.

Truly, I don’t recommend anybody trying to learn anything from these tapes. They’re mostly interesting as museum pieces (and to see how young the participants look).

Peter
OSR

13

> Hmmmm… Is this the one that was done PRIOR to the release of NT…
> done some time in 1992 or MAYBE early 1993? Because, damn it, I
> missed that conference (I was still at Digital at the time).

The initial DDK Conference was put on twice, a few weeks apart. Same
agenda and the same speakers as the later Disneyland one. Maybe a
speaker or two changed, or a talk or two was slightly updated. But not
two different events.

There was enough concern about speakers and agenda of the DDK con that
they did a dry run, locally. There was also the issue that a lot of
MSFTies wanted to attend, and the event organizers only wanted to
restrict MSFTie attendence at Disneyland to staff/speakers, and leave
seats for IHVs/OEMs. The local Bellevue event was also more for the
local MSFTies that couldn’t make it to Disneyland…

But it wasn’t videoed, IIRC (I was there, this conference took place
> AFTER we started OSR).

Hmm, I’m pretty sure one of these DDK Confs and NT PDC were both videotaped.

I seem to recall having a few boxes of these, as well as IFS event.
There were a few sets in the building 20 porting lab, eventually
attendees walked off with all the videos. :frowning:

MSFTies might be able to order them via internal MS AV methods. And I’ve
heard some of these events, maybe just the IFS event, were converted to
DVD-based media by MSDN a while ago. Not sure how to get those.