CcFlushCache under SRV2

Slava_Imameev · October 31, 2016, 3:30pm

This is why I asked about RDBS related completion routine for read request on the server side.

anatoly_mikhailov · October 31, 2016, 3:41pm

> This is why I asked about RDBS related completion routine for read
request on the server side.

You made assumption there is a file system above. But if server side open
files as usual client, for what such implementation of server is needed?
Especially do own caching instead of give it up to FSD where files are read
from.

2016-10-31 22:30 GMT+03:00 :

>

>
> This is why I asked about RDBS related completion routine for read request
> on the server side.
>
> —
> NTFSD is sponsored by OSR
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer>
></http:>

anatoly_mikhailov · October 31, 2016, 3:45pm

Moreover, localhost is target. No need to send two IRP instead of one.

2016-10-31 22:40 GMT+03:00 Anatoly Mikhailov :

> > This is why I asked about RDBS related completion routine for read
> request on the server side.
> >
> You made assumption there is a file system above. But if server side open
> files as usual client, for what such implementation of server is needed?
> Especially do own caching instead of give it up to FSD where files are read
> from.
>
> 2016-10-31 22:30 GMT+03:00 :
>
>>

>>
>> This is why I asked about RDBS related completion routine for read
>> request on the server side.
>>
>> —
>> NTFSD is sponsored by OSR
>>
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at <
>> http://www.osronline.com/page.cfm?name=ListServer>
>>
>
></http:>

Bill_Zissimopoulos · October 31, 2016, 4:15pm

Anatoly Mikhailov wrote:

You can do test 2 with not your FSD also just to make sure it’s a bug in
your FSD.

Here is another twist to this story. If I run the test against NTFS (via redirector/SRV) the test succeeds. If however I add the FlushViewOfFile call after the modifying for loop and then run the test against NTFS (via redirector/SRV), the FlushViewOfFile call fails with ERROR_INVALID_PARAMETER (87).

<<
srand(seed);
for (PUINT8 Bgn = MappedView + FileSize1 / 2, End = Bgn + FileSize0; End > Bgn; Bgn++)
*Bgn = rand() & 0xff;

Success = FlushViewOfFile(MappedView, 0);
tlib_printf("FlushViewOfFile LastError=%ld ", GetLastError());
ASSERT(Success);

>

There is definitely something weird that this test triggers when going through redirector/SRV, even on NTFS.

Slava Imameev wrote:

RDBSS is used by network redirectors

We are talking about the server side while network redirector( namely mrxsmb )
is on the client side.

I have been perplexed about this. I would expect all calls to come to me via SRV2. But sometimes it seems as if the redirector is calling my FSD directly. I must admit complete ignorance about the true workings of the redirector/SRV/whathaveyou.

Here is the IRP stack and FILE_OBJECT at the time of the READ call. Notice the page fault that is triggered by the for loop, which comes to my FSD via RDBSS directly (why?). I note here that all my tests are done using a \localhost\winfsp-tests-share share.
<<
kd> kc

kd> ?? (PVOID)FileObject
PVOID 0xfffffa80`04590820

kd> !fileobj @@(FileObject)

winfsp-tests\file0

Related File Object: 0xfffffa8003a64f20

Device Object: 0xfffffa8003d59820 \FileSystem\WinFsp
Vpb: 0xfffffa8004b08830
Event signalled
Access: Read Write SharedRead SharedWrite

Flags: 0x1041008
No Intermediate Buffering
Modified
Handle Created
Remote Origin

FsContext: 0xfffff9800158ae00 FsContext2: 0xfffff98001598fb0
CurrentByteOffset: 0
Cache Data:
Section Object Pointers: fffff980090d2fc8
Shared Cache Map: 00000000

>

And here is the IRP stack and FILE_OBJECT at the time of the CREATE call. Notice that this came via SRV (srv2!SrvCreateFile).
<<
kd> kc

kd> ?? (PVOID)FileObject
PVOID 0xfffffa80`04590820

kd> !fileobj @@(FileObject)

winfsp-tests\file0

Related File Object: 0xfffffa8003a64f20

Device Object: 0xfffffa8003d59820 \FileSystem\WinFsp
Vpb is NULL

Flags: 0x8
No Intermediate Buffering

CurrentByteOffset: 0

>

Since I run this test for \localhost only, I will set things up so that I can try it across two machines and see what I get there. But IMO it should work on the \localhost case regardless.

Bill

anatoly_mikhailov · October 31, 2016, 4:37pm

> There is definitely something weird that this test triggers when going
through redirector/SRV, even on NTFS.

File mapping via redirector is special case. There are races that couldn’t
be solved. Imagine to your self you mapped remote file, and modified its
mapped view. Meanwhile some other client write to the same file, but the
mapped view wouldn’t be updated and more over if it’ll be sent to remote
FSD the recent data will be overwritten by stale one.

Just to be sure. You did test 2 with NTFS the same way you did it with your
FSD and test was ok?

I have been perplexed about this. I would expect all calls to come to me
via SRV2. But sometimes it seems as if the redirector is calling my FSD
directly.

Your target is locahost. There is no nothing weird. By call stack we see
there was page fault and memory manager sent read request which passed MUP,
smb miniredirector, smb miniredirector uses RDBSS to manage remote file
caching and mapping, RDBSS forward IRP to FILE_OBJECT of your FSD. No need
to do it via SRV2 because target machine is the same as source. There is no
only network traffic in this chain. Everything else is the same.

Bill_Zissimopoulos · October 31, 2016, 4:49pm

> Just to be sure. You did test 2 with NTFS the same way you did it with your

FSD and test was ok?

When my FSD is not involved and I am running the test against NTFS I get the following behavior:

Run the test against NTFS directly (i.e. on C: drive). PASS
Run the test against NTFS directly (i.e. on C: drive). Do a FlushViewOfFile call prior to UnmapViewOfFile. PASS
Run the test against NTFS but via redirector/SRV (i.e. through a network share). PASS
Run the test against NTFS but via redirector/SRV (i.e. through a network share). Do a FlushViewOfFile call prior to UnmapViewOfFile. FAIL

Because of the FAIL in (4) I am now beginning to suspect that the issue may not be with my FSD.

Bill

anatoly_mikhailov · October 31, 2016, 4:53pm

> Because of the FAIL in (4) I am now beginning to suspect that the issue
may not be with my FSD.

Run the test against NTFS but via redirector/SRV (i.e. through a network
share). PASS

Will i be right if i say the case 3 doesn’t wotk with your FSD?

2016-10-31 23:49 GMT+03:00 :

> > Just to be sure. You did test 2 with NTFS the same way you did it with
> your
> > FSD and test was ok?
>
> When my FSD is not involved and I am running the test against NTFS I get
> the following behavior:
>
> 1. Run the test against NTFS directly (i.e. on C: drive). PASS
> 2. Run the test against NTFS directly (i.e. on C: drive). Do a
> FlushViewOfFile call prior to UnmapViewOfFile. PASS
> 3. Run the test against NTFS but via redirector/SRV (i.e. through a
> network share). PASS
> 4. Run the test against NTFS but via redirector/SRV (i.e. through a
> network share). Do a FlushViewOfFile call prior to UnmapViewOfFile. FAIL
>
> Because of the FAIL in (4) I am now beginning to suspect that the issue
> may not be with my FSD.
>
> Bill
>
>
> —
> NTFSD is sponsored by OSR
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer>
></http:>

Bill_Zissimopoulos · October 31, 2016, 4:53pm

I wrote:

Run the test against NTFS but via redirector/SRV (i.e. through a network
share). Do a FlushViewOfFile call prior to UnmapViewOfFile. FAIL

BTW, the FAIL I am talking about is not a failed test, but a failed API call. Specifically the FlushViewOfFile call fails in this case with ERROR_INVALID_PARAMETER (87).

My call looks like:
FlushViewOfFile(MappedView, 0);
With MappedView the return value of MapViewOfFile. So I cannot see why the FlushViewOfFile should fail and I am suspecting a deeper issue.

Bill

Bill_Zissimopoulos · October 31, 2016, 4:54pm

> Will i be right if i say the case 3 doesn’t wotk with your FSD?

You are right, yes.

Cases 1, 2 work with my FSD. Cases 3 and 4 do not.

anatoly_mikhailov · October 31, 2016, 4:58pm

Then there is the issue with your FSD. At least for case 3. For some reason
network redirector purge the data instead of flushing. I’ll take a look at
memory manager soon.

2016-10-31 23:54 GMT+03:00 :

> > Will i be right if i say the case 3 doesn’t wotk with your FSD?
>
> You are right, yes.
>
> Cases 1, 2 work with my FSD. Cases 3 and 4 do not.
>
>
> —
> NTFSD is sponsored by OSR
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer>
></http:>

anatoly_mikhailov · October 31, 2016, 5:46pm

I’m not sure there wasn’t attempts to write modified data by redirector.
Possibly the write request wasn’t forward to your FSD and just failed
somewhere before. To find out it, you can attach filter to network
redirector to see if there were paging IRP_MJ_WRITE and if it was so, see
where it fails. Once failed IRP_MJ_WRITE forces memory manager to fail
further I/O on mapped file. Thus, after unmapping all filled buffer data is
lost.

2016-10-31 23:58 GMT+03:00 Anatoly Mikhailov :

> Then there is the issue with your FSD. At least for case 3. For some
> reason network redirector purge the data instead of flushing. I’ll take a
> look at memory manager soon.
>
> 2016-10-31 23:54 GMT+03:00 :
>
>> > Will i be right if i say the case 3 doesn’t wotk with your FSD?
>>
>> You are right, yes.
>>
>> Cases 1, 2 work with my FSD. Cases 3 and 4 do not.
>>
>>
>> —
>> NTFSD is sponsored by OSR
>>
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at <
>> http://www.osronline.com/page.cfm?name=ListServer>
>>
>
></http:>

OSR_Community_User · October 31, 2016, 6:37pm

That doesn’t mesh with my experience – in my experience, Mm will retry the writes specifically on the off chance it was a transient (recoverable) error. Last I looked, Mm knows if the file backing the section is on the network, or not, and if it is on the network it will eventually give up and discard the data.

Tony
OSR

Bill_Zissimopoulos · October 31, 2016, 7:10pm

> I’m not sure there wasn’t attempts to write modified data by redirector.

Possibly the write request wasn’t forward to your FSD and just failed
somewhere before. To find out it, you can attach filter to network
redirector to see if there were paging IRP_MJ_WRITE and if it was so, see
where it fails.

Anatoly, I think you are onto something. Here is an excerpt from a FileSpy log:
<<
43 winfsp-tests-x64.exe IRP_MJ_CREATE 40000884 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS FILE_OVERWRITE_IF CreOpts: 00000068 Access: 0012019F Share: 00000003 Attrib: 00000080 Result: FILE_CREATED
35 System IRP_MJ_CREATE 40000884 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FILE_OVERWRITE_IF CreOpts: 00000148 Access: 0012019B Share: 00000003 Attrib: 00000080 Result: FILE_CREATED
36 System FASTIO_QUERY_NETWORK_OPEN_INFO FFFFF98001416E00 Z:\winfsp-tests\file0 FAILURE
37 System IRP_MJ_QUERY_INFORMATION 40060874 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FileNetworkOpenInformation CreateTime: 01D12188-9016B21F LastAccTime: 01D12188-9016B21F LastWrtTime: 01D12188-9016B21F ChangeTime: 01D12188-9016B21F AllocationSize: 00000000-00000000 EndOfFile: 00000000-00000000 Attrib: 00000020
38 System IRP_MJ_QUERY_SECURITY 40000004 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS
39 System IRP_MJ_QUERY_INFORMATION 40060874 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FileInternalInformation IndexNumber: 00000000-00000005
40 System IRP_MJ_QUERY_INFORMATION 40060874 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FileStandardInformation AllocationSize: 00000000-00000000 EndOfFile: 00000000-00000000 NumberOfLinks: 1 DeletePending: FALSE
41 System IRP_MJ_QUERY_INFORMATION 40060874 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FileBasicInformation CreateTime: 01D12188-9016B21F LastAccTime: 01D12188-9016B21F LastWrtTime: 01D12188-9016B21F ChangeTime: 01D12188-9016B21F Attrib: 00000020
42 System IRP_MJ_QUERY_INFORMATION 40060874 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FileStandardInformation AllocationSize: 00000000-00000000 EndOfFile: 00000000-00000000 NumberOfLinks: 1 DeletePending: FALSE
44 winfsp-tests-x64.exe FASTIO_QUERY_STANDARD_INFO FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 FAILURE
46 winfsp-tests-x64.exe IRP_MJ_QUERY_INFORMATION 40000042 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS FileStandardInformation AllocationSize: 00000000-00000000 EndOfFile: 00000000-00000000 NumberOfLinks: 1 DeletePending: FALSE
45 System IRP_MJ_QUERY_INFORMATION 40060874 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FileStandardInformation AllocationSize: 00000000-00000000 EndOfFile: 00000000-00000000 NumberOfLinks: 1 DeletePending: FALSE
48 winfsp-tests-x64.exe IRP_MJ_SET_INFORMATION 40000042 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS FileEndOfFileInformation EndOfFile: 00000000-00020064
47 System IRP_MJ_SET_INFORMATION 40060834 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FileEndOfFileInformation EndOfFile: 00000000-00020064
50 winfsp-tests-x64.exe IRP_MJ_READ 40060043 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS Offset: 00000000-00000000 ToRead: 8000 Read: 8000
49 winfsp-tests-x64.exe IRP_MJ_READ 40000001 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS Offset: 00000000-00000000 ToRead: 8000 Read: 8000
52 winfsp-tests-x64.exe IRP_MJ_READ 40060043 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS Offset: 00000000-00008000 ToRead: 8000 Read: 8000
51 winfsp-tests-x64.exe IRP_MJ_READ 40000001 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS Offset: 00000000-00008000 ToRead: 8000 Read: 8000
54 winfsp-tests-x64.exe IRP_MJ_READ 40060043 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS Offset: 00000000-00010000 ToRead: 10000 Read: 10000
53 winfsp-tests-x64.exe IRP_MJ_READ 40000001 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS Offset: 00000000-00010000 ToRead: 10000 Read: 10000
56 winfsp-tests-x64.exe IRP_MJ_READ 40060043 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS Offset: 00000000-00020000 ToRead: 1000 Read: 64
55 winfsp-tests-x64.exe IRP_MJ_READ 40000001 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS Offset: 00000000-00020000 ToRead: 1000 Read: 64
60 winfsp-tests-x64.exe IRP_MJ_CLEANUP 40000404 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS
57 System IRP_MJ_QUERY_INFORMATION 40060874 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS FileBasicInformation CreateTime: 01D12188-9016B21F LastAccTime: 01D12188-9016B21F LastWrtTime: 01D12188-9016B21F ChangeTime: 01D12188-9016B21F Attrib: 00000020
58 winfsp-tests-x64.exe IRP_MJ_WRITE 40060043 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_INVALID_PARAMETER Offset: 00000000-00000000 ToWrite: 20064 Written: 0
59 winfsp-tests-x64.exe IRP_MJ_WRITE 40060043 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_INVALID_PARAMETER Offset: 00000000-00000000 ToWrite: 20064 Written: 0
70 winfsp-tests-x64.exe IRP_MJ_CLOSE 40000404 FFFFF8A00243C010 \Device\Mup\localhost\winfsp-tests-share\winfsp-tests\file0 STATUS_SUCCESS
69 System IRP_MJ_CLEANUP 40000404 FFFFF98001416E00 Z:\winfsp-tests\file0 STATUS_SUCCESS

>

Notice that after \Device\Mup receives CLEANUP (line 60) there are 2 WRITE IRP’s sent to \Device\Mup that are failed with STATUS_INVALID_PARAMETER. My FSD never sees these IRP’s and cannot act on them. [Actually the line numbers are: 58,59 on the WRITE IRP’s and 60 on the CLEANUP IRP. Perhaps this means that the WRITE IRP’s are attempted from within the redirector CLEANUP handler.]

There is also an IRP_MJ_QUERY_INFORMATION but it succeeds and it looks reasonable. The FileAttrib means FILE_ATTRIBUTE_ARCHIVE and the date/times are reasonable: Tuesday, November 17, 2015 10:37:39pm (when this VM was snapshotted).

So I have no clue why the WRITE IRP’s would fail with STATUS_INVALID_PARAMETER. There must something about my FSD they don’t like in this case, but I am not wiser as to what.

Bill

Bill_Zissimopoulos · October 31, 2016, 7:12pm

And I apologize for the badly formatted log, but I do not how to make it format properly when posting through the web interface.

Bill_Zissimopoulos · October 31, 2016, 7:41pm

Ok, I am at a loss. The test now fails under NTFS with FILE_FLAG_NO_BUFFERING under all circumstances.

I am going to call a timeout on this thread, because I think I may be wasting everyone’s time. Please allow me a couple of days to examine this with a clear head and report back.

Thanks everyone for your help so far.

Bill

anatoly_mikhailov · October 31, 2016, 10:28pm

> Last I looked, Mm knows if the file backing the section is on the
network, or not, and if it is on the network it will eventually give up and
discard the data.

That’s right. Mm consider if the file is on the network. If so then Mm
marks it to itself to fail all I/O if there was at least one write error.
Except for STATUS_FILE_LOCK_CONFLICT. For this status Mm will try to flush
data on remote file again later. If Mm gets STATUS_FILE_INVALID or
STATUS_MEDIA_WRITE_PROTECTED statuses on write, the Mm will fail all
further I/O on file whether it’s remote or not.

2016-11-01 2:40 GMT+03:00 :

> Ok, I am at a loss. The test now fails under NTFS with
> FILE_FLAG_NO_BUFFERING under all circumstances.
>
> I am going to call a timeout on this thread, because I think I may be
> wasting everyone’s time. Please allow me a couple of days to examine this
> with a clear head and report back.
>
> Thanks everyone for your help so far.
>
> Bill
>
>
> —
> NTFSD is sponsored by OSR
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer>
></http:>

anatoly_mikhailov · October 31, 2016, 10:40pm

> So I have no clue why the WRITE IRP’s would fail with
STATUS_INVALID_PARAMETER.

You opened file with FILE_FLAG_NO_BUFFERING. That means you can read or
write file only on sectors boundary. Also reading or writing size must be
of multiple sectors. Otherwise you’ll get STATUS_INVALID_PARAMETER error.
The log says there was attempt to write 20064 bytes. It’s not multiple of
512 byte sectors. It’s not multiple of 4096 byte sectors also.

Try to map file which has exactly 4096 byte length to see what changes. I
think it’ll be written ok.

anatoly_mikhailov · November 1, 2016, 3:24am

By the way. Since you have seen test 2 succeeded against NTFS then don’t be
confused. Likely remote file was opened with no FILE_FLAG_NO_BUFFERING set
before opening the file with the flag. And if it’s so, there could be file
reading and memory manager referenced the opened file instance with no the
flag. Therefore when you opened the file with the flag set and did tests,
writing was ok because memory manager was using to I/O not your opened file
instance but previous one.

Bill_Zissimopoulos · November 1, 2016, 5:07pm

Here is an update. What I say below applies only to the discussed test when run with FILE_FLAG_NO_BUFFERING against NTFS via network share. My FSD is *not* involved at all.

On Win8 using \localhost: in all test runs this morning the test failed. Here is a typical run:

C:\NtfsTest>\Users\billziss\Downloads\winfsp\winfsp-tests-x64.exe --ntfs --resilient --share=winfsp-tests-share=C:\ rdwr_mmap_test
rdwr_mmap_test… KO
ASSERT(*Bgn == (rand() & 0xff)) failed at rdwr-test.c:585:rdwr_mmap_dotest

On Win10 using \localhost: in all test runs this morning the test succeeded. Here is a typical run:

C:\NtfsTest>\Users\billziss\Projects\winfsp\build\VStudio\build\Debug\winfsp-tests-x64.exe --ntfs --resilient --share=winfsp-tests-share=C:\ rdwr_mmap_test
rdwr_mmap_test… OK 1s
— COMPLETE —

From Win10 to Win8 using a \win8dbg\users share: in all test runs this morning the test succeeded. Here is a typical run:

C:\>\Users\billziss\Projects\winfsp\build\VStudio\build\Debug\winfsp-tests-x64.exe --ntfs --resilient --share=\win8dbg\users\billziss\downloads\winfsp rdwr_mmap_test
rdwr_mmap_test… OK 2s
— COMPLETE —

Full FileSpy logs for all cases are available upon request.

This suggests to me that there is indeed an issue with the Win8 OS when using the \localhost share. This issue does not have anything to do with my FSD, and can be exhibited against NTFS.

OS version: 6.2 Build 9200

Bill

Bill_Zissimopoulos · November 1, 2016, 5:31pm

A final note. My FSD exhibits the same behavior as NTFS:

FAILURE on Win8 when using \localhost.
SUCCESS on Win10 when using \localhost.
SUCCESS on Win10 to Win8 using a \win8dbg\users share.

I think I can safely say that this is a bug in Win8 (version 6.2 Build 9200).

Bill

CcFlushCache under SRV2

Call Site

Call Site