VM-entry failure due to invalid guest state.

Sergey_Pisarev · July 27, 2018, 8:02pm

Hello !

This is cpu related question and not OS related one. But since some people here do virtualisation software development I hope it’s still ok to post here.

I am developing driver that converts currently running OS to run under hypervisor(something like SimpleVisor). After vmlaunch cpu jumps to vm-exit dispatcher with exit reason 0x80000021 (VM-entry failure due to invalid guest state). I have checked everything in 26.3.1 Checks on the Guest State Area and fixed all the errors I have found. I am still getting this error. Below caps of my cpu and values in guest area. Maybe someone can glance over it and notice something invalid.

IA32_VMX_BASIC (0x480) hex:0x00da0400`0000000f bin:00000000 11011010 00000100 00000000 00000000 00000000 00000000 00001111

IA32_VMX_CR0_FIXED0 (0x486) hex:0x80000021 bin:10000000 00000000 00000000 00100001
IA32_VMX_CR0_FIXED1 (0x487) hex:0xffffffff bin:11111111 11111111 11111111 11111111
guest_cr0 (0x6800) hex:0x80050031 bin:10000000 00000101 00000000 00110001

IA32_VMX_CR4_FIXED0 (0x488) hex:0x2000 bin:00000000 00000000 00100000 00000000
IA32_VMX_CR4_FIXED1 (0x489) hex:0x227ff bin:00000000 00000010 00100111 11111111
guest_cr4 (0x6804) hex:0x26f8 bin:00000000 00000000 00100110 11111000

ia32_vmx_true_pinbased_ctls(0x48D) hex:0x0000007f`00000016 bin:00000000 00000000 00000000 01111111 00000000 00000000 00000000 00010110
pin_based_vm_execution_controls(0x4000) hex:0x16 bin:00000000 00000000 00000000 00010110

ia32_vmx_true_procbased_ctls(0x48e) hex:0xfff9fffe`04006172 bin:11111111 11111001 11111111 11111110 00000100 00000000 01100001 01110010
primary_processor_based_vm_execution_controls(0x4002) hex:0x94006172 bin:10010100 00000000 01100001 01110010

ia32_vmx_procbased_ctls2(0x48b) hex:0x000000ff`00000000 bin:00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000
secondary_processor_based_vm_execution_controls(0x401e) hex:0xaa bin:00000000 00000000 00000000 10101010

ia32_vmx_true_exit_ctls(0x48f) hex:0x007fffff`00036dfb bin:00000000 01111111 11111111 11111111 00000000 00000011 01101101 11111011
vm_exit_controls(0x400c) hex:0x3efff bin:00000000 00000011 11101111 11111111

ia32_vmx_true_entry_ctls(0x490) hex:0x0000ffff`000011fb bin:00000000 00000000 11111111 11111111 00000000 00000000 00010001 11111011
vm_entry_controls(0x4012) hex:0x13ff bin:00000000 00000000 00010011 11111111

guest_cr3(0x6802) hex:0x1aa000

guest_dr7 (0x681a) hex:0x400
guest_rflags (0x6820) hex:0x286 bin:00000000 00000000 00000000 00000000 00000000 00000000 00000010 10000110

guest_rsp(0x681c) hex:ffff8202cf325858
guest_rip(0x681e) hex:fffff80116c61058

guest_cs_selector(0x802) hex:0x10 bin:00000000 00010000
guest_cs_base(0x6808) hex:0
guest_cs_limit(0x4802) hex:0
guest_cs_access_rights(0x4816) hex:0x209b bin:00000000 00000000 00100000 10011011

guest_es_selector(0x800) hex:000000000000002b bin:00000000 00101011
guest_es_base(0x6806) hex:0
guest_es_limit(0x4800) hex:00000000ffffffff
guest_es_access_rights(0x4814) hex:0xcff3 bin:00000000 00000000 11001111 11110011

guest_ss_selector(0x804) hex:0x18 bin:00000000 00011000
guest_ss_base(0x680a) hex:0
guest_ss_limit(0x4804) hex:0
guest_ss_access_rights(0x4818) hex:0x4093 bin:00000000 00000000 01000000 10010011

guest_ds_selector(0x806) hex:0x2b bin:00000000 00101011
guest_ds_base(0x680c) hex:0
guest_ds_limit(0x4806) hex:0xffffffff
guest_ds_access_rights(0x481a) hex:0xcff3 bin:00000000 00000000 11001111 11110011

guest_fs_selector(0x808) hex:0x53 bin:00000000 00000000 00000000 01010011
guest_fs_base(0x680e) hex:0
guest_fs_limit(0x4808) hex:0x3c00
guest_fs_access_rights(0x481c) hex:0x40f3 bin:00000000 00000000 01000000 11110011

guest_gs_selector(0x80a) hex:0x2b bin:00000000 00101011
guest_gs_base(0x6810) hex:fffff8024b822000
guest_gs_limit(0x480a) hex:0xffffffff
guest_gs_access_rights(0x481e) hex:0xcff3 bin:00000000 00000000 11001111 11110011

guest_gdtr_base(0x6816) hex:fffff8024ea53fb0
guest_gdtr_limit(0x4810) hex:0x57

guest_ldtr_selector(0x80c) hex:0
guest_ldtr_limit(0x480c) hex:0
guest_ldtr_base(0x6812) hex:0
guest_ldtr_access_rights(0x4820) hex:0x10000 bin:00000000 00000001 00000000 00000000

guest_tr_selector(0x80e) hex:0x40 bin:00000000 01000000
guest_tr_limit(0x480e) hex:0x67
guest_tr_base(0x6814) hex:fffff8024ea52000
guest_tr_access_rights(0x4822) hex:0x8b bin:00000000 00000000 00000000 10001011

guest_idtr_limit(0x4812) hex:0xfff
guest_idtr_base(0x6818) hex:fffff8024ea51000

guest_ia32_debugctl(0x2802) hex:0
guest_ia32_sysenter_cs(0x482a) hex:0
guest_ia32_sysenter_esp(0x6824) hex:0
guest_ia32_sysenter_eip(0x6826) hex:0

exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001
exit_qualification(0x6400) hex:0

Mike_Larkin · July 28, 2018, 11:30am

On Sat, Jul 28, 2018 at 03:02:43AM +0300, xxxxx@gmail.com wrote:

Hello !

This is cpu related question and not OS related one. But since some people here do virtualisation software development I hope itâ€™s still ok to post here.

I am developing driver that converts currently running OS to run under hypervisor(something like SimpleVisor). After vmlaunch cpu jumps to vm-exit dispatcher with exit reason 0x80000021 (VM-entry failure due to invalid guest state). I have checked everything in 26.3.1 Checks on the Guest State Area and fixed all the errors I have found. I am still getting this error. Below caps of my cpu and values in guest area. Maybe someone can glance over it and notice something invalid.

What about EFER settings (and the rules associated with IA32e_MODE_GUEST setting)?

-ml

IA32_VMX_BASIC (0x480) hex:0x00da0400`0000000f bin:00000000 11011010 00000100 00000000 00000000 00000000 00000000 00001111

IA32_VMX_CR0_FIXED0 (0x486) hex:0x80000021 bin:10000000 00000000 00000000 00100001
IA32_VMX_CR0_FIXED1 (0x487) hex:0xffffffff bin:11111111 11111111 11111111 11111111
guest_cr0 (0x6800) hex:0x80050031 bin:10000000 00000101 00000000 00110001

IA32_VMX_CR4_FIXED0 (0x488) hex:0x2000 bin:00000000 00000000 00100000 00000000
IA32_VMX_CR4_FIXED1 (0x489) hex:0x227ff bin:00000000 00000010 00100111 11111111
guest_cr4 (0x6804) hex:0x26f8 bin:00000000 00000000 00100110 11111000

ia32_vmx_true_pinbased_ctls(0x48D) hex:0x0000007f`00000016 bin:00000000 00000000 00000000 01111111 00000000 00000000 00000000 00010110
pin_based_vm_execution_controls(0x4000) hex:0x16 bin:00000000 00000000 00000000 00010110

ia32_vmx_true_procbased_ctls(0x48e) hex:0xfff9fffe`04006172 bin:11111111 11111001 11111111 11111110 00000100 00000000 01100001 01110010
primary_processor_based_vm_execution_controls(0x4002) hex:0x94006172 bin:10010100 00000000 01100001 01110010

ia32_vmx_procbased_ctls2(0x48b) hex:0x000000ff`00000000 bin:00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000
secondary_processor_based_vm_execution_controls(0x401e) hex:0xaa bin:00000000 00000000 00000000 10101010

ia32_vmx_true_exit_ctls(0x48f) hex:0x007fffff`00036dfb bin:00000000 01111111 11111111 11111111 00000000 00000011 01101101 11111011
vm_exit_controls(0x400c) hex:0x3efff bin:00000000 00000011 11101111 11111111

ia32_vmx_true_entry_ctls(0x490) hex:0x0000ffff`000011fb bin:00000000 00000000 11111111 11111111 00000000 00000000 00010001 11111011
vm_entry_controls(0x4012) hex:0x13ff bin:00000000 00000000 00010011 11111111

guest_cr3(0x6802) hex:0x1aa000

guest_dr7 (0x681a) hex:0x400
guest_rflags (0x6820) hex:0x286 bin:00000000 00000000 00000000 00000000 00000000 00000000 00000010 10000110

guest_rsp(0x681c) hex:ffff8202cf325858
guest_rip(0x681e) hex:fffff80116c61058

guest_cs_selector(0x802) hex:0x10 bin:00000000 00010000
guest_cs_base(0x6808) hex:0
guest_cs_limit(0x4802) hex:0
guest_cs_access_rights(0x4816) hex:0x209b bin:00000000 00000000 00100000 10011011

guest_es_selector(0x800) hex:000000000000002b bin:00000000 00101011
guest_es_base(0x6806) hex:0
guest_es_limit(0x4800) hex:00000000ffffffff
guest_es_access_rights(0x4814) hex:0xcff3 bin:00000000 00000000 11001111 11110011

guest_ss_selector(0x804) hex:0x18 bin:00000000 00011000
guest_ss_base(0x680a) hex:0
guest_ss_limit(0x4804) hex:0
guest_ss_access_rights(0x4818) hex:0x4093 bin:00000000 00000000 01000000 10010011

guest_ds_selector(0x806) hex:0x2b bin:00000000 00101011
guest_ds_base(0x680c) hex:0
guest_ds_limit(0x4806) hex:0xffffffff
guest_ds_access_rights(0x481a) hex:0xcff3 bin:00000000 00000000 11001111 11110011

guest_fs_selector(0x808) hex:0x53 bin:00000000 00000000 00000000 01010011
guest_fs_base(0x680e) hex:0
guest_fs_limit(0x4808) hex:0x3c00
guest_fs_access_rights(0x481c) hex:0x40f3 bin:00000000 00000000 01000000 11110011

guest_gs_selector(0x80a) hex:0x2b bin:00000000 00101011
guest_gs_base(0x6810) hex:fffff8024b822000
guest_gs_limit(0x480a) hex:0xffffffff
guest_gs_access_rights(0x481e) hex:0xcff3 bin:00000000 00000000 11001111 11110011

guest_gdtr_base(0x6816) hex:fffff8024ea53fb0
guest_gdtr_limit(0x4810) hex:0x57

guest_ldtr_selector(0x80c) hex:0
guest_ldtr_limit(0x480c) hex:0
guest_ldtr_base(0x6812) hex:0
guest_ldtr_access_rights(0x4820) hex:0x10000 bin:00000000 00000001 00000000 00000000

guest_tr_selector(0x80e) hex:0x40 bin:00000000 01000000
guest_tr_limit(0x480e) hex:0x67
guest_tr_base(0x6814) hex:fffff8024ea52000
guest_tr_access_rights(0x4822) hex:0x8b bin:00000000 00000000 00000000 10001011

guest_idtr_limit(0x4812) hex:0xfff
guest_idtr_base(0x6818) hex:fffff8024ea51000

guest_ia32_debugctl(0x2802) hex:0
guest_ia32_sysenter_cs(0x482a) hex:0
guest_ia32_sysenter_esp(0x6824) hex:0
guest_ia32_sysenter_eip(0x6826) hex:0

exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001
exit_qualification(0x6400) hex:0

NTDEV is sponsored by OSR

Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

Sergey_Pisarev · July 28, 2018, 11:47am

Load IA32_EFER bit is clear in vm-entry controls, and EFER field is not initialised (must be zero).
IA-32e mode guest bit is set.
According to manual EFER checks performed only if load EFER bit is set.

Forgot to say I am testing on win10x64

On 28 Jul 2018, at 18:30, xxxxx@azathoth.net wrote:
>
> On Sat, Jul 28, 2018 at 03:02:43AM +0300, xxxxx@gmail.com mailto:xxxxx wrote:
>> Hello !
>>
>> This is cpu related question and not OS related one. But since some people here do virtualisation software development I hope it’s still ok to post here.
>>
>> I am developing driver that converts currently running OS to run under hypervisor(something like SimpleVisor). After vmlaunch cpu jumps to vm-exit dispatcher with exit reason 0x80000021 (VM-entry failure due to invalid guest state). I have checked everything in 26.3.1 Checks on the Guest State Area and fixed all the errors I have found. I am still getting this error. Below caps of my cpu and values in guest area. Maybe someone can glance over it and notice something invalid.
>>
>
> What about EFER settings (and the rules associated with IA32e_MODE_GUEST setting)?
>
> -ml
>
>> IA32_VMX_BASIC (0x480) hex:0x00da04000000000f bin:00000000 11011010 00000100 00000000 00000000 00000000 00000000 00001111 >> >> >> IA32_VMX_CR0_FIXED0 (0x486) hex:0x80000021 bin:10000000 00000000 00000000 00100001 >> IA32_VMX_CR0_FIXED1 (0x487) hex:0xffffffff bin:11111111 11111111 11111111 11111111 >> guest_cr0 (0x6800) hex:0x80050031 bin:10000000 00000101 00000000 00110001 >> >> IA32_VMX_CR4_FIXED0 (0x488) hex:0x2000 bin:00000000 00000000 00100000 00000000 >> IA32_VMX_CR4_FIXED1 (0x489) hex:0x227ff bin:00000000 00000010 00100111 11111111 >> guest_cr4 (0x6804) hex:0x26f8 bin:00000000 00000000 00100110 11111000 >> >> ia32_vmx_true_pinbased_ctls(0x48D) hex:0x0000007f00000016 bin:00000000 00000000 00000000 01111111 00000000 00000000 00000000 00010110
>> pin_based_vm_execution_controls(0x4000) hex:0x16 bin:00000000 00000000 00000000 00010110
>>
>> ia32_vmx_true_procbased_ctls(0x48e) hex:0xfff9fffe04006172 bin:11111111 11111001 11111111 11111110 00000100 00000000 01100001 01110010 >> primary_processor_based_vm_execution_controls(0x4002) hex:0x94006172 bin:10010100 00000000 01100001 01110010 >> >> ia32_vmx_procbased_ctls2(0x48b) hex:0x000000ff00000000 bin:00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000
>> secondary_processor_based_vm_execution_controls(0x401e) hex:0xaa bin:00000000 00000000 00000000 10101010
>>
>> ia32_vmx_true_exit_ctls(0x48f) hex:0x007fffff00036dfb bin:00000000 01111111 11111111 11111111 00000000 00000011 01101101 11111011 >> vm_exit_controls(0x400c) hex:0x3efff bin:00000000 00000011 11101111 11111111 >> >> ia32_vmx_true_entry_ctls(0x490) hex:0x0000ffff000011fb bin:00000000 00000000 11111111 11111111 00000000 00000000 00010001 11111011
>> vm_entry_controls(0x4012) hex:0x13ff bin:00000000 00000000 00010011 11111111
>>
>> guest_cr3(0x6802) hex:0x1aa000
>>
>> guest_dr7 (0x681a) hex:0x400
>> guest_rflags (0x6820) hex:0x286 bin:00000000 00000000 00000000 00000000 00000000 00000000 00000010 10000110
>>
>> guest_rsp(0x681c) hex:ffff8202cf325858
>> guest_rip(0x681e) hex:fffff80116c61058
>>
>> guest_cs_selector(0x802) hex:0x10 bin:00000000 00010000
>> guest_cs_base(0x6808) hex:0
>> guest_cs_limit(0x4802) hex:0
>> guest_cs_access_rights(0x4816) hex:0x209b bin:00000000 00000000 00100000 10011011
>>
>> guest_es_selector(0x800) hex:000000000000002b bin:00000000 00101011
>> guest_es_base(0x6806) hex:0
>> guest_es_limit(0x4800) hex:00000000ffffffff
>> guest_es_access_rights(0x4814) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>
>> guest_ss_selector(0x804) hex:0x18 bin:00000000 00011000
>> guest_ss_base(0x680a) hex:0
>> guest_ss_limit(0x4804) hex:0
>> guest_ss_access_rights(0x4818) hex:0x4093 bin:00000000 00000000 01000000 10010011
>>
>> guest_ds_selector(0x806) hex:0x2b bin:00000000 00101011
>> guest_ds_base(0x680c) hex:0
>> guest_ds_limit(0x4806) hex:0xffffffff
>> guest_ds_access_rights(0x481a) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>
>> guest_fs_selector(0x808) hex:0x53 bin:00000000 00000000 00000000 01010011
>> guest_fs_base(0x680e) hex:0
>> guest_fs_limit(0x4808) hex:0x3c00
>> guest_fs_access_rights(0x481c) hex:0x40f3 bin:00000000 00000000 01000000 11110011
>>
>> guest_gs_selector(0x80a) hex:0x2b bin:00000000 00101011
>> guest_gs_base(0x6810) hex:fffff8024b822000
>> guest_gs_limit(0x480a) hex:0xffffffff
>> guest_gs_access_rights(0x481e) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>
>> guest_gdtr_base(0x6816) hex:fffff8024ea53fb0
>> guest_gdtr_limit(0x4810) hex:0x57
>>
>> guest_ldtr_selector(0x80c) hex:0
>> guest_ldtr_limit(0x480c) hex:0
>> guest_ldtr_base(0x6812) hex:0
>> guest_ldtr_access_rights(0x4820) hex:0x10000 bin:00000000 00000001 00000000 00000000
>>
>> guest_tr_selector(0x80e) hex:0x40 bin:00000000 01000000
>> guest_tr_limit(0x480e) hex:0x67
>> guest_tr_base(0x6814) hex:fffff8024ea52000
>> guest_tr_access_rights(0x4822) hex:0x8b bin:00000000 00000000 00000000 10001011
>>
>> guest_idtr_limit(0x4812) hex:0xfff
>> guest_idtr_base(0x6818) hex:fffff8024ea51000
>>
>> guest_ia32_debugctl(0x2802) hex:0
>> guest_ia32_sysenter_cs(0x482a) hex:0
>> guest_ia32_sysenter_esp(0x6824) hex:0
>> guest_ia32_sysenter_eip(0x6826) hex:0
>>
>> exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001
>> exit_qualification(0x6400) hex:0
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: http:>
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
>> Details at http:>
>>
>> To unsubscribe, visit the List Server section of OSR Online at http:>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:>
>
> To unsubscribe, visit the List Server section of OSR Online at http:></http:></http:></http:></http:></http:></http:></mailto:xxxxx>

Sergey_Pisarev · July 28, 2018, 2:08pm

The problem is misconfigured GDT entry. To be precise flag System set incorrectly in some segment descriptors. For example it is not set for descriptor referenced by cs.

kd> r cs
cs=0010

kd> r gdtr
gdtr=fffff80144c53fb0

kd> ?(0xfffff80144c53fb0 + 0x10)
Evaluate expression: -8790644277312 = fffff801`44c53fc0

kd> dt -r nt!_KGDTENTRY64 0xfffff80144c53fc0 +0x000 LimitLow : 0 +0x002 BaseLow : 0 +0x004 Bytes : <unnamed-tag> +0x000 BaseMiddle : 0 '' +0x001 Flags1 : 0x9b '' +0x002 Flags2 : 0x20 ' ' +0x003 BaseHigh : 0 '' +0x004 Bits : <unnamed-tag> +0x000 BaseMiddle : 0y00000000 (0) +0x000 Type : 0y11011 (0x1b) +0x000 Dpl : 0y00 +0x000 Present : 0y1 +0x000 LimitHigh : 0y0000 +0x000 System : 0y0 +0x000 LongMode : 0y1 +0x000 DefaultBig : 0y0 +0x000 Granularity : 0y0 +0x000 BaseHigh : 0y00000000 (0) +0x008 BaseUpper : 0 +0x00c MustBeZero : 0x409300 +0x000 DataLow : 0n9177623557046272 +0x008 DataHigh : 0n18176026718765056 And here is the note I have found in SimpleVisor sources: // NOTE: The Windows definition of KGDTENTRY64 is WRONG. The "System" field // is incorrectly defined at the position of where the AVL bit should be. // The actual location of the SYSTEM bit is encoded as the highest bit in // the "Type" field. However S flag is set correctly if I use lar instruction instead of parsing gdt by hand (thats what I did before). > On 28 Jul 2018, at 18:48, Sergey Pisarev <xxxxx> wrote: > > Load IA32_EFER bit is clear in vm-entry controls, and EFER field is not initialised (must be zero). > IA-32e mode guest bit is set. > According to manual EFER checks performed only if load EFER bit is set. > > Forgot to say I am testing on win10x64 > >> On 28 Jul 2018, at 18:30, xxxxx@azathoth.net <mailto:xxxxx> <xxxxx>> wrote: >> >> On Sat, Jul 28, 2018 at 03:02:43AM +0300, xxxxx@gmail.com <mailto:xxxxx> wrote: >>> Hello ! >>> >>> This is cpu related question and not OS related one. But since some people here do virtualisation software development I hope it’s still ok to post here. >>> >>> I am developing driver that converts currently running OS to run under hypervisor(something like SimpleVisor). After vmlaunch cpu jumps to vm-exit dispatcher with exit reason 0x80000021 (VM-entry failure due to invalid guest state). I have checked everything in 26.3.1 Checks on the Guest State Area and fixed all the errors I have found. I am still getting this error. Below caps of my cpu and values in guest area. Maybe someone can glance over it and notice something invalid. >>> >> >> What about EFER settings (and the rules associated with IA32e_MODE_GUEST setting)? >> >> -ml >> >>> IA32_VMX_BASIC (0x480) hex:0x00da04000000000f bin:00000000 11011010 00000100 00000000 00000000 00000000 00000000 00001111
>>>
>>>
>>> IA32_VMX_CR0_FIXED0 (0x486) hex:0x80000021 bin:10000000 00000000 00000000 00100001
>>> IA32_VMX_CR0_FIXED1 (0x487) hex:0xffffffff bin:11111111 11111111 11111111 11111111
>>> guest_cr0 (0x6800) hex:0x80050031 bin:10000000 00000101 00000000 00110001
>>>
>>> IA32_VMX_CR4_FIXED0 (0x488) hex:0x2000 bin:00000000 00000000 00100000 00000000
>>> IA32_VMX_CR4_FIXED1 (0x489) hex:0x227ff bin:00000000 00000010 00100111 11111111
>>> guest_cr4 (0x6804) hex:0x26f8 bin:00000000 00000000 00100110 11111000
>>>
>>> ia32_vmx_true_pinbased_ctls(0x48D) hex:0x0000007f00000016 bin:00000000 00000000 00000000 01111111 00000000 00000000 00000000 00010110 >>> pin_based_vm_execution_controls(0x4000) hex:0x16 bin:00000000 00000000 00000000 00010110 >>> >>> ia32_vmx_true_procbased_ctls(0x48e) hex:0xfff9fffe04006172 bin:11111111 11111001 11111111 11111110 00000100 00000000 01100001 01110010
>>> primary_processor_based_vm_execution_controls(0x4002) hex:0x94006172 bin:10010100 00000000 01100001 01110010
>>>
>>> ia32_vmx_procbased_ctls2(0x48b) hex:0x000000ff00000000 bin:00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000 >>> secondary_processor_based_vm_execution_controls(0x401e) hex:0xaa bin:00000000 00000000 00000000 10101010 >>> >>> ia32_vmx_true_exit_ctls(0x48f) hex:0x007fffff00036dfb bin:00000000 01111111 11111111 11111111 00000000 00000011 01101101 11111011
>>> vm_exit_controls(0x400c) hex:0x3efff bin:00000000 00000011 11101111 11111111
>>>
>>> ia32_vmx_true_entry_ctls(0x490) hex:0x0000ffff`000011fb bin:00000000 00000000 11111111 11111111 00000000 00000000 00010001 11111011
>>> vm_entry_controls(0x4012) hex:0x13ff bin:00000000 00000000 00010011 11111111
>>>
>>> guest_cr3(0x6802) hex:0x1aa000
>>>
>>> guest_dr7 (0x681a) hex:0x400
>>> guest_rflags (0x6820) hex:0x286 bin:00000000 00000000 00000000 00000000 00000000 00000000 00000010 10000110
>>>
>>> guest_rsp(0x681c) hex:ffff8202cf325858
>>> guest_rip(0x681e) hex:fffff80116c61058
>>>
>>> guest_cs_selector(0x802) hex:0x10 bin:00000000 00010000
>>> guest_cs_base(0x6808) hex:0
>>> guest_cs_limit(0x4802) hex:0
>>> guest_cs_access_rights(0x4816) hex:0x209b bin:00000000 00000000 00100000 10011011
>>>
>>> guest_es_selector(0x800) hex:000000000000002b bin:00000000 00101011
>>> guest_es_base(0x6806) hex:0
>>> guest_es_limit(0x4800) hex:00000000ffffffff
>>> guest_es_access_rights(0x4814) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>>
>>> guest_ss_selector(0x804) hex:0x18 bin:00000000 00011000
>>> guest_ss_base(0x680a) hex:0
>>> guest_ss_limit(0x4804) hex:0
>>> guest_ss_access_rights(0x4818) hex:0x4093 bin:00000000 00000000 01000000 10010011
>>>
>>> guest_ds_selector(0x806) hex:0x2b bin:00000000 00101011
>>> guest_ds_base(0x680c) hex:0
>>> guest_ds_limit(0x4806) hex:0xffffffff
>>> guest_ds_access_rights(0x481a) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>>
>>> guest_fs_selector(0x808) hex:0x53 bin:00000000 00000000 00000000 01010011
>>> guest_fs_base(0x680e) hex:0
>>> guest_fs_limit(0x4808) hex:0x3c00
>>> guest_fs_access_rights(0x481c) hex:0x40f3 bin:00000000 00000000 01000000 11110011
>>>
>>> guest_gs_selector(0x80a) hex:0x2b bin:00000000 00101011
>>> guest_gs_base(0x6810) hex:fffff8024b822000
>>> guest_gs_limit(0x480a) hex:0xffffffff
>>> guest_gs_access_rights(0x481e) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>>
>>> guest_gdtr_base(0x6816) hex:fffff8024ea53fb0
>>> guest_gdtr_limit(0x4810) hex:0x57
>>>
>>> guest_ldtr_selector(0x80c) hex:0
>>> guest_ldtr_limit(0x480c) hex:0
>>> guest_ldtr_base(0x6812) hex:0
>>> guest_ldtr_access_rights(0x4820) hex:0x10000 bin:00000000 00000001 00000000 00000000
>>>
>>> guest_tr_selector(0x80e) hex:0x40 bin:00000000 01000000
>>> guest_tr_limit(0x480e) hex:0x67
>>> guest_tr_base(0x6814) hex:fffff8024ea52000
>>> guest_tr_access_rights(0x4822) hex:0x8b bin:00000000 00000000 00000000 10001011
>>>
>>> guest_idtr_limit(0x4812) hex:0xfff
>>> guest_idtr_base(0x6818) hex:fffff8024ea51000
>>>
>>> guest_ia32_debugctl(0x2802) hex:0
>>> guest_ia32_sysenter_cs(0x482a) hex:0
>>> guest_ia32_sysenter_esp(0x6824) hex:0
>>> guest_ia32_sysenter_eip(0x6826) hex:0
>>>
>>> exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001
>>> exit_qualification(0x6400) hex:0
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> Visit the list online at: http:>
>>>
>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
>>> Details at http:>
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at http:>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: http:>
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
>> Details at http:>
>>
>> To unsubscribe, visit the List Server section of OSR Online at http:>
></http:></http:></http:></http:></http:></http:></mailto:xxxxx></mailto:xxxxx>

Mike_Larkin · July 30, 2018, 2:35am

On Sat, Jul 28, 2018 at 09:09:16PM +0300, xxxxx@gmail.com wrote:

The problem is misconfigured GDT entry. To be precise flag System set incorrectly in some segment descriptors. For example it is not set for descriptor referenced by cs.

Nice find, thanks for sharing the result of your diagnostic.

-ml

kd> r cs
cs=0010

kd> r gdtr
gdtr=fffff80144c53fb0

kd> ?(0xfffff80144c53fb0 + 0x10)
Evaluate expression: -8790644277312 = fffff801`44c53fc0

kd> dt -r nt!_KGDTENTRY64 0xfffff80144c53fc0 +0x000 LimitLow : 0 +0x002 BaseLow : 0 +0x004 Bytes : <unnamed-tag> > +0x000 BaseMiddle : 0 '' > +0x001 Flags1 : 0x9b '' > +0x002 Flags2 : 0x20 ' ' > +0x003 BaseHigh : 0 '' > +0x004 Bits : <unnamed-tag> > +0x000 BaseMiddle : 0y00000000 (0) > +0x000 Type : 0y11011 (0x1b) > +0x000 Dpl : 0y00 > +0x000 Present : 0y1 > +0x000 LimitHigh : 0y0000 > +0x000 System : 0y0 > +0x000 LongMode : 0y1 > +0x000 DefaultBig : 0y0 > +0x000 Granularity : 0y0 > +0x000 BaseHigh : 0y00000000 (0) > +0x008 BaseUpper : 0 > +0x00c MustBeZero : 0x409300 > +0x000 DataLow : 0n9177623557046272 > +0x008 DataHigh : 0n18176026718765056 > > And here is the note I have found in SimpleVisor sources: > // NOTE: The Windows definition of KGDTENTRY64 is WRONG. The "System" field > // is incorrectly defined at the position of where the AVL bit should be. > // The actual location of the SYSTEM bit is encoded as the highest bit in > // the "Type" field. > > However S flag is set correctly if I use lar instruction instead of parsing gdt by hand (thats what I did before). > > > > On 28 Jul 2018, at 18:48, Sergey Pisarev <xxxxx> wrote: > > > > Load IA32_EFER bit is clear in vm-entry controls, and EFER field is not initialised (must be zero). > > IA-32e mode guest bit is set. > > According to manual EFER checks performed only if load EFER bit is set. > > > > Forgot to say I am testing on win10x64 > > > >> On 28 Jul 2018, at 18:30, xxxxx@azathoth.net <mailto:xxxxx> <xxxxx>> wrote: > >> > >> On Sat, Jul 28, 2018 at 03:02:43AM +0300, xxxxx@gmail.com <mailto:xxxxx> wrote: > >>> Hello ! > >>> > >>> This is cpu related question and not OS related one. But since some people here do virtualisation software development I hope itâ€™s still ok to post here. > >>> > >>> I am developing driver that converts currently running OS to run under hypervisor(something like SimpleVisor). After vmlaunch cpu jumps to vm-exit dispatcher with exit reason 0x80000021 (VM-entry failure due to invalid guest state). I have checked everything in 26.3.1 Checks on the Guest State Area and fixed all the errors I have found. I am still getting this error. Below caps of my cpu and values in guest area. Maybe someone can glance over it and notice something invalid. > >>> > >> > >> What about EFER settings (and the rules associated with IA32e_MODE_GUEST setting)? > >> > >> -ml > >> > >>> IA32_VMX_BASIC (0x480) hex:0x00da04000000000f bin:00000000 11011010 00000100 00000000 00000000 00000000 00000000 00001111
> >>>
> >>>
> >>> IA32_VMX_CR0_FIXED0 (0x486) hex:0x80000021 bin:10000000 00000000 00000000 00100001
> >>> IA32_VMX_CR0_FIXED1 (0x487) hex:0xffffffff bin:11111111 11111111 11111111 11111111
> >>> guest_cr0 (0x6800) hex:0x80050031 bin:10000000 00000101 00000000 00110001
> >>>
> >>> IA32_VMX_CR4_FIXED0 (0x488) hex:0x2000 bin:00000000 00000000 00100000 00000000
> >>> IA32_VMX_CR4_FIXED1 (0x489) hex:0x227ff bin:00000000 00000010 00100111 11111111
> >>> guest_cr4 (0x6804) hex:0x26f8 bin:00000000 00000000 00100110 11111000
> >>>
> >>> ia32_vmx_true_pinbased_ctls(0x48D) hex:0x0000007f00000016 bin:00000000 00000000 00000000 01111111 00000000 00000000 00000000 00010110 > >>> pin_based_vm_execution_controls(0x4000) hex:0x16 bin:00000000 00000000 00000000 00010110 > >>> > >>> ia32_vmx_true_procbased_ctls(0x48e) hex:0xfff9fffe04006172 bin:11111111 11111001 11111111 11111110 00000100 00000000 01100001 01110010
> >>> primary_processor_based_vm_execution_controls(0x4002) hex:0x94006172 bin:10010100 00000000 01100001 01110010
> >>>
> >>> ia32_vmx_procbased_ctls2(0x48b) hex:0x000000ff00000000 bin:00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000 > >>> secondary_processor_based_vm_execution_controls(0x401e) hex:0xaa bin:00000000 00000000 00000000 10101010 > >>> > >>> ia32_vmx_true_exit_ctls(0x48f) hex:0x007fffff00036dfb bin:00000000 01111111 11111111 11111111 00000000 00000011 01101101 11111011
> >>> vm_exit_controls(0x400c) hex:0x3efff bin:00000000 00000011 11101111 11111111
> >>>
> >>> ia32_vmx_true_entry_ctls(0x490) hex:0x0000ffff`000011fb bin:00000000 00000000 11111111 11111111 00000000 00000000 00010001 11111011
> >>> vm_entry_controls(0x4012) hex:0x13ff bin:00000000 00000000 00010011 11111111
> >>>
> >>> guest_cr3(0x6802) hex:0x1aa000
> >>>
> >>> guest_dr7 (0x681a) hex:0x400
> >>> guest_rflags (0x6820) hex:0x286 bin:00000000 00000000 00000000 00000000 00000000 00000000 00000010 10000110
> >>>
> >>> guest_rsp(0x681c) hex:ffff8202cf325858
> >>> guest_rip(0x681e) hex:fffff80116c61058
> >>>
> >>> guest_cs_selector(0x802) hex:0x10 bin:00000000 00010000
> >>> guest_cs_base(0x6808) hex:0
> >>> guest_cs_limit(0x4802) hex:0
> >>> guest_cs_access_rights(0x4816) hex:0x209b bin:00000000 00000000 00100000 10011011
> >>>
> >>> guest_es_selector(0x800) hex:000000000000002b bin:00000000 00101011
> >>> guest_es_base(0x6806) hex:0
> >>> guest_es_limit(0x4800) hex:00000000ffffffff
> >>> guest_es_access_rights(0x4814) hex:0xcff3 bin:00000000 00000000 11001111 11110011
> >>>
> >>> guest_ss_selector(0x804) hex:0x18 bin:00000000 00011000
> >>> guest_ss_base(0x680a) hex:0
> >>> guest_ss_limit(0x4804) hex:0
> >>> guest_ss_access_rights(0x4818) hex:0x4093 bin:00000000 00000000 01000000 10010011
> >>>
> >>> guest_ds_selector(0x806) hex:0x2b bin:00000000 00101011
> >>> guest_ds_base(0x680c) hex:0
> >>> guest_ds_limit(0x4806) hex:0xffffffff
> >>> guest_ds_access_rights(0x481a) hex:0xcff3 bin:00000000 00000000 11001111 11110011
> >>>
> >>> guest_fs_selector(0x808) hex:0x53 bin:00000000 00000000 00000000 01010011
> >>> guest_fs_base(0x680e) hex:0
> >>> guest_fs_limit(0x4808) hex:0x3c00
> >>> guest_fs_access_rights(0x481c) hex:0x40f3 bin:00000000 00000000 01000000 11110011
> >>>
> >>> guest_gs_selector(0x80a) hex:0x2b bin:00000000 00101011
> >>> guest_gs_base(0x6810) hex:fffff8024b822000
> >>> guest_gs_limit(0x480a) hex:0xffffffff
> >>> guest_gs_access_rights(0x481e) hex:0xcff3 bin:00000000 00000000 11001111 11110011
> >>>
> >>> guest_gdtr_base(0x6816) hex:fffff8024ea53fb0
> >>> guest_gdtr_limit(0x4810) hex:0x57
> >>>
> >>> guest_ldtr_selector(0x80c) hex:0
> >>> guest_ldtr_limit(0x480c) hex:0
> >>> guest_ldtr_base(0x6812) hex:0
> >>> guest_ldtr_access_rights(0x4820) hex:0x10000 bin:00000000 00000001 00000000 00000000
> >>>
> >>> guest_tr_selector(0x80e) hex:0x40 bin:00000000 01000000
> >>> guest_tr_limit(0x480e) hex:0x67
> >>> guest_tr_base(0x6814) hex:fffff8024ea52000
> >>> guest_tr_access_rights(0x4822) hex:0x8b bin:00000000 00000000 00000000 10001011
> >>>
> >>> guest_idtr_limit(0x4812) hex:0xfff
> >>> guest_idtr_base(0x6818) hex:fffff8024ea51000
> >>>
> >>> guest_ia32_debugctl(0x2802) hex:0
> >>> guest_ia32_sysenter_cs(0x482a) hex:0
> >>> guest_ia32_sysenter_esp(0x6824) hex:0
> >>> guest_ia32_sysenter_eip(0x6826) hex:0
> >>>
> >>> exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001
> >>> exit_qualification(0x6400) hex:0
> >>> —
> >>> NTDEV is sponsored by OSR
> >>>
> >>> Visit the list online at: http:>
> >>>
> >>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> >>> Details at http:>
> >>>
> >>> To unsubscribe, visit the List Server section of OSR Online at http:>
> >>
> >> —
> >> NTDEV is sponsored by OSR
> >>
> >> Visit the list online at: http:>
> >>
> >> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> >> Details at http:>
> >>
> >> To unsubscribe, visit the List Server section of OSR Online at http:>
> >
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:></http:></http:></http:></mailto:xxxxx></mailto:xxxxx>

Sergey_Pisarev · August 1, 2018, 3:50pm

I feel that my last post was a little haphazard.
So to clarify:
Windows segment descriptors are totally good (as far as I know).
It was my mistake in access rights configuration.
Format of _KGDTENTRY64 is wrong, but GDT entry format in memory is correct.

P.S.
Still I would like to have decent error reporting in VMX, no the error code that points you to 5 pages of possible reasons for failure.

On 30 Jul 2018, at 09:35, xxxxx@azathoth.net wrote:
>
> On Sat, Jul 28, 2018 at 09:09:16PM +0300, xxxxx@gmail.com mailto:xxxxx wrote:
>> The problem is misconfigured GDT entry. To be precise flag System set incorrectly in some segment descriptors. For example it is not set for descriptor referenced by cs.
>>
>
> Nice find, thanks for sharing the result of your diagnostic.
>
> -ml
>
>> kd> r cs
>> cs=0010
>>
>> kd> r gdtr
>> gdtr=fffff80144c53fb0
>>
>> kd> ?(0xfffff80144c53fb0 + 0x10)
>> Evaluate expression: -8790644277312 = fffff80144c53fc0 >> >> kd> dt -r nt!_KGDTENTRY64 0xfffff80144c53fc0
>> +0x000 LimitLow : 0
>> +0x002 BaseLow : 0
>> +0x004 Bytes :
>> +0x000 BaseMiddle : 0 ‘’
>> +0x001 Flags1 : 0x9b ‘’
>> +0x002 Flags2 : 0x20 ’ '
>> +0x003 BaseHigh : 0 ‘’
>> +0x004 Bits :
>> +0x000 BaseMiddle : 0y00000000 (0)
>> +0x000 Type : 0y11011 (0x1b)
>> +0x000 Dpl : 0y00
>> +0x000 Present : 0y1
>> +0x000 LimitHigh : 0y0000
>> +0x000 System : 0y0
>> +0x000 LongMode : 0y1
>> +0x000 DefaultBig : 0y0
>> +0x000 Granularity : 0y0
>> +0x000 BaseHigh : 0y00000000 (0)
>> +0x008 BaseUpper : 0
>> +0x00c MustBeZero : 0x409300
>> +0x000 DataLow : 0n9177623557046272
>> +0x008 DataHigh : 0n18176026718765056
>>
>> And here is the note I have found in SimpleVisor sources:
>> // NOTE: The Windows definition of KGDTENTRY64 is WRONG. The “System” field
>> // is incorrectly defined at the position of where the AVL bit should be.
>> // The actual location of the SYSTEM bit is encoded as the highest bit in
>> // the “Type” field.
>>
>> However S flag is set correctly if I use lar instruction instead of parsing gdt by hand (thats what I did before).
>>
>>
>>> On 28 Jul 2018, at 18:48, Sergey Pisarev wrote:
>>>
>>> Load IA32_EFER bit is clear in vm-entry controls, and EFER field is not initialised (must be zero).
>>> IA-32e mode guest bit is set.
>>> According to manual EFER checks performed only if load EFER bit is set.
>>>
>>> Forgot to say I am testing on win10x64
>>>
>>>> On 28 Jul 2018, at 18:30, xxxxx@azathoth.net mailto:xxxxx mailto:xxxxx> mailto:xxxxx>> wrote:
>>>>
>>>> On Sat, Jul 28, 2018 at 03:02:43AM +0300, xxxxx@gmail.com mailto:xxxxx mailto:xxxxx> wrote:
>>>>> Hello !
>>>>>
>>>>> This is cpu related question and not OS related one. But since some people here do virtualisation software development I hope it’s still ok to post here.
>>>>>
>>>>> I am developing driver that converts currently running OS to run under hypervisor(something like SimpleVisor). After vmlaunch cpu jumps to vm-exit dispatcher with exit reason 0x80000021 (VM-entry failure due to invalid guest state). I have checked everything in 26.3.1 Checks on the Guest State Area and fixed all the errors I have found. I am still getting this error. Below caps of my cpu and values in guest area. Maybe someone can glance over it and notice something invalid.
>>>>>
>>>>
>>>> What about EFER settings (and the rules associated with IA32e_MODE_GUEST setting)?
>>>>
>>>> -ml
>>>>
>>>>> IA32_VMX_BASIC (0x480) hex:0x00da04000000000f bin:00000000 11011010 00000100 00000000 00000000 00000000 00000000 00001111 >>>>> >>>>> >>>>> IA32_VMX_CR0_FIXED0 (0x486) hex:0x80000021 bin:10000000 00000000 00000000 00100001 >>>>> IA32_VMX_CR0_FIXED1 (0x487) hex:0xffffffff bin:11111111 11111111 11111111 11111111 >>>>> guest_cr0 (0x6800) hex:0x80050031 bin:10000000 00000101 00000000 00110001 >>>>> >>>>> IA32_VMX_CR4_FIXED0 (0x488) hex:0x2000 bin:00000000 00000000 00100000 00000000 >>>>> IA32_VMX_CR4_FIXED1 (0x489) hex:0x227ff bin:00000000 00000010 00100111 11111111 >>>>> guest_cr4 (0x6804) hex:0x26f8 bin:00000000 00000000 00100110 11111000 >>>>> >>>>> ia32_vmx_true_pinbased_ctls(0x48D) hex:0x0000007f00000016 bin:00000000 00000000 00000000 01111111 00000000 00000000 00000000 00010110
>>>>> pin_based_vm_execution_controls(0x4000) hex:0x16 bin:00000000 00000000 00000000 00010110
>>>>>
>>>>> ia32_vmx_true_procbased_ctls(0x48e) hex:0xfff9fffe04006172 bin:11111111 11111001 11111111 11111110 00000100 00000000 01100001 01110010 >>>>> primary_processor_based_vm_execution_controls(0x4002) hex:0x94006172 bin:10010100 00000000 01100001 01110010 >>>>> >>>>> ia32_vmx_procbased_ctls2(0x48b) hex:0x000000ff00000000 bin:00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000
>>>>> secondary_processor_based_vm_execution_controls(0x401e) hex:0xaa bin:00000000 00000000 00000000 10101010
>>>>>
>>>>> ia32_vmx_true_exit_ctls(0x48f) hex:0x007fffff00036dfb bin:00000000 01111111 11111111 11111111 00000000 00000011 01101101 11111011 >>>>> vm_exit_controls(0x400c) hex:0x3efff bin:00000000 00000011 11101111 11111111 >>>>> >>>>> ia32_vmx_true_entry_ctls(0x490) hex:0x0000ffff000011fb bin:00000000 00000000 11111111 11111111 00000000 00000000 00010001 11111011
>>>>> vm_entry_controls(0x4012) hex:0x13ff bin:00000000 00000000 00010011 11111111
>>>>>
>>>>> guest_cr3(0x6802) hex:0x1aa000
>>>>>
>>>>> guest_dr7 (0x681a) hex:0x400
>>>>> guest_rflags (0x6820) hex:0x286 bin:00000000 00000000 00000000 00000000 00000000 00000000 00000010 10000110
>>>>>
>>>>> guest_rsp(0x681c) hex:ffff8202cf325858
>>>>> guest_rip(0x681e) hex:fffff80116c61058
>>>>>
>>>>> guest_cs_selector(0x802) hex:0x10 bin:00000000 00010000
>>>>> guest_cs_base(0x6808) hex:0
>>>>> guest_cs_limit(0x4802) hex:0
>>>>> guest_cs_access_rights(0x4816) hex:0x209b bin:00000000 00000000 00100000 10011011
>>>>>
>>>>> guest_es_selector(0x800) hex:000000000000002b bin:00000000 00101011
>>>>> guest_es_base(0x6806) hex:0
>>>>> guest_es_limit(0x4800) hex:00000000ffffffff
>>>>> guest_es_access_rights(0x4814) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>>>>
>>>>> guest_ss_selector(0x804) hex:0x18 bin:00000000 00011000
>>>>> guest_ss_base(0x680a) hex:0
>>>>> guest_ss_limit(0x4804) hex:0
>>>>> guest_ss_access_rights(0x4818) hex:0x4093 bin:00000000 00000000 01000000 10010011
>>>>>
>>>>> guest_ds_selector(0x806) hex:0x2b bin:00000000 00101011
>>>>> guest_ds_base(0x680c) hex:0
>>>>> guest_ds_limit(0x4806) hex:0xffffffff
>>>>> guest_ds_access_rights(0x481a) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>>>>
>>>>> guest_fs_selector(0x808) hex:0x53 bin:00000000 00000000 00000000 01010011
>>>>> guest_fs_base(0x680e) hex:0
>>>>> guest_fs_limit(0x4808) hex:0x3c00
>>>>> guest_fs_access_rights(0x481c) hex:0x40f3 bin:00000000 00000000 01000000 11110011
>>>>>
>>>>> guest_gs_selector(0x80a) hex:0x2b bin:00000000 00101011
>>>>> guest_gs_base(0x6810) hex:fffff8024b822000
>>>>> guest_gs_limit(0x480a) hex:0xffffffff
>>>>> guest_gs_access_rights(0x481e) hex:0xcff3 bin:00000000 00000000 11001111 11110011
>>>>>
>>>>> guest_gdtr_base(0x6816) hex:fffff8024ea53fb0
>>>>> guest_gdtr_limit(0x4810) hex:0x57
>>>>>
>>>>> guest_ldtr_selector(0x80c) hex:0
>>>>> guest_ldtr_limit(0x480c) hex:0
>>>>> guest_ldtr_base(0x6812) hex:0
>>>>> guest_ldtr_access_rights(0x4820) hex:0x10000 bin:00000000 00000001 00000000 00000000
>>>>>
>>>>> guest_tr_selector(0x80e) hex:0x40 bin:00000000 01000000
>>>>> guest_tr_limit(0x480e) hex:0x67
>>>>> guest_tr_base(0x6814) hex:fffff8024ea52000
>>>>> guest_tr_access_rights(0x4822) hex:0x8b bin:00000000 00000000 00000000 10001011
>>>>>
>>>>> guest_idtr_limit(0x4812) hex:0xfff
>>>>> guest_idtr_base(0x6818) hex:fffff8024ea51000
>>>>>
>>>>> guest_ia32_debugctl(0x2802) hex:0
>>>>> guest_ia32_sysenter_cs(0x482a) hex:0
>>>>> guest_ia32_sysenter_esp(0x6824) hex:0
>>>>> guest_ia32_sysenter_eip(0x6826) hex:0
>>>>>
>>>>> exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001
>>>>> exit_qualification(0x6400) hex:0
>>>>> —
>>>>> NTDEV is sponsored by OSR
>>>>>
>>>>> Visit the list online at: http: http:>>
>>>>>
>>>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
>>>>> Details at http: http:>>
>>>>>
>>>>> To unsubscribe, visit the List Server section of OSR Online at http: http:>>
>>>>
>>>> —
>>>> NTDEV is sponsored by OSR
>>>>
>>>> Visit the list online at: http: http:>>
>>>>
>>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
>>>> Details at http: http:>>
>>>>
>>>> To unsubscribe, visit the List Server section of OSR Online at http: http:>>
>>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: http:
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at http:
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx>

Mike_Larkin · August 1, 2018, 3:52pm

On Wed, Aug 01, 2018 at 10:51:11PM +0300, xxxxx@gmail.com wrote:

I feel that my last post was a little haphazard.
So to clarify:
Windows segment descriptors are totally good (as far as I know).
It was my mistake in access rights configuration.
Format of _KGDTENTRY64 is wrong, but GDT entry format in memory is correct.

P.S.
Still I would like to have decent error reporting in VMX, no the error code that points you to 5 pages of possible reasons for failure.

Not that it’s *too* much help, but I found that bochs does a good job
explaining (via the debug console) why various entry conditions failed
to allow launch. But it’s pretty slow for regular development, usually I
break it out when I’ve exhausted all other scenarios.

-ml

> On 30 Jul 2018, at 09:35, xxxxx@azathoth.net wrote:
> >
> > On Sat, Jul 28, 2018 at 09:09:16PM +0300, xxxxx@gmail.com mailto:xxxxx wrote:
> >> The problem is misconfigured GDT entry. To be precise flag System set incorrectly in some segment descriptors. For example it is not set for descriptor referenced by cs.
> >>
> >
> > Nice find, thanks for sharing the result of your diagnostic.
> >
> > -ml
> >
> >> kd> r cs
> >> cs=0010
> >>
> >> kd> r gdtr
> >> gdtr=fffff80144c53fb0
> >>
> >> kd> ?(0xfffff80144c53fb0 + 0x10)
> >> Evaluate expression: -8790644277312 = fffff80144c53fc0 > >> > >> kd> dt -r nt!_KGDTENTRY64 0xfffff80144c53fc0
> >> +0x000 LimitLow : 0
> >> +0x002 BaseLow : 0
> >> +0x004 Bytes :
> >> +0x000 BaseMiddle : 0 ‘’
> >> +0x001 Flags1 : 0x9b ‘’
> >> +0x002 Flags2 : 0x20 ’ '
> >> +0x003 BaseHigh : 0 ‘’
> >> +0x004 Bits :
> >> +0x000 BaseMiddle : 0y00000000 (0)
> >> +0x000 Type : 0y11011 (0x1b)
> >> +0x000 Dpl : 0y00
> >> +0x000 Present : 0y1
> >> +0x000 LimitHigh : 0y0000
> >> +0x000 System : 0y0
> >> +0x000 LongMode : 0y1
> >> +0x000 DefaultBig : 0y0
> >> +0x000 Granularity : 0y0
> >> +0x000 BaseHigh : 0y00000000 (0)
> >> +0x008 BaseUpper : 0
> >> +0x00c MustBeZero : 0x409300
> >> +0x000 DataLow : 0n9177623557046272
> >> +0x008 DataHigh : 0n18176026718765056
> >>
> >> And here is the note I have found in SimpleVisor sources:
> >> // NOTE: The Windows definition of KGDTENTRY64 is WRONG. The “System” field
> >> // is incorrectly defined at the position of where the AVL bit should be.
> >> // The actual location of the SYSTEM bit is encoded as the highest bit in
> >> // the “Type” field.
> >>
> >> However S flag is set correctly if I use lar instruction instead of parsing gdt by hand (thats what I did before).
> >>
> >>
> >>> On 28 Jul 2018, at 18:48, Sergey Pisarev wrote:
> >>>
> >>> Load IA32_EFER bit is clear in vm-entry controls, and EFER field is not initialised (must be zero).
> >>> IA-32e mode guest bit is set.
> >>> According to manual EFER checks performed only if load EFER bit is set.
> >>>
> >>> Forgot to say I am testing on win10x64
> >>>
> >>>> On 28 Jul 2018, at 18:30, xxxxx@azathoth.net mailto:xxxxx mailto:xxxxx> mailto:xxxxx>> wrote:
> >>>>
> >>>> On Sat, Jul 28, 2018 at 03:02:43AM +0300, xxxxx@gmail.com mailto:xxxxx mailto:xxxxx> wrote:
> >>>>> Hello !
> >>>>>
> >>>>> This is cpu related question and not OS related one. But since some people here do virtualisation software development I hope itâ€™s still ok to post here.
> >>>>>
> >>>>> I am developing driver that converts currently running OS to run under hypervisor(something like SimpleVisor). After vmlaunch cpu jumps to vm-exit dispatcher with exit reason 0x80000021 (VM-entry failure due to invalid guest state). I have checked everything in 26.3.1 Checks on the Guest State Area and fixed all the errors I have found. I am still getting this error. Below caps of my cpu and values in guest area. Maybe someone can glance over it and notice something invalid.
> >>>>>
> >>>>
> >>>> What about EFER settings (and the rules associated with IA32e_MODE_GUEST setting)?
> >>>>
> >>>> -ml
> >>>>
> >>>>> IA32_VMX_BASIC (0x480) hex:0x00da04000000000f bin:00000000 11011010 00000100 00000000 00000000 00000000 00000000 00001111 > >>>>> > >>>>> > >>>>> IA32_VMX_CR0_FIXED0 (0x486) hex:0x80000021 bin:10000000 00000000 00000000 00100001 > >>>>> IA32_VMX_CR0_FIXED1 (0x487) hex:0xffffffff bin:11111111 11111111 11111111 11111111 > >>>>> guest_cr0 (0x6800) hex:0x80050031 bin:10000000 00000101 00000000 00110001 > >>>>> > >>>>> IA32_VMX_CR4_FIXED0 (0x488) hex:0x2000 bin:00000000 00000000 00100000 00000000 > >>>>> IA32_VMX_CR4_FIXED1 (0x489) hex:0x227ff bin:00000000 00000010 00100111 11111111 > >>>>> guest_cr4 (0x6804) hex:0x26f8 bin:00000000 00000000 00100110 11111000 > >>>>> > >>>>> ia32_vmx_true_pinbased_ctls(0x48D) hex:0x0000007f00000016 bin:00000000 00000000 00000000 01111111 00000000 00000000 00000000 00010110
> >>>>> pin_based_vm_execution_controls(0x4000) hex:0x16 bin:00000000 00000000 00000000 00010110
> >>>>>
> >>>>> ia32_vmx_true_procbased_ctls(0x48e) hex:0xfff9fffe04006172 bin:11111111 11111001 11111111 11111110 00000100 00000000 01100001 01110010 > >>>>> primary_processor_based_vm_execution_controls(0x4002) hex:0x94006172 bin:10010100 00000000 01100001 01110010 > >>>>> > >>>>> ia32_vmx_procbased_ctls2(0x48b) hex:0x000000ff00000000 bin:00000000 00000000 00000000 11111111 00000000 00000000 00000000 00000000
> >>>>> secondary_processor_based_vm_execution_controls(0x401e) hex:0xaa bin:00000000 00000000 00000000 10101010
> >>>>>
> >>>>> ia32_vmx_true_exit_ctls(0x48f) hex:0x007fffff00036dfb bin:00000000 01111111 11111111 11111111 00000000 00000011 01101101 11111011 > >>>>> vm_exit_controls(0x400c) hex:0x3efff bin:00000000 00000011 11101111 11111111 > >>>>> > >>>>> ia32_vmx_true_entry_ctls(0x490) hex:0x0000ffff000011fb bin:00000000 00000000 11111111 11111111 00000000 00000000 00010001 11111011
> >>>>> vm_entry_controls(0x4012) hex:0x13ff bin:00000000 00000000 00010011 11111111
> >>>>>
> >>>>> guest_cr3(0x6802) hex:0x1aa000
> >>>>>
> >>>>> guest_dr7 (0x681a) hex:0x400
> >>>>> guest_rflags (0x6820) hex:0x286 bin:00000000 00000000 00000000 00000000 00000000 00000000 00000010 10000110
> >>>>>
> >>>>> guest_rsp(0x681c) hex:ffff8202cf325858
> >>>>> guest_rip(0x681e) hex:fffff80116c61058
> >>>>>
> >>>>> guest_cs_selector(0x802) hex:0x10 bin:00000000 00010000
> >>>>> guest_cs_base(0x6808) hex:0
> >>>>> guest_cs_limit(0x4802) hex:0
> >>>>> guest_cs_access_rights(0x4816) hex:0x209b bin:00000000 00000000 00100000 10011011
> >>>>>
> >>>>> guest_es_selector(0x800) hex:000000000000002b bin:00000000 00101011
> >>>>> guest_es_base(0x6806) hex:0
> >>>>> guest_es_limit(0x4800) hex:00000000ffffffff
> >>>>> guest_es_access_rights(0x4814) hex:0xcff3 bin:00000000 00000000 11001111 11110011
> >>>>>
> >>>>> guest_ss_selector(0x804) hex:0x18 bin:00000000 00011000
> >>>>> guest_ss_base(0x680a) hex:0
> >>>>> guest_ss_limit(0x4804) hex:0
> >>>>> guest_ss_access_rights(0x4818) hex:0x4093 bin:00000000 00000000 01000000 10010011
> >>>>>
> >>>>> guest_ds_selector(0x806) hex:0x2b bin:00000000 00101011
> >>>>> guest_ds_base(0x680c) hex:0
> >>>>> guest_ds_limit(0x4806) hex:0xffffffff
> >>>>> guest_ds_access_rights(0x481a) hex:0xcff3 bin:00000000 00000000 11001111 11110011
> >>>>>
> >>>>> guest_fs_selector(0x808) hex:0x53 bin:00000000 00000000 00000000 01010011
> >>>>> guest_fs_base(0x680e) hex:0
> >>>>> guest_fs_limit(0x4808) hex:0x3c00
> >>>>> guest_fs_access_rights(0x481c) hex:0x40f3 bin:00000000 00000000 01000000 11110011
> >>>>>
> >>>>> guest_gs_selector(0x80a) hex:0x2b bin:00000000 00101011
> >>>>> guest_gs_base(0x6810) hex:fffff8024b822000
> >>>>> guest_gs_limit(0x480a) hex:0xffffffff
> >>>>> guest_gs_access_rights(0x481e) hex:0xcff3 bin:00000000 00000000 11001111 11110011
> >>>>>
> >>>>> guest_gdtr_base(0x6816) hex:fffff8024ea53fb0
> >>>>> guest_gdtr_limit(0x4810) hex:0x57
> >>>>>
> >>>>> guest_ldtr_selector(0x80c) hex:0
> >>>>> guest_ldtr_limit(0x480c) hex:0
> >>>>> guest_ldtr_base(0x6812) hex:0
> >>>>> guest_ldtr_access_rights(0x4820) hex:0x10000 bin:00000000 00000001 00000000 00000000
> >>>>>
> >>>>> guest_tr_selector(0x80e) hex:0x40 bin:00000000 01000000
> >>>>> guest_tr_limit(0x480e) hex:0x67
> >>>>> guest_tr_base(0x6814) hex:fffff8024ea52000
> >>>>> guest_tr_access_rights(0x4822) hex:0x8b bin:00000000 00000000 00000000 10001011
> >>>>>
> >>>>> guest_idtr_limit(0x4812) hex:0xfff
> >>>>> guest_idtr_base(0x6818) hex:fffff8024ea51000
> >>>>>
> >>>>> guest_ia32_debugctl(0x2802) hex:0
> >>>>> guest_ia32_sysenter_cs(0x482a) hex:0
> >>>>> guest_ia32_sysenter_esp(0x6824) hex:0
> >>>>> guest_ia32_sysenter_eip(0x6826) hex:0
> >>>>>
> >>>>> exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001
> >>>>> exit_qualification(0x6400) hex:0
> >>>>> —
> >>>>> NTDEV is sponsored by OSR
> >>>>>
> >>>>> Visit the list online at: http: http:>>
> >>>>>
> >>>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> >>>>> Details at http: http:>>
> >>>>>
> >>>>> To unsubscribe, visit the List Server section of OSR Online at http: http:>>
> >>>>
> >>>> —
> >>>> NTDEV is sponsored by OSR
> >>>>
> >>>> Visit the list online at: http: http:>>
> >>>>
> >>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> >>>> Details at http: http:>>
> >>>>
> >>>> To unsubscribe, visit the List Server section of OSR Online at http: http:>>
> >>>
> >>
> >>
> >> —
> >> NTDEV is sponsored by OSR
> >>
> >> Visit the list online at: http:
> >>
> >> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> >> Details at http:
> >>
> >> To unsubscribe, visit the List Server section of OSR Online at http:
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > Visit the list online at: http:
> >
> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> > Details at http:
> >
> > To unsubscribe, visit the List Server section of OSR Online at http:
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx>

VM-entry failure due to invalid guest state.

exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001 exit_qualification(0x6400) hex:0

exit_reason(0x4402) hex:0x80000021 bin:10000000 00000000 00000000 00100001
exit_qualification(0x6400) hex:0