Data structure alignment

NTDEV
1

Hi,

This is not a Windows driver specific question, however it is related with the driver i’m working because it has a main data structure shared across different operating systems.

I have realized very recently that compilers in Windows have a different struct data alignment, and structure data padding.

What I have realized is that my main structure has different size when compiled in Windows then the size when compiled in Linux.

I think that is because in Linux, word size is 4 bytes and and in windows 8 bytes, please correct me if i’m wrong.

What should I do to have a uniform structure data size across different operating systems?

I discover this while transmiting my main structure data, from a windows system to a unix based system.

I have also read this article about data alignment and padding:

http://en.wikipedia.org/wiki/Data_structure_alignment

And it speaks about praga pack. I wonder if I should use a pragma pack of 1 byte to solve my problems.

What is your opinion about this?

Thanks,

With my best regards,

Nuno Santos

2

Well I guess my first question is what do you mean by word size. The
main types in Windows have sizes of:

Char 1 byte
Short 2 bytes
Int 4 bytes
Int64 8 bytes

My own approach to dealing with same struct for two different systems is
to:

  1. Define my own types for all values with #if to resolve the
    definitions for the right system types
  2. Completely pack the structure, i.e. no gaps such as short followed
    by a long assume that all data aligns on the byte boundary represented
    by its size and use that.
  3. Use #pragma pack with 1 only if you can’t do the alignment as above
    because of some stupidity by one or both ends.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“xxxxx@imaginando.net” wrote in message
news:xxxxx@ntdev:

> Hi,
>
> This is not a Windows driver specific question, however it is related with the driver i’m working because it has a main data structure shared across different operating systems.
>
> I have realized very recently that compilers in Windows have a different struct data alignment, and structure data padding.
>
> What I have realized is that my main structure has different size when compiled in Windows then the size when compiled in Linux.
>
> I think that is because in Linux, word size is 4 bytes and and in windows 8 bytes, please correct me if i’m wrong.
>
> What should I do to have a uniform structure data size across different operating systems?
>
> I discover this while transmiting my main structure data, from a windows system to a unix based system.
>
> I have also read this article about data alignment and padding:
>
> http://en.wikipedia.org/wiki/Data_structure_alignment
>
> And it speaks about praga pack. I wonder if I should use a pragma pack of 1 byte to solve my problems.
>
> What is your opinion about this?
>
> Thanks,
>
> With my best regards,
>
> Nuno Santos

3

Don Burn wrote:

  1. Define my own types for all values with #if to resolve the
    definitions for the right system types
  2. Completely pack the structure, i.e. no gaps such as short followed
    by a long assume that all data aligns on the byte boundary represented
    by its size and use that.
  3. Use #pragma pack with 1 only if you can't do the alignment as above
    because of some stupidity by one or both ends.

I would add:
4. Write the code so that it doesn't depend on the layout of the
structures at all if possible.

In particular I always prefer to avoid coding tricks which overlay a C
structure on device registers or a data packet. While this looks elegant
in theory it becomes a recipe for disaster as you port your code to
different architectures.

On both Linux and windows the sizes of certain structures will change
depending on architecture if the structure includes a pointer type of
any sort. On linux the structures will grow on a 64bit architecture if
you have use long or unsigned long anywhere in the structure.

However if you absolutely have to do it Don's approach is also the one I
would take.

Mike

Mike Pumford, Senior Software Engineer
MPC Data Limited
e-mail: xxxxx@mpcdata.com web: www.mpcdata.com
tel: +44 (0) 1225 710600 fax: +44 (0) 1225 710601
ddi: +44 (0) 1225 710635

4

xxxxx@imaginando.net wrote:

I have realized very recently that compilers in Windows have a different struct data alignment, and structure data padding.

What I have realized is that my main structure has different size when compiled in Windows then the size when compiled in Linux.

I think that is because in Linux, word size is 4 bytes and and in windows 8 bytes, please correct me if i’m wrong.

You’re wrong. There are rules in the language standards about packing,
and there are aspects that are implementation-dependent. In both cases
(Visual C++ and gcc), the actual packing depends on compiler
command-line parameters and pragmas. You probably have different
command-line settings and just don’t realize it.

What should I do to have a uniform structure data size across different operating systems?

You need to understand the rules. In general, if you set up your
structures so that you never include a misaligned data type, then this
problem never comes up. For example:

struct one {
unsigned char a1; // needs 1-byte alignment
unsigned char a2;
unsigned short a3; // needs 2-byte alignment, has 2-byte alignment
unsigned int a4; // needs 4-byte alignment, has 4-byte alignment
};

That structure will never have any padding. All of the types are
correctly aligned. But if a scramble the ordering, padding will be
inserted:

struct one {
unsigned char a1; // needs 1-byte alignment
unsigned short a3; // needs 2-byte alignment, so gets 1 byte padding
unsigned char a2;
unsigned int a4; // needs 4-byte alignment, so gets 3 bytes padding
};

You can override that requirement with the pragmas. If I use pragma
pack(1), then there will be no padding inserted.

There is one other thing that might be biting you. In 64-bit gcc,
“long” is a 64-bit type. In 64-bit Visual C++, “long” is a 32-bit type.

And it speaks about praga pack. I wonder if I should use a pragma pack of 1 byte to solve my problems.

We can’t answer that. If you have poorly thought out structures with
bad alignment, then “pragma pack” can hide those issues.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

5

I have a really complex set of structures in my code. Some of those structures are composed of other structures. I will probably need to take into account each substructure size in order to correctly align the whole structure.

Before posting this question, I thought the solution should be to fill the structures to make them always multiple of 4. That’s why I have “NotUsed” variables on this code. This is just a small example of the structures I rely on:

typedef struct _DPX_TRACKING_STATE
{
DPX_CALIBRATION_ZONE CalibrationZones[DPX_MAX_CALIBRATION_ZONES];
int NumZones;
int NotUsed;
} DPX_TRACKING_STATE, *PDPX_TRACKING_STATE;

typedef struct _DPX_CALIBRATION_ZONE
{
poly Poly;
long long Matrix[3][3];
DPX_POINT T0Center;
DPX_POINT T1Center;
int valid;
int notused;
} DPX_CALIBRATION_ZONE, *PDPX_CALIBRATION_ZONE;

typedef struct _DPX_POINT
{
long X, Y;
} DPX_POINT, *PDPX_POINT;

typedef struct
{
double u, v;
double sx, sy;
double x, y, z;
} poly_vert;

typedef struct
{
poly_vert vert[4];
int n;
int notused;
} poly;

Should I start to analyze them from the bottom up? What should be the procedure to such level of structure dependency?

Imagine the following…

poly depends on poly_vert. poly_vert is 8*7 bytes long which is = 56 bytes.

poly declares 4*poly_vert which means 4*56 = 224 bytes

but then I need the n variable which is an int.

How should I procedure in order to correctely structure this structures?

Thanks,

With my best regards,

Nuno

6

xxxxx@imaginando.net wrote:

I have a really complex set of structures in my code. Some of those structures are composed of other structures. I will probably need to take into account each substructure size in order to correctly align the whole structure.

Before posting this question, I thought the solution should be to fill the structures to make them always multiple of 4.

That’s not quite enough. If a structure contains an 8-byte variable
(double, or long on Linux 64), then the structure pads out to a multiple
of 8.

By the way, the “long” issue is going to bite you. In Windows (always)
and in 32-bit Linux gcc, “long” is 4 bytes. In 64-bit Linux gcc, “long”
is 8 bytes. So, your “DPX_POINT” structure will be 8 bytes on Windows
and on Linux 32, but 16 bytes on Linux 64. You might want to change
those to ints instead of longs.

Should I start to analyze them from the bottom up? What should be the procedure to such level of structure dependency?

The structures you showed us have no padding issues. They produce the
same sizes with pragma pack of 1, 2, 4, or 8, with Visual C++ or with
gcc. I created the following test program with your definitions:

int
main()
{
DPX_TRACKING_STATE d1;
DPX_TRACKING_STATE d2[10];
DPX_CALIBRATION_ZONE z1;
DPX_CALIBRATION_ZONE z2[10];
printf( “%d %d %d %d\n”,
sizeof(d1),
sizeof(d2),
sizeof(z1),
sizeof(z2)
);
return 0;
}

How should I procedure in order to correctely structure this structures?

I would say those were correctly structured, except for the issue with
“long”.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

7

Basically, in structures intended for interop you use only explicitly
sized typedefs: int32_t, int64_t, int16_t.
(defined in stdint.h, available in VC++ since 2010)
Explicit sizes help tallying the alignment.
– pa

8

Also:

No pointers; not only might the interoperability have to be between
machines, it may need to be between 32-bit and 64-bit versions.

When dealing with endianness, establish a standard (such as Network
Standard Byte Order) and honor it everywhere.

When dealing across compilers, recognize that structure layout my be
idiosyncratic to that compiler. If you are lucky enough to discover two
compilers for the same target allow control of the packing, and as a
result the byte layouts are compatible between the two compilers, think of
this as a Happy Accident.

Never load or store data to your structure. In C++ this is easy; all data
members are protected and functions are called to do load-store and handle
any marshalling requirements. Never, ever use “efficiency” as an excuse
not to do this; the Microsoft compilers frequently convert these “calls”
into a single ‘mov’ instruction if no complex marshalling is required, so
the “inefficiency” is at best a fantasy based on folklore and innuendo.
In C, you provide subroutines, and with care these will optimize to one
instruction if that is sufficient, but it takes a little more work on your
part to accomplish this.

When transmitting text, (a) assume Unicode is a requirement (b) choose an
encoding that works well for transmission, such as UTF-8 (particularly
good if you need interop between different endianness)

Depending on the total amount of information transmitted, do not rule out
XML or other textual representations until you have evaluated the actual
cost/benefit tradeoffs. Appeals to folklore or techniques that made sense
on the PDP-11 are inappropriate.
joe

Basically, in structures intended for interop you use only explicitly
sized typedefs: int32_t, int64_t, int16_t.
(defined in stdint.h, available in VC++ since 2010)
Explicit sizes help tallying the alignment.
– pa

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

9

> I have a really complex set of structures in my code. Some of those

structures are composed of other structures. I will probably need to take
into account each substructure size in order to correctly align the whole
structure.

Before posting this question, I thought the solution should be to fill the
structures to make them always multiple of 4. That’s why I have “NotUsed”
variables on this code. This is just a small example of the structures I
rely on:

typedef struct _DPX_TRACKING_STATE
{
DPX_CALIBRATION_ZONE CalibrationZones[DPX_MAX_CALIBRATION_ZONES];
int NumZones;
int NotUsed;
*****
I perceive no reason this should cause any structure in any compiler to be
a multiple of 4 bytes. If you discovered this in one compiler, it is at
best an accident of time and space. The next release of the compiler is
free to do something entirely different
*****
} DPX_TRACKING_STATE, *PDPX_TRACKING_STATE;

typedef struct _DPX_CALIBRATION_ZONE
{
poly Poly;
long long Matrix[3][3];
DPX_POINT T0Center;
DPX_POINT T1Center;
int valid;
int notused;
} DPX_CALIBRATION_ZONE, *PDPX_CALIBRATION_ZONE;

typedef struct _DPX_POINT
{
long X, Y;
****
Personally, I detest the use of commas in declaration lists. I consider
it a bad programmin practice
****
} DPX_POINT, *PDPX_POINT;

typedef struct
{
double u, v;
double sx, sy;
double x, y, z;
} poly_vert;

typedef struct
{
poly_vert vert[4];
int n;
int notused;
} poly;

Should I start to analyze them from the bottom up? What should be the
procedure to such level of structure dependency?

Imagine the following…

poly depends on poly_vert. poly_vert is 8*7 bytes long which is = 56
bytes.

poly declares 4*poly_vert which means 4*56 = 224 bytes

but then I need the n variable which is an int.

How should I procedure in order to correctely structure this structures?
*****
If both targets are 32-bit or 64-bit (but not both) x86, I’d suggest
setting the packing to 1 and hope for the best. If you need to support
other platforms, such as RISC machines or big-endian machines, Abandon All
Hope. Note that your data types are not platform-independent; ‘int’ and
‘long’ are types defined by the compiler writer, and you should use
platform-independent names like INT32, which for variuos compilers may
have different typedefs but which guarantee a 32-bit integer.

A friend of ine who wad lead designer built a conforming C compiler, for
which
sizeof(char) == sizeof(short)
sizeof(short) == sizeof(int)
sizeof(int) == sizeof(long)
sizeof(long) == sizeof(long long)
sizeof(long)== sizeof(float)
sizeof(float) == sizeof(double)
sizeof(long double) == sizeof(double)
sizeof(char) == 1

It was for a word-addressed DSP processor and the addressing unit was a
word, and a word had 80 bits.

The C standard gives compiler writers great flexibility in how they choose
sizeof() for any particular type, and how they pack data structures. The
result is that interoperability is more like an accident than a planned
condition.
joe

Thanks,

With my best regards,

Nuno

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer