Delphi Pascal - Using SetFilePointerEx and GetFileSizeEx, Getting Physical Media exact size when reading as a file

I do not know how to use any API that is not in the RTL. I have been using SetFilePointer and GetFileSize to read a Physical Disk into a buffer and dump it to a file, something like this in a loop does the job for flash memory cards under 2GB:

SetFilePointer(PD,0,nil,FILE_BEGIN);
SetLength(Buffer,512);
ReadFile(PD,Buffer[0],512,BytesReturned,nil);

However GetFileSize has a limit at 2GB and so does SetFilePointer. I have absolutley no idea how to delcare an external API, I have looked at the RTL and googled for many examples and have found no correct answer.

I tried this

function GetFileSizeEx(hFile: THandle; lpFileSizeHigh: Pointer): DWORD; 
    external 'kernel32';

and as suggested this

function GetFileSizeEx(hFile: THandle; var FileSize: Int64): DWORD;
    stdcall; external 'kernel32';

But the function returns a 0 even though I am using a valid disk handle which I have confirmed and dumped data from using the older API's.

I am using SetFilePointer to jump every 512 bytes 开发者_StackOverflow中文版and ReadFile to write into a buffer, in reverse I can use it to set when I am using WriteFile to write Initial Program Loader Code or something else to the disk. I need to be able to set the file pointer beyond 2gb well beyond.

Can someone help me make the external declarations and a call to both GetFileSizeEx and SetFilePointerEx that work so I can modify my older code to work with say 4 to 32gb flash cards.

---

I suggest that you take a look at this Primoz Gabrijelcic blog article and his GpHugeFile unit which should give you enough pointers to get the file size.

Edit 1 This looks rather a daft answer now in light of the edit to the question.

Edit 2 Now that this answer has been accepted, following a long threads of comments to jachguate's answer, I feel it incumbent to summarise what has been learnt.

• GetFileSize and SetFilePointer have no 2GB limitation, they can be used on files of essentially arbitrary size.

• GetFileSizeEx and SetFilePointerEx are much easier to use because they work directly with 64 bit quantities and have far simpler error condition signals.

• The OP did not in fact need to calculate the size of his disk. Since the OP was reading the entire contents of the disk the size was not needed. All that was required was to read the contents sequentially until there was nothing left.

• In fact GetFileSize/GetFileSizeEx do not support handles to devices (e.g. a physical disk or volume) as was requested by the OP. What's more, SetFilePointer/SetFilePointerEx cannot seek to the end of such device handles.

• In order to obtain the size of a disk, volume or partition, one should pass the the IOCTL_DISK_GET_LENGTH_INFO control code to DeviceIoControl.

Finally, should you need to use GetFileSizeEx and SetFilePointerEx then they can be declared as follows:

function GetFileSizeEx(hFile: THandle; var lpFileSize: Int64): BOOL;
    stdcall; external 'kernel32.dll';
function SetFilePointerEx(hFile: THandle; liDistanceToMove: Int64;
    lpNewFilePointer: PInt64; dwMoveMethod: DWORD): BOOL;
    stdcall; external 'kernel32.dll';

One easy way to obtain these API imports them is through the excellent JEDI API Library.

---

The GetFileSizeEx routine expects a pointer to a LARGE_INTEGER data type, and documentation says:

If your compiler has built-in support for 64-bit integers, use the QuadPart member to store the 64-bit integer

Lucky you, Delphi has built-in support for 64 bit integers, so use it:

var
  DriveSize: LongWord;
begin
  GetFilePointerSizeEx(PD, @DriveSize);
end;

SetFilePointerEx, on the other hand, expects parameters for liDistanceToMove, lpNewFilePointer, both 64 bit integers. My understanding is it wants signed integers, but you have the UInt64 data type for Unsingned 64 bit integers if I'm missunderstanding the documentation.

---

Alternative coding

Suicide, first of all your approach is wrong, and because of your wrong approach you ran into some hairy problems with the way Windows handles Disk drives opened as files. In pseudo code your approach seems to be:

Size = GetFileSize;
for i=0 to (Size / 512) do
begin
  Seek(i * 512);
  ReadBlock;
  WriteBlockToFile;
end;

That's functionally correct, but there's a simpler way to do the same without actually getting the SizeOfDisk and without seeking. When reading something from a file (or a stream), the "pointer" is automatically moved with the ammount of data you just read, so you can skip the "seek". All the functions used to read data from a file return the amount of data that was actually read: you can use that to know when you reached the end of the file without knowing the size of the file to start with!

Here's an idea of how you can read an physical disk to a file, without knowing much about the disk device, using Delphi's TFileStream:

var DiskStream, DestinationStream:TFileStream;
    Buff:array[0..512-1] of Byte;
    BuffRead:Integer;
begin
  // Open the disk for reading
  DiskStream := TFileStream.Create('\\.\PhysicalDrive0', fmOpenRead);
  try
    // Create the file
    DestinationStream := TFileStream.Create('D:\Something.IMG', fmCreate);
    try

      // Read & write in a loop; This is where all the work's done:
      BuffRead := DiskStream.Read(Buff, SizeOf(Buff));
      while BuffRead > 0 do
      begin
        DestinationStream.Write(Buff, BuffRead);
        BuffRead := DiskStream.Read(Buff, SizeOf(Buff));
      end;

    finally DestinationStream.Free;
    end;
  finally DiskStream.Free;
  end;
end;

You can obviously do something similar the other way around, reading from a file and writing to disk. Before writing that code I actually attempted doing it your way (getting the file size, etc), and immediately ran into problems! Apparently Windows doesn't know the exact size of the "file", not unless you read from it.

Problems with disks opened as files

For all my testing I used this simple code as the base:

var F: TFileStream;
begin
  F := TFileStream.Create('\\.\PhysicalDrive0', fmOpenRead);
  try
    // Test code goes here...
  finally F.Free;
  end;
end;

The first (obvious) thing to try was:

ShowMessage(IntToStr(DiskStream.Size));

That fails. In the TFileStream implementation that depends on calling FileSeek, and FileSeek can't handle files larger then 2Gb. So I gave GetFileSize a try, using this code:

var RetSize, UpperWord:DWORD;
RetSize := GetFileSize(F.Handle, @UpperWord);
ShowMessage(IntToStr(UpperWord) + ' / ' + IntToStr(RetSize));

That also fails, even those it should be perfectly capable of returning file size as an 64 bit number! Next I tried using the SetFilePointer API, because that's also supposed to handle 64bit numbers. I thought I'd simply seek to the end of the file and look at the result, using this code:

var RetPos, UpperWord:DWORD;
UpperWord := 0;
RetPos := SetFilePos(F.Handle, 0, @UpperWord, FILE_END);
ShowMessage(IntToStr(UpperWord) + ' / ' + IntToStr(RetPos));

This code also fails! And now I'm thinking, why did the first code work? Apparently reading block-by-block works just fine and Windows knows exactly when to stop reading!! So I thought maybe there's a problem with the implementation of the 64 bit file handling routines, let's try seeking to end of the file in small increments; When we get an error seeking we know we reached the end we'll stop:

var PrevUpWord, PrevPos: DWORD;
    UpWord, Pos: DWORD;
UpWord := 0;
Pos := SetFilePointer(F.Handle, 1024, @UpWord, FILE_CURRENT); // Advance the pointer 512 bytes from it's current position
while (UpWord <> PrevUpWord) or (Pos <> PrevPos) do
begin
  PrevUpWord := UpWord;
  PrevPos := Pos;
  UpWord := 0;
  Pos := SetFilePointer(F.Handle, 1024, @UpWord, FILE_CURRENT);
end;

When trying this code I had a surprise: It doesn't stop at the of the file, it just goes on and on, for ever. It never fails. To be perfectly honest I'm not sure it's supposed to ever fail... It's probably not supposed to fail. Anyway, doing a READ in that loop fails when we're past the end of file so we can use a VERY hacky mixed approach to handle this situation.

Ready-made routines that work around the problem

Here's the ready-made routine that gets the size of the physical disk opened as a file, even when GetFileSize fails, and SetFilePointer with FILE_END fails. Pass it an opened TFileStream and it will return the size as an Int64:

function Hacky_GetStreamSize(F: TFileStream): Int64;
var Step:DWORD;

    StartPos: Int64;
    StartPos_DWORD: packed array [0..1] of DWORD absolute StartPos;

    KnownGoodPosition: Int64;
    KGP_DWORD: packed array [0..1] of DWORD absolute KnownGoodPosition;

    Dummy:DWORD;

    Block:array[0..512-1] of Byte;
begin
  // Get starting pointer position
  StartPos := 0;
  StartPos_DWORD[0] := SetFilePointer(F.Handle, 0, @StartPos_DWORD[1], FILE_CURRENT);
  try
    // Move file pointer to the first byte
    SetFilePointer(F.Handle, 0, nil, FILE_BEGIN);
    // Init
    KnownGoodPosition := 0;
    Step := 1024 * 1024 * 1024; // Initial step will be 1Gb
    while Step > 512 do
    begin
      // Try to move
      Dummy := 0;
      SetFilePointer(F.Handle, Step, @Dummy, FILE_CURRENT);
      // Test: Try to read!
      if F.Read(Block, 512) = 512 then
        begin
          // Ok! Save the last known good position
          KGP_DWORD[1] := 0;
          KGP_DWORD[0] := SetFilePointer(F.Handle, 0, @KGP_DWORD[1], FILE_CURRENT);
        end
      else
        begin
          // Read failed! Move back to the last known good position and make Step smaller
          SetFilePointer(F.Handle, KGP_DWORD[0], @KGP_DWORD[1], FILE_BEGIN);
          Step := Step div 4; // it's optimal to devide by 4
        end;
    end;
    // From here on we'll use 512 byte steps until we can't read any more
    SetFilePointer(F.Handle, KGP_DWORD[0], @KGP_DWORD[1], FILE_BEGIN);
    while F.Read(Block, 512) = 512 do
      KnownGoodPosition := KnownGoodPosition + 512;
    // Done!
    Result := KnownGoodPosition;
  finally
    // Move file pointer back to starting position
    SetFilePointer(F.Handle, StartPos_DWORD[0], @StartPos_DWORD[1], FILE_BEGIN);
  end;
end;

To be complete, here are two routines that may be used to set and get the file pointer using Int64 for positioning:

function Hacky_SetStreamPos(F: TFileStream; Pos: Int64):Int64;
var aPos:Int64;
    DWA:packed array[0..1] of DWORD absolute aPos;
const INVALID_SET_FILE_POINTER = $FFFFFFFF;
begin
  aPos := Pos;
  DWA[0] := SetFilePointer(F.Handle, DWA[0], @DWA[1], FILE_BEGIN);
  if (DWA[0] = INVALID_SET_FILE_POINTER) and (GetLastError <> NO_ERROR) then
    RaiseLastOSError;
  Result := aPos;
end;

function Hacky_GetStreamPos(F: TFileStream): Int64;
var Pos:Int64;
    DWA:packed array[0..1] of DWORD absolute Pos;
begin
  Pos := 0;
  DWA[0] := SetFilePointer(F.Handle, 0, @DWA[1], FILE_CURRENT);
  Result := Pos;
end;

Last notes

The 3 routines I'm providing take as a parameter an TFileStream, because that's what I use for file reading and writing. They obviously only use TFileStream.Handle, so the parameter can simply be replaced with an file handle: the functionality would stay the same.

---

I know this thread is old, but...

One small suggestion - if you use the Windows DeviceIoControl(...) function you can get Drive Geometry and/or Partition Information, and use them to get the total size/length of the opened drive or partition. No more messing around with incrementally seeking to the end of the device.

Those IOCTLs can also be used to give you the correct volume sector size, and you could use that instead of defaulting to 512 everywhere.

---

Very very useful. But I got a problem for disks greater then 4 GB. I solved replacing:

   // Ok! Save the last known good position
      KGP_DWORD[1] := 0;
      KGP_DWORD[0] := SetFilePointer(F.Handle, 0, @KGP_DWORD[1], FILE_CURRENT);

with the following:

  // Ok! Save the last known good position
  KnownGoodPosition := KnownGoodPosition + Step;

Many thanks again...

And many thanks also to James R. Twine. I followed the advice of using IOCTL_DISK_GET_DRIVE_GEOMETRY_EX and got disk dimension with no problem and no strange workaround. Here is the code:

TDISK_GEOMETRY = record
  Cylinders : Int64; //LargeInteger
  MediaType : DWORD; //MEDIA_TYPE
  TracksPerCylinder: DWORD ;
  SectorsPerTrack: DWORD ;
  BytesPerSector : DWORD ;
end;
TDISK_GEOMETRY_EX = record
  Geometry: TDISK_GEOMETRY ;
  DiskSize:  Int64; //LARGE_INTEGER ;
  Data : array[1..1000] of byte; // unknown length
end;
function get_disk_size(handle: thandle): int64;
var
  BytesReturned: DWORD;
  DISK_GEOMETRY_EX : TDISK_GEOMETRY_EX;
begin
  result := 0;
  if DeviceIOControl(handle,IOCTL_DISK_GET_DRIVE_GEOMETRY_EX,
     nil,0,@DISK_GEOMETRY_EX, sizeof(TDISK_GEOMETRY_EX),BytesReturned,nil)
  then result := DISK_GEOMETRY_EX.DiskSize;
end;