Contents


Technical Notes

Internal Data Structures:

Windows:

AirPhotoSE uses a Windows Multiple Document Interface (MDI) structure which permits simultaneous independent display of two images, limited of course by visibility and the physical dimensions of the display. Dual displays are also supported, so that multiple images may be viewed and worked on in two screens simultaneously.  The user must set this up by right clicking on the Windows Desktop, then selecting Properties, Settings and if the proper driver for the display controller has been installed and two monitors are connected and turned on, check the box "Extend my Windows desktop onto this monitor" to activate the secondary display after checking the second monitor image.

The programme detects the number of available processors and where the algorithm and image geometry permit, it uses multi-threading whereby different portions of an image are processed simultaneously.



Images:

AirPhotoSE stores images as Windows Memory Mapped Files on disk if they are very large, otherwise they are stored in memory. This overcomes the operating system limit on the size of images which can be displayed and processed. Windows Bitmaps are stored in shared Paged Pool Memory.  Under WinNT/2000/XP/Vista/Win7, the maximum available paged pool space in a 32 bit Windows workstation system is 491.875 MB and 650 MB on a Windows 2003 server. This limited space is insufficient to store the large satellite, orthophoto and digital map images now available.  file mapping object can consist of all or only part of the file. It is backed by the file on disk. This means that when the system swaps out pages of the file mapping object, any changes made to the file mapping object are written to the file. When the pages of the file mapping object are swapped back in, they are restored from the file. The file view can consist of all or only part of the file mapping object. A process manipulates the file through the file views. A process can create multiple views for a file mapping object. The file views created by each process reside in the virtual address space of that process. When multiple processes use the same file mapping object to create views for a local file, the data is coherent. That is, the views contain identical copies of the file on disk. The file cannot reside on a remote computer.

Control and Calibration Points:

AirPhotoSE stores control and calibration points for each image object in two self-balancing binary trees. One is indexed on the point ID numbers, the other is indexed on the x-y pixel coordinates.  

Each control point in each  tree has data stored with a pointer to a record:

  PControlPtRecord = ^ControlPtRecord;
  ControlPtRecord = record
    Id: Integer;
    X: Integer;
    Y: Integer;
    Z: Integer;
    CoordX: Extended;
    CoordY: Extended;
    CoordZ: Extended;
    IsEnabled: Boolean;
    BackgroundBitmap: TIEBitmap;
    ForegroundBitmap: TIEBitmap;
  end;

Id is the control point's identification number. X and Y are pixel coordinates of the position of the control point in the image, Z is a 32 bit integer whose bits are used for specialised flags, e.g. for noting that the coordinates of the control point lie outside the image itself if it has been entered using the line intersection option in control point entry or specifying an EPSG grid number.  CoordX, CoordY, and CoordZ are included for compatibility with control point records in AirPhoto where they contain geographic coordinates in decimal latitude, longitude and height above the chosen ellipsoid. In AirPhotoSE all values are zero. IsEnabled is a flag which is turned off if the control point is disabled.  The Background Bitmap is a small image of the area where the control point data will be located, copied from location in the memory mapped file window. The Foreground Bitmap is the same area with the control point number and symbol written over the background image data.  This is copied to the memory mapped file to display a control point. To hide a control point, the background is copied instead. If an image is rotated, scaled, or transformed, all control point images are first rendered invisible by copying all backgrounds for all points to the memory mapped file window. Then, after the operation is completed, the backgrounds and then the foregrounds are refreshed at all the image points with appropriate orientation changes if required.

Calibration points are stored with a similar record:

PCalPtRecord = ^CalPtRecord;
  CalPtRecord = record
    Id: Integer;
    X: Integer;
    Y: Integer;
    Z: Integer;
    CoordX: Extended;
    CoordY: Extended;
    CoordZ: Extended;
    IsEnabled: Boolean;
    CalCoordinates: CoordinateType;
    BackgroundBitmap: TIEBitmap;
    ForegroundBitmap: TIEBitmap;
  end;

except for the addition of the coordinate type variable which specifies the grid coordinates.  WGS84-UTM, Decimal Longitude/Latitude and arbitrary X-Y coordinates are available. Coordinate types for points in a given image may not be mixed.  GIS input and output is supported for ArcGis and MapInfo. However, X-Y coordinates are not available for GIS output in MapInfo format.

All the records in both trees are streamed to a binary file when either the user specifically request saving control points or optionally when an image is closed.  These files have the same name as the image file but with a .ctl extension. They are automatically re-read into the trees when an image object is loaded and the background and foreground bitmaps are re-created and loaded with image data from the referenced points.  

Algorithms and hardware:

Image transformation and calibration use SVD (Singular Value Decomposition) for a least-squares implementation with an unlimited number of points for computation of a homography between an image and a map or between the positions of the calibration points and their geographic values.

Up to four multiple-processors are supported and images are processed in up to four patches simultaneously (Multi-threading).

Speed in making orthophotos is achieved by automatic computation of keypoints for matching on a reduced size version of each image, the degree of reduction being under user control.  The original images are rectified and applied to a terrain model computed from a point cloud created with the help of bundle adjustment.  

Control Point Algorithms:

A minimum of four control points in a source and target (map, satellite, geoportal image) is required to establish correspondence. The main rectification algorithm used by the programme adapts automatically to the number of points entered.  A modified projective transformation is used to produce approximate correspondence between source and target based on Singular Value Decomposition least squares optimisation 

If there are between 4 and 6 points, a pure projective transformation is computed, If there are more, additional bi-variate polynomial correction factors are introduced, the order of the correction polynomial depending on the available number of points up to a maximum order of five (Scollar et al. 1990), with: 7 to 9, order 2, 10 to 12, order 3, 13 to 17 order 4, 18 or more, order 5. The relative quality of the transformation is obtained by computing the differences between forward and inverse transformation at each control point (Reprojection Error, displayed with Shift-F9).  Higher order corrections aid in reducing the effects of radial lens distortion, atmospheric refraction and hilly terrain.

The Google Earth and the GeoPortal Viewers:

AirPhotoSE uses the Delphi wrapper TWebBrowser around the Microsoft Windows dynamic link library shdocvw.dll. Details are given at:

http://msdn.microsoft.com/en-us/library/aa741313%28v=vs.85%29.aspx <http://msdn.microsoft.com/en-us/library/aa741313(v=vs.85).aspx>


It is also used in the Internet Explorer from version 4.0 onward.  In turn, the AirPhotoSE code for Google Earth calls a JavaScript programme GEDemo.htm which manages Google Earth's GEPlugin and interfaces it to the WebBrowser environment.  GEDemo.htm is created afresh each time Google Earth is loaded in AirPhotoSE to permit any changes in the display format which may have been made to be used.  The Shdocvw.dll defaults to Internet Explorer version 7.  This may result in an error message reporting that the web browser is out of date from a site if it checks the browser version and finds that a version later than 7 is required.  The problem can be overcome with the UseIE menu item in AirPhotoSE's GeoPortal implementation.  

Warning:

If a registry change is made to force the use of a later version of the Internet Explorer with AirPhotoSE to prevent error messages coming from a GeoPortal website, then the JavaScript used to run Google Earth may fail and Google Earth may not open.  Unfortunately, no way has been found to prevent this incompatibility.

Orthophoto and Point Cloud Generation:

Orthophotos from multiple images are created with a series of separate programmes which are spawned from AirPhotoSE.  When a 64 bit version of Windows is detected, an optimized 64 bit version is used to permit processing of a large number of images.

Keypoint Detection:

Keypoint (salient points) are detected using code written by Andrea Vedaldi. The VLFeat open source library implements popular computer vision algorithms including HOG, SIFT, MSER, k-means, hierarchical k-means, agglomerative information bottleneck, SLIC superpixels, and quick shift. It is written in C for efficiency and compatibility

http://www.vlfeat.org/

This contains a much improved and augmented version of the SIFT algorithm due to David Lowe:

http://www.cs.ubc.ca/~lowe/keypoints/

Approximate Nearest Neighbor Searching:

The ANN library by David M. Mount and Sunil Arya is used for creating a list of keypoint nearest neibhbour keypoints.

http://www.cs.umd.edu/~mount/ANN/

Bundle Adjustment:

Bundle adjustment using the keypoint neighbour lists is carried out with code written by Noah Snavely 

http://www.cs.cornell.edu/~snavely/bundler/bundler-v0.3-manual.html

to create a Stanford 3D PLY file.

http://graphics.stanford.edu/data/3Dscanrep/

It in turn uses the code of the sparse Bundle Adjustment package of Lourakis and Argyros

M.I.A. Lourakis and A.A. Argyros.  The Design and Implementation of a Generic Sparse Bundle Adjustment Software Package Based on the Levenberg-Marquardt Algorithm.  Tech.  Rep. 340, Inst. of Computer Science-FORTH, Heraklion, Crete, Greece. Available from

http://www.ics.forth.gr/~lourakis/sba

Point Clouds

If a Point Cloud is desired, the output from Bundle adjustment is refined using the algorithms CMVS and PMVS2  written by Yasutaka Furukawa:

http://www.di.ens.fr/cmvs/

as programmed for 32 and 64 bit Windows by Pierre Moulon:

http://francemapping.free.fr/Portfolio/Prog3D/CMVS.html

in a version from the Open Source Photogrammetry library:

http://opensourcephotogrammetry.blogspot.de/search?updated-min=2013-01-01T00:00:00-08:00&updated-max=2014-01-01T00:00:00-08:00&max-results=1

Cloud Compare:

The PLY output from Bundle adjustment or the Point Cloud is used to create orthophotos from each image and if chosen, a digital terrain model is made using the program Cloud Compare by Daniel Girardeau Montaud:

http://www.danielgm.net/cc/

It offers a huge variety of additional treatments for point clouds which it spawns in turn.

Stitching:

Rough image stitching is done using output of the corner coordinartes of the orthophoto quadrilaterals created in Cloud Compare and with blending and secondary position adjustment using Microsoft's ICE by Matt Uyttendaele and team:

http://research.microsoft.com/en-us/um/redmond/groups/ivm/ice/


Richard Szeliski, Simon Winder, and Matt Uyttendaele, High-quality multi-pass image resampling, no. MSR-TR-2010-10, February 2010

http://research.microsoft.com/pubs/119069/msr-tr-2010-10c.pdf

Programming:

AirPhotoSE was programmed in Delphi 2010 by Irwin Scollar using the Imageen libraries of Fabrizio diVittorio from

HiComponents for many low-level functions. 

It also uses many routines from the Turbo Power libraries.   

Routines from the JEDI Jedi Code Library and JEDI Visiual Component libraries are also widely used.