Homographies in Adobe Photoshop

Marion Massachusetts Progression
Marion, Massachusetts

This animation demonstrates various stages of construction of an orthographically rendered mapping of a scene's light field. 50+ images were taken from within a plane parallel to the wall. Because of parallax, no one transformation could map these images onto one another for all depth at the same time. Each image could contribute to multiple layers in the final composite, each with separate mappings (e.g. wall between legs scaled up to offset foreshortening). Artistic license was taken with the floor and ceiling, which would not be seen in a normal orthographic rendering.

Project Goal

The main goal of this project was to add new methods for manually defining homographies in Adobe Photoshop to better facilitate multi-viewpoint image integration. Although Adobe Photoshop offers a few homography related methods, these are geared more towards working with an image or set of images shot from a single point in space. When trying to combine images taken from different positions in relation to a space, parallax almost always means that there will be no single homography that will cause all corresponding points to map onto each other. The philosophy taken with this project is that an artist is better equipped than a computer algorithm to weigh the tradeoffs between various mappings and that the computers role should be to allow the artist to visualize possible mappings as quickly, intuitively and non-destructively as possible.

New Homography Features

  1. Allow selection of any 4 points of interest for defining a homography. Photoshop only allows one to use a layer's bounding box corners, and these are only rarely the points of highest priority in mapping decisions.
  2. Provide non-destructive perspective transformations; this is achieved by embedding "recipes" for recreating a transformed layer from the original source image.
  3. Offer option to load new images with transform of current layer. This feature is handy when images taken from different positions share the same direction vector. This can also be helpful when HDR compositing for an already tranformed image is desired (i.e. multiple images shot from fixed point with different shutter speeds).
  4. Optimize large file size - Instead of masking out unused pixels for layers, pixels that won't be visible can be aggressively deleted; if these pixels later need to be revealed again, they can be restored by running the transformation recipe on the original source image.
  5. Ability to define a homography by connecting corresponding points between images:

An overarching benefit of all these features is that they allow an artist to work in a more rapid, experimental and iterative fashion. The speed and accuracy with which one can "converge" upon a desired transform will allow for the creation of images of greater complexity.



Embedding Processing Metadata

A key building block to these homography related additions is the ability to embed inside Photoshop's built-in structures the steps needed to recreate a transformed layer. Two pieces of information needs to be stored in order to reload a layer from source with all applied perspective transformations:

  1. The file path of the source image.
  2. A set of 4 pairs of points defining the homography applied to the source image.

The first part of this recipe can be stored in a layer's name using the following form:

layer_description { loadLayer('DSC_1045.jpg'); }

Since most of the code for these additions interact with Photoshop's DOM using Javascript, the above form makes it easy to use regular expressions to pull out multiple commands from a layer name (between curly brackets, separated by semicolons). Once the commands were isolated it was simple to pass them to the Javascripts eval() function. Duplicating layers by normal means preserves this metadata. As a side note, in the future other functions will also be added to handle filter calls such as those that would correct for barrel or pincushion distortion.

The second part of the recipe is a real key to this project. As mentioned above, 4 pairs of points are required to define a homography. Transformations in Photoshop are done by moving the 4 bounding box corners of a layer or layer set to new positions. If we can always reload an image from source, the original bounding box corners for an image represent 4 of the points needed to define a homography; with a source image (e.g. *.jpg, *.dng) this just means knowing the dimensions of the image in pixels. To store the information needed to fully capture a transformed layer's homography, it is only necessary to keep track of where the original 4 bounding box corners have ended up.

This can be tracked by creating a vector path with 4 corner points placed at the 4 bounding box corners of the original source image. This is done when the image is loaded, at the same time as the layer name is altered to include the layer source metadata. This path is stored as a vector mask for this layer. A vector mask is a special type of vector path that is associated, and by default "linked", with a particular layer (in much the same way as a layer mask is a channel attached to a particular layer). A vector mask has many properties which make it ideal for the task of tracking a homography:

  1. When left in its default linked state, normal perspective free transforms, perspective cropping, and Auto-Align Layers affect the points in the vector mask in the same way as the pixels in the layer. That is, the points over the corner bounds of the layer will be moved to wherever there corresponding bound corners are moved.
  2. Vector path points, like all path points, are floating point and thus have sub-pixel accuracy.
  3. Multiple transformations can be applied to a layer, but the vector mask will always store the information for the single homography that goes from source to cumulative transformed state. Since each transformation degrades image quality slightly, it is advantageous to replace a layer with versions which has only once been transformed.
  4. Path points can lie outside of the document bounds (e.g. path points won't get clipped)

Allowing Selection of Points of Interest for Transforms

This feature takes advantage of two of the undocumented action events for cropping with perspective and interactive free transforming with initial transform corner positions. The crop feature in Photoshop has a perspective option, which, when checked, allows one to define 4 points that will become the corner points of the newly cropped and rectified image. By duplicating a selected layer into a new document, we can use this crop feature to force points of interest to be at the bounding box corners. Duplicating this newly cropped image back to the original composition document gives a layer that can be transformed with control points lying at the points of defined interest.

It should be noted that the closer a point of interest is to a control handle, the easier it is to place it. And just as important, being closer to a handle means that image points are less likely to be affected by subsequently moving the other three control points (better local control).

Defining Point Correspondences

Two quadrilaterals used to define a homography. Single quadrilateral sub-paths can be used to define control points for transformations as well.

Normally this procedure wouldn't be of much help, since these steps involve deleting much of the layer we wanted to transform. Also, these two transformation processes can result in strong image degradation. However, our vector path and layer source metadata make it possible to reload the entire layer with a single computed homography, preserving as much image detail as possible.

Defining Homographies through Point Correspondences

This feature relies upon a user defined 'Work Path' to define point correspondences between a base image - or set of images - and the image to be mapped onto it. The Javascript code that handles this feature makes an outside call to another executable I wrote. This takes a list of input points, a list of corresponding output points, and the 4 bounding points of the image to be transformed:


The point correspondences (4 or more pairs) allow for a homography to be computed using singular value decomposition. Passing the bounding corner points through this homography yields the coordinates, in the Photoshop documents space, to which the corner points should be moved when calling the transform command. As with the other features added in this project, the free transform destination points are not committed until confirmed by the user; they may be tweaked further. In the end, multiple transformations often need to be done on different regions of the same image because of parallax. These tools allow for hierarchies of transformations to be made with no image degradation penalty from its depth.

Rich MacDonald - 2008