Stitching

The stitching system is used to stitch a set of tiled images into a single mosaic.

The Stitch Manager

The Stitch Manager is used to control the stitching process. It can be opened by pressing the Stitch Manager button in the Alignment tab.

stitch_manager_icon

stitch_manager

The combo box at the top is used to select the active image layer from the list of visible image layers. There are four buttons under the combo box:

  • Stitch - Starts the stitching process for the active layer. During the stitching process, this button turns into a stop button.
  • Clear Stitch - Clears the stitching information that is associated with the active layer.
  • Open Stitch Viewer - Opens the Stitch Viewer window, which can be used to visualize and edit the selected stitch object.
  • Settings - Opens the Settings dialog for the Stitch Manager.

The rest of the window is used to show the list of stitching errors. If the layer has not yet been stitched, then this list will be empty.

Whenever the Stitch Manager window is open, a stitch grid for the active layer will be displayed in the overlay:

white_stitch_grid

The grid will initially be white, but during the stitching process, the lines and dots will turn green or red.

The Stitching Process

To start the stitching process, press the Stitch button in the Stitch Manager. The stitch grid will change colors throughout the stitching process. During the stitching process, it is perfectly fine to change the active layer or close the Stitch Manager window. The stitching process will continue to run in the background, even if the stitch grid is not displayed in the overlay.

Once the stitching process is complete, the stitch grid can be inspected with the Stitch Viewer. First, a stitch object must be selected. A stitch object can be selected by using the Select Stitch tool:

select_stitch_icon

After activating the Select Stitch tool, click on a line or dot in the stitch grid to select it.

The lines in the stitch grid represent tile pairs, which are two tiles that share a common edge:

selected_tile_pair

The dots in the stitch grid represent tile quads, which are four tiles that share a common corner:

selected_tile_quad

Finally, click on the Open Stitch Viewer button to start viewing the selected tile pair or quad.

The Stitch Viewer

The Stitch Viewer has three modes: Tile Pair mode, Tile Quad mode, and Row/Column mode.

Tile Pairs

If a tile pair is selected, then the Stitch Viewer will start in Tile Pair mode:

tile_pair_viewer

The purpose of the Tile Pair viewer is to allow the user to view the results of searches that were run during the stitching process, and to allow the user to run their own searches. What is a “search”? In the stitching system, a “search” is when Pix2Net attempts to find valid stitch points for a pair of tiles.

Clicking the Search button will bring up the Search dialog:

broad_search_dialog targeted_search_dialog

There are two types of searches: Broad and Targeted. The purpose of a broad search is to determine if there is only one possible way to stitch a tile pair or not, and if there is, to find that stitch point. The purpose of a targeted search is to find the best stitch point near a target point.

Let’s start with the broad search. A search always has a major axis, which points in the direction of stage movement, and a minor axis, which is perpendicular to the major axis. Since we are dealing with a vertical pair of tiles, the major axis is the Y axis and the minor axis is the X axis.

The most important part of the broad search is defining the search boundaries. If we assume that there is some overlap, then there are two possibilities: Either the top tile’s lower-left corner is overlapping with the bottom tile’s upper-right, or the top tile’s lower-right corner is overlapping with the bottom tile’s upper-left, as shown in this figure:

broad_two_cases

In a broad search, the “minimum overlap” and “maximum overlap” determine the worst case scenarios for both cases. A broad search will perform two tests: In the first test, it will take the bottom tile’s image in the “minimum overlap” scenario of case 1 (the “template” image), and it will try to find it in the top tile’s image in the “maximum overlap” scenario of case 1 (the “reference” image). In the second test, it does the same thing, but for case 2. If these two tests produce only a single stitch point, then the broad search has successfully determined that there is only one way to stitch these two tiles. However, if the broad search finds either no stitch points or multiple stitch points, then the broad search’s result will be considered invalid.

The “maximum slice length” in each axis determines how the template image will be sliced up. The reason that a template image is divided into slices during a search is because, if either tile is warped, it may not be possible to find the entire template image in the reference image. The default parameters will divide the template image into three slices along the minor axis.

The “minimum correlation” determines the lowest amount that a template image is allowed to match a reference image and still be considered valid. If there are multiple places where a template image matches a reference image, then “correlation margin” is used to determine if the lower correlations are real matches or not. For example, if the highest stitch point has a correlation of 98%, and the “correlation margin” is 2%, then another stitch point of 97% would also be considered valid (because it is 1% away from 98%), but a stitch point of 95% would not (because it is 3% away from 98%).

The “distance penalty” doesn’t make sense for a broad search, so it should always be zero. The “maximum warp” is the percentage of tile width that the stitch points of the slices are allowed to vary from each other, under the assumption that the tiles can’t possibly be warped enough for both of those slices to be correct. For example, let’s assume a tile width of 2048, a “maximum warp” of 1% (i.e. about 20 pixels), and a slice with a stitch point at 0, 1591. In that case, a second slice point at 0, 1593 would be considered valid (because it is only 2 pixels away), but a third slice point at 0, 1621 would be considered invalid (because it is 30 pixels away).

A targeted search, on the other hand, always returns the best set of stitch points it can find. A targeted search requires the user to specify two target points, at opposite sides along the minor axis. In the case of a vertical tile pair, the first target point is specified on the left side and the second target point is specified on the right side. Pix2Net then chooses a target point for each slice by interpolating between the two extremes. Finally, the “Allowed Error” determines the maximum distance, as a percentage of tile width, that a valid stitch point may be from the interpolated target point.

The targeted search will always return the stitch points with the highest correlation, provided that correlation is higher than the minimum correlation and the slices are within the “maximum warp” constraint. There is just one exception: If the “distance penalty” is greater than 0, then the correlation at each possible stitch point will be adjusted by subtracting the distance penalty times the distance from the target point. The distance penalty is very important for tiles that have many valid stitch points, because the distance penalty will force Pix2Net to lean towards stitch points that are closer to the target point, instead of choosing stitch points solely based on their correlation.

At the top of the Search dialog, there is a “Load settings” button. This button can be used to populate the Search dialog with the standard settings used in each pass during the stitching process. Here are the three passes the occur during the stitching process:

  • Pass 1 - In this pass, a broad search is run on each tile pair in order to find a list of known stitch points for each row and column.

  • Pass 2 - In this pass, the list of known points found in pass 1 are used to find a line of best fit for each row and column. This line of best fit

    is then used to run a targeted search, with 3% error and 0% distance penalty, on each tile pair.

  • Pass 3 - In this pass, the tile pairs that still have invalid stitch points are fixed by running a targeted search with 1% error and a 0.1%

    distance penalty. The assumption is that the searches that failed in pass 2 are the searches in which distance needs to be considered, instead of just looking at the correlation.

That covers the search button.

The table on the lower left, beneath the Search button, shows the list of results from the various searches that have been run. It also includes a Verification result, which will be covered in just a little bit. For now, click on a “Pass 2” or “Pass 1” result to see what a search result looks like.

The Clear button can be used to clear the selected search result. The Left Quad and Right Quad buttons are equivalent to selecting the left or right quad in the stitch grid. The Row/Column button will put the Stitch Viewer in Row/Column mode, so that the user can see how this pair’s stitch points compare to the other stitch points along the same row or column.

The table on the lower right contains the list of stitch points for each slice in the selected search result. For targeted searches, the combo box above the table will always be “Correlation”, but for broad searches, it can be changed to “Test stage moved left” or “Test stage moved right”, to specify which of the broad search tests to view.

The table will always contain an entry that does not have an “X” or “Y” coordinate, which is selected by default. This entry is an overview of the slices. It shows the two tiles separately, with a green rectangle around the template image, and a yellow rectangle around the reference image. It also shows a mosaic on the right, which is generated from the stitch points of the valid slices.

Clicking on the other entries will show the template and reference rectangles for that particular slice. It will also show a correlation image, which has darker pixels for the low correlations and brighter pixels for the high correlations. If the user hovers the mouse over the image, then the corresponding correlation value is displayed in the status bar. For targeted searches, a green dot represents the target point for the slice (for broad searches, the green dot is meaningless). For targeted searches, a red dot represents the best correlation point. For broad searches, a single red dot represents a valid stitch point. The lack of a red dot indicates no valid correlation points were found, and two red dots indicate that there are at least two possible ways to stitch the tiles.

A mosaic shows the template image on top of the reference image at the highest possible correlation point. Finally, a difference image shows the distribution of the differing pixels between the template and reference images. The smallest differences are represented by dark pixels, and the largest differences are represented by white pixels.

Finally, the Verification entry in the lower left table is a special correlation that the stitching process runs on the final, warped tiles. If any new searches are run, the verification entry will immediately vanish, because the final warped images may have changed, so a new verification needs to be run. A verification consists of doing a correlation in which the template image is divided into five slices, and the reference image is 3 pixels bigger than the template. The “score” that the verification reports is the lowest correlation among the five slices. The “potential” is calculated by looking at the worst slice’s current correlation score, and the highest score that can be obtained by moving template image around by just 3 pixels. A verification with a low score and a high potential means that at least one of the tile’s stitch points is definitely wrong.

Tile Quads

If a tile quad is selected, then the Stitch Viewer will be in Tile Quad mode:

tile_quad_viewer

The purpose of the Tile Quad viewer is to show the user what four tiles look like, given the stitch points currently defined. Clicking on the Row or Column button will put the viewer in Row/Column mode.

The score, which is displayed to the right of the Row and Column buttons, is calculated by checking the consistency of the four tile pairs. A lower score is better; a score of zero indicates perfect agreement. A score of 5 or greater indicates that the tile pairs disagree by at least 5 pixels, and the quad will be colored red in the stitch grid.

The stitch points for each tile pair are displayed in the table in the lower left. The user can visit any of these tile pairs by double-clicking on them. The “Expected” column is the stitch point for each tile pair that would be required to give the quad a perfect score of zero. If pass 3 finds a tile quad with 3 valid tile pairs and 1 invalid tile pair, then it will use the expected point when it tries to fix the invalid tile pair.

Rows/Columns

The Stitch Viewer can be placed in Row/Column mode by clicking on the Row or Column buttons in the Tile Pair and Tile Quad modes:

row_column_viewer

The purpose of the Row/Column viewer is to visualize the trend of stitch points along a column or row. The top graph represents the X values of the stitch points, and the bottom graph represents the Y values of the bottom graph. In both graphs, the tile number is the x axis, and the stitch point’s X/Y value in pixels is the y axis. In this view, the tile number is not the usual index of the tile in the layer, but instead, it is the index of the tile along this particular row or column.

In the lower left, the view can be changed from “Active Result” to “Manual”, “Pass 3”, “Pass 2”, or “Pass 1”, in order to look at the results of a particular pass. The trend line, which is determined by the results of “Pass 1”, can be toggled on and off.

The table in the middle contains all of the stitch objects along this row or column. The user can visit these stitch objects by double-clicking on them.

Clear Stitch Grid

The Clear Stitch Grid dialog can be opened from the Stitch Manager:

clear_stitch_grid

This dialog can be used to clear the stitch grid. It can also be used more selectively, so that only certain passes are cleared, or so that only the warped tiles are deleted.

Settings

The Settings dialog can be opened from the Stitch Manager:

settings

The settings dialog controls the verifications that are shown in the Stitch Manager. By default, verifications are only shown if they have a score lower than 97%, and if they are not along the border of the layer.

The final thing that can be changed in the Settings dialog is the type of stitch grid that is drawn. By default, the stitch grid is white when stitch information is missing, green if stitch objects are valid, and red if stitch objects are invalid. However, if the stitch grid is changed to Degree of Warp, then the colors will represent the extent to which tiles have been warped along their edges. White indicates almost no warp, green indicates a small amount of warp, yellow indicates medium amount of warp, and red indicates a large amount of warp.

Fixing Stitch Errors

During the stitching process, Pix2Net will run three passes. Each pass runs a search that returns a valid result (a green line) or an invalid result (a red line). There are times when pass 1 and pass 2 return a red line, but that does not necessarily mean something bad happened; there are many times where pass 1 and pass 2 are supposed to return invalid results, because it is expected that a later pass will find a valid result. Therefore, search results can be divided into four categories: True positives, true negatives, false positives, and false negatives.

In general, false positives are much worse than false negatives in the stitching process. A false negative is simply a piece of information that could be used, but is ignored by Pix2Net. A false positive, however, represents a bad assumption that may be used by later passes, resulting in more bad assumptions and stitch errors. Therefore, the search parameters for each pass have been chosen with two goals in mind: 1) Eliminate false positives completely and 2) avoid an excessive amount of false negatives.

Pass 1 Errors

If the stitching completes and a row or column has a large amount of red lines, that’s a pretty good sign that pass 1 failed on that row or column. In order to verify this, select a tile pair on that row/column, open the Stitch Viewer, and switch to the Row/Column view. If you set the view to ‘Pass 1’, and you notice that the trend line is tilted due to some outliers, then those outliers are most likely false positives.

In pass 1, a false positive is when pix2net believes there is only one valid way to stitch a tile pair, but there are in fact either multiple ways to stitch the pair, or there is no valid way to stitch the pair. A false negative, on the other hand, is when pix2net believes there is not a single valid way to stitch a tile pair, but there actually is only one valid way to stitch the pair.

False positives are very bad, because it only takes one false positive to throw off a trend line. False negatives aren’t nearly as harmful, because a false negative simply means there is one fewer data point to help guide the trend line.

Currently, there’s no easy way to simply run pass 1 with different parameters on an entire row or column, so fixing the results of a pass 1 error may be tedious. Pix2Net tries to reduce the chances of a false positive by using a high minimum correlation (98%) and a high correlation margin (1%). On a side note, if pass 1 does not find any valid stitch points along a row or column, then pass 2 will default to using an ideal trend line.

Pass 2 Errors

Pass 2 will attempt to stitch tile pairs along the trend line from pass 1. Pass 2 will use a distance penalty of zero, so pass 2 will only work if the highest correlation point in each slice is the correct stitch point.

A false positive in pass 2 is when a search result is marked as valid, but the stitch points are actually incorrect. This can happen if the highest correlation point in each slice is wrong, but the points are still within the allowed warping distance, so each slice is consistent with the other slices. A pass 2 false positive is not as bad as a pass 1 false positive, but it can still be pretty bad if a lot of nearby pass 3 stitches rely on it in order to generate their target points.

A false negative in pass 2 is when the highest correlation point in each slice does in fact represent a valid way to stitch the pair, but pass 2 incorrectly marks the search result as invalid. This can happen if the tiles are so warped that the resulting stitch points violate the maximum allowed warping distance. It can also happen if the slices have a low correlation, due to charging effects or excessive tile warp. Pass 2 uses a minimum correlation of 95%, which is lower than pass 1’s 98%. However, 95% may still not be low enough to prevent all false negatives.

In contrast to a false negative, a true negative in pass 2 is when a search result correctly identifies a situation where the highest correlation point in each slice does not represent a valid way to stitch a tile pair. A good example of this case is a pair of tiles that have nothing but horizontal white lines, or nothing but vertical white lines. There are many possible, valid stitch points, so pass 2 will not be able to determine the correct way to stitch the pair. A pass 3 search, on the other hand, will work, because pass 3 includes a distance penalty that can be used to sort the valid stitch points by their distance from a target point.

Pass 3 Errors

Since pass 3 has a low minimum correlation of 90%, and a distance penalty of 0.1%, it will almost certainly work if valid target points are passed to it. However, the target points used may end up being invalid target points if they were calculated from bad results from pass 2 or pass 1.

Also, pass 3 will only run searches on invalid pass 2 results. Pass 3 will not run on valid pass 2 results, because it does not know which pass 2 results are true positives, and which results are false positives. This means that pass 3 will not fix a red dot that is connected to four green lines, because pass 3 does not know which of those lines is a false positive. However, you as a human may be able to determine this by selecting the red dot, opening the stitch viewer, and looking at the row and column views. Look closely at the stitch points of the pairs that are adjacent to the selected quad. If one or more of those stitch points seems to be out of place in comparison to its neighbors, then that is probably the false positive. You may be able to fix this false positive by simply running a pass 3 search on it.

One final thing to notice is a difference between the way pass 2 and pass 3 choose their targets points. When pass 2 calculates its target points, it uses the trend line from pass 1. When pass 3 calculates its target points, however, it may use the nearby pass 2 and pass 3 search results. Therefore, a pass 3 result may be bad because of other bad pass 3 results, but a pass 2 result will never be bad because of other bad pass 2 results.