Let's take a look at the first image AAsb1.jpg, which has been enlarged to the pixel level. In the top left is the original selection. It is just an irregular shape dragged out with the Freehand Selection tool without any aliasing or feathering. What exactly has been selected is obvious when we paste the selection as a new layer over a contrasting background. This is shown at bottom left.

If we take the original selection and run the Shape Based Antialias filter with a setting of Outside, we get the image at top right. The selection marquee has expanded by one pixel. However, to understand what happened in the expanded region we paste the result as a new layer over a contrasting background (bottom right). You can see that, in the region added to the outside of the original selection, the degree of selection has been modified according to the object shape. Where the sides of the object are straight very little is added outside the original selection. However, where the edges of the object are jagged more is selected so that the edge appears smoother. You can get an impression of this smoothing effect by setting the zoom factor to 8% or so.

A setting of Inside for Shape Based Antialias works very similarly. The difference is that, instead of extending the selection outwards, the antialiasing is carried out inside the edge of the current selection. Though this is not really what happens, you could could think of the Inside setting as first contracting the selection by one pixel and then applying the Outside antialiasing scheme. (It couldn't quite be done this way in reality since you would lose some pieces of the selection but it gives the general gist.) When the image outside the initial selection is a different color from the content of the selection you should use the Inside method. Otherwise your antialiased selection will have a halo of a different color - the outside color.

Now let's look at a more complicated situation involving transparency
shown in AAsb2.jpg (below)

On the extreme left is the original selection. There is a pink layer and, on a second layer above it, we have a green selected object surrounded by transparency.

The top row on the right illustrates what happens with Inside Shape Based Antialias. There is no change in the marquee because all the antialiasing is inside the edge of the original selection. However, when we paste the result as a new layer over a fresh piece of background we get the result at top right. The antialiasing is clearly apparent.

The bottom row shows Outside Shape Based Antialiasing. As expected, the selection marquee expands. However, there is nothing in the expanded region because outside the original selection there was only transparency. Consequently the antialias region contains varying degrees of selection of a transparent part of the image. These varying degrees of selection are readily visualized by flood filling the processed selection with a different color. The results of a blue fill are shown at bottom right. The smoothing effect of antialiasing can be seen by setting a zoom of 8% or so.

When a selection is modified by Inside or Outside Shape Based Antialiasing there is no immediate indication that something has happened (expect that in the Outside case the marquee expands by one pixel). Colors don't change. Nor to transparent areas of the image magically fill with color. It is only when you do something to the selection, e.g. copy and paste, flood fill, paint, etc. that the change becomes apparent.

Note that the antialiasing in this filter is determined completely by the shape of the object. The filter pays absolutely no attention to color or transparency. It doesn't care what is in the image - only how the boundary of the selection changes direction.