Antialiasing the edges of filled polygons is similar to antialiasing points and lines. However, antialiasing polygons in color index mode isn't practical since object intersections are more prevalent and you really need to use OpenGL blending to get decent results.
To enable polygon antialiasing call glEnable() with GL_ POLYGON_SMOOTH. This causes pixels on the edges of the polygon to be assigned fractional alpha values based on their coverage. Also, if you want, you can supply a value for GL_ POLYGON_SMOOTH_HINT.
In order to get the polygons blended correctly when they overlap, you need to sort the polygons in front to back order in eye space. This method does not work without sorting and requires an alpha buffer. Before rendering, disable depth testing, enable blending and set the blending factors to GL_ SRC_ALPHA_SATURATE (source) and GL_ ONE (destination). The final color will be the sum of the destination color and the scaled source color; the scale factor is the smaller of either the incoming source alpha value or one minus the destination alpha value. This means that for a pixel with a large alpha value, successive incoming pixels have little effect on the final color because one minus the destination alpha is almost zero.
Since the accumulated coverage is stored in the color buffer, destination alpha is required for this algorithm to work. Thus you must request a visual or pixel format with destination alpha. OpenGL does not require implementations to support a destination alpha buffer so visual selection may fail.