I’ve looked at a dozen tutorials on removing background in Photoshop, and they all seem to do the same thing. I’ve followed the usual route of “create a new layer – select your subject – invert selection – press Delete” as described, for example, here. In itself, it does what it should no problem. However, the sharp cut-off from 0 % transparency (the subject) to 100% transparency (background) creates a pixelated edge, extremely visible when placing a dark subject on solid background.
How can I blur/antialias this edge with a small radius, 1-2 pixels, so that there is fade-out rather than cut-off? Note that this blur should only be present in the transparency channel, not the RGB image.
The short answer to your question would be to include a step between making your selection and inverting it: choose Select > Modify > Feather, and choose a 1~2px value.
Do yourself a huge favour, however, and rather than inverting your selection and hitting Delete, click on the ‘Add Layer Mask’ icon in the Layers palette. It looks like a rectangle with a dot in the middle. This will (visually) have the same effect, but it hides the background rather than deleting it, allowing you yo edit the mask to correct mistakes later.