Some layer styles can cause problems when the layer subsequently gets a layer mask. This quick technique will get around the problem.

Figure 1: The original image.

Figure 1 shows a foreground object on its own layer and a background (which, coincidentally, is a shot of rural southeast Iowa where I grew up). The top layer, with the foreground object, has a strong Outer Glow layer style. I sometimes face a problem when I add a layer mask to the top layer and mask it with a soft brush: the layer style affects all pixels, so unless I mask all pixels completely in an area I’ll end up with a thick Outer Glow that I can’t mask effectively by masking the layer. See Figure 2.

Figure 2: I masked the object, but the Outer Glow remains.

What I need is a way to mask both the pixels and the layer style, and it can be achieved by adding the layer to a group and masking the group. Photoshop’s masking capabilities are more complex than we often realize: we can mask layers, groups and even vector masks. We can even mask a layer with a layer mask and then mask that combination with a vector mask!

Pressing Cmd/Ctrl-G when your layer (or layers) are selected will place them in a group (you’ll see a folder icon in the Layers palette). Once that’s done you can apply a layer mask to the group and mask away. The result is something like Figure 3 below.

Figure 3: The final image.