What are Layers and Layer Masks?
Layers are either pixel images or adjustments "stacked" on top of the background layer that appears when you open an image in Photoshop or Elements. Learning to use layers is a very important step in advancing through image editing as layers allow you to modify your image without damaging the initial state (non-destructive editing). Layers are not available to you in Lightroom, so advanced editing must be done in Photoshop or Elements. Layers allow you to modify the image or to permit such things as compositing images. They are valuable for detailed and specific editing of local areas of an image using adjustment layers. Many global corrections are possible within ACR and Lightroom and some general local corrections can be done with tools like the adjustment brush, but very specific local corrections are better done in Photoshop using layers and their attendant masks.
Masks are attachments to layers that can restrict the application of the pixels in an image composite or the corrections in an adjustment layer to the desired parts of an image while keeping them from occurring to other areas. A white mask is the default and permits the content of the layer to show completely. An example would be a second different image on a layer above the original background layer. A white mask on that layer would permit the image to show, and it would completely hide the original image under it. It would be like putting one print on top of another on your desk as the bottom print would no longer be visible. If the mask were black the top image would not show at all.
In the image at the right the Curves 2 layer shows a mask that is black at the bottom and white at the top and is based on a selection of a portion of the sky. This allows the top of the image (the white in the mask) to be adjusted by the curves and keeps that adjustment from affecting the portion of the image where black appears on the mask. Masks can be made from selections or painted in with the brush tool and can be modified in many ways to get the desired results.
A black mask hides the content of a layer, preventing anything on the layer or any correction made by an adjustment layer from having any affect on the layer below. It is as if the top layer did not exist. Any shade of gray lighter than pure black will allow a percentage of the top layer to be seen or to affect the image below. Varying the value of the gray on a mask allows you to vary the intensity of the current layer in relation to the background or other layer below it. These modifications can be added to the mask using the Gradient Tool, or the Brush Tool or even by filling a portion of the mask from a selection. Modifications to the mask density can also be made using Levels or Curves.
Masks can be created from selections made from the image. With a selection active, adding an adjustment layer will automatically create a mask which allows the selected area to be adjusted and hide the adjustment from the remainder of the image. This can be used to make local color corrections to skin tones, for example, or to control the contrast of a local area such as a sky without allowing those adjustments to show on other parts of the image. A mask created from the sky area of a background layer can be applied to a second image on top to allow the sky from the top image to replace the sky in the background image. Getting the idea?
Layers can be stacked on top of other layers and will affect all the layers below them or can be "clipped" to the immediate layer below to modify only that layer. With these controls you can modify various aspects of your image to get the desired affect. Initially, this concept can be a little hard to wrap your head around, but once you see the effects on an image you should quickly grasp the idea. Without layers and their masks compositing images or making discrete modifications to an image would be very difficult, and impossible to remove without starting over from scratch. Since any layer can be removed without damaging the background layer below it, any mistakes or changes you want to make are easily redone or simply removed. This means that great decision you made last night can be dismissed in the morning light and a better one made in its place.
Pixel and Non-Pixel Layers
Pixel layers are layers which have actual content (pixels) rather than being adjustment layers. A Levels or Curves layer, for example, does not contain pixel information but is a modification of the layer(s) below it controlled by changes in the default for that layer which is typically no change at all. An adjustment layer contains an icon rather than an image at the left of the mask indicating what type of layer it is. In the image at the top of the page there are two pixel layers, the Background layer and Layer 1 which is a duplicate of the Background layer. All the other layers above are adjustment layers and not pixels layers.
My practice is to keep pixel layers at the bottom of the stack as much as possible with adjustment layers added above. This prevents a pixel based layer, such as a merged visible layer, from rendering an adjustment layer ineffective as an adjustment under a pixel layer is effectively hidden by the layer content and no longer usable. This is not as difficult as you might think once you get used to the idea. Doing so allows you to make adjustments to the image and return to the bottom pixel layers to make changes in retouching.
Typical practice is to make a duplicate layer using Control/Command J immediately above the background layer to be used for retouching. This immediately doubles the file size as it doubles the pixels in the image. While this is not a really big problem in most cases, retouching can also be done on blank layers which will reduce the added file size. Add a blank layer using the icon just to the left of the trash icon at the bottom of the Layers dialog or by using the keyboard shortcut Shift Control/Command N. The new layer will be added above the active layer so make sure you are on the background layer when you do this. When you use the keyboard shortcut a dialog will open allowing you to give the layer a name, mark it with a color and/or change the blending mode of the layer. Simply press the Enter key to dismiss the dialog is you have no need for the options.
Most retouching is done with the Clone Stamp tool or a Healing Brush Tool which will both open with the options bar showing a drop down for "Current Layer", "Current and Below", or "All Layers". Choose the appropriate one for your needs which typically is "Current and Below" to allow the tool to access the content of the background layer as the source material for the work you will do, assuming that the layer immediately below is the desired source layer. Choosing "All Layers" often gives you unwanted results if an adjustment layer above has noticeably modified the tones in the image as the retouched content will reflect those changes as well, adding the changes which are already being modified above, resulting in doubling the effect which is rarely what you are looking for. If you retouch on a layer added at the top of a stack which has adjustment layers below it choosing "All Layers" might seem to make more sense, but now you are asking for trouble as added adjustments above this layer might result in undesired effects. This is why I suggest retouching, being a pixel layer, be at the bottom of the stack.
The third icon from the left at the bottom of the Layers pallet is a rectangle with a dot in the middle. This is the "Add Layer Mask" icon. A layer without a mask, such as a blank retouching layer or a composite layer will not by default have a mask attached to it and this is how you add one if you need it. Masks are added to pixel layers to allow compositing of information from one layer to another.
Copying another image into an existing image will position the new image above the active layer. Again, I suggest doing this at the bottom of the stack so that adjustment layers above intended for the composited image are not compromised. Since the new layer is a pixel layer its immediate affect is to hide the content of the layer below it. This is where the power of the layer mask comes into play with a pixel layer and what makes image compositing possible. We need a selection of the area we want to reveal and a layer mask to make that happen. That is often done by simply painting on the mask with a brush with white as the foreground color as seen in the image at the right.
While anything can be composited into another image, the most common and usually the first composite someone makes is to add a better sky to an otherwise good shot. The trick here is to make a selection of the sky area of the good shot. My usual process is to select the sky area loosely with the lasso tool. This restricts the next step to the sky area rather than the entire image. Using Color Range, select the sky area adding to the selection with the plus eyedropper until you get something useful.
Now you select the new layer and use the "add layer mask" button at the bottom of the layers pallet. The active selection creates the mask allowing your new sky to appear in the original layer. This should give you a good insight into the value of masking rather quickly. There will be problems with the mask, but you are off to a good start.
Selections in Photoshop are never perfect. Anti-aliasing of subject edges means selections are usually a pixel or two short of what we would prefer to have selected. This is not a masking tutorial so the details will not be resolved here, but Refine Edge is the next step to molding the mask edges to perform in a more pleasing manner.
You can use selections in the same way to add adjustment layers to an image and make modifications to local areas. With an active selection choose your adjustment and the mask will be added along with the adjustment layer. Refine edge or other manual manipulation of the mask will probably be needed and this is a skill that can take a while to master.
Adjusting the mask
Typically a layer mask made from a selection such as color range can be too specific. Mathematically derived selections can make one pixel black and an adjacent pixel white depending on their values. This will make the adjustment visually obvious when the intention is to blend it into the image. Refine Edge will appear in the Porperties dilaog whenever a mask is selected (active) or you can manipulate the mask manually with tools like the paint brush, blurring and even Levels.
Blurring the mask using Gaussian blue is one common means of softening the selection into a more usable mask. Blurring creates a gray transition area between the white and black pixels and can be done with a low or high radius depending on the desired effect. Using Gaussian blur is better than "feathering" a selection made with a lasso or similar tool as it gives you better feedback. Feathering can be a good means of controlling a selection that might be part of an action as it locks in a specific amount of control.
Another interesting way to modifiy and control a mask is using Levels. Since a mask is generally blacks and whites with a grayscale blending transition after blurring, running levels allows you to modify the mask and the way the adjustment blends into the image. Again, this is a control that becomes way more obvious once you see it used, and is a powerful means of making masks meet your needs.
Viewing the Mask
You can see the mask instead of the image by holding down the Alt/Option key and clicking on the mask itself. This will toogle the mask view and the image view so you can see what the mask looks like and also check for holes in the mask that should not be there and otherwise make changes to the mask as you need. Masks almost always need to be blurred to be blended into the image without making themselves obvious. Working with a mask is more than simply making a selection and in some ways is an art unto itself.
So, now you have an idea of why Layers and Masks are important to working in Photoshop or Elements. Start teaching yourself to work with them rather than trying to work on the original image so you can preserve the integrity of the capture and as a foundation for better image editing.