A toggle button is a button which stays down when clicked once and goes up when clicked a second time.
When to use
Use a toggle button to indicate a state.
Example: A word processor should use toggle buttons to indicate the state of "Bold", "Italic" or "Underline" formatting.
When not to use
Do not use a toggle button to indicate an action.
Example: A music player should not use a toggle button to implement a Play/Pause button. It should use a normal button and adjust the icon and label to represent the action which would be performed when clicked.
Icon and label
Never change the icon or label of a toggle button. A toggle button is very similar to a check box. It should thus be used in the same way.
The fact that it is pressed down is enough to indicate the current state of the element it represents.
Classic buttons represents an action and their label usually starts with a verb. Label of toggle buttons on the other hand can often be reduced to a noun only. The "noun only" form should be favored over the "verb + noun" form whenever it does not cause any ambiguity as this form reduces clutter.
Example: A toggle button to show/hide a sidebar could be labeled either "Show Sidebar" or simply "Sidebar".