When you use the = (equals) operator, PHP performs a "copy equals" - it takes the value from operand two and copies it into operand one. While this is fine for most purposes, it doesn't work when you want to be able to change operand two later on and have operand one also change.
In this situation, references are helpful - they allow you to have two variables pointing to the same data. Using references, setting operand one to equal operand two is instantaneous. Furthermore, once two variables are pointing to the same data, you can change either variable and the other one will also update. To assign by reference you need to use the reference operator, &, after the equals operator, giving =&. Here's how it looks:
$a = 10;
$b =& $a;
Here we're using the reference operator to make $b point to the same value as $a, as can be seen in the first two print statements. After incrementing $a, both variables are printed out again, and both are 11, as expected. Finally, to prove that the relationship is two-way, $b is incremented, and again both $a and $b have been updated with the one call.
In the example above, simple integers are used to demonstrate referencing, however references aren't any faster than normal copy assigns for such data because there is so little to copy. Assigning by reference is generally only important if you want to take advantage of the "two variables, one value" paradigm.
When it comes to functions, references are also used to allow a function to work directly on a variable rather than on a copy - more on that later.
In PHP, objects are always copied by reference - more on that later.