For the past century, there has been much discussion and debate about the equations of motion satisfied by a classical point charge when the effects of its own electromagnetic field are taken into account. Derivations by Abraham (1903), Lorentz (1904), Dirac (1938) and others suggest that the "self-force" (or "radiation reaction force") on a point charge is given in the non-relativistic limit by a term proportional to the time derivative of the acceleration of the charge. However, the resulting equations of motion then become third order in time, and they admit highly unphysical "runaway" solutions. During the past century, there also has been much discussion and debate about the interpretation of these equations of motion and the conditions that can/should be imposed to eliminate the runaway behavior. We argue that the above difficulties stem from that fact that the usual notion of a point charge is mathematically ill defined. However, a mathematically rigorous notion of a point charge arises in a perturbative description of a body if one considers a limit wherein not only the size of a body but its charge and mass go to zero in an asymptotically self-similar manner. We show how the Abraham-Lorentz-Dirac self-force then arises in a perturbative description of the body's motion, but does not give rise to runaway behavior. As a biproduct of this work, we also rigorously derive dipole forces and resolve some paradoxes of elementary physics, such as how a magnetic dipole placed in a non-uniform magnetic field can gain kinetic energy despite the fact that the magnetic field can "do no work" on the body.