By representing himself in the penalty phase of his trial for the church massacre in Charleston, S.C., Dylann Roof joins a roster of notorious killers who have gone that route — a list that could serve as a caution to anyone considering following their lead.

Some criminal defendants who act as their own lawyers want a stage to promote an ideology; some apparently want the spotlight or think they can fare better than a real lawyer; some are too controlling to let anyone else be in charge; some are too paranoid to trust lawyers; and some are just delusional.

Whatever the motive, it rarely ends well for the defendant. Judges routinely advise against it, and often insist that court-appointed counsel be on hand as a backup.

Nidal Malik Hasan, an Army major and a psychiatrist, killed 13 people and injured 30 in a shooting spree at Fort Hood, Tex., in 2009. He represented himself in his military trial, telling the court that he had, in effect, switched sides in the war in Afghanistan, and that his motive had been to defend the Taliban.

But Mr. Hasan made no attempt to grandstand at his trial, barely questioning prosecution witnesses, calling none of his own, and making no major statements. The lawyers appointed by the court to advise him indicated that his goal was to be executed and become a martyr.

In 2013, he was convicted on all counts and sentenced to death.

John Allen Muhammad masterminded and carried out a series of seemingly random shootings that terrorized the Washington area in 2002, and he and his teenage accomplice, Lee Boyd Malvo, were suspected in as many as two dozen killings in several states around the country.

Mr. Muhammad insisted on representing himself at his two trials, in Virginia and Maryland, maintaining that he had nothing to do with the attacks and that…