A Cape May Connection

Black Hotelier Had a Pivotal Role in the Fight Against Slavery

Thomas Dorsey, one of the wealthiest Black businessmen in Philadelphia around the time of the Civil War, was a co-founder of the Pennsylvania Anti-Slavery Society. He was a close friend of abolitionist Frederick Douglass. And he provided counsel to John Brown, whose raid on the Harpers Ferry arsenal helped spark the Civil War.

He also was co-owner of one of Cape May’s ground-breaking hotels. Along with his partner, Stephen Smith—an African American coal and lumber magnate and leading abolitionist whose summer house on Lafayette Street still stands—the hotel they owned for a number of years, the Banneker House, hosted a wide range of Black leaders who changed the course of American history.

Smith, who played a crucial role in the pre-Civil War Underground Railroad, helped hundreds flee slavery in the secret compartments he and his business partner built in the cars that transported his goods. (See “Stephen Smith, Cape May’s Underground Railroad Leader,” by Barbara Dreyfuss, Cape May Magazine, Fall 2015).

After the Civil War, in 1867, Smith and Dorsey joined with two others to buy the renowned Banneker House, the property on Lafayette Street adjoining Smith’s. Since the 1840s it had been a resort where leading African Americanabolitionists and Underground Railroad leaders often stayed. From its steps, they issued proclamations against slavery, circulated nationally in Frederick Douglass’ Paper. The PhiladelphiaEvening Telegraph declared, “With Dorsey at the head, it cannot fail to become a popular resort.”

At Dorsey’s Philadelphia home, he was often joined around his table by such men as the legendary orator and newspaperman Frederick Douglass; William Lloyd Garrison, the founder of the American Anti-Slavery Society; and U.S. Senator Charles Sumner of Massachusetts, a fiery foe of slavery. And what a wonderful table it must have been. Dorsey was a renowned caterer, known for serving the haute cuisine of the time such as lobster salad, terrapin, deviled crabs, and chicken croquettes at the city’s elite parties and events. W.E.B. DuBois, a founder of the NAACP, described him as someone who had “little education but great refinement of manner.”

While Dorsey could have focused on living a very comfortable life, he was determined to do what he could to end slavery. Dorsey himself knew its horrors. He and his three brothers had freed themselves, and it was their story of evading slave catchers that was a critical impetus for the creation of the organized Underground Railroad throughout Pennsylvania, New Jersey, New York, and Delaware.

Dorsey was 24 when he and his three brothers fled from a Maryland planter in 1836. They had been promised their freedom upon the death of their enslaver, reportedly their own father, but when he died, their white half-brother attempted to sell them.

The Dorsey brothers made their way to Philadelphia and to the home of Robert Purvis, son of a white British businessman and a free woman of color. Purvis was a prominent anti-slavery activist. Three years earlier, he had joined Garrison in establishing the American Anti-Slavery Society, which demanded the immediate end to slavery. After contacting Purvis, Thomas Dorsey decided to stay in Philadelphia, while his brother Basil Dorsey went to work on Purvis’ farm nearby. Their two other brothers located employment on neighboring farms.

But trouble soon found all of the brothers when the man who laid claim to them tracked them down. Thomas Dorsey was kidnapped and taken to Baltimore, to be shipped to a slave market in New Orleans. Purvis soon learned that the three other men were also being hunted. As Purvis later recalled, “I had just returned from the city and was in the act of eating my supper when a neighbor’s son came in great excitement to tell me that Basil had been carried off. I sprang from the table.”

Thus began a complicated and dangerous series of maneuvers by Purvis and his friends to guarantee the freedom of all four Dorsey brothers.

A group of the kidnappers surrounded the Purvis house. Purvis said later, “By my advice, Charles [Dorsey], in whose hands I placed a double-barreled gun heavily charged, walked out in front of the house and defied them. The slave catchers, thinking doubtless ‘discretion the better part of valor’ instantly departed.” Purvis had his brother secretly convey Charles and William Dorsey to a friend’s home in New Jersey, some 40 miles away.

Basil Dorsey was apprehended by slave catchers outside Philadelphia, but they were pressed to go before a judge to prove under the law their right to seize Basil. Rather than go back to slavery, Basil threatened to cut his throat if the decision of the judge went against him. Some clever legal maneuvers by Purvis’ friends secured his release. As for Thomas Dorsey, awaiting a boat to slavery in the South, Purvis was able to quickly raise $1,000 (the equivalent of $28,000 today) to purchase his freedom. 

It was a complicated and difficult multi-pronged effort to ensure that the four men maintained their freedom. Purvis would later write that “among the hundreds of cases which came under my notice, none excited my interest more deeply” than that of the four Dorsey brothers. All of this made clear to Robert Purvis that helping freedom seekers could not be haphazard. There needed to be an organized network of people who could hide, transport, feed, and clothe those who had fled slavery.

Therefore in 1837 Purvis organized and became president of the Vigilant Association of Philadelphia, a group dedicated to helping freedom seekers. The Vigilant Association coordinated the Underground Railroad. Soon after, Robert Purvis, Stephen Smith, and Thomas Dorsey helped establish the Pennsylvania Anti-Slavery Society, an arm of the American Anti-Slavery Society. Purvis was its president or vice president for many years.

It was the same time that the white activist John Brown, enraged by the murder of an abolitionist by a pro-slavery mob, consecrated his life to the destruction of slavery. He viewed slavery as a “barbarous, unprovoked, and unjustifiable war” against Black Americans, and came to believe the only way to end it was by the use of force. Over the years his campaign in Kansas attacking pro-slavery settlers headlined newspapers across the country.

By the late 1850s Brown planned to establish a refuge for people fleeing slavery in the mountains of Virginia, and to use it as a base for raids to liberate enslaved people elsewhere. He believed these raids would destabilize the slave system and ultimately bring about its end.

John Brown toured the North seeking men and money to aid his plan. He was particularly active in Philadelphia, working with the wealthy African American community there. Thomas Dorsey was one of the men who led the effort to back Brown, setting up meetings for him with those who might provide financing and support.

In August 1859 a newly formed Black militia company, organized by several men connected with the Underground Railroad, was scheduled to take part in a Philadelphia street parade. Brown feared it would draw attention to his plans, so went to the city the day before in hopes of preventing their public display.

That evening, Brown attended a public meeting at a local Black church, where it was announced that “there was a grand project afoot to invade the South with an army of armed northern Negroes and free the slaves.” A call was made for volunteers. Brown was aghast at the revelation and believed it had ruined all his plans. 

Thomas Dorsey called an emergency meeting at his home late that night. Among those in attendance were John Brown, Frederick Douglass, and a dozen other Black leaders. They tried to stem the damage but it was too late. The authorities began to watch the militia closely, thwarting efforts to recruit volunteers for Brown.

A few days later, Douglass secretly met with Brown at an old stone quarry in Chambersburg, Pennsylvania, and learned of a new plan. Brown wanted to begin his operation by seizing the federal arsenal at Harpers Ferry, Virginia, and use the weapons stored there to arm locally enslaved people for self-defense. Brown pleaded with Douglass to go to Harpers Ferry with him. Douglass was sympathetic to his friend, but refused to take part in the raid, warning Brown that he was going into a trap.

When Douglass’ friends in Philadelphia learned that he had refused to join in the raid, they implored him to go to Harpers Ferry with Brown, and even offered to provide well for his family until he safely returned to them. They believed that Douglass’ participation in the raid would inspire others to join and increase its chance of success. 

William H. Johnson, an African-American barber and activist, later remembered meeting Brown and Douglass at Thomas Dorsey’s house on October 13, 1859, “where a meeting of the leaders was held again. Captain Stevens, who was hung with him [Brown] at Charleston, in the following month, was with him that night. He told us that he was about to make a raid into Virginia.  ‘But,’ said he, ‘it will be a failure. Our little band will be but the forlorn hope of what might have been a grand expedition, but blood must be shed or the bonds of the colored men will never be broken.’”

Johnson offered to fight with Brown. But Brown forbade him to join in. “’Are you married?’ he asked, turning to me. I told him I had been married for seven years. ‘Have you children?’ he then asked. I replied that I had not, but expected to be a father in a few weeks.  ‘Then you can’t go,’ said he with a sad smile. I did not go, but would have done so if I could.”

Three nights later, Brown led 18 men—white and Black—into Harpers Ferry and took control of the armory with little resistance. Citizens and local militias soon appeared and prevented any escape until federal troops arrived under the command of Robert E. Lee. By the time it was over, a badly wounded Brown was taken into custody and 10 of his men, including two of his sons, were dead.

When news of the raid broke, it shocked the nation. Douglass was still in Philadelphia for a speaking engagement. He was staying at Thomas Dorsey’s house. He soon learned that a letter found at Brown’s headquarters had implicated him, and that a warrant had been issued for his arrest. Douglass quickly departed for the safety of Canada. Others who feared extradition to Virginia soon followed, aided by the Underground Railroad.

Brown was quickly put on trial and found guilty of all charges. On December 2, 1859, after shaking hands with his captors and thanking them for their kindness, an unflinching and unapologetic John Brown was hanged. It was his final act of defiance.

Historians debate whether John Brown expected his plan to actually succeed or whether he saw himself as a martyr whose death would increase anti-slavery activism. But what is clear is that the raid on Harpers Ferry hastened the Civil War. Fearing more such attacks, many southerners who had resisted calls for secession were now more open to it. Many northerners were animated by Brown. John Brown was a hero to most African Americans, those enslaved and those nominally free. When war finally erupted, Union soldiers marched off to battle, honoring his memory singing “John Brown’s Body.”

During the Civil War, Thomas Dorsey and other prominent Philadelphia Black abolitionists led the recruitment of Black troops. Dorsey’s wife, Louise, helped raise funds to care for sick and wounded soldiers. The eldest daughter of Frederick Douglass, Rosetta, boarded with the Dorseys during the war before accepting a teaching position in Salem, New Jersey. In 1865, Frederick Douglass asked the Dorseys to accompany him to Lincoln’s second inaugural. That evening, Douglass was invited to the inaugural ball and Mrs. Dorsey agreed to accompany him. Upon arrival at the White House they were prevented from entering because of their color. When Lincoln heard that Douglass and Mrs. Dorsey were outside, he had them ushered into the East Room and warmly greeted them. Lincoln was eager to know what Douglass thought of his inaugural address. Douglass replied, “Mr. Lincoln, that was a sacred effort.” It was the first time African Americans who were not servants were at an inaugural ball.

In 1865, Frederick Douglass asked the Dorseys to accompany him to Lincoln’s second inaugural. That evening, Douglass was invited to the inaugural ball and Mrs. Dorsey agreed to accompany him. Upon arrival at the White House they were prevented from entering because of their color. When Lincoln heard that Douglass and Mrs. Dorsey were outside, he had them ushered into the East Room and warmly greeted them. Lincoln was eager to know what Douglass thought of his inaugural address. Douglass replied, “Mr. Lincoln, that was a sacred effort.” It was the first time African Americans who were not servants were at an inaugural ball.

In the years when Smith and Dorsey and their associates owned the Banneker House, it continued to be frequented by members of Philadelphia’s Banneker Institute, an African American literary and educational organization, where literature, cultural and social issues—especially regarding equal rights—were discussed. They talked baseball, too.

Members of the Banneker Institute formed the Pythian Base Ball Club, which became one of the best Black baseball teams in Philadelphia. White and Black teams did not play each other, not until the Pythians in September 1869 challenged the Philadelphia Olympics. In August the captain of the Pythians and its star player, Octavius Catto, was planning tactics for the game. But he was doing so from the beach in Cape May, staying at Dorsey’s hotel. He wrote a fellow teammate, “My boy you made a great mistake in not coming down—we are having a high season and blankets are plenty in Room No. 8 – Banneker House, Cape May. We have many things to say but…Let the musicians discourse!”

In 1871 civil rights activists in Philadelphia suffered a severe blow when Octavius Catto was assassinated on Election Day. Catto had led the fight to desegregate the city’s streetcars and for the right of Black Americans to vote. The baseball team never played again.

The following summer, a correspondent for the New National Era, a newspaper edited by Frederick Douglass, stayed at the Banneker House and described Thomas Dorsey as “irrepressible.” He wrote that Dorsey kept the company alive with his odd sayings, and by “his frequent and emphatic declaration that he was a Charles Sumner man.”

Sumner had been an ardent opponent of slavery in Congress and a champion for equal rights. In 1856, just days after delivering an impassioned speech against slavery, he was nearly beaten to death with a cane on the Senate floor by a pro-slavery congressman. Sumner came to Cape May to recuperate from his injuries.

Today all that remains of the Banneker House is an open grassy space on Lafayette and Franklin Streets, next to the Stephen Smith house. But the area around it, now known as Freedom’s Corner, which includes the Smith House, the new Harriet Tubman Museum, the Franklin Street School—once a segregated building for African Americans—and the historic Allen AME Church are all in the process of being restored. They will tell the stories of Smith, Dorsey and other African American leaders in the struggle for freedom and equality.