Stephen Smith: Cape May’s Underground Railroad Leader

You’ve probably walked past the unimposing pale yellow house at 645 Lafayette Street many times, and perhaps you read the historical marker out front—the one saying that Stephen Smith was a businessman, philanthropist, abolitionist and founder of Cape May’s AME Church.

But what you didn’t know is that Smith, who spent many summers in Cape May, was one of the most important figures of the pre-Civil War Underground Railroad.

Long obscured by shrubs and trees, Smith’s second home has stood for nearly 170 years as a testament to a remarkable man who risked his fortune and his life to help hundreds of enslaved people reach freedom.

In the 1840s and 1850s, Smith, born a slave, was the richest black man in America, according to the newspaper of famed abolitionist Frederick Douglass. In addition to his efforts on the Underground Railroad, he collaborated with leading abolitionists, using his money and influence to create education programs, libraries, and schools for free blacks. He helped build churches and later contributed to facilities to aid poor and neglected children, and he founded the first home for aged African-Americans.

Yet today few people know Stephen Smith’s name. That may be about to change. The current owners of his summer home plan to restore it to the way it was when Smith built it in 1846, claw foot tub, kitchen and fireplace room and all. They want to retell his achievements through tours, presentations, and other events.

Four Hampton family siblings now own the house that their grandparents bought in 1932 from a close friend, Marie Vidal, Smith’s grandniece. Over the years, the siblings heard stories about Smith that were passed down from Vidal.

“I feel I am breathing new life into the philanthropy, the work Smith did against such great odds,” says Jo-Anne Hampton, one of the owners. “The community needs to know this was much more than just a house. I want to maintain the legacy of this house and man, not just for my family, but for the community.”

Hampton talks about her efforts while sitting in Smith’s parlor next to the original fireplace. She has flown in from California. Her brothers and sister are scattered over four states; only one lives full-time in Cape May.

But they are committed to the project. They have cut back the vegetation that was hiding the building and cleaned out much of the odd assortment of furniture the family collected over the past few decades.

It is a hot summer day and the eight-foot high French doors are shuttered. Still, the room is cool thanks to its thick plaster walls. Hampton says she is busy applying for grants and seeking contributions for the restoration. She has even made application to HGTV. She has set up a website and is working on other social media outlets.

If the family is successful in raising the money to restore the house, it will be the second time the Hamptons have secured the Stephen Smith home.

The siblings’ mother, Amelia Hampton, saved it from the wrecking ball in the 1960s when it was slated for demolition under the town’s controversial urban renewal plan.

When Amelia Hampton failed to persuade local officials to remove Smith’s house from a list of buildings to be knocked down, she sent an emergency telegram directly to then-President Lyndon Johnson. She pleaded with him, “Please save the Stephen Smith House.”

Acknowledging her concerns, an assistant to the President spoke to officials at the U.S. Department of Housing and Urban Development. Although a large swath of black-owned businesses and homes in Cape May were destroyed as the redevelopment plan was put into effect, Stephen Smith’s house was not touched.

“My mother made it clear she wanted to maintain the house for its historical significance,” says her son, Trevor Hampton. And the Hamptons are determined to keep it that way. It’s not just the history of one man, he says, but of a movement to free and educate and empower blacks. “He had a lot to do with making things happen in the country; it’s an emotional issue to us as a race.”

So who exactly was Stephen Smith? And how did he accomplish so much at a time when the odds against him were staggering?

“In Philadelphia, Mr. Stephen Smith was one of our main men,” Reverend Clement Oliver, a key coordinator of the Underground Railroad throughout south Jersey, revealed to an historian in a long-buried 1895 interview. “He did more than any other colored man I knew of.”

When Smith built his summer retreat in Cape May, he was already a rich man, running a very profitable lumber and coal concern in Columbia, Pennsylvania, with his partner and close friend, William Whipper. Smith and Whipper eventually came to own 22 railroad cars and several canal barges, which they used to transport their products east to Philadelphia and west to Pittsburgh.

The two men could have been content to focus solely on their company. In an era when it was hard enough for a black man to get a job, let alone own a successful business, that would have would have been remarkable enough.

But with a flood of former slaves braving enormous odds to make their way to the free state of Pennsylvania, Smith and Whipper saw it as a sacred duty to help. Because of their proximity to the slave state of Maryland, and to the foot of a major bridge across the Susquehanna River, many freedom-seekers came their way. Word was out that they could find help in Columbia from the town’s many Quakers and its large free black population.

Smith and Whipper personally sent many escapees by wagon or foot on to other stops of the Underground Railroad. But as large groups of slaves started to arrive, it was too dangerous to send them that way.

So the two men built a false end to their rail cars and hid runaway slaves in a secret compartment. These Underground Railroad conductors used their flesh and blood, as well as their steel and lumber railroad to convey people north quickly and safely.

Smith never faced reprisal for his work helping runaway slaves, but he did for his business success. He was a target of white mobs that resented his wealth.

That began in the 1830s, when violence erupted in many northern cities, including Columbia, Pennsylvania. It was a time of intense competition among free blacks, recent immigrants, and struggling white workers for jobs. White mobs attacked black homes, churches and businesses in Columbia.

Smith was singled out and his office ransacked. Tensions were so high that many black families fled to nearby hills for safety. Smith continued to receive threats even after the violence subsided. The pressure was so great that at one point he put his business up for sale. In the end he kept it, but around 1842 he put Whipper in charge of the firm and moved to Philadelphia.

But racial tensions were no less severe there. Philadelphia experienced its worst racial attacks ever around that time. During the most violent days of the conflict, gunshots rang out, blacks were assaulted, and their homes, churches and meeting places were destroyed. A large building built by Smith to serve as a school, lecture hall, and a place for religious services was burned to the ground.

It was after this that Smith built his retreat in Cape May, with lumber from his own firm, and where the ocean breezes and sunshine provided a respite from the conflicts and tensions in the city. That’s not to say Cape May was immune from the great debate over slavery. The resort catered to both Philadelphia abolitionists and southern planters, and at times open clashes broke out.

Throughout the 1850s to well after the Civil War, Smith continued to expand his business activities, particularly his real estate ventures. He bought and sold many properties, owning as many as fifty houses in Philadelphia. In Cape May he was well known at real estate auctions, buying properties on Decatur, Mansion and Jackson Streets, as well as others near Congress Hall.

During the Civil War, Smith and three other members of Philadelphia’s prestigious Banneker Institute, which sponsored lectures, debates and other events to enrich and educate African-Americans, bought the property next to his Lafayette Street house. They opened a hotel where, for $8 a week, African-Americans from Philadelphia could enjoy a beach vacation.

Smith’s achievements are even more remarkable given his beginnings. His mother was a slave when he was born in 1795, which meant he was, too. His father was a white man, but his name is not known.

When Smith was about nine years old, a Revolutionary War officer who owned a lumber business in Columbia, Pennsylvania bought him. Smith worked hard and was able to buy his freedom six years before he would have been freed under Pennsylvania’s gradual abolition law. At the same time he married a free mulatto woman, Harriet Lee, and opened his own lumber and coal business, while his wife ran an oyster and refreshment stand.

Smith was a deeply religious man, and a national leader in the African Methodist Episcopal Church. He founded Cape May’s AME Church, now located on Franklin Street, around the corner and a few doors down from his house.

The AME Church, founded in Philadelphia in 1794 by Richard Allen, provided social services and educational programs, along with religious guidance to free blacks. And it was known as a haven for fleeing slaves, providing them with food, shelter, and other help.

A vocal abolitionist, Smith was a charter member of the Pennsylvania Anti-Slavery Society, a follower of William Lloyd Garrison, and a close associate of abolitionist leaders Lucretia Mott, William Still, and Robert Purvis.

After the Civil War, William Whipper summed up what it meant to him and Smith to have helped liberate people. “I feel thankful for having had the glorious privilege of laboring with others for the redemption of my race from oppression and thralldom; and I would prefer today to be penniless in the streets, rather than to have withheld a single hour’s labor or a dollar from the sacred cause of liberty, justice and humanity.” n