Shortly after the Civil War, abolitionist and U.S. Army surgeon John Milton Hawks joined forces with other Union Army officers to form the Florida Land and Lumber Company, which was to be the economic basis for a settlement of five hundred freed slaves. Originally located on the Halifax River and named Orange Port, Hawks eventually moved the settlement to the town's current location and renamed it Port Orange.
By 1867, however, poor planning and unproductive harvests had plunged the community into an economic decline and, two years later, there were only nine families remaining. Today, the only remains of the freed-slave settlement, known locally as Freemanville and located at Orange Avenue and Charles Street, are its African American roots and the 1911 Mount Moriah Baptist Church. Besides the church, visitors can also view the state historic marker that was placed on the site in recognition of this post-Civil War community.