Swanendael was bought from the native Americans by a certain Gillis Hossett, working for Samuel Godijn. The latter was a director of the Noordse Compagnie, a whaling company operating mostly in the Arctic region around Spitsbergen. Swanendael was one of the first patroonships in New Netherland. The first colonists arrived here in 1631, but the village was destroyed by the indigenous population the next year.
In roughly the same location, a fort was established in 1659 that is generally known as the 'Compagniesfort' The location of the fort was called Whorekill. The surrounding area was bought by the Dutch in 1663, but conquered by the English the next year.