Benson Lossing describes an island near Castleton in 1866:
"Upon the island in the Hudson, at the mouth of this stream- a noted place of encampment and trade for the Iroquois- the Dutch built their first fort on the Hudson in 1614, and placed it in command of Captain Christians. The island was named Kasteel, or Castle, and from it the little village just mentioned received its name."
The Hudson By Daylight Map published in 1878 names the islands to the west of Castleton (from south to north) as Shad Island, Schermerhorn, Cow Island, Campbells island and Pixtaway Island.