Highdown Hill was used by ancient immigrants as an Iron Age fort, Bronze Age warehouse, Roman bath house, and a Saxon cemetery. The arrival of the Normans meant that this area, now called the Garinges, was owned by the Earl of Arundel. Later the De Goring family took over this estate, giving the parish its name.  It then became the site of windmills. The tomb of mischievous 18^th century miller John Olliver is on one path to the 85-metre-high summit.

The Goring estate was bought by David Lyon Jr in 1835 using slave compensation funds for his 13 estates in Jamaica. He demolished the manor house at Northbrook to build Goring Hall. There is confusion about the origins of Highdown Tower. There are hints it might have been built as a Dower House from the 1840s. It was a strange location to build a house next to chalk quarry pits. William Lyons, the nephew of David, was the owner of the Arundel Estate when Frederick Stern arrived in 1909 to rent Highdown Tower.

The landowners of the main Highdown Hill and Ferring parish (next to Goring) were the Henty family. In the 1860s their workers accidentally discovered the first Anglo-Saxon graves, when planting trees, which started a series of archaeological survey digs. In 1930 the Henty family put Highdown Hill up for sale. By 1938, following a local campaign, the summit was rescued by the National Trust. During the 20^th century, many people have walked up this hill: jockeys, shepherds, archaeologists, Jewish cockneys, soldiers, and clubbers. Today, it is part of the South Downs National Park.

Sir Frederick Stern’s horticultural obsession was inspired by this porous landscape. Highdown is part of a chain of chalk hills formed by ancient sea algae called coccoliths. Over millions of years, they were squashed and squeezed into calcium carbonate, forming limestone chalk. Despite the thin soil on top of the chalk, there can be up to 40 different wild plant species here. Sainfoin and Pyramidal Orchids grow around the Hill. Sheep grazing is vital to make these plants thrive without being smothered by longer grasses. Many butterflies, moths, invertebrates, rabbits and skylarks are attracted to this rare chalk grassland.