Wayne Bartram said the eastern end of the island was so well known as a meeting place for gay men that he would tell visitors looking to contact others to "go to the beach and turn right".
But offensive behaviour was not on and it was disgusting to think that men went there to have sex.
He had many gay friends who visited Rabbit Island to play cards, swim, read or talk, with or without clothes.
"We don't need animosity because we are not all dirty people, we are just gay," he said.
A nudist who agreed to be photographed but not named said he had been practising nudity there for about five years and was part of a group of "genuine folk who don't want to make a scene".
There was enough space at the beach to enjoy a nudist lifestyle, he said.
But there was a group of "hide and seek guys who disappear into the bushes" and he found their behaviour offensive.
They often hid behind bushes to stare at other beach users, he said.
Wellington labourer Josh Scott told The Nelson Mail that he wasn't too bothered when he glimpsed a nudist while walking back from a swim at the eastern end of the beach, but "wondered what was going on" when he walked past a group of five naked men.
A woman who frequently rides her horse on the beach said she often saw naked people and had been told by other beach users that "all sorts" went on under the pine trees.
Sergeant Craig Barker of Richmond police said there were one or two complaints about nudity at Rabbit Island each year and an occasional report of offensive behaviour.
A number of people had been issued trespass notices over the years, he said.
The Tasman District Council has not had any complaints about nudity this summer.
If it receives any, it is ready to put up signs reminding people that the beach is not clothes optional.
Mayor Richard Kempthorne said he was aware the eastern end of the beach was used as a gathering place by the gay community, but not of any complaints.
Members of the public should contact the police if they saw anything objectionable happening in the area, he said.