The youngster was in the play area near the boating lake last Thursday at 2.45pm when a man approached and grabbed his wrist.
The family's shouting persuaded the man to release his hold and walk away leaving the boy unhurt, but his shaken family are now asking Hastings council to make the area safer.
The boy's grandmother, who wishes to remain anonymous, said: "I want to warn other families and to highlight to the council the bad planning in the play area.
"The man was able to get right up to my grandson without anyone seeing him and grab his arm. He appeared to come from nowhere.
"The fencing makes it impossible to see children in most parts of the play area so anything could happen. It was so frightening."
The grandmother, her sister and the little boy were walking through the park when the attempt happened last Thursday afternoon.
The youngster had run a few yards ahead into the playground when the man appeared from behind the fencing and grabbed his arm.
The grandmother said: "I ran at him shouting to let go and my sister ran at him with her dogs.
"But he calmly looked back at us, let go and walked off without saying a word. He was so cool and callous.
"My grandson thought he had done something wrong and kept saying 'sorry nanny'. My daughter-in-law is distraught and even asked if she had made her son too friendly."
The man is described as white, late 50s, of large build, with greased back receding black hair.
He was wearing grey trousers, a red polo shirt and a grey v neck jumper.
The play area by the boating lake is enclosed with small wooden fencing, but inside is a different matter with wooden, maze-like fencing standing at more than five and a half feet in some places, obscuring most adults and children from sight.
The enclosed and secluded area is also a popular spot for drug users and since Operation Vertigo, the police operation to crack down on drug abuse in the town, the area has been marked for increased visibility presence with Police Community Support Officers regularly patrolling.
Inspector Mark Bright said: "We don't have the volume of incidents within the play area to say to the borough council you should relook at your design, but we are investigating last Thursday's incident and extra patrols were in the area immediately afterwards."
A spokesman for Hastings council said: "Millions of pounds have been spent on restoring the park and we have worked hard to make it an open, safe place to play and walk.
"Some of the wooden fencing in the playground acts as a retaining wall for changes in levels on the play equipment.
"However, some of it is there for safety reasons and it is these sections we will be happy to look at again.
"Our officers will be investigating this and we are keen to talk again to the police crime prevention officer about the issue.
"We want this park to be a real community asset and are open to receive comments from community groups and park users."
Police are appealing for anyone with any information relevant to the incident or the identity of the man to contact them on 0845 60 70 999, quoting serial 823 of 7/10/04.