The Lewes Tourist Information Office is understood to be the latest building Lewes District Council wants to make available for housing as part of the controversial 49 sites scheme.
Sustainable travel expert Chris Smith, from Lewes Travel Log, said alternative sites for the Tourist Information Office had been sought but none was on the ground floor in a prominent position.
But the district council said no decision had been taken on the future of individual Tourist Information Centres.
The decision to move the office was criticised by Mr Smith who said it threatened the growth of tourism in the town.
Mr Smith said: “Promoting tourism is vital to the economy of Lewes. We must have a prominently situated tourist office to do this.
“There are no other sites available so far as I am aware which would enable the tourist office to do its job properly. This is a financially disastrous proposal.”
But a spokesperson for the council said: “Lewes District Council is committed to making sure we remain a vibrant and attractive destination for visitors.
“We are looking to improve and modernise the services on offer to further enhance the strong visitor economy that results in spending of in excess of £50 million a year.
“We have a four year Tourism Strategy that seeks to plan the future of visitor services provision across the district.
“No decision has been taken on the future of individual Tourist Information Centres.”
The district council is making available 49 sites across Lewes District for housing, with the ambition of building around 450 homes, 40 per cent of which would be affordable housing.
It has gone into partnership with Karis Developments to build these homes but has so far refused to release the list of sites to the public due to legal and contractual matters which needed to be concluded. But it said it would announce the sites as soon as possible.
Mr Smith said he had seen a copy of the secret list, which revealed that the building, on the corner of Fisher Street and the High Street was to be sold.
It is expected that its new use would be as a shop with flats above, he said.
The Western Road toilets is on the list too, despite a petition with more than 1,500 signatures. It has been earmarked for housing.
A dark cloud also hangs over St Mary’s Social Centre in Lewes where volunteers said it was on the 49 sites list and that a new social centre would be built on the site, together with some homes.
Around 450 houses could be built on land across Lewes District in a partnership between Lewes District Council and a consortium.
Southern Housing Group, Conran and Partners Architects and Karis Developments which make up the consortium, are the preferred bidders to build on the 49 sites across Seaford, Newhaven, Peacehaven and Lewes.
Planning applications could begin by the end of this year and there will also be public consultation.
Building on some of the schemes in Lewes district could start within two years.