Amid Lib Dem fury over the home secretary Theresa May's handling of the controversy, Clegg announced that open parliamentary hearings would be held to examine draft clauses of a new bill.
The deputy prime minister, who said he has had to act as a restraining influence on the security services, made the announcement as he pledged to ensure the coalition did not repeat the mistakes of the last government. "I saw the appalling populist excesses of authoritarian home secretaries, like John Reid, under Labour. This total casual disregard for people who care about privacy and civil liberties – I am not going to allow this government to make the same mistake," Clegg said in a Guardian interview.
Clegg spoke out as the government sought to lower the temperature after a furious backlash against plans to give the Government Communications Headquarters (GCHQ) real-time access to Skype, instant message and email data. Sixteen Lib Dem MPs, more than a quarter of his parliamentary party, signed a letter warning of "worrying reports" about the proposals.
Downing Street and Clegg announced that draft clauses in a new parliamentary bill would be published around the time of the Queen's speech on 9 May. These will be scrutinised by a parliamentary committee, probably the House of Commons home affairs select committee, which will hold public hearings.
Clegg said: "We are not simply going to bounce new legislation through parliament. We will publish draft clauses and subject them to proper pre-legislative scrutiny. The route to legislation will be a deliberative, open one which can be subject to real stress testing and scrutiny. That is what proper pre-legislative scrutiny is all about."
He said it was important to act because he had been advised by the security services that criminals and terrorists were circumventing the rules and the Regulation of Investigatory Powers Act barred them from monitoring the date and time of suspect instant messages and Skype calls.
Clegg said: "There is a gap opening up in the application of existing statutory powers for the police because of the increasing volume of email and telephone traffic that is now directed via voice over internet protocol means … I am keen to lower the temperature by reassuring people that we are not doing what we are accused of wanting to do, which is to create new databases and create new powers of surveillance over the contents of people's emails."
The authorities would still need a warrant to access the contents of an email or a call, he said. The changes would simply allow the authorities to access information they were entitled to access on more traditional forms of communication – where a call is made and when and by whom.
The Lib Dems are frustrated with May because they believe she allowed the row to be stoked after failing to give a proper response to a story in the Sunday Times which Clegg described as "wildly hyperbolic". The Lib Dems are annoyed that the Home Office issued a standard law and order response and failed to explain the intricacies of the policy.
In a rare insight into the tensions between ministers and the security services, Clegg said: "The security establishment will always say they need new powers tomorrow. It is the role of the politicians and parliament to make sure that requests for new powers, updated powers, made by the security services, are properly scrutinised and checked. That is the checks and balances in a democracy.
"In a rough and ready way, they are being played out semi-publicly now. You always have this push and pull. Under the last government, not only did they give everything the police and security services ever asked for, they gave them more than they needed.
"We in this government are setting the bar much higher. We are saying we will only think of legislating if you can prove to us that it really is necessary. And I am persuaded there is a dilemma. There just is an issue."
Clegg was shown the strength of anger within his party when 16 Lib Dem MPs, led by Julian Huppert, a member of the home affairs select committee, signed a letter to the Guardian, that said: "Following worrying reports of possible government proposals to collect real-time information on people's activity online, including from social media sites, we are pleased to hear yesterday the deputy prime minister making clear his commitment to civil liberties and protecting privacy, and confirming that the government will publish draft legislation with sufficient time for consultation and pre-legislative scrutiny.
"It is absolutely vital that the public get a chance to see and debate the details of any proposals to extend state surveillance, not just being presented with a Home Office fait accompli. It is also essential that the initial plans include adequate safeguards, which should be stronger than the current weak controls.
"Liberal Democrats in government will not follow the last Labour government by sounding the retreat on the protection of civil liberties in the United Kingdom. It continues to be essential that our civil liberties are safeguarded, and that the state is not given the powers to snoop on its citizens at will."
One MP said the letter was issued following discussions between the MPs and senior party figures: "We put pen to paper because we wanted to put in writing the guarantees that we have been given in private. This is a marker."
The letter in Wednesday's Guardian is signed by Huppert, Annette Brooke, Malcolm Bruce, Mike Crockart, Andrew George, Mike Hancock, John Leech, Greg Mulholland, John Pugh, Alan Reid, Adrian Sanders, Ian Swales, David Ward, Mark Williams, Stephen Gilbert and Roger Williams.
In his first comments on the proposals, David Cameron denied that the government would be able to monitor calls, emails, texts and website visits at will. He said the moves were needed to keep up with changes in technology and were vital in the effort to tackle serious crime and terrorism.
Asked why the Tories appeared to have changed their position on GCHQ accessing information, the prime minister said: "Let's be absolutely clear, this is not what the last government proposed and we opposed. And let's be clear, this is not about extending the reach of the state into people's data; it's about trying to keep up with modern technology.
"But we should remember that this sort of data, used at the moment, through the proper processes, is absolutely vital in stopping serious crime and some of the most serious terrorist incidents that could kill people in our country, so it's essential we get this right.
On a local election campaign stop in Leeds, the Labour leader Ed Miliband said: "Once again we see a very sensitive issue being spectacularly mishandled by this government. If we are talking about stopping people committing murders and terrorist acts, there have always been powers in place and we will look at any proposal the government has.
"When you are dealing with sensitive privacy issues, with people's fears about what a government database might mean or what the government will be looking at, you must be incredibly sensitive about the way you handle the proposals, the way you set them out and clarity about what they mean. The government has failed that test."
