The criticism is contained in a House of Commons Public Accounts Committee report, Implementing the Transparency Agenda, published on Wednesday 1 August.
Committee chairman Margaret Hodge said: "This committee fully supports the principle of greater openness and its potential to strengthen accountability and drive improvements in public services.
"But the government has a lot more work to do before that potential is realised."
Ms Hodge added: "It is simply not good enough to dump large quantities of raw data into the public domain. It must be accessible, relevant and easy for us all to understand.
"Otherwise the public cannot use it to make comparisons and exercise choice, which is the key objective of the transparency agenda."
Too much data was poorly presented and difficult to interpret, said Ms Hodge.
In some sectors, such as adult social care, there were big gaps in the information provided so users could not use it to make informed choices.
"As more and more different providers are involved in providing our public services, there must be a level playing field in terms of transparency," said Ms Hodge.
"At the moment individual academies do not make available information on spending per pupil that allows value for money to be compared fully between different types of school."
The report voices particular concern that private providers are able to hide behind 'commercial confidentiality' to block the disclosure of relevant information.
"We must be able to follow the taxpayers' pound wherever it is spent," it says.
Data is also being issued by government and other public bodies without any clear idea of the costs, benefits and risks of doing so, the report says.
"The government should develop a comprehensive analysis of what it actually costs to release data, and of the real benefits and risks."
Those without access to the internet must not be forgotten, said Ms Hodge.
"They are often the very people who rely most on public services and could benefit most from access to better information.
"Further steps to ensure universal access to public data should be developed and set out."
The report recognises that public bodies in both local and central government have significantly increased the volume and range of information released
It also recognises that the case for transparency is inherently strong – although it warns that further work needs to be done to realise the full benefits of transparency.
It does not help government to meet the objectives of the transparency agenda when large quantities of raw data are released without ensuring that the data are fit for purpose, says the report.
Some data is hard to interpret, such as on local government spending, and there are important gaps in information, such as incomplete price and performance information on adult social care.
The committee also voices concern about some information not being presented on a consistent basis, again for example in local government.
Poor or incomplete data hinders the ability of users to exercise effective choice. It also undermines the ability of service deliverers and policy makers to focus on improving quality.
The full document can be downloaded here.