Chapter 24. Information and data management

Portfolios, programmes and projects generate a large volume of information and data, held in digital or physical form as documents or as collections of data. These need to be organised to create an information repository for the work, establishing and maintaining a clear picture of:

• what information and data are held in relation to the work
• who is the information owner for each item
• how and where they are held
• who has access to them, and under what conditions
• the handling arrangements for specific pieces of information or data

Most information and data in project delivery is held as documents, either created or received by the team. Establishing an information repository where documents and other materials can be stored and managed is an important early task. For smaller or simpler work, this can be done by creating a dedicated area on the organisation’s electronic file system. For larger or more complex work, specialist document or information management software may be needed. 

The repository and the material it contains should be established, maintained and managed in line with government data standards (see 24.6.1.4 on meeting data management standards) and cyber security requirements (see 24.6.1.5 on protecting information and data security). This includes ensuring that access to material in the repository is controlled appropriately through the life cycle, to ensure that: 

• people k can access the information and data they need

• sensitive and personal data is protected in line with legal and government requirements, with access limited to those with a legitimate business need and withdrawn when no longer needed 

• document versions are controlled, so that they cannot be changed once approved or baselined in that version 

A simple repository is typically structured according to the work breakdown structure. For larger portfolios, programmes and projects, repositories where content can be manipulated using meta data are more efficient and less prone to errors. Specialist systems can control versions and access, record ownership and log changes, including who makes changes and who accesses the information. Physical storage should also be considered, including ensuring continuing access to essential material if digital systems are unavailable. Where the work forms part of a wider portfolio or programme, a consistent approach should be used and, where appropriate, a common repository. 

A disciplined approach  to the creation or collection, management and retention or disposal of each information and data asset held is important. This avoids confusion over different versions of a document and ensures effective change control and traceability (see Chapter 22: Change control and Chapter 23: Traceability management). Information can either be created within the team as part of the work or received from outside the team. A clear and simple procedure should be established covering both, to ensure correct storage and handling.

Minutes should be recorded for all formal meetings relating to the governance and management of the work and held alongside the documents and data submitted to and considered at the meeting. The minutes should not repeat information in the supporting papers but should include:

• the date and time of the meeting and who attended, both name and role
• the agenda, matters considered and the key points made in discussion
• the chair’s summing up on each matter considered
• the decisions taken
• the actions to be taken, by when, by whom and who needs to be informed when the action is complete

Minutes should be written up promptly by the meeting secretary and circulated to those present for comment before approval by the chair and then retained as the approved record of the meeting. Decisions and actions should be logged separately and cross-referenced, for example in a decision or action register, and monitored through to completion.

Many types of information is added to, used and closed as part of controlling the work (see 24.4 on what is information and data management). These include:

• risks
• issues
• change requests
• lessons learned
• actions
• decisions.

Such information can be held in proprietary or bespoke project delivery management systems or, for smaller work, in spreadsheets. Spreadsheets can work well, but have a risk. Unless they are stored in a formal document management system, there maybe record of who changed what and when, so measures need to be taken to manage updates through version control.

Decisions and actions support the management of work and provide traceability. They can also serve as evidence in the event of a commercial, legal or other dispute.

Decisions and actions can sometimes be agreed outside formal meetings, for example where urgent action is needed. These should be recorded in the decision or action register with the reasons for taking them and discussed at the next formal meeting, being recorded in the minutes.

The decision register should include:

• a unique reference,
• a description of the decision,
• who made the decision,
• date the decision is required by or was made.
• status, such as pending, made, deferred or cancelled, which can be used to filter the register.

An action register should include:

• a unique reference
• a description of the action
• who agreed to take the action
• date the action is to be completed by
• who is the recipient of the action
• a cross-reference to the source of the action (for example to a decision, risk, issue or meeting)
• status, such as not started, in progress, completed, cancelled, which can be used to filter the register

In government, information and data management must be carried out in line with the government’s legal obligations and other requirements, for example to protect national security (see Chapter 7: Health safety and security).

Specific legal requirements apply in respect of government transparency and the protection of, and access to, personal data held by public authorities.

The Public Records Act 1958 (as amended by the Public Records Act 1967) provides for the selection, retention and transfer to the National Archives of information and data constituting public records

The Freedom of Information Act 2000 provides for the release of recorded information (including drafts, emails and recordings) held by public authorities on written request, subject to specific exemptions and provisions on cost

The Data Protection Act 2018, which implements the General Data Protection Regulation (GDPR) provide specific data protection principles governing the way personal data is used, maintained and protected

Personal data is a legally defined category of data which requires particular care in handling, in line with the Data Protection Act 2018. Where personal data is to be shared, this needs to be done in a fair and transparent way, ensuring that there is a lawful basis for sharing the data and that it is processed securely. A data protection impact assessment should be conducted, and a written data sharing agreement put in place. Further guidance is provided in Data sharing: a code of practice.

Information should be published where appropriate, in line with the government’s transparency commitments. The Freedom of Information Act 2000 provides that information should be shared on request, subject to specified exemptions (for example, intended future publication, national security, formulation of government policy and effective conduct of public affairs, personal information, professional legal privilege, or prejudice to commercial interests), and the cost threshold established under the Act. Where it is unclear whether data should be released, the organisation’s chief data officer should be consulted. More information is provided in the Guide to managing an FOI request and other guidance published by the Information Commissioner’s Office.

Where information or data is considered suitable for publication, it should be quality assured, anonymised and made available with appropriate documentation including details on its quality. Open data published by public authorities should be released in consistent and accessible formats, to improve its utility, as set out in the Service standard.

Where specified, publication of data and analysis should be done through approved routes. For example, publication of official statistics should follow the protocols set out in the Code of practice for statistics, which provides the framework to ensure that statistics are trustworthy, good quality and valuable.

The Data ethics framework sets out expectations for responsible data use across the public sector. It is based on the principles of transparency, accountability and fairness.

Portfolio, programme and project teams often generate or collect large volumes of information and data. How information and data is collected, stored and managed affects both how well it can be used and shared and how well it is protected.

The Programme and project data standard sets the expectation for the creation of programme and project data across government. This helps organisations create consistent and high-quality data.

Where new data is to be collected or generated, planning ahead ensures that its collection is designed appropriately and meets the relevant standards.

As a minimum, the information manager should:

• start from understanding why the team needs specific information and data
• keep an original copy of data as it is received
• ensure compliance with the Programme and project data standard
• ensure the data is backed up
• ensure the data includes appropriate metadata, for example information about the characteristics of the data and how it should be handled
• keep an audit trail of changes throughout the life cycle of data
• ensure data can be rolled back to an ‘as-received’ state if necessary
• actively manage, review and improve data quality

The Government Functional Standard for Digital and the Service standard set out the core principles and standards for data management, and the Government Functional Standard for Analysis and the Aqua book (requires sign in) set out standards for data use in analysis, and these should be followed where such activities are part of the scope of the work. Detailed guidance on data quality is provided in the Government data quality framework and in the Data ethics framework and other guidance published by Government Digital Service.

Data standards for the collection and retention of major project data are set by the National Infrastructure and Service Transformation Authority and should be observed.

Specific data standards also apply in certain fields.

Infrastructure and construction work should use the Information management initiative framework, formerly the UK Building Information Management (UK BIM) Framework, which standardises the approach to generating and classifying data, data security and data exchange across construction, supporting adoption of the ‘digital twin’ approach.

Health and social care work should consult the NHS Data standards directory for guidance on data standards for health and social care in England.

The purpose of cyber security is to ensure the security of data and information. To operate effectively, the government needs to maintain the confidentiality, integrity and availability of its information, systems and infrastructure, and the services it provides. All organisations handling government data and information are required to meet government standards, as set out in the Government Functional Standard for Security and Government Functional Standard for Digital (see Chapter 7: Health, safety and security), in particular, there is a responsibility to:

• protect systems that store or process sensitive information or deliver operational services from exploitation of vulnerabilities
• provide a statement of assurance for all project delivery work to show evidence of assessing information and cyber risks, and of the controls put in place to manage them.
• provide organisational boards assessing risk with information to be able to identify programmes or projects with high information and cyber security risks
• provide clear information and cyber security guidance and standards for new work

Cyber security also comprises the protection of end-user devices and emails, digital services and cyber threats from, for example, identity theft, breaches of access and intellectual property theft.

Everyone involved in managing information and data should be aware of the importance of cyber security and of following security requirements, including:

• ensuring access to classified, sensitive or critical information and data is only provided to identified, authenticated and authorised users or systems, and where necessary seeking advice on applying proportionate risk responses.
• classifying information assets according to the Government Security Classification Policy and following the appropriate handling, storage, sharing, and destruction of information based on its marking
• induction and training for employees and people employed under contract on appropriate information security measures, including when working remotely
• roles and responsibilities in the information handling chain, including how to report data security breaches and cyber security risks and incidents

Further guidance is provided in the Government Functional Standard for Security, the Minimum cyber security standard and on the UK Government Security and National Cyber Security Centre websites. Additional considerations apply to the handling and sharing of data with international partners, and further advice should be sought.

Knowledge assets are valuable public assets which support the effective delivery of public services which need to be managed to ensure that this value is maximised and not lost or underused.

All public sector organisations should have a strategy in place to support the effective management of knowledge assets, including how to identify, protect and support their exploitation where appropriate.

Portfolios, programmes and projects can often create new knowledge assets as part of their work.  These can have considerable value and impact beyond their original purpose, whether social, economic, financial or in combination. Making best use of knowledge assets is part of maximising the benefits realised as a result of the work, in line with the principles of Managing public money (requires sign in).

The Rose Book explains the different types of knowledge asset to consider, including information and data assets, and the relevant intellectual property rights and other methods of protection associated with them, such as database rights, copyright and Crown copyright, and the law on confidence, contract and trade secrets. The Rose book provides guidance on identifying, protecting and exploiting knowledge assets and where to find further help in doing so.

Preparation for managing information and data should start as early as possible as in the life cycle. This includes identifying the requirements for information and data management, and determining the tools and processes to be used.

The management of information and data must meet legal and government standards and align with the relevant organisational controls. Specific central government data standards apply in reporting on work in the Government Major Projects Portfolio, and these should be observed, further information is available from the National Infrastructure and Service Transformation Authority.

Requirements can vary significantly, depending on the nature, scale and complexity of the work, and the types of information and data involved. For example:

• sensitive military work involves highly classified information using secure platforms and demands high standards of control over access to documents and data handling
• work to transform digital services typically involves use of open data standards but can present significant risks in handling personal data and the potential for cyber-attacks
• infrastructure and construction work generates information in multiple digital and physical formats, from documents, plans, maps, artefacts, video footage, models and simulations, as well as large bodies of technical data, some highly sensitive

Understanding the likely requirements of the work at the start to determine the approach, processes and tools needed and inform the design of an appropriate information and data repository.

The tools and facilities used should be appropriate to the needs of the work. This usually  means using existing organisational electronic document management platforms and standard data management and visualisation tools, but sometimes additional software or storage is needed, for example for specialised requirements. Physical storage needs should also be identified.

When deciding what tools and facilities needed, consider:

• what users need in terms of functionality and usability
• the likely scale, complexity and duration of the requirement
• the security, access and permission requirements
• the storage capacity and bandwidth requirements
• the need for application programming interfaces (APIs) to allow data to be moved between applications
• the potential for scalability or can be reused by other work
• the opportunities for automation, for example of repetitive rule-based processes
• the cost, resources and timescales to set up the functionality, including training and support
• requirements for data back up and retrieval, and to support business continuity

Where new functionality or facilities are under consideration, the organisation’s digital, data and security functions should be consulted and involved.

Managing information and data comprises the activities summarised in Figure 24.1 and is needed throughout the life of the work.

The information management framework, forms part of the overall governance and management framework for the work. The development should start with the requirements identified in preparation and set out how information and data are to be managed through the life cycle of the work. It includes:

• the nature and expected extent of the information and data to be managed, including any new knowledge assets created
• any requirements for handling, including relevant legal, government and organisational requirements, and for ensuring traceability
• arrangements for organisation, classification, handling, storage, review, transfer or disposal
• arrangements for assigning and managing information, data and knowledge asset ownership
• procedures for recording critical information, for example meetings, decisions and actions
• cyber security and business continuity, including handling of exceptions and incidents

Back up and contingency arrangements should be considered, particularly where access is needed for business continuity.

The information management framework should be updated as and when needed, to keep it fit for purpose.

Overseeing includes: 

• being satisfied that the information and data being used in the work is being adequately managed both collectively and at individual information and data elements or asset levels

• legal obligations and government standards on information and data management are being met

• information and data being managed securely, and shared and distributed appropriately. 

• resolving escalations for risks, issues or incidents relating to information and data

• ensuring that procedures and tools are used appropriately and that arrangements for managing information and data remain fit for purpose as the work progresses, prompting updates to the framework, if needed

Information or data created within the team should have an identified owner, usually the person creating it or for whom it is being created (for example, the senior responsible owner for a business case).

The team should use standard templates where possible, incorporating details about the creation and status of the information (known as metadata). For documents, this should include as a minimum:

• the document name and reference number, as appropriate
• the document classification in accordance with the government security classification guidance (for example SECRET), included as a header on every page (see 24.6.1.5 on protecting information and data security)
• the status of the document (for example, draft, for review, approved, withdrawn)
• who issued the document, the date of issue and version number, using major and minor version numbering system (for example, major version 1.0 and 2.0, minor version 1.1 and 2.2) to support traceability
• the page number (or sheet number in a worksheet) and total number of pages (for example 4 of 37), shown on each page

Typically, a document should be:

• defined in terms of its purpose, content, review panel, approver(s) and acceptance criteria (this is often referred to as a product description)
• drafted by the author until ready for review
• reviewed by informed and competent people against the acceptance criteria, and where necessary, iteratively updated and reissued as a draft for review, until ready for approval
• approved, or returned for update until it can be approved
• stored and distributed
• withdrawn when no longer needed and held in accordance with the retention policy
• disposed of once the retention period has lapsed

Other information and data created, for example user data, survey data or web-scraped data, should be handled in line with data standards (see 24.6.1.4 on meeting data management standards). The quality of the data can be improved at source by validation rules and capturing appropriate metadata. For further guidance on data collection and validation, see the Government data quality framework.

Information and data received from outside the team should be registered on receipt, identified by date, originator and title, with the original or equivalent classification (if applicable) and other existing metadata retained.

Where information and data is received from outside, or generated from multiple sources, it can be less obvious who the information owner should be and a decision needs to be taken by the information manager to assign ownership, usually for particular types or bodies of information. The information or data should be checked for completeness, security and sensitivity considerations, and a decision made on who it needs to be distributed to.

Ensure that handling requirements are clearly indicated, so that team members understand what needs formal handling or particular care and what can be treated informally.

Once information or data has been created or received, validated and assigned an owner, it should be prepared for storage. The information owner should check that its status, security classification and provenance is clear, decide where, and in what format the information should be stored, and who should be able to access it, so that appropriate access controls can be applied in storing the material.

Where sensitive information and data involved, for example, user data or survey responses, the data might need to be anonymised, to reduce risk or to enable the data to be used more easily.

Check for duplication and issues of consistency should also occur.

Information and data should then be stored by, or with the agreement of, the information owner.

All documentation relating to the work should be stored in the main information repository, other than where security or data handling requirements require material to be stored elsewhere, for example in a more highly classified environment, in other organisational data repositories, or in physical storage. The appropriate access controls and traceability management should be applied.

The reliability of the information and data should be checked and made available so it can be used appropriately in undertaking the work and making decisions. This can involve exploration, analysis, and further processing using specialised tools and software, with information and data feeding directly into the work and solution (for example, in 4D or5D design, planning software, and digital twins) as well as supporting the production of the business case, plans and reports, statistical outputs and performance information.

The information owner should ensure that appropriate standards and practices are followed in processing and/or using information and data. This includes ensuring that known issues with quality or reliability are understood and that the information or data is used appropriately. Where necessary, a risk assessment should be carried out on whether to use the data and if so, how to respond for known risks.

Any information or data that is subject to change should be regularly monitored for its data quality to ensure it continues to be fit for purpose.

Consideration should be given on how and with whom, information and data should be shared to support the conduct of the work, decision-making and communications. Sharing of information and data must meet the government’s legal obligations (see 24.6.1.3 on meeting legal and regulatory requirements) and security requirements (see 24.6.1.5 on protecting informaton and data security), as set out in the information management framework.

Information and data should be retained in accordance with the organisation’s data retention policy and legal requirements. Where information and data are not transferred to the service owner or operator at the end of the work, the information owner should determine, from the policy, whether it should be retained or archived elsewhere within the sponsoring organisation, or destroyed.

Specific legal requirements apply to the retention of public records including their retention by the organisation and eventual transfer, where appropriate, to the National Archives. Legal requirements also apply to the retention of contractual records, typically at least 6 years. For further information see the Code of practice on the management of records. Where necessary the chief data officer within the sponsoring organisation should be consulted.

Where superseded by a newer version, the information or data should be withdrawn and clearly marked as such. It is, however, important to ensure withdrawn versions are retained so that they can be referred to if necessary in future, including for traceability, control and audit purposes or possible legal disputes.

Where information or data relates to the design, development and implementation of the solution, it is essential that this is retained and, where possible, handed over to the future service owner or operator, to support use and disposal of the solution in future. Specific requirements apply in building information modelling and digital twin approaches, and these should be followed.

Where information and data are to be archived, the information owner should ensure that material is held in an appropriate format so that its integrity is maintained, and any relevant metadata or contextual information can be retained alongside.

When the work is completed, the information manager should ensure that information and data held in relation to the work has been reviewed and:

• handed over as part of the transition of the work into use
• transferred to an appropriate repository for use or retention in the sponsoring organisation
• archived in line with requirements for retention of public records, or
• destroyed

The information and data management framework and repository should then be closed.