Engineering data

What do we mean by data engineering?  

The design, development and management of data systems, processes, and infrastructure for the purposes of integration and/or analysis. 

Data engineering translates architectural designs into reality. It includes designing and implementing processes for data collection, preparation, storage, and analysis and creating data pipelines that enable data to be extracted from multiple sources, transformed, and aggregated for the purposes of analysis.

Knowledge 

• How to implement / follow effective data governance.
• Knowledge of data security.
• Understanding data architecture best practice.
• Knowledge of best practice data quality management.
• Knowledge of programming languages like Python and R, data visualisation tools like PowerBI and Tableau and big data technologies like Apache and Spark.
• Understanding of Cloud platforms e.g.  Azure, AWS, Google Cloud.
• Understanding of different project delivery approaches, including Agile and waterfall.

Skills 

Able to: 

• Input to, and implement, data architecture and modelling: Designing and implementing the detailed structure of databases to handle large volumes of data and supports efficient querying and analysis. 
• Support data cleansing and preparation: Using tools and techniques to cleanse data, remove duplicates, ensure completion and readiness for use.  
• Manage extraction, transformation and loading of data (ETL): Taking data from its source, converting it, and loading it into data platforms for analysis, or into other solutions in its transformed state. 
• Create data pipelines: Enabling data to flow from its originating source to data platforms for analysis. 
• Apply technical knowledge: Including programming languages like Python and R, data visualisation tools like PowerBI and Tableau and big data technologies like Apache and Spark to query, analyse and present data. 
• Manage databases efficiently: Ensuring effective structuring of data, appropriate security controls and optimising performance. 
• Communicate and collaborate effectively: With stakeholders at all levels, including technical and non-technical teams. 
• Manage risk: Identifying and assessing the risks associated with data engineering processes, ensuring they reflected in the risk register appropriately and have effective mitigation within risk appetite.