Senior Data Engineer

Senior Data Engineer

The Senior Data Engineer, with expertise in Azure Databricks, is a key player in our data engineering team, responsible for designing, implementing, and maintaining scalable data solutions. This role involves collaborating with cross-functional teams to understand data requirements, architecting efficient data pipelines, and ensuring the integrity and availability of our data infrastructure. The Senior Data Engineer will leverage their proficiency in Azure Databricks to implement advanced analytics, optimize data workflows, and contribute to the overall success of our data-driven initiatives.

Main responsibilities:

Data Architecture and Design:

• Collaborate with stakeholders to understand data requirements and design scalable data architectures.
• Implement and maintain efficient data models and structures using Azure Databricks.

  Data Pipeline Development:

• Design, develop, and optimize end-to-end data pipelines, ensuring the timely and accurate flow of data across the organization.
• Implement ETL processes to transform raw data into actionable insights.

Azure Databricks Expertise:

• Leverage in-depth knowledge of Azure Databricks to implement and optimize advanced analytics and machine learning workflows.
• Utilize Databricks notebooks for data exploration, analysis, and code collaboration.

Data Quality and Governance:

• Implement data quality checks and ensure adherence to data governance and security standards /Collaborate with data stewards to establish and maintain data quality best practices.

Performance Optimization:

• Optimize data workflows for performance and scalability, ensuring efficient processing and storage of large datasets.

Qualifications: 

Education: Bachelor or Master Degree in Computer Science, Software Engineering, or a related field;

Languages: Italian and English Fluent.

Technical Skills:

• Proven experience as a Data Engineer with a focus on Azure Databricks.
• In-depth knowledge of data modeling, ETL processes, and data warehousing concepts.
• Proficiency in programming languages such as Python or Scala for data engineering tasks.
• Experience with big data technologies, such as Apache Spark.
• Strong understanding of data governance, security, and quality best practices.
• Familiarity with cloud-based data storage and processing services, particularly Azure Data Lake Storage and Azure SQL Data Warehouse.
• Excellent SQL skills for data manipulation and analysis.

Soft Skills:

• Ability to collaborate effectively in a cross-functional and dynamic team environment.
• Strong problem-solving and troubleshooting skills.
• Previous experience in implementing advanced analytics and machine learning workflows is a plus.

#LI-IC1