Job Description

Collibra Data Lineage Automation Engineer – Contractor

hackajob on-demand focuses on matching talented contractors like you with organisations seeking specific skills for their projects. We use our platform to connect you with exciting contract opportunities and discuss projects on behalf of the companies we partner with.

Role Overview

We are seeking a Data Lineage Automation Engineer to design and implement automated end-to-end data lineage solutions across a complex enterprise data ecosystem. This role combines data engineering, metadata management, and AI/ML to enable lineage capture, traceability, and intelligent metadata extraction across cloud, legacy, and reporting systems.

Key Responsibilities

• Lead implementation of automated data lineage across hybrid environments:
• Cloud platforms (Snowflake, AWS)
• Legacy relational databases and ETLs
• NoSQL stores
• BI/reporting platforms (Tableau, Power BI)
• Implement/extend lineage frameworks like Spline, OpenLineage, or Marquez .
• Build connectors, extractors, or agents to bridge gaps in lineage capture.
• Integrate with metadata platforms (e.g., Collibra) to publish lineage in a usable format.
• Apply AI/ML techniques to infer lineage where automation is incomplete.
• Develop reusable lineage components for cross-domain operational use.
• Advise stakeholders on lineage standardization, storage, and operational best practices.

Required Skills & Experience

• Proven experience delivering automated data lineage solutions in hybrid architectures .
• Hands-on expertise with Spline, OpenLineage, Marquez, or comparable frameworks .
• Deep understanding of metadata capture, ETL tracing, and query execution mapping .
• Strong AI/ML background in metadata intelligence , code parsing (NLP), or pattern detection.
• Experience integrating lineage with governance tools (Collibra, Alation).
• Strong programming skills: Python, Scala, or Java.
• SQL expertise across systems: Snowflake, SQL Server, Oracle, MongoDB, etc.

Preferred Skills (Big Plus)

• Experience with commercial data lineage tools.
• Prior work in regulated industries (finance, healthcare).
• Familiarity with event-based architectures for real-time lineage.
• Knowledge of data mesh or domain-driven lineage approaches.

Ideal Candidate Profile

• Enterprise-scale automated lineage implementation experience.
• Operates at the intersection of data engineering, governance, and AI .
• Acts as a technical thought partner for architecture and governance teams.
• Automation-first, reuse-oriented mindset.

Job Tags

Similar Jobs