Drove the migration of the GCP Data Platform to Azure, leveraging Terraform, Kubernetes, Dagster, and Python to enhance platform performance and scalability. Designed and implemented migration strategies to ensure a seamless transition, minimizing downtime and maintaining data integrity. Utilized Terraform to automate infrastructure provisioning, ensuring consistency and efficiency across environments. Leveraged Kubernetes to manage containerized applications, improving scalability and resource management. Implemented Dagster for orchestrating data workflows, enhancing visibility and control over data pipelines.
Development and Maintenance of Data Platform
Contributed to the development and maintenance of the current data platform, utilizing Dataproc, GKE, and BigQuery to support data workflows and process optimization. Optimized data processing workflows using Dataproc, ensuring efficient and scalable data handling. Managed and scaled GKE clusters to support containerized applications, enhancing performance and reliability. Leveraged BigQuery for large-scale data analytics, providing insights and supporting data-driven decision-making. Collaborated with data engineering teams to address technical challenges and enhance platform capabilities.
Migration of Data Models to DBT and Databricks
Supported the migration of data models to DBT, improving data transformation processes and ensuring consistency. Set up a Databricks environment for processing data models, leveraging its powerful analytics and machine learning capabilities. Scheduled data workflows using Dagster, ensuring timely and efficient data processing. Collaborated with stakeholders to define migration strategies and ensure alignment with business objectives. Provided technical guidance and support throughout the migration process, addressing any issues and optimizing performance.