Databricks Platform Engineer

Databricks Platform EngineerLocation: Toronto, CanadaSalary $180-200K CADOverview: Our client is seeking an experienced Databricks Platform Engineer to join their Data/AI team. This team will be dedicated to building a modern data infrastructure for advanced analytics and the development of responsible AI. Get in on the ground floor of the strategic investment this client is making to develop cutting-edge Generative AI and analytical capabilities that will be the basis for their next generation research enablement platform and data products. Responsibilities: Design and implement cloud solutions, build machine learning operations (MLOps) on cloud (Azure and Databricks) Build CI/CD pipelines orchestration using tools such as Azure DevOps Use MLFlow and handle model serving and monitoring on Databricks Test, validate, and automate tests for ML models Tune Databricks for spark cluster performance and utilization optimization Provide guidance on Databricks capabilities to accommodate business/technical use cases Automate and manage provisioning needs, including Databricks Cluster Policies, Unity Catalog, Secrets Management, Role-Based Access Control model, and permissions Monitor and manage data and processes to control costs and optimize compute spend Configure and manage monitoring/alerting around latency performance on the Databricks platform, including Delta Live Tables and Databricks Workflows and machine learning model performance Coordinate and collaborate with dependent infrastructure teams and Azure services to integrate Databricks securely. Provide technical expertise and troubleshooting as well as support for change management, governance, compliance, internal audits, and remediations Communicate with a team of data scientists, ML engineers, data engineers, and architects, and document the processes Requirements: Bachelor’s Degree 2+ years of relevant experience on Databricks, including performance tuning, infrastructure guidance, workflow automation and cost optimization strategies Experience in cloud solutions, preferably on Azure Skills in MLFlow, containerization, and deployment practices (Preferred) Excellent communication and documentation skills to collaborate effectively with cross-functional teams