“LLM-as-a-judge frameworks” jobs in Canada

No matching filters. Try another name.

Results

• 2d

  Applied AI Engineer

  Remote in Canada

  RemoteFull Time

  

  Jobgether

  Candidates need experience shipping LLM or machine learning software to real production users, plus a systematic approach to evaluating model and agent quality. They must treat cost and latency as engineering constraints and communicate technical trade-offs clearly to engineers, product teams and customers.

• 3d

  Ingénieur·e IA – Skills & Systèmes agentiques

  Montréal, QC

  Full Time

  

  Publicis Groupe Holdings B.V

  Formation ou expérience en développement logiciel ou ingénierie IA; Expérience concrète de solutions basées sur les LLM en production; Maîtrise de Python et des API

  Python, OpenAI, Azure OpenAI, Anthropic Claude, Google Gemini, Mistral AI

• 10d

  Senior AI Quality Developer, Personalization

  Toronto, ON

  HybridFull Time

  A

  Autodesk

  6+ YOEAutodesk needs candidates with 6+ years of software or test automation experience and hands-on TypeScript and Python skills. Candidates should have experience testing LLM-based applications, knowledge of RAG and agent architectures, and familiarity with CI/CD and AWS.

  Playwright, Puppeteer, Pytest, AWS, CI/CD pipelines

• 12d

  Senior Product Manager, AI Platform

  Waterloo, ON

  HybridFull Time

  VH

  Verily Health Inc: Verily Health is a data platform and technology company purpose-built to power AI-enabled precision health.

  5+ YOEVerily Health Inc. needs 5+ years of platform product management, hands-on experience shipping AI evaluation tools such as LLM-as-a-judge or golden datasets, and a technical background in software, data science or clinical informatics. Healthcare compliance experience such as HIPAA or FDA SaMD is preferred.

  LLM-as-a-judge frameworks, golden datasets, code assertions, trace tooling

• 16d

  Senior/Lead Machine Learning Engineer

  Montréal, QC +1

  RemoteFull Time

  

  Mimica: Mimica’s process intelligence highlights inefficiencies, prioritizes improvements based on ROI, recommends the optimal technology for automation…

  Candidates need a solid machine learning and data science background, hands-on experience evaluating LLMs in production, and strong Python engineering skills. They should work independently on ambiguous problems and communicate clearly in English.

  Python

• 17d

  Senior AI Engineer for Observability

  Toronto, ON

  Full Time

  

  Bank of Montreal

  7+ YOECandidates need a bachelor's degree in a relevant field and 7 to 10 years of site reliability engineering experience. They must have AI observability and platform engineering experience, and strong skills in Python, SQL, and monitoring tools.

  Prometheus, Grafana, Datadog, OpenTelemetry, Python, SQL

• 28d

  Senior AI Applied Scientist II (Agentic)

  Vancouver, BC

  $150k–$177k/yr

  

  Prenuvo: Prenuvo currently operates in 26 clinics across North America, Australia, and Europe.

  4+ YOEPrenuvo needs a candidate with an MSc or PhD in a related field and at least 4 years of ML/NLP experience with first-author publications in top venues. Candidates must have expert PyTorch skills, hands-on LLM fine-tuning (SFT, LoRA, RLHF, DPO), agentic AI and RAG experience, and be legally authorized to work in Canada.

  PyTorch, Hugging Face Transformers, Unsloth, vLLM, DeepSpeed, Axolotl

• 28d

  Staff Machine Learning Engineer, SecureAI

  Toronto, ON

  

  Okta

  8+ YOEOkta needs candidates with 8+ years of software development, strong Python skills, and hands-on experience with applied machine learning and generative AI platforms. A Bachelor's or Master's in computer science or a related field is required.

  Python, PyTorch, TensorFlow, FastAPI, Airflow, AWS Bedrock

  28d

  Staff Machine Learning Engineer, Generative AI (Auth0)

  Toronto, ON

  

  Okta

  7+ YOECandidates need 7+ years of software development experience, strong Python skills and hands-on work with generative AI platforms and RAG. Experience with AI agent frameworks, ML frameworks and defining ML evaluation metrics is also required.

  AWS Bedrock, OpenAI, Anthropic Claude, LangChain, LangGraph, LlamaIndex

  28d

  Principal Machine Learning Engineer, SecureAI

  Toronto, ON

  

  Okta

  10+ YOECandidates need 10+ years of software development experience, strong Python skills, and hands-on applied machine learning and generative AI experience. A bachelor's or master's degree in computer science or a related field is expected.

  Python, PyTorch, TensorFlow, FastAPI, Airflow, AWS Bedrock

• 1mo

  Senior Machine Learning Engineer

  Remote in Canada

  RemoteFull Time

  

  Cresta

  5+ YOECandidates need 5 to 8 or more years of building production machine learning systems, with significant LLM experience. Strong skills in NLP, transformers, RAG and agentic workflows are required, plus technical leadership.

  PyTorch, TensorFlow, Hugging Face

• 1mo

  Senior ML Data Processing Developer

  Montréal, QC

  

  LawZero

  5+ YOECandidates need a computer science or related degree, 5+ years in data processing, ML engineering or NLP on trillion-token-scale text, and strong Python. Experience with distributed frameworks like Spark, Ray or Flink and orchestration tools like Airflow, Prefect or Dagster is also required.

  Spark, Ray, Flink, Airflow, Prefect, Dagster

• 1mo

  AI Engineer

  Toronto, ON

  Hybrid

  

  NetBrain

  3+ YOECandidates need a bachelor's degree in a technical field and 3+ years in software or applied AI, including 2+ years running LLM or agent applications in production. Strong Python and distributed backend skills, plus experience with LLM evaluation and LLM security risks, are required.

  LangSmith, LangChain, LangGraph, AutoGen, LlamaIndex

11 companies