Posted 7 days ago
QA Developer, AI Assistant
@ Poka Inc.
Montréal, Quebec
SIGNALS
Overview
Poka Inc. is hiring a QA Developer for its AI assistant team, building the assistant used by manufacturing workers and supervisors. The role is full-time, focused on automated evals, tests and tools that judge answer quality, with Python work weekly. Hybrid in Québec City or Montréal (two office days per week within the stated radius), with 100% remote otherwise.
Responsibilities:
- Automate QA for AI assistant features and tools
- Build and maintain eval suites from real usage
- Define scoring rubrics and use LLM-as-judge
- Run regression checks in CI on prompt, tool and model changes
- Contribute to adversarial and exploratory testing
Requirements summary:
- 2 to 4 years of relevant experience
- Test automation or SDET background
- Python and TypeScript proficiency
- Curiosity about LLM behavior and failure modes
- Ability to judge quality without single correct answers
- Strong English
Tools & skills
Source and dates
Source type not verified · apify-linkedin
- Employer posting date
- 2026-11-06
- First seen by our source
- 2026-11-14
- Source record updated
- 2026-11-14
These dates describe the source record, not a live verification that the employer is still hiring. Check the original posting.
Job details
| Seniority | Mid level |
| Department | Software development |
| Experience | 2+ years |
| Hours per week | 40 (estimated) |
| Travel | air: none · ground: none |
| Work setting | Office, Home |
| Position | Mostly sitting |
| Physical demands | physical low · computer high |
| Languages | English |
| Perks mentioned | Generous paid time off, Flexible hours, Retirement plan, Wellness, Home office allowance |
Summary and details written by AI from the posting; the employer’s posting is the authority.
Poka Inc.
| Field | Value |
|---|---|
| Open jobs on Jobsy | 1 |
Job description
At Poka, we're changing how the world's industrial teams work — capturing the knowledge of their best people and putting it in every operator's hands.
Our connected worker platform is purpose-built for manufacturing and the wider industrial world, with industrial AI inside the shift, not on top of it. The result: frontline performance that becomes compounding knowledge and measurable impact — every shift, every site, every region.
We're proud to support global leaders like Nestlé, Bosch, Mars, and Danone across 3,000+ sites in 70+ countries. And as a fast-growing part of the IFS family, you'll join a team whose daily work directly shapes the platform behind how modern manufacturing gets done.
Role overview
Our team develops the AI assistant built into Poka. Workers and supervisors use it every day on manufacturing floors. We build it with prompts, tools, information retrieval and agents.
As a quality assurance (QA) specialist on this team, you make sure what we ship works: the assistant's answers and the features around it. You do it with automation, AI and tools you build, not with manual testing. You work with the devs, the product team and the design team to define what "good" means, then you build the evals, automated tests and tools that check it.
How this is different from traditional QA
What you'll be responsible for
Above all, we're looking for someone curious and resourceful. You care about the success of the product and your team. You like complex problems, especially the ones without an obvious solution. You want to write clean, high-quality code and keep learning.
If that sounds like you, you're probably the person we're looking for!
What we offer:
Please note: An understanding of English is essential at Poka, as it is the language used during company-wide meetings and many internal communications, ensuring effective collaboration and decision-making. In addition, as Poka operates in an international environment, English is the primary language for communicating with our clients and partners, enabling us to better serve them and represent Poka in a professional manner.
Our connected worker platform is purpose-built for manufacturing and the wider industrial world, with industrial AI inside the shift, not on top of it. The result: frontline performance that becomes compounding knowledge and measurable impact — every shift, every site, every region.
We're proud to support global leaders like Nestlé, Bosch, Mars, and Danone across 3,000+ sites in 70+ countries. And as a fast-growing part of the IFS family, you'll join a team whose daily work directly shapes the platform behind how modern manufacturing gets done.
Role overview
Our team develops the AI assistant built into Poka. Workers and supervisors use it every day on manufacturing floors. We build it with prompts, tools, information retrieval and agents.
As a quality assurance (QA) specialist on this team, you make sure what we ship works: the assistant's answers and the features around it. You do it with automation, AI and tools you build, not with manual testing. You work with the devs, the product team and the design team to define what "good" means, then you build the evals, automated tests and tools that check it.
How this is different from traditional QA
- No manual test passes. If something needs to be checked more than once, you automate it, with code, AI or whatever tool fits.
- A large part of what you test has no single answer. The same question can get 50 good answers phrased differently, and the assistant can answer differently from one run to the next. You help define what "good" means and measure it.
- You write Python every week.
What you'll be responsible for
- Automating QA for the team's features and building the tools that make it easier, including ways for non-developers to add evals and analyze the results.
- Building and maintaining eval suites based on real usage: actual user questions, expected behavior and the edge cases that show up in production.
- Defining with the team what a good answer looks like for our users, writing scoring rubrics and using LLM-as-judge where it makes sense, calibrated against human-labeled samples.
- Running regression checks in CI every time we change a prompt, a tool, retrieval or the Claude model version, covering answer quality, cost and speed. You build the guardrails, evals and tests that tell us whether a release gets the green light or not.
- Reviewing real conversations and grouping failures into patterns: made-up answers, wrong tool calls, missed context, inappropriate tone. Then working with developers to turn them into fixes.
- Contributing to adversarial testing: prompt injection, data leaking between customers, getting the assistant to do things it shouldn't.
- Doing timeboxed exploratory testing on features that change what the assistant can decide or do, like new agents, tools or actions, including checking that actions stay safe when something fails or gets retried.
- Joining feature kickoffs to help define acceptance criteria, including AI-specific ones.
- AEC, DEC or BAC in a field related to the position, or equivalent experience
- 2 to 4 years of relevant experience
- A background in test automation or as an SDET
- You're comfortable with several programming languages, including Python and TypeScript
- Real curiosity about how LLMs behave and why they fail
- You're comfortable judging quality when there's no single answer, and explaining your reasoning
- Curiosity about our users' world. You can't judge an answer without understanding what a supervisor on a manufacturing floor actually needs.
- You've helped set up an eval harness, even in a personal project
- Experience using AI tools to speed up testing
- Basic statistics: pass rates, variance, sample size
- Experience in security or adversarial testing
- Knowledge of the test pyramid and the shift-left approach
- B2B experience or experience on industrial platforms
- Knowledge of Jira
- Python
- TypeScript
- Claude Code
- Eval framework (promptfoo)
- AWS
- Docker
- And many others
Above all, we're looking for someone curious and resourceful. You care about the success of the product and your team. You like complex problems, especially the ones without an obvious solution. You want to write clean, high-quality code and keep learning.
If that sounds like you, you're probably the person we're looking for!
What we offer:
- Scale-up energy with huge backing and a global yet close-knit team.
- Generous vacation policy, additional paid days, and charity day.
- Hybrid work policy - Employees who live within a 40 km radius of our Québec City office or a 15 km radius of our Montréal office are required to work in-office two days per week.
- 100% remote work if you do not live within the radiuses mentioned above.
- A truly flexible schedule that allows you to adapt your work hours.
- Group insurance from day one (dental, medication, disability, travel, etc.) with employer contributions, including access to telemedicine and an Employee Assistance Program (EAP).
- A retirement savings plan with employer contributions.
- A MacBook Pro and a $500 home office allowance upon hiring.
- A generous annual learning and development budget to support your growth, plus the tools and opportunities you need to keep progressing in your career.
- A rewarding employee referral program for successful hires.
- Tasty snacks available at all times in our offices (fresh fruits, snacks, pastries, coffee, tea, etc.)
- Certain benefits are exclusively accessible to employees in Canada.
Please note: An understanding of English is essential at Poka, as it is the language used during company-wide meetings and many internal communications, ensuring effective collaboration and decision-making. In addition, as Poka operates in an international environment, English is the primary language for communicating with our clients and partners, enabling us to better serve them and represent Poka in a professional manner.
Journal
Only you can see this. Kept on this device, and in your account when you sign in.