Posted 5 days ago
Big Data Engineer (PySpark / Spark / Kafka)
@ Astra North Infoteck Inc.
Toronto, Ontario
Overview
Astra North Infoteck Inc. is hiring a contract Big Data Engineer to design, build and support enterprise-scale data platforms. The work covers batch and real-time pipelines, Kafka streaming and ETL/ELT workflows built on the Spark and Hadoop ecosystem. The role is hybrid, with 3 days a week in the office.
Responsibilities:
- Design and optimize scalable data pipelines with PySpark, Spark, Hadoop and NiFi
- Build and support Kafka streaming and event-driven applications
- Create and tune ETL/ELT workflows for large datasets
- Monitor and troubleshoot Spark jobs and data pipelines
Requirements summary:
- Hands-on PySpark and Apache Spark experience
- Apache Kafka and Hadoop ecosystem experience
- Apache NiFi experience
- Strong SQL and Python skills
Tools & skills
Source and dates
Source type not verified · zoho
- Employer posting date
- 2026-11-01
- First seen by our source
- 2026-11-02
- Source record updated
- 2026-11-02
These dates describe the source record, not a live verification that the employer is still hiring. Check the original posting.
Job details
| Seniority | Mid level |
| Department | Software development |
| Travel | air: none · ground: none |
| Work setting | Office |
| Position | Mostly sitting |
| Physical demands | physical low · computer high |
Summary and details written by AI from the posting; the employer’s posting is the authority.
Astra North Infoteck Inc.
| Field | Value |
|---|---|
| Open jobs on Jobsy | 427 |
Job Description: Big Data Engineer (PySpark / Spark / Kafka)
Work Mode: Hybrid – 3 Days a Week in Office
Job Overview
We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.
Key Responsibilities
- Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
- Build and maintain batch and real-time data processing solutions.
- Develop and support Kafka-based streaming applications and event-driven architectures.
- Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
- Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
- Implement data ingestion solutions from databases, APIs, files, and streaming sources.
- Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
- Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
- Support platform upgrades, deployments, testing, certification, and production releases.
- Ensure data quality, governance, security, and operational excellence across data platforms.
Mandatory Skills
- PySpark
- Apache Spark (Spark SQL, DataFrames)
- Apache Kafka
- Hadoop Ecosystem (HDFS, Hive, YARN)
- Apache NiFi
- SQL
- Python
Preferred Skills
- Spark Streaming
- Airflow / Oozie
- Hive
- Scala
- Jenkins, Bitbucket, Git
- JIRA, Confluence
- Cloud Platforms (GCP/AWS/Azure)
- Data Warehousing Concepts and Dimensional Modeling
Journal
Only you can see this. Kept on this device, and in your account when you sign in.