JOB DETAILS

JOB DETAIL

1. Define the technical designs and solutions which meet functional requirements for complex data acquisition and integration pipelines, ensuring scalability and reliability within a committed timeline.

2. Manage the end-to-end life cycle for the production of data extraction systems, including scraping implementations, ensuring alignment to the functional specification.

3. Provide estimates for tasks, including the design, development, and testing of ETL workflows using Pandas or PySpark.

4. Collaborate with cross-functional teams of product and development to deliver automated data feeds and product features, working closely with product owners to understand data schema requirements.

5. Developing backend services following standard and best practices, specifically utilizing Python for heavy data transformation and Node.js for asynchronous web interaction.

6. Seeing through a project from conception to finished product, taking ownership of data from external source extraction to final loading into OLTP (Firestore/CloudSQL) and OLAP (BigQuery) systems.

7. Designing and developing APIs with standard, quality, and security, ensuring robust integration with 3rd-party services and internal data consumption layers. Provide new techniques or suggestions to team members, particularly regarding rapid adaptation to new technologies and self-learning strategies for emerging cloud tools.

REQUIRED WORK EXPERIENCE

- Bachelor's Degree in Computer Sciences, Software Engineering, or equivalent experience

- Strong proficiency in Python and Node.js and a deep understanding of their paradigms (e.g., Python for ETL/Data Processing and Node.js)
.
- Extensive experience with relational databases (e.g., PostgreSQL, CloudSQL) and NoSQL databases like Firestore, including data modeling for OLTP systems.

- Familiarity with RESTful APIs, XML/HTML parsing, and integration patterns for consuming third-party data sources

- In-depth knowledge of containerization technologies like Docker and orchestration workflows using Apache Airflow.

- Fast learning, up to date with technology, detail-oriented code, high soft and hard skills