About the Role

As a Data Engineer you will be part of the Core Data Engineering team driving the change and technical innovation to build big data cloud applications supporting reporting, analytics, AI and machine learning. We look for self-starters who thrive in a fast-paced, agile environment being able to change direction quickly, and showing an eagerness to learn new technologies as the need arises. This position requires hands-on coding & strong development background. In this role you will join the exciting DriveWealth team working on the latest Big Data technologies (Spark, AWS stack, EMR, Glue, DynamoDB) to lay out the application framework and infrastructure foundation using your Python programming & Big Data processing skills.

What You'll Do

• Build Robust data pipelines primarily using Pyspark and other Big Data technologies.
• Automate routine data analysis reporting and presentation generation with Python
• Develop automation code (Python Scripts) to create and configure infrastructure resource monitoring needed to run applications.
• Help design the best Data Lake platform to support the capacity and performance requirements.
• Design and implement data processing and machine learning pipelines.
• Performance tuning of Spark jobs.
• Create an end to end fully automated analytics production flow. From Data ingestion to data and analytics ready to be consumed by customer facing API.
• Support the team to ensure all committed stories for the sprint are completed per the sprint goal

What You'll Need

• 3+ years of experience in Data Engineering with Apacha Spark/PySpark.
• 5+ years of experience with services like S3, EMR, Kinesis, Glue, Athena, Glue, DynamoDB, Lambda, github, jenkins.
• 5+ years of experience in Python (analysis, application development, or scripting) with Python libraries: Sci-kit, Pandas, Numpy, Matplotlib, PyTorch, etc.
• Excellent understanding of BigData concepts and at-least one RDBMS.
• Proficient with SQL, BI & data visualization techniques
• Detail oriented – demonstrate thoroughness and strong ownership of your work
• Bachelor or Master degree in Computer Science preferred.

Special Knowledge (Nice To Have, But Not Required)

• Experience working in the finance industry, understanding of stock trading workflow/architecture.