hotskills.tech

About Job

Job Title: Senior Data Engineer Generalist – Python Expert Location: Manhattan , NY - 3 days a week FTE Role US citizens and Green Card Holders and those authorized to work in the US are encouraged to apply. We are unable to sponsor H1b candidates at this time Job Description: Looking for - A Data Engineer Generalist who is a python expert and can code without AI.. Key Responsibilities: Design, implement, and maintain scalable data engineering pipelines using Python, Spark, and SQL, applying clean coding principles, and optimizing for performance in Azure and Databricks.Use good software engineering principles to enable modularity, reliability, and reusability of code.Orchestrate complex data workflows using frameworks like Airflow, Databricks workflows (or) equivalent, adopting a code-first approach.Provide data engineering support for trading analysts in NY.Use CI/CD tools like GitHub actions or equivalent for deploying data pipeline to higher environments efficiently.Implement data quality checks, validation frameworks, and monitoring systems to ensure data integrity and reliability.Design effective data models, optimize storage and retrieval processes, and efficiently handle semi-structured and unstructured data.Create and manage APIs for data access and integration and develop web scraping solutions using leading frameworks when necessary.Build scalable data ingestion pipelines for various external data feeds with different formats, velocity, and patterns.Collaborate closely with trading analysts in an agile environment, iterating data products in an efficient manner demonstrating strong understanding of the domain context.Apply strong analytical and logical problem-solving skills to address complex data engineering challenges and develop innovative solutions.Ensure data security and compliance with relevant regulations, implementing best practices for data protection and access control.Continuously optimize pipeline performance, evaluate new technologies, and drive innovation in the data engineering ecosystem.Maintain comprehensive documentation and contribute to the team's knowledge base, fostering a culture of continuous learning and improvement.Collaborate closely with data engineers in London to implement as per standardized guidelines and guardrails. Qualifications and Requirements: Degree educated in either Computer Science, Mathematics, Engineering, or a related field.Strong hands-on experience with Python, Spark, SQL and strong knowledge of clean coding principles.Strong understanding of Data Governance best practices (DQ handling, Lineage..etc)Experience in handling various data ingestion patterns.Strong accountability and Entrepreneurialism in day-to-day work.Storing working experience with Data Orchestration (Airflow), Containerization (Docker, K8) and CI/CD tools.Desirable to have working knowledge in Microsoft Azure and Databricks.Advantageous to have experience working with Commodity/Energy trading data.Demonstrated ability to work effectively both independently and as part of a collaborative global team following internal Data engineering standards.Excellent problem-solving skills and a strong attention to detail.Strong stakeholder communication skills. A reasonable, good faith estimate of the minimum and maximum base salary for this position is $120 K to $180 K per year with Bonus. Employee benefits will also be available, and details are available like Pension/401K/ Paid Vacation/ Life, Medical & Dental insurance etc. * The pay range listed above reflects the expected starting salary /Pay rate for this role. This range may be adjusted based on market conditions, location, and other relevant factors. The Company will determine the final starting salary/Pay rate in consultation with the selected candidate(s), in full compliance with applicable laws job perk: Posted At: 23 hours ago, job perk: Salary: 120K–180K a year, job perk: Schedule Type: Full-time