Job description

Title: Big Data Lead

Location: Schenectady, New York

Duration: 6 Months plus possible extension

Job Description:

• Design & Implement Data ingestion and Data lakes-based solutions using Big Data Technologies.

• The Tech Lead should be highly proficient in the use of Big Data / Open- Source Technologies and standard techniques of Data Integration, Data Manipulation.

• Should be able to design and develop cost efficient and performant data pipelines in the cloud platform

• Create data environment to support our data analytics, reporting and data science teams

• Experience with integration of data from multiple data sources

• Knowledge of various Data Pipeline techniques and frameworks

• Performance optimization - need to monitor the complete process and apply necessary infrastructure changes to speed up the query execution.

• Efficient data ingestion - Discovering patterns in data sets with data mining techniques and using different data ingestion APIs and inject data into the data lake as per need.

Essential Skills:

• 6+ Years hands on knowledge on SQL as well as SQL/NoSQL databases

• Proficient in programming languages such as Python, PySpark, Scala and Java

• Experience with Spark

• Working knowledge of XML, ETL, API and Web Services

• Experience with integration of data from multiple data sources

• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB

• Knowledge of various ETL techniques and frameworks, such as Flume

• Experience with various messaging systems, such as Kafka or RabbitMQ

• Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming

• Working knowledge and experience in Big data services in one of the Cloud Provider will be good (AWS or Azure or GCP)

Essential Qualification:

• BE/Btech in Computer Science, Engineering or relevant field

Experience:

9 Years - 12 Years

Job Type: Contract

Schedule:

• 8 hour shift

Work Location: On the road

jjbodyshop.com is the go-to platform for job seekers looking for the best job postings from around the web. With a focus on quality, the platform guarantees that all job postings are from reliable sources and are up-to-date. It also offers a variety of tools to help users find the perfect job for them, such as searching by location and filtering by industry. Furthermore, jjbodyshop.com provides helpful resources like resume tips and career advice to give job seekers an edge in their search. With its commitment to quality and user-friendliness, jjbodyshop.com is the ideal place to find your next job.