Experience
- Experience in building and managing data pipelines.
- experience with development and operations of data pipelines in the cloud (Preferably Azure.)
- Experience with distributed data/computing tools: Map/Reduce, Hadoop, Hive, Spark
- Deep expertise in architecting and data pipelines in cloud using cloud native technologies.
- Good experience in both ETL and ELT Ingestion patterns
- Hands-on experience working on large volumes of data(Petabyte scale) with distributed compute frameworks.
- Good Understanding of container platforms Kubernetes and docker
- Excellent knowledge and experience with object-oriented programming
- Familiarity developing with RESTful API interfaces.
- Experience in markup languages such as JSON and YAML
- Proficient in relational database design and development
- Good knowledge on Data warehousing concepts
- Working experience with agile scrum methodology


Technical Skills
- Strong skills in distributed cloud Data analytics platforms like Databricks, HD insight, EMR cluster etc.
- Strong in Programming Skills -Python/Java/R/Scala etc.
- Experience with stream-processing systems: Kafka, Apache Storm, Spark-Streaming, Apache Flink, etc.
- Hands-on working knowledge in cloud data lake stores like Azure Data Lake Storage.
- Data pipeline orchestration with Azure Data Factory, Amazon Data Pipeline
- Good Knowledge on File Formats like ORC, Parquet, Delta, Avro etc.
- Good Experience in using SQL and No-SQL. databases like MySQL, Elasticsearch, MongoDB, PostgreSQL and Cassandra running huge volumes of data
- Strong experience in networking and security measures
- Proficiency with CI/CD automation, and specifically with DevOps build and release pipelines
- Proficiency with Git, including branching/merging strategies, Pull Requests, and basic command line functions
- Strong experience in networking and security measures
- Good Data Modelling skills


Job Responsibilities

- Cloud Analytics, Storage, security, resiliency and governance
- Building and maintaining the data architecture for data Engineering and data science projects
- Extract Transform and Load data from sources Systems to data lake or Datawarehouse leveraging combination of various IaaS or SaaS components
- Perform compute on huge volume of data using open-source projects like Databricks/spark or Hadoop
- Define table schema and quickly adapt with the pipeline
- Working with High volume unstructured and streaming datasets
- Responsible to manage NoSQL Databases on Cloud (AWS, Azure etc.)
- Architect solutions to migrate projects from On-premises to cloud
- Research, investigate and implement newer technologies to continually evolve security capabilities
- Identify valuable data sources and automate collection processes
- Implement adequate networking and security measures for the data pipeline
- Implement monitoring solution for the data pipeline
- Support the design, and implement data engineering solutions
- Maintain excellent documentation for understanding and accessing data storage
- Work independently as well as in teams to deliver transformative solutions to clients
- Be proactive and constantly pay attention to the scalability, performance and availability of our systems
- Establishes privacy/security hierarchy and regulates access
- Collaborate with engineering and product development teams
- Systematic problem-solving approach with strong communication skills and a sense of ownership and drive