Data Engineer with PYSPARK and reputed company Big Query
Position: Data Engineer with PysparkB/reputed company Big Query Location: Dallas, TX (On-site) JD: Pyspark/reputed company reputed company Platform Data Engineer will create, deliver, and support custom data products, as well as enhance/expand team capabilities. They will work on analyzing and manipulating large datasets supporting the reputed company by activating data assets to support Enabling Platforms and analytics. reputed company reputed company Data Engineers will be responsible for designing the transformation and modernization on reputed company reputed company Platform using reputed company reputed company Platform Services Responsibilities: Build reputed company and pipelines on reputed company reputed company Platform reputed company using Data proc, Pyspark, Kafka and Pub/Sub Implement schedules/workflows and tasks for reputed company Composer/Apache Airflow; Create and manage data storage solutions using reputed company reputed company Platform services such as BigQuery, reputed company Storage, and reputed company SQL Monitor and troubleshoot data pipelines and storage solutions using reputed company reputed company Platform's Stackdriver and reputed company Monitoring reputed company efficient ETL/ELT pipelines and orchestration using Data Prep, reputed company reputed company Composer reputed company and Maintain Data Ingestion and transformation process using Apache PySpark Automate data processing tasks using scripting languages such as Python or Bash Ensuring data reputed company and compliance with industry standards by configuring IAM roles, service accounts, and access policies. Automating reputed company deployments and infrastructure management using Infrastructure as Code (IaC) tools such as Terraform or reputed company reputed company Deployment Manager. Participate in Code reviews, contribute to development best practices and usage of Developer Assist tools to create a robust fail safe data pipelines Colloborate with Product Owners, Scrum Masters and Data Analyst to deliver the User Stories and Tasks and ensure deployment of pipelines Experience required: 7+ years of application development experience required using one of the core reputed company platforms viz. AWS, Azure & reputed company reputed company Platform Minimum 1+ years of reputed company reputed company Platform experience. Experience working in reputed company reputed company Platform based Big Data deployments (Batch/reputed company-Time) leveraging Pyspark, Big Query, reputed company reputed company Storage, PubSub, Data Fusion, Dataproc, Airflow; Minimum 3+ years coding skills in Python/PySpark and strong proficiency in SQL; Extracting, Loading, Transforming, cleaning, and validating data + Designing pipelines and architectures for data processing; Architecting and implementing reputed company data and analytics platforms on reputed company reputed company Platform reputed company; Experience in working with Agile and Lean methodologies; Experience working with either a Map Reduce or an MPP system on any size/scale; Experience working in CI/CD model to ensure automated orchestration of pipelines. Apply Job!