Building Batch Data Pipelines on Google Cloud (BBDP)

Ziele der Schulung

Data pipelines typically fall under one of the Extract and Load (EL), Extract, Load and Transform (ELT) or Extract, Transform and Load (ETL) paradigms. This course describes which paradigm should be used and when for batch data. Furthermore, this course covers several technologies on Google Cloud for data transformation including BigQuery, executing Spark on Dataproc, pipeline graphs in Cloud Data Fusion and serverless data processing with Dataflow. Learners get hands-on experience building data pipeline components on Google Cloud using Qwiklabs.

Zielgruppe Seminar

This course is intended for developers who are responsible for designing pipelines and architectures for data processing.

Voraussetzungen

• Experience with data modeling and ETL (extract, transform, load) activities.
• Experience with developing applications by using a common programming language such as Python or Java.

Seminarinhalt

Module 1 - Introduction to Building Batch Data Pipelines

Topics:

• EL, ELT, ETL
• Quality considerations
• How to conduct operations in BigQuery
• Shortcomings
• ETL to solve data quality issues

Objectives:

• Review different methods of loading data into your data lakes and warehouses: EL, ELT, and ETL.

Module 2 - Executing Spark on Dataproc

Topics:

• The Hadoop ecosystem
• Run Hadoop on Dataproc
• Cloud Storage instead of HDFS
• Optimizing Dataproc

Objectives:

• Review the Hadoop ecosystem.
• Discuss how to lift and shift your existing Hadoop workloads to the cloud using Dataproc.
• Explain when to use Cloud Storage instead of HDFS storage.
• Explain how to optimize your Dataproc jobs.

Module 3 - Serverless Data Processing with Dataflow

Topics:

• Introduction to Dataflow
• Why customers value Dataflow
• Dataflow pipelines
• Aggregate with GroupByKey and Combine
• Side inputs and windows
• Dataflow templates

Objectives:

• Identify the features that customers value in Dataflow.
• Discuss core concepts in Dataflow.
• Review the use of Dataflow templates and SQL.
• Write a simple Dataflow pipeline and run it both locally and on the cloud.
• Identify map and reduce operations, execute the pipeline, and use command line parameters.
• Read data from BigQuery into Dataflow and use the output of a pipeline as a side input to another pipeline.

Module 4 - Manage Data Pipelines with Cloud Data Fusion and Cloud Composer

Topics:

• Building batch data pipelines visually with Cloud Data Fusion
• Components
• UI overview
• Building a pipeline
• Exploring data using Wrangler
• Orchestrating work between Google Cloud services with Cloud Composer
• Apache Airflow environment
• DAGs and operators
• Workflow scheduling
• Monitoring and logging

Objectives:

• Discuss how to manage your data pipelines with Data Fusion and Cloud Composer.
• Summarize how Cloud Data Fusion allows data analysts and ETL developers to wrangle data and build pipelines in a visual way.
• Describe how Cloud Composer can help to orchestrate the work across multiple Google Cloud services.

Hinweise

Partner

Dieses Seminar bieten wir in Kooperation mit unserem Google Cloud Learning Partner Fast Lane Institute for Knowledge Transfer GmbH an.

Open Badge für dieses Seminar - Ihr digitaler Kompetenznachweis

Durch die erfolgreiche Teilnahme an einem Kurs bei IT-Schulungen.com erhalten Sie zusätzlich zu Ihrem Teilnehmerzertifikat ein digitales Open Badge (Zertifikat) – Ihren modernen Nachweis für erworbene Kompetenzen.

Ihr Open Badge ist jederzeit in Ihrem persönlichen und kostenfreien Mein IT-Schulungen.com-Konto verfügbar. Mit wenigen Klicks können Sie diesen digitalen Nachweis in sozialen Netzwerken teilen, um Ihre Expertise sichtbar zu machen und Ihr berufliches Profil gezielt zu stärken.

Übersicht: Google Cloud Schulungen Portfolio

Mehr zu den Vorteilen von Badges

Gesicherte Kurstermine

Seminare kurz vor der Durchführung