Forwarded from How to DWH with Python
Подготовил конспект курса по Spark. Поможет освежить важное в памяти или сэкономить время на просмотре. Сам курс к изучению рекомендую.
#briefly #spark Spark Starter Kit
https://telegra.ph/Udemy-Spark-Starter-Kit-part-1-06-19
What's inside:
— Hadoop and Spark comparison: storage, MapReduce, speed, resources management.
— Challenges Spark tries to address.
— How Spark achieves high efficiency.
— How Spark achieves fault-tolerance.
— What is RDD.
Ссылка на курс: Spark Starter Kit
#briefly #spark Spark Starter Kit
https://telegra.ph/Udemy-Spark-Starter-Kit-part-1-06-19
What's inside:
— Hadoop and Spark comparison: storage, MapReduce, speed, resources management.
— Challenges Spark tries to address.
— How Spark achieves high efficiency.
— How Spark achieves fault-tolerance.
— What is RDD.
Ссылка на курс: Spark Starter Kit
Telegraph
Udemy: Spark Starter Kit, part 1
Spark vs Hadoop: who wins? Link to lecture. Hadoop = HDFS + MapReduce. Spark is not a replacement for Hadoop. In particular, Spark does not come with its own storage: it leverages existing one like HDFS, S3, etc. Distributed filesystem are preferred to accelerate…
Forwarded from How to DWH with Python
Подготовил конспект статьи от Shopify о сетапе Airflow на 10 тысяч DAG'ов со 150 тысячами запусков в день. Сэкономит вам время на прочтении и поможет освежить в памяти в будущем.
#briefly #airflow Airflow: scaling out recommendations by Shopify
https://telegra.ph/Airflow-scaling-out-recommendations-by-Shopify-06-03
What's inside:
— Cloud Storage vs Network File System.
— Metadata retention policy.
— Manifest file.
— Consistent distribution of load.
— Concurrency management.
— Using different execution environments.
Origin: Lessons Learned From Running Apache Airflow at Scale
#briefly #airflow Airflow: scaling out recommendations by Shopify
https://telegra.ph/Airflow-scaling-out-recommendations-by-Shopify-06-03
What's inside:
— Cloud Storage vs Network File System.
— Metadata retention policy.
— Manifest file.
— Consistent distribution of load.
— Concurrency management.
— Using different execution environments.
Origin: Lessons Learned From Running Apache Airflow at Scale
Telegraph
Airflow: scaling out recommendations by Shopify
Shopify runs over 10k DAGs. 150k runs per day. Over 400 tasks at a given moment on average. This is a brief overview of their approach. Link to source article. Fast file access Problem: reading DAGs files from Google Cloud Storage (through GCSFuse as a filesystem…
🔥10👍4