{"id":717,"date":"2024-09-28T18:17:04","date_gmt":"2024-09-28T16:17:04","guid":{"rendered":"https:\/\/www.nicedata.fr\/?p=717"},"modified":"2024-10-07T15:07:21","modified_gmt":"2024-10-07T13:07:21","slug":"zone-gold-et-pipeline-dintegration-synapse","status":"publish","type":"post","link":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/","title":{"rendered":"Zone Gold et Pipeline d&rsquo;int\u00e9gration synapse"},"content":{"rendered":"\n<p>Apr\u00e8s avoir aliment\u00e9 notre zone bronze : <a href=\"https:\/\/www.nicedata.fr\/index.php\/2022\/08\/19\/parametrer-les-pipeline-azure-synapse-analytics-sans-ouvrir-synapse\/\">Param\u00e9trer les Pipeline Azure Synapse Analytics sans ouvrir Synapse !<\/a> ;<\/p>\n\n\n\n<p>Apr\u00e8s avoir aliment\u00e9 notre zone sliver : <a href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/08\/17\/notre-premier-notebook-spark-dans-synapse\/\">Notre premier notebook Spark dans Synapse<\/a> ;<\/p>\n\n\n\n<p>Nous allons maintenant alimenter notre zone gold et cr\u00e9er notre pipeline d&rsquo;int\u00e9gration complet de l&rsquo;ingestion de nos donn\u00e9es On-Prem jusqu&rsquo;\u00e0 l&rsquo;\u00e9criture de notre zone de donn\u00e9e \u00ab\u00a0Gold\u00a0\u00bb.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_81 ez-toc-wrap-center counter-hierarchy ez-toc-counter ez-toc-transparent ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Sommaire<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#Alimentation_de_la_zone_Gold\" >Alimentation de la zone Gold<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#Creation_du_pipeline\" >Cr\u00e9ation du pipeline<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#Creation_des_%C2%AB_sous-pipelines_%C2%BB\" >Cr\u00e9ation des \u00ab\u00a0sous-pipelines\u00a0\u00bb<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#Alimentation_Bronze\" >Alimentation Bronze<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#Alimentation_Sliver\" >Alimentation Sliver<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#Alimentation_Gold\" >Alimentation Gold<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#Pipeline_general\" >Pipeline g\u00e9n\u00e9ral<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\" id=\"AlimGold\"><span class=\"ez-toc-section\" id=\"Alimentation_de_la_zone_Gold\"><\/span>Alimentation de la zone Gold<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>A la mani\u00e8re de ce que nous avons fait pour notre premier notebook (<a href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/08\/17\/notre-premier-notebook-spark-dans-synapse\/\">Notre premier notebook Spark dans Synapse<\/a>), nous allons cr\u00e9er un nouveau notebook tr\u00e8s simple qui va s\u00e9lectionner et renommer proprement nos donn\u00e9es de la zone silver pour les \u00e9crire dans la zone Gold.<\/p>\n\n\n\n<p>Dans un nouveau Notebook, n&rsquo;ayant pas d&rsquo;import \u00e0 faire, nous commen\u00e7ons par d\u00e9clarer nos diff\u00e9rents chemins :<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\nsilverUrl = &#039;abfss:\/\/&lt;name of your container&gt;@&lt;name of your storage account&gt;.dfs.core.windows.net&#039;\ngoldUrl = &#039;abfss:\/\/&lt;name of your container&gt;@&lt;name of your storage account&gt;.dfs.core.windows.net&#039;\n\nsilverInvoicesFile = &#039;\/Invoices&#039;\ngoldSaleFile = &#039;\/Sale&#039;\n<\/pre><\/div>\n\n\n<p>Nous allons cette fois-ci lire la zone Silver<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\ndfSilverInvoices = spark.read.load(f&#039;{silverUrl}{silverInvoicesFile}&#039;, format=&#039;parquet&#039;)\n<\/pre><\/div>\n\n\n<p>Nous arrivons au moment o\u00f9 nous allons utiliser la fonction selectExpr (<a href=\"https:\/\/spark.apache.org\/docs\/latest\/api\/python\/reference\/pyspark.sql\/api\/pyspark.sql.DataFrame.selectExpr.html\">pyspark.sql.DataFrame.selectExpr \u2014 PySpark 3.5.3 documentation<\/a>), pour s\u00e9lectionner et renommer nos colonnes de fichiers parquet comme nous le ferions dans un SELECT SQL.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\ndfGoldSale = dfSilverInvoices.selectExpr(\n    &#039;InvoiceID&#039;,\n    &#039;line_InvoiceLineID as InvoiceLineID&#039;,\n    &#039;line_StockItemID as StockItemID&#039;,\n    &#039;CustomerID&#039;,\n    &#039;BillToCustomerID&#039;,\n    &#039;OrderID&#039;,\n    &#039;InvoiceDate&#039;,\n    &#039;line_Description as Description&#039;,\n    &#039;line_Quantity as Quantity&#039;,\n    &#039;line_UnitPrice as UnitPrice&#039;,\n    &#039;line_TaxRate as TaxRate&#039;,\n    &#039;line_TaxAmount as TaxAmount&#039;,\n    &#039;line_LineProfit as Profit&#039;\n)\n<\/pre><\/div>\n\n\n<p>selectExpr prend en argument une expression SQL. C&rsquo;est ainsi que nous pouvons utiliser la syntaxe SQL pour renommer nos colonnes en utilisant le mot clef \u00ab\u00a0as\u00a0\u00bb.<\/p>\n\n\n\n<p>Nous pourrions faire ici certaines transformations business mais pour notre objectif nous nous arr\u00eaterons ici et allons \u00e9crire notre dataset dans notre zone gold.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\ndfGoldSale.write.mode(&quot;overwrite&quot;).parquet(goldUrl+goldSaleFile)\n<\/pre><\/div>\n\n\n<p>Voici l&rsquo;aper\u00e7u du notebook final :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"892\" height=\"861\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-1.png?resize=892%2C861&#038;ssl=1\" alt=\"\" class=\"wp-image-724\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-1.png?w=892&amp;ssl=1 892w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-1.png?resize=300%2C290&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-1.png?resize=768%2C741&amp;ssl=1 768w\" sizes=\"auto, (max-width: 892px) 100vw, 892px\" \/><figcaption class=\"wp-element-caption\">Notebook de transformation complet<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Creation_du_pipeline\"><\/span>Cr\u00e9ation du pipeline<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Une fois tous nos traitements d\u00e9velopp\u00e9s, nous avons maintenant pour mission de les ex\u00e9cuter \u00e0 la chaine. Pour ce faire, nous allons cr\u00e9er un pipeline d&rsquo;int\u00e9gration qui ex\u00e9cutera nos diff\u00e9rents traitements dans l&rsquo;ordre et suivant les conditions d\u00e9finies.<\/p>\n\n\n\n<p>Nous avons rendez-vous dans la section \u00ab\u00a0int\u00e9gration\u00a0\u00bb de Synapse ou nous allons designer notre processus d&rsquo;int\u00e9gration.<\/p>\n\n\n\n<p>Afin de rester le plus futur proof possible, nous n&rsquo;allons pas directement ex\u00e9cuter nos traitements les uns \u00e0 la suite des autres mais plut\u00f4t des sous-ensemble de traitements. Cela implique que pour notre pipeline d&rsquo;int\u00e9gration, nous n&rsquo;allons pas ex\u00e9cuter des notebook les uns apr\u00e8s les autres mais plut\u00f4t d&rsquo;autres pipelines. Nous allons faire une sorte de pipelineception, qui nous permettra de modifier et tester uniquement un sous-ensemble lors de nos travaux futurs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Creation_des_%C2%AB_sous-pipelines_%C2%BB\"><\/span>Cr\u00e9ation des \u00ab\u00a0sous-pipelines\u00a0\u00bb<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Dans notre projet toujours tr\u00e8s simpliste, nous allons donc cr\u00e9er un pipeline qui enchainera trois sous-pipelines qui alimenteront chacun une des zones de notre datalake.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"917\" height=\"275\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-2.png?resize=917%2C275&#038;ssl=1\" alt=\"\" class=\"wp-image-725\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-2.png?w=917&amp;ssl=1 917w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-2.png?resize=300%2C90&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-2.png?resize=768%2C230&amp;ssl=1 768w\" sizes=\"auto, (max-width: 917px) 100vw, 917px\" \/><figcaption class=\"wp-element-caption\">Pipeline complet<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Alimentation_Bronze\"><\/span>Alimentation Bronze<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Pour cette partie, nous allons encapsuler les travaux d&rsquo;un pr\u00e9c\u00e9dent article qui traite justement de notre int\u00e9gration de donn\u00e9e en zone bronze : <a href=\"https:\/\/www.nicedata.fr\/index.php\/2022\/08\/19\/parametrer-les-pipeline-azure-synapse-analytics-sans-ouvrir-synapse\/\">Param\u00e9trer les Pipeline Azure Synapse Analytics sans ouvrir Synapse !<\/a><\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"571\" height=\"340\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-3.png?resize=571%2C340&#038;ssl=1\" alt=\"\" class=\"wp-image-726\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-3.png?w=571&amp;ssl=1 571w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-3.png?resize=300%2C179&amp;ssl=1 300w\" sizes=\"auto, (max-width: 571px) 100vw, 571px\" \/><figcaption class=\"wp-element-caption\">Cr\u00e9ation d&rsquo;un nouveau pipeline<\/figcaption><\/figure>\n\n\n\n<p>Apr\u00e8s avoir donn\u00e9 un nom et une description \u00e0 notre pipeline :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"386\" height=\"379\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-4.png?resize=386%2C379&#038;ssl=1\" alt=\"\" class=\"wp-image-727\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-4.png?w=386&amp;ssl=1 386w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-4.png?resize=300%2C295&amp;ssl=1 300w\" sizes=\"auto, (max-width: 386px) 100vw, 386px\" \/><figcaption class=\"wp-element-caption\">D\u00e9finition du nom et de la description d&rsquo;un pipeline<\/figcaption><\/figure>\n\n\n\n<p>Nous allons ajouter les traitements \u00e0 ex\u00e9cuter. Dans le cas pr\u00e9sent, nous avons d\u00e9j\u00e0 un pipeline qui alimente notre zone bronze. Nous allons donc ajouter celui-ci dans la zone d&rsquo;\u00e9dition de notre pipeline. Nous avons deux possibilit\u00e9s pour ce faire.<\/p>\n\n\n\n<p>La 1\u00e8re consiste \u00e0 cliquer \/ d\u00e9placer notre pipeline existant dans la zone de notre pipeline :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"275\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-5.png?resize=1024%2C275&#038;ssl=1\" alt=\"\" class=\"wp-image-728\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-5.png?resize=1024%2C275&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-5.png?resize=300%2C80&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-5.png?resize=768%2C206&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-5.png?w=1040&amp;ssl=1 1040w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><figcaption class=\"wp-element-caption\">Cliquer \/ d\u00e9poser d&rsquo;un pipeline<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"617\" height=\"312\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-6.png?resize=617%2C312&#038;ssl=1\" alt=\"\" class=\"wp-image-729\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-6.png?w=617&amp;ssl=1 617w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-6.png?resize=300%2C152&amp;ssl=1 300w\" sizes=\"auto, (max-width: 617px) 100vw, 617px\" \/><figcaption class=\"wp-element-caption\">Pipeline ajout\u00e9 \u00e0 notre pipeline<\/figcaption><\/figure>\n\n\n\n<p>La 2\u00e8me consiste \u00e0 cliquer \/ d\u00e9placer l&rsquo;activit\u00e9 \u00ab\u00a0Execute pipeline\u00a0\u00bb depuis le panneau d&rsquo;activit\u00e9 :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"641\" height=\"591\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-7.png?resize=641%2C591&#038;ssl=1\" alt=\"\" class=\"wp-image-730\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-7.png?w=641&amp;ssl=1 641w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-7.png?resize=300%2C277&amp;ssl=1 300w\" sizes=\"auto, (max-width: 641px) 100vw, 641px\" \/><figcaption class=\"wp-element-caption\">Cliquer \/ d\u00e9placer d&rsquo;une activit\u00e9 \u00ab\u00a0Execute pipeline\u00a0\u00bb<\/figcaption><\/figure>\n\n\n\n<p>Puis d&rsquo;aller param\u00e9trer cette activit\u00e9 pour invoquer notre pipeline existant :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"629\" height=\"429\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-8.png?resize=629%2C429&#038;ssl=1\" alt=\"\" class=\"wp-image-731\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-8.png?w=629&amp;ssl=1 629w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-8.png?resize=300%2C205&amp;ssl=1 300w\" sizes=\"auto, (max-width: 629px) 100vw, 629px\" \/><figcaption class=\"wp-element-caption\">D\u00e9finition du pipeline \u00e0 invoquer d&rsquo;une activit\u00e9 \u00ab\u00a0Execute pipeline\u00a0\u00bb<\/figcaption><\/figure>\n\n\n\n<p>La seule diff\u00e9rence dans cette deuxi\u00e8me approche est que notre activit\u00e9 \u00e0 un nom g\u00e9n\u00e9rique, que l&rsquo;on peu malgr\u00e9 tout modifier manuellement (comme pour toutes les activit\u00e9s) dans l&rsquo;onglet g\u00e9n\u00e9ral de notre activit\u00e9 :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"612\" height=\"481\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-9.png?resize=612%2C481&#038;ssl=1\" alt=\"\" class=\"wp-image-732\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-9.png?w=612&amp;ssl=1 612w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-9.png?resize=300%2C236&amp;ssl=1 300w\" sizes=\"auto, (max-width: 612px) 100vw, 612px\" \/><figcaption class=\"wp-element-caption\">Nommage d&rsquo;une activit\u00e9 de pipeline<\/figcaption><\/figure>\n\n\n\n<p>Notre pipeline pour la zone bronze ne contient pour l&rsquo;instant rien de plus, nous pouvons donc passer \u00e0 la suite.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Alimentation_Sliver\"><\/span>Alimentation Sliver<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Pour nos chargements de zone silver, nous allons ex\u00e9cuter les notebook Spark tels que cr\u00e9\u00e9s dans un article pr\u00e9c\u00e9dant : <a href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/08\/17\/notre-premier-notebook-spark-dans-synapse\/\">Notre premier notebook Spark dans Synapse<\/a><\/p>\n\n\n\n<p>De la m\u00eame mani\u00e8re que nous avons fait pour ajouter l&rsquo;activit\u00e9 d&rsquo;ex\u00e9cution de pipeline, nous allons ajouter nos diff\u00e9rents notebook :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"452\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-10.png?resize=1024%2C452&#038;ssl=1\" alt=\"\" class=\"wp-image-733\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-10.png?resize=1024%2C452&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-10.png?resize=300%2C132&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-10.png?resize=768%2C339&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-10.png?w=1049&amp;ssl=1 1049w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><figcaption class=\"wp-element-caption\">Ajout des notebook \u00e0 un pipeline<\/figcaption><\/figure>\n\n\n\n<p>Sans autre action de notre part, les notebook s&rsquo;ex\u00e9cuteront en parall\u00e8le. C&rsquo;est-\u00e0-dire qu&rsquo;ils d\u00e9marreront en m\u00eame temps&#8230; \u00e0 condition d&rsquo;avoir assez de ressource sur notre cluster Spark !<\/p>\n\n\n\n<p>En effet, chaque notebook va correspondre \u00e0 une application Spark, qui va s&rsquo;ex\u00e9cuter sur notre pool et va avoir besoin d&rsquo;un driver et d&rsquo;un ex\u00e9cuteur au minimum. Par d\u00e9faut un notebook est configur\u00e9 pour utiliser 2 ex\u00e9cuteurs, ce qui demande au total 3 n\u0153uds de cluster.<\/p>\n\n\n\n<p>En ayant un cluster Spark avec seulement 3 n\u0153uds au maximum, nous allons remarquer que nos notebook ne seront pas vraiment ex\u00e9cut\u00e9s en parall\u00e8le.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"922\" height=\"79\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-17.png?resize=922%2C79&#038;ssl=1\" alt=\"\" class=\"wp-image-740\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-17.png?w=922&amp;ssl=1 922w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-17.png?resize=300%2C26&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-17.png?resize=768%2C66&amp;ssl=1 768w\" sizes=\"auto, (max-width: 922px) 100vw, 922px\" \/><figcaption class=\"wp-element-caption\">Pool Spark avec un maximum 3 n\u0153uds<\/figcaption><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Notebooks presque en parall\u00e8les<\/h4>\n\n\n\n<p>Afin de lancer l&rsquo;ex\u00e9cution de notre pipeline et de le tester en condition r\u00e9elle, nous allons devoir le publier dans le mode \u00ab\u00a0Live\u00a0\u00bb de synapse. Un trigger ex\u00e9cute toujours le code publi\u00e9. Autrement, il faut \u00ab\u00a0d\u00e9buguer\u00a0\u00bb le pipeline.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"818\" height=\"395\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-19.png?resize=818%2C395&#038;ssl=1\" alt=\"\" class=\"wp-image-742\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-19.png?w=818&amp;ssl=1 818w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-19.png?resize=300%2C145&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-19.png?resize=768%2C371&amp;ssl=1 768w\" sizes=\"auto, (max-width: 818px) 100vw, 818px\" \/><\/figure>\n\n\n\n<p>Puis il suffit de lancer un \u00ab\u00a0trigger maintenant\u00a0\u00bb<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"841\" height=\"483\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-18.png?resize=841%2C483&#038;ssl=1\" alt=\"\" class=\"wp-image-741\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-18.png?w=841&amp;ssl=1 841w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-18.png?resize=300%2C172&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-18.png?resize=768%2C441&amp;ssl=1 768w\" sizes=\"auto, (max-width: 841px) 100vw, 841px\" \/><figcaption class=\"wp-element-caption\">D\u00e9marrage d&rsquo;un pipeline en condition normale<\/figcaption><\/figure>\n\n\n\n<p>A l&rsquo;ex\u00e9cution de notre pipeline, nous pouvons remarquer que nos notebook semblent s&rsquo;ex\u00e9cuter en m\u00eame temps. Les deux sont marqu\u00e9s \u00ab\u00a0In Progress\u00a0\u00bb avec les m\u00eames dates\/heures de d\u00e9marrage.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"544\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-14.png?resize=1024%2C544&#038;ssl=1\" alt=\"\" class=\"wp-image-737\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-14.png?resize=1024%2C544&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-14.png?resize=300%2C159&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-14.png?resize=768%2C408&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-14.png?w=1102&amp;ssl=1 1102w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><figcaption class=\"wp-element-caption\">Activit\u00e9s de pipeline d\u00e9marr\u00e9s en m\u00eame temps<\/figcaption><\/figure>\n\n\n\n<p>Cependant, la v\u00e9rit\u00e9 est que nos activit\u00e9s n&rsquo;ont fait que demander \u00e0 notre cluster l&rsquo;ex\u00e9cution de nos notebook (donc des applications Spark). Si l&rsquo;on regarde cette fois dans le monitoring des applications Spark, nous allons voir que l&rsquo;une de nos applications est \u00ab\u00a0Queued\u00a0\u00bb. Elle est en fait mise dans la file d&rsquo;ex\u00e9cution et ne sera trait\u00e9e que lorsque notre cluster aura la capacit\u00e9 de la traiter.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"193\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-13.png?resize=1024%2C193&#038;ssl=1\" alt=\"\" class=\"wp-image-736\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-13.png?resize=1024%2C193&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-13.png?resize=300%2C56&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-13.png?resize=768%2C145&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-13.png?resize=1536%2C289&amp;ssl=1 1536w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-13.png?w=1763&amp;ssl=1 1763w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/figure>\n\n\n\n<p>Une fois la premi\u00e8re application finie, la deuxi\u00e8me prendra automatiquement le relais.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"298\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-15.png?resize=1024%2C298&#038;ssl=1\" alt=\"\" class=\"wp-image-738\" style=\"width:1024px;height:auto\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-15.png?resize=1024%2C298&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-15.png?resize=300%2C87&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-15.png?resize=768%2C223&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-15.png?resize=1536%2C446&amp;ssl=1 1536w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-15.png?w=1714&amp;ssl=1 1714w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><figcaption class=\"wp-element-caption\">La deuxi\u00e8me application d\u00e9marre apr\u00e8s la fin de la pr\u00e9c\u00e9dente.<\/figcaption><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">V\u00e9ritable ex\u00e9cution en parall\u00e8le<\/h4>\n\n\n\n<p>Afin de voir une r\u00e9elle ex\u00e9cution en parall\u00e8le de nos notebook, nous allons dans un premier temps augmenter le nombre de n\u0153uds maximum de notre pool Spark \u00e0 6 afin de combler le besoin de deux notebook en parall\u00e8le.<\/p>\n\n\n\n<p>Pour autoriser plus de n\u0153uds \u00e0 notre cluster, nous allons ouvrir le panneau de configuration d\u00e9di\u00e9 \u00e0 partir de la partie management de notre cluster Spark.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"392\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-21.png?resize=1024%2C392&#038;ssl=1\" alt=\"\" class=\"wp-image-744\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-21.png?resize=1024%2C392&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-21.png?resize=300%2C115&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-21.png?resize=768%2C294&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-21.png?w=1163&amp;ssl=1 1163w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"929\" height=\"84\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-16.png?resize=929%2C84&#038;ssl=1\" alt=\"\" class=\"wp-image-739\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-16.png?w=929&amp;ssl=1 929w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-16.png?resize=300%2C27&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-16.png?resize=768%2C69&amp;ssl=1 768w\" sizes=\"auto, (max-width: 929px) 100vw, 929px\" \/><figcaption class=\"wp-element-caption\">Spark pool \u00e0 maximum 6 n\u0153uds<\/figcaption><\/figure>\n\n\n\n<p>Il nous suffit d&rsquo;autoriser jusqu&rsquo;\u00e0 6 n\u0153uds \u00e0 notre pool dans ce menu :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"631\" height=\"610\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-22.png?resize=631%2C610&#038;ssl=1\" alt=\"\" class=\"wp-image-745\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-22.png?w=631&amp;ssl=1 631w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-22.png?resize=300%2C290&amp;ssl=1 300w\" sizes=\"auto, (max-width: 631px) 100vw, 631px\" \/><figcaption class=\"wp-element-caption\">Configuration du nombre de n\u0153uds maximum de notre pool<\/figcaption><\/figure>\n\n\n\n<p>Apr\u00e8s un petit temps de propagation, nous avons maintenant notre pool Spark autorisant jusqu&rsquo;\u00e0 6 n\u0153uds :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"929\" height=\"84\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-23.png?resize=929%2C84&#038;ssl=1\" alt=\"\" class=\"wp-image-746\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-23.png?w=929&amp;ssl=1 929w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-23.png?resize=300%2C27&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-23.png?resize=768%2C69&amp;ssl=1 768w\" sizes=\"auto, (max-width: 929px) 100vw, 929px\" \/><figcaption class=\"wp-element-caption\">Pool Spark avec un maximum 3 n\u0153uds<\/figcaption><\/figure>\n\n\n\n<p>Maintenant nous allons relancer notre pipeline et dans la vue de monitoring de nos applications Spark nous allons voir que nos deux Notebook sont maintenant lanc\u00e9es en parall\u00e8le.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"202\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-20.png?resize=1024%2C202&#038;ssl=1\" alt=\"\" class=\"wp-image-743\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-20.png?resize=1024%2C202&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-20.png?resize=300%2C59&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-20.png?resize=768%2C152&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-20.png?resize=1536%2C303&amp;ssl=1 1536w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-20.png?w=1706&amp;ssl=1 1706w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><figcaption class=\"wp-element-caption\">Ex\u00e9cution en parall\u00e8le de deux notebook Spark<\/figcaption><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Ex\u00e9cution en s\u00e9quentielle<\/h4>\n\n\n\n<p>Imaginons maintenant que nous souhaitions forcer une ex\u00e9cution s\u00e9quentielle de nos pipelines. Cela peut se faire pour nous assurer d&rsquo;un ordre d&rsquo;ex\u00e9cution. Nous allons dans ce cas nous servir des \u00ab\u00a0sorties\u00a0\u00bb d&rsquo;activit\u00e9s. Si l&rsquo;on passe notre souris sur une activit\u00e9, on remarque que trois nouvelles ic\u00f4nes apparaissent \u00e0 c\u00f4t\u00e9 de la coche verte qui est toujours pr\u00e9sente.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"328\" height=\"226\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-24.png?resize=328%2C226&#038;ssl=1\" alt=\"\" class=\"wp-image-747\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-24.png?w=328&amp;ssl=1 328w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-24.png?resize=300%2C207&amp;ssl=1 300w\" sizes=\"auto, (max-width: 328px) 100vw, 328px\" \/><figcaption class=\"wp-element-caption\">Visualisation des sorties d&rsquo;activit\u00e9s<\/figcaption><\/figure>\n\n\n\n<p>Ces sorties vont nous permettre de lier les activit\u00e9s les unes avec les autres. Par exemple, si l&rsquo;on fait un lien entre la sortie \u00ab\u00a0On success\u00a0\u00bb d&rsquo;un notebook pour aller \u00e0 un autre notebook, cela se traduira par : \u00ab\u00a0Si ce notebook fini avec un \u00e9tat OK, alors, ex\u00e9cute le notebook suivant\u00a0\u00bb.<\/p>\n\n\n\n<p>Ce lien se cr\u00e9er en faisant un cliquer \/ d\u00e9poser de la sortie du premier notebook, jusqu&rsquo;au notebook suivant :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"284\" height=\"244\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-25.png?resize=284%2C244&#038;ssl=1\" alt=\"\" class=\"wp-image-748\"\/><figcaption class=\"wp-element-caption\">Cr\u00e9ation d&rsquo;un lien s\u00e9quentiel entre deux notebooks<\/figcaption><\/figure>\n\n\n\n<p>Une fois cr\u00e9\u00e9, ce lien s\u00e9quentiel se mat\u00e9rialisera par une fl\u00e8che (verte pour une sortie \u00ab\u00a0On success\u00a0\u00bb) entre les deux activit\u00e9s :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"365\" height=\"318\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-26.png?resize=365%2C318&#038;ssl=1\" alt=\"\" class=\"wp-image-749\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-26.png?w=365&amp;ssl=1 365w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-26.png?resize=300%2C261&amp;ssl=1 300w\" sizes=\"auto, (max-width: 365px) 100vw, 365px\" \/><figcaption class=\"wp-element-caption\">Notebooks li\u00e9s par un lien s\u00e9quentiel de succ\u00e8s<\/figcaption><\/figure>\n\n\n\n<p>Nous venons donc de cr\u00e9er un lien fort n\u00e9cessitant le succ\u00e8s du premier notebook pour ex\u00e9cuter le second. Autrement dit, le deuxi\u00e8me notebook ne se lancera que si le premier s&rsquo;est compl\u00e8tement ex\u00e9cut\u00e9 ET n&rsquo;est pas en erreur.<\/p>\n\n\n\n<p>Lors du monitoring de l&rsquo;ex\u00e9cution, nous voyons cette fois que seule notre premi\u00e8re activit\u00e9 s&rsquo;est lanc\u00e9e. La deuxi\u00e8me n&rsquo;est m\u00eame pas en attente :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"850\" height=\"562\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-27.png?resize=850%2C562&#038;ssl=1\" alt=\"\" class=\"wp-image-750\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-27.png?w=850&amp;ssl=1 850w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-27.png?resize=300%2C198&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-27.png?resize=768%2C508&amp;ssl=1 768w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><figcaption class=\"wp-element-caption\">Ex\u00e9cution du premier notebook<\/figcaption><\/figure>\n\n\n\n<p>Ensuite, lors de la fin de notre premier notebook (sans erreurs), le deuxi\u00e8me se lance automatiquement :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"833\" height=\"549\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-28.png?resize=833%2C549&#038;ssl=1\" alt=\"\" class=\"wp-image-751\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-28.png?w=833&amp;ssl=1 833w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-28.png?resize=300%2C198&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-28.png?resize=768%2C506&amp;ssl=1 768w\" sizes=\"auto, (max-width: 833px) 100vw, 833px\" \/><figcaption class=\"wp-element-caption\">Ex\u00e9cution du deuxi\u00e8me notebook<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Alimentation_Gold\"><\/span>Alimentation Gold<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Pour notre zone gold, nous allons simplement ajouter le notebook cr\u00e9\u00e9 au d\u00e9but de cet article (<a href=\"#AlimGold\">ici<\/a>).<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"758\" height=\"278\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-29.png?resize=758%2C278&#038;ssl=1\" alt=\"\" class=\"wp-image-752\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-29.png?w=758&amp;ssl=1 758w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-29.png?resize=300%2C110&amp;ssl=1 300w\" sizes=\"auto, (max-width: 758px) 100vw, 758px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Pipeline_general\"><\/span>Pipeline g\u00e9n\u00e9ral<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Maintenant que nos sous-pipelines sont cr\u00e9\u00e9s, nous pouvons cr\u00e9er notre pipeline maitre, qui appellera tous les autres.<\/p>\n\n\n\n<p>En suivant exactement le m\u00eame sch\u00e9ma que pr\u00e9c\u00e9demment, nous allons ajouter toutes nos activit\u00e9s :<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"827\" height=\"367\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-30.png?resize=827%2C367&#038;ssl=1\" alt=\"\" class=\"wp-image-753\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-30.png?w=827&amp;ssl=1 827w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-30.png?resize=300%2C133&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-30.png?resize=768%2C341&amp;ssl=1 768w\" sizes=\"auto, (max-width: 827px) 100vw, 827px\" \/><figcaption class=\"wp-element-caption\">Pipeline d&rsquo;alimentation du Datalake complet<\/figcaption><\/figure>\n\n\n\n<p>Dans le monitoring de nos pipelines, nous pouvons maintenant suivre les ex\u00e9cutions de notre pipeline et ses sous-pipelines.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"314\" src=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-31.png?resize=1024%2C314&#038;ssl=1\" alt=\"\" class=\"wp-image-754\" srcset=\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-31.png?resize=1024%2C314&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-31.png?resize=300%2C92&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-31.png?resize=768%2C235&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/image-31.png?w=1179&amp;ssl=1 1179w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><figcaption class=\"wp-element-caption\">Ex\u00e9cution de l&rsquo;int\u00e9gralit\u00e9 de notre pipeline et ses sous pipelines<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Nous avons cr\u00e9\u00e9 un pipeline simple d&rsquo;alimentation complet de notre datalake en utilisant des sous-pipelines. Nous pouvons maintenant compl\u00e9ter l&rsquo;alimentation de nos diff\u00e9rentes zones et ajouter petit \u00e0 petit chaque notebook dans le pipeline d\u00e9di\u00e9.<\/p>\n\n\n\n<p>Ici, nous n&rsquo;avons utilis\u00e9 uniquement des sorties \u00ab\u00a0On success\u00a0\u00bb et des activit\u00e9s d&rsquo;ex\u00e9cution de pipeline et de notebook, mais il est \u00e9vident que pour un datalake de production nous devrons g\u00e9rer les cas d&rsquo;erreurs avec les sorties \u00ab\u00a0On Fail\u00a0\u00bb par exemple. Dans tous les cas nous pouvons continuer notre construction pour avoir une alimentation compl\u00e8te gr\u00e2ce \u00e0 l&rsquo;ex\u00e9cution d&rsquo;un seul pipeline !<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Apr\u00e8s avoir aliment\u00e9 notre zone bronze : Param\u00e9trer les Pipeline Azure Synapse Analytics sans ouvrir Synapse ! ; Apr\u00e8s avoir aliment\u00e9 notre zone sliver : Notre premier notebook Spark dans Synapse ; Nous allons maintenant alimenter notre zone gold et cr\u00e9er notre pipeline d&rsquo;int\u00e9gration complet de l&rsquo;ingestion de nos donn\u00e9es On-Prem jusqu&rsquo;\u00e0 l&rsquo;\u00e9criture de notre [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":721,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[76],"tags":[9,32,82],"class_list":["post-717","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-synapsedatalake","tag-azure-synapse-analytics","tag-datalake","tag-pipeline"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Zone Gold et Pipeline d&#039;int\u00e9gration synapse - NiceData<\/title>\n<meta name=\"description\" content=\"Nous allons maintenant alimenter notre zone gold et cr\u00e9er notre pipeline d&#039;int\u00e9gration complet de l&#039;ingestion de nos donn\u00e9es On-Prem jusqu&#039;\u00e0 l&#039;\u00e9criture de notre zone de donn\u00e9e &quot;Gold&quot;.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/\" \/>\n<meta property=\"og:locale\" content=\"fr_FR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Zone Gold et Pipeline d&#039;int\u00e9gration synapse - NiceData\" \/>\n<meta property=\"og:description\" content=\"Nous allons maintenant alimenter notre zone gold et cr\u00e9er notre pipeline d&#039;int\u00e9gration complet de l&#039;ingestion de nos donn\u00e9es On-Prem jusqu&#039;\u00e0 l&#039;\u00e9criture de notre zone de donn\u00e9e &quot;Gold&quot;.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/\" \/>\n<meta property=\"og:site_name\" content=\"NiceData\" \/>\n<meta property=\"article:published_time\" content=\"2024-09-28T16:17:04+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-10-07T13:07:21+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1700\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Jean-Laurent Ferralis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@JLFerralis\" \/>\n<meta name=\"twitter:site\" content=\"@JLFerralis\" \/>\n<meta name=\"twitter:label1\" content=\"\u00c9crit par\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jean-Laurent Ferralis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Dur\u00e9e de lecture estim\u00e9e\" \/>\n\t<meta name=\"twitter:data2\" content=\"14 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/\"},\"author\":{\"name\":\"Jean-Laurent Ferralis\",\"@id\":\"https:\/\/www.nicedata.fr\/#\/schema\/person\/8d1ad38004d3b0cf6bff7c200c795e19\"},\"headline\":\"Zone Gold et Pipeline d&rsquo;int\u00e9gration synapse\",\"datePublished\":\"2024-09-28T16:17:04+00:00\",\"dateModified\":\"2024-10-07T13:07:21+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/\"},\"wordCount\":1766,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.nicedata.fr\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1\",\"keywords\":[\"Azure Synapse Analytics\",\"datalake\",\"pipeline\"],\"articleSection\":[\"Azure Synapse - Datalake\"],\"inLanguage\":\"fr-FR\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/\",\"url\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/\",\"name\":\"Zone Gold et Pipeline d'int\u00e9gration synapse - NiceData\",\"isPartOf\":{\"@id\":\"https:\/\/www.nicedata.fr\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1\",\"datePublished\":\"2024-09-28T16:17:04+00:00\",\"dateModified\":\"2024-10-07T13:07:21+00:00\",\"description\":\"Nous allons maintenant alimenter notre zone gold et cr\u00e9er notre pipeline d'int\u00e9gration complet de l'ingestion de nos donn\u00e9es On-Prem jusqu'\u00e0 l'\u00e9criture de notre zone de donn\u00e9e \\\"Gold\\\".\",\"breadcrumb\":{\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#breadcrumb\"},\"inLanguage\":\"fr-FR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#primaryimage\",\"url\":\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1\",\"width\":2560,\"height\":1700,\"caption\":\"people running\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.nicedata.fr\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Zone Gold et Pipeline d&rsquo;int\u00e9gration synapse\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.nicedata.fr\/#website\",\"url\":\"https:\/\/www.nicedata.fr\/\",\"name\":\"NiceData\",\"description\":\"L&#039;expertise Data du sud\",\"publisher\":{\"@id\":\"https:\/\/www.nicedata.fr\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.nicedata.fr\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"fr-FR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.nicedata.fr\/#organization\",\"name\":\"NiceData\",\"url\":\"https:\/\/www.nicedata.fr\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\/\/www.nicedata.fr\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/11\/NICE-DATA_JLFMod.webp?fit=2493%2C1249&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/11\/NICE-DATA_JLFMod.webp?fit=2493%2C1249&ssl=1\",\"width\":2493,\"height\":1249,\"caption\":\"NiceData\"},\"image\":{\"@id\":\"https:\/\/www.nicedata.fr\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/JLFerralis\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.nicedata.fr\/#\/schema\/person\/8d1ad38004d3b0cf6bff7c200c795e19\",\"name\":\"Jean-Laurent Ferralis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/3cbfdae273d44fb82b902a3451eb0db37485119e34cb70c1fdd186c9b731b9f3?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/3cbfdae273d44fb82b902a3451eb0db37485119e34cb70c1fdd186c9b731b9f3?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/3cbfdae273d44fb82b902a3451eb0db37485119e34cb70c1fdd186c9b731b9f3?s=96&d=mm&r=g\",\"caption\":\"Jean-Laurent Ferralis\"},\"description\":\"French Data Professionnal - BI consultant and #sql lover. I also #swimbikerun when possible ! Living in @villedenice\",\"sameAs\":[\"http:\/\/xp-it.com\"],\"url\":\"https:\/\/www.nicedata.fr\/index.php\/author\/jlf\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Zone Gold et Pipeline d'int\u00e9gration synapse - NiceData","description":"Nous allons maintenant alimenter notre zone gold et cr\u00e9er notre pipeline d'int\u00e9gration complet de l'ingestion de nos donn\u00e9es On-Prem jusqu'\u00e0 l'\u00e9criture de notre zone de donn\u00e9e \"Gold\".","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/","og_locale":"fr_FR","og_type":"article","og_title":"Zone Gold et Pipeline d'int\u00e9gration synapse - NiceData","og_description":"Nous allons maintenant alimenter notre zone gold et cr\u00e9er notre pipeline d'int\u00e9gration complet de l'ingestion de nos donn\u00e9es On-Prem jusqu'\u00e0 l'\u00e9criture de notre zone de donn\u00e9e \"Gold\".","og_url":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/","og_site_name":"NiceData","article_published_time":"2024-09-28T16:17:04+00:00","article_modified_time":"2024-10-07T13:07:21+00:00","og_image":[{"width":2560,"height":1700,"url":"https:\/\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg","type":"image\/jpeg"}],"author":"Jean-Laurent Ferralis","twitter_card":"summary_large_image","twitter_creator":"@JLFerralis","twitter_site":"@JLFerralis","twitter_misc":{"\u00c9crit par":"Jean-Laurent Ferralis","Dur\u00e9e de lecture estim\u00e9e":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#article","isPartOf":{"@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/"},"author":{"name":"Jean-Laurent Ferralis","@id":"https:\/\/www.nicedata.fr\/#\/schema\/person\/8d1ad38004d3b0cf6bff7c200c795e19"},"headline":"Zone Gold et Pipeline d&rsquo;int\u00e9gration synapse","datePublished":"2024-09-28T16:17:04+00:00","dateModified":"2024-10-07T13:07:21+00:00","mainEntityOfPage":{"@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/"},"wordCount":1766,"commentCount":0,"publisher":{"@id":"https:\/\/www.nicedata.fr\/#organization"},"image":{"@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1","keywords":["Azure Synapse Analytics","datalake","pipeline"],"articleSection":["Azure Synapse - Datalake"],"inLanguage":"fr-FR","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/","url":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/","name":"Zone Gold et Pipeline d'int\u00e9gration synapse - NiceData","isPartOf":{"@id":"https:\/\/www.nicedata.fr\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#primaryimage"},"image":{"@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1","datePublished":"2024-09-28T16:17:04+00:00","dateModified":"2024-10-07T13:07:21+00:00","description":"Nous allons maintenant alimenter notre zone gold et cr\u00e9er notre pipeline d'int\u00e9gration complet de l'ingestion de nos donn\u00e9es On-Prem jusqu'\u00e0 l'\u00e9criture de notre zone de donn\u00e9e \"Gold\".","breadcrumb":{"@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#breadcrumb"},"inLanguage":"fr-FR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/"]}]},{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#primaryimage","url":"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1","width":2560,"height":1700,"caption":"people running"},{"@type":"BreadcrumbList","@id":"https:\/\/www.nicedata.fr\/index.php\/2024\/09\/28\/zone-gold-et-pipeline-dintegration-synapse\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.nicedata.fr\/"},{"@type":"ListItem","position":2,"name":"Zone Gold et Pipeline d&rsquo;int\u00e9gration synapse"}]},{"@type":"WebSite","@id":"https:\/\/www.nicedata.fr\/#website","url":"https:\/\/www.nicedata.fr\/","name":"NiceData","description":"L&#039;expertise Data du sud","publisher":{"@id":"https:\/\/www.nicedata.fr\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.nicedata.fr\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"fr-FR"},{"@type":"Organization","@id":"https:\/\/www.nicedata.fr\/#organization","name":"NiceData","url":"https:\/\/www.nicedata.fr\/","logo":{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/www.nicedata.fr\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/11\/NICE-DATA_JLFMod.webp?fit=2493%2C1249&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/11\/NICE-DATA_JLFMod.webp?fit=2493%2C1249&ssl=1","width":2493,"height":1249,"caption":"NiceData"},"image":{"@id":"https:\/\/www.nicedata.fr\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/JLFerralis"]},{"@type":"Person","@id":"https:\/\/www.nicedata.fr\/#\/schema\/person\/8d1ad38004d3b0cf6bff7c200c795e19","name":"Jean-Laurent Ferralis","image":{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/secure.gravatar.com\/avatar\/3cbfdae273d44fb82b902a3451eb0db37485119e34cb70c1fdd186c9b731b9f3?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/3cbfdae273d44fb82b902a3451eb0db37485119e34cb70c1fdd186c9b731b9f3?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/3cbfdae273d44fb82b902a3451eb0db37485119e34cb70c1fdd186c9b731b9f3?s=96&d=mm&r=g","caption":"Jean-Laurent Ferralis"},"description":"French Data Professionnal - BI consultant and #sql lover. I also #swimbikerun when possible ! Living in @villedenice","sameAs":["http:\/\/xp-it.com"],"url":"https:\/\/www.nicedata.fr\/index.php\/author\/jlf\/"}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.nicedata.fr\/wp-content\/uploads\/2024\/10\/runningpeoples-scaled.jpg?fit=2560%2C1700&ssl=1","jetpack-related-posts":[],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/posts\/717","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/comments?post=717"}],"version-history":[{"count":7,"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/posts\/717\/revisions"}],"predecessor-version":[{"id":758,"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/posts\/717\/revisions\/758"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/media\/721"}],"wp:attachment":[{"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/media?parent=717"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/categories?post=717"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.nicedata.fr\/index.php\/wp-json\/wp\/v2\/tags?post=717"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}