621 lines
4.3 MiB
HTML
621 lines
4.3 MiB
HTML
|
<!DOCTYPE html>
|
|||
|
|
<!--
|
|||
|
|
==============================================================================
|
|||
|
|
"GitHub HTML5 Pandoc Template" v2.2 — by Tristano Ajmone
|
|||
|
|
==============================================================================
|
|||
|
|
Copyright © Tristano Ajmone, 2017-2020, MIT License (MIT). Project's home:
|
|||
|
|
|
|||
|
|
- https://github.com/tajmone/pandoc-goodies
|
|||
|
|
|
|||
|
|
The CSS in this template reuses source code taken from the following projects:
|
|||
|
|
|
|||
|
|
- GitHub Markdown CSS: Copyright © Sindre Sorhus, MIT License (MIT):
|
|||
|
|
https://github.com/sindresorhus/github-markdown-css
|
|||
|
|
|
|||
|
|
- Primer CSS: Copyright © 2016-2017 GitHub Inc., MIT License (MIT):
|
|||
|
|
http://primercss.io/
|
|||
|
|
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
The MIT License
|
|||
|
|
|
|||
|
|
Copyright (c) Tristano Ajmone, 2017-2020 (github.com/tajmone/pandoc-goodies)
|
|||
|
|
Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
|
|||
|
|
Copyright (c) 2017 GitHub Inc.
|
|||
|
|
|
|||
|
|
"GitHub Pandoc HTML5 Template" is Copyright (c) Tristano Ajmone, 2017-2020,
|
|||
|
|
released under the MIT License (MIT); it contains readaptations of substantial
|
|||
|
|
portions of the following third party softwares:
|
|||
|
|
|
|||
|
|
(1) "GitHub Markdown CSS", Copyright (c) Sindre Sorhus, MIT License (MIT).
|
|||
|
|
(2) "Primer CSS", Copyright (c) 2016 GitHub Inc., MIT License (MIT).
|
|||
|
|
|
|||
|
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|||
|
|
of this software and associated documentation files (the "Software"), to deal
|
|||
|
|
in the Software without restriction, including without limitation the rights
|
|||
|
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|||
|
|
copies of the Software, and to permit persons to whom the Software is
|
|||
|
|
furnished to do so, subject to the following conditions:
|
|||
|
|
|
|||
|
|
The above copyright notice and this permission notice shall be included in all
|
|||
|
|
copies or substantial portions of the Software.
|
|||
|
|
|
|||
|
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|||
|
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|||
|
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|||
|
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|||
|
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|||
|
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|||
|
|
SOFTWARE.
|
|||
|
|
==============================================================================-->
|
|||
|
|
<html>
|
|||
|
|
<head>
|
|||
|
|
<meta charset="utf-8" />
|
|||
|
|
<meta name="generator" content="pandoc" />
|
|||
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
|||
|
|
<meta name="author" content="Julien RIOU" />
|
|||
|
|
<meta name="dcterms.date" content="2024-02-13" />
|
|||
|
|
<title>Real-Time Feeding of a Data Lake with PostgreSQL and Debezium</title>
|
|||
|
|
<style type="text/css">
|
|||
|
|
@charset "UTF-8";.markdown-body{-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%;color:#24292e;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16px;line-height:1.5;word-wrap:break-word;box-sizing:border-box;min-width:200px;max-width:980px;margin:0 auto;padding:45px}.markdown-body a{color:#0366d6;background-color:transparent;text-decoration:none;-webkit-text-decoration-skip:objects}.markdown-body a:active,.markdown-body a:hover{outline-width:0}.markdown-body a:hover{text-decoration:underline}.markdown-body a:not([href]){color:inherit;text-decoration:none}.markdown-body strong{font-weight:600}.markdown-body h1,.markdown-body h2,.markdown-body h3,.markdown-body h4,.markdown-body h5,.markdown-body h6{margin-top:24px;margin-bottom:16px;font-weight:600;line-height:1.25}.markdown-body h1{font-size:2em;margin:.67em 0;padding-bottom:.3em;border-bottom:1px solid #eaecef}.markdown-body h2{padding-bottom:.3em;font-size:1.5em;border-bottom:1px solid #eaecef}.markdown-body h3{font-size:1.25em}.markdown-body h4{font-size:1em}.markdown-body h5{font-size:.875em}.markdown-body h6{font-size:.85em;color:#6a737d}.markdown-body img{border-style:none}.markdown-body svg:not(:root){overflow:hidden}.markdown-body hr{box-sizing:content-box;height:.25em;margin:24px 0;padding:0;overflow:hidden;background-color:#e1e4e8;border:0}.markdown-body hr::before{display:table;content:""}.markdown-body hr::after{display:table;clear:both;content:""}.markdown-body input{margin:0;overflow:visible;font:inherit;font-family:inherit;font-size:inherit;line-height:inherit}.markdown-body [type=checkbox]{box-sizing:border-box;padding:0}.markdown-body *{box-sizing:border-box}.markdown-body blockquote{margin:0}.markdown-body ol,.markdown-body ul{padding-left:2em}.markdown-body ol ol,.markdown-body ul ol{list-style-type:lower-roman}.markdown-body ol ol,.markdown-body ol ul,.markdown-body ul ol,.markdown-body ul ul{margin-top:0;margin-bottom:0}.markdown-body ol ol ol,.markdown-body ol ul ol,.markdown-body ul ol ol,.markdown-body ul ul ol{list-style-type:lower-alpha}.markdown-body li>p{margin-top:16px}.markdown-body li+li{margin-top:.25em}.markdown-body dd{margin-left:0}.markdown-body dl{padding:0}.markdown-body dl dt{padding:0;margin-top:16px;font-size:1em;font-style:italic;font-weight:600}.markdown-body dl dd{padding:0 16px;margin-bottom:16px}.markdown-body code{font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace}.markdown-body pre{font:12px SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;word-wrap:normal}.markdown-body blockquote,.markdown-body dl,.markdown-body ol,.markdown-body p,.markdown-body pre,.markdown-body table,.markdown-body ul{margin-top:0;margin-bottom:16px}.markdown-body blockquote{padding:0 1em;color:#6a737d;border-left:.25em solid #dfe2e5}.markdown-body blockquote>:first-child{margin-top:0}.markdown-body blockquote>:last-child{margin-bottom:0}.markdown-body table{display:block;width:100%;overflow:auto;border-spacing:0;border-collapse:collapse}.markdown-body table th{font-weight:600}.markdown-body table td,.markdown-body table th{padding:6px 13px;border:1px solid #dfe2e5}.markdown-body table tr{background-color:#fff;border-top:1px solid #c6cbd1}.markdown-body table tr:nth-child(2n){background-color:#f6f8fa}.markdown-body img{max-width:100%;box-sizing:content-box;background-color:#fff}.markdown-body code{padding:.2em 0;margin:0;font-size:85%;background-color:rgba(27,31,35,.05);border-radius:3px}.markdown-body code::after,.markdown-body code::before{letter-spacing:-.2em;content:" "}.markdown-body pre>code{padding:0;margin:0;font-size:100%;word-break:normal;white-space:pre;background:0 0;border:0}.markdown-body .highlight{margin-bottom:16px}.markdown-body .highlight pre{margin-bottom:0;word-break:normal}.markdown-body .highlight pre,.markdown-body pre{padding:16px;overflow:auto;font-size:85%;line-height:1.45;background-color:#f6f8fa;border-radius:3px}.markdown-body pre code{display:inline;max-width:aut
|
|||
|
|
</style>
|
|||
|
|
<style type="text/css">code{white-space: pre;}</style>
|
|||
|
|
<!--[if lt IE 9]>
|
|||
|
|
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
|||
|
|
<![endif]-->
|
|||
|
|
</head>
|
|||
|
|
<body>
|
|||
|
|
<article class="markdown-body">
|
|||
|
|
<header>
|
|||
|
|
<h1 class="title followed-by-subtitle">Real-Time Feeding of a Data Lake with PostgreSQL and Debezium</h1>
|
|||
|
|
<p class="subtitle">PostgreSQL Users Group Belgium</p>
|
|||
|
|
<p class="author">Julien RIOU</p>
|
|||
|
|
<p class="date">February 13, 2024</p>
|
|||
|
|
</header>
|
|||
|
|
<hr>
|
|||
|
|
<nav id="TOC">
|
|||
|
|
<h1 class="toc-title">Contents</h1>
|
|||
|
|
<ul>
|
|||
|
|
<li><a href="#speaker">Speaker</a></li>
|
|||
|
|
<li><a href="#summary">Summary</a></li>
|
|||
|
|
<li><a href="#who-are-we">Who are we?</a></li>
|
|||
|
|
<li><a href="#internal-databases">Internal databases</a></li>
|
|||
|
|
<li><a href="#data-lake">Data Lake</a></li>
|
|||
|
|
<li><a href="#extract-transform-load-etl">Extract Transform Load (ETL)</a>
|
|||
|
|
<ul>
|
|||
|
|
<li><a href="#sqoop">Sqoop</a></li>
|
|||
|
|
</ul></li>
|
|||
|
|
<li><a href="#change-data-capture-cdc">Change Data Capture (CDC)</a>
|
|||
|
|
<ul>
|
|||
|
|
<li><a href="#debezium-1">Debezium</a></li>
|
|||
|
|
<li><a href="#kafka">Kafka</a></li>
|
|||
|
|
</ul></li>
|
|||
|
|
<li><a href="#other-uses">Other uses</a></li>
|
|||
|
|
<li><a href="#the-future">The future</a></li>
|
|||
|
|
<li><a href="#do-you-really-need-cdc">Do you really need CDC?</a></li>
|
|||
|
|
<li><a href="#questions">Questions</a></li>
|
|||
|
|
</ul>
|
|||
|
|
</nav>
|
|||
|
|
<hr>
|
|||
|
|
<h1 id="speaker">Speaker</h1>
|
|||
|
|
<p><img src="data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEBLAEsAAD/4RwARXhpZgAASUkqAAgAAAALAA8BAgAGAAAAkgAAABABAgAPAAAAmAAAABIBAwABAAAAAQAAABoBBQABAAAAqAAAABsBBQABAAAAsAAAACgBAwABAAAAAgAAADEBAgANAAAAuAAAADIBAgAUAAAAxgAAADsBAgAMAAAA2gAAAJiCAgAMAAAA5gAAAGmHBAABAAAA8gAAAHoDAABDYW5vbgBDYW5vbiBFT1MgNjAwRAAALAEAAAEAAAAsAQAAAQAAAEdJTVAgMi4xMC4zMAAAMjAyMzoxMjoyNyAxNDozMjoxOQBKdWxpZW4gUmlvdQBKdWxpZW4gUmlvdQAkAJqCBQABAAAAqAIAAJ2CBQABAAAAsAIAACKIAwABAAAAAgAAACeIAwABAAAAkAEAADCIAwABAAAAAgAAADKIBAABAAAAkAEAAACQBwAEAAAAMDIzMAOQAgAUAAAAuAIAAASQAgAUAAAAzAIAAAGRBwAEAAAAAQIDAAGSCgABAAAA4AIAAAKSBQABAAAA6AIAAASSCgABAAAA8AIAAAWSBQABAAAA+AIAAAeSAwABAAAABQAAAAmSAwABAAAACQAAAAqSBQABAAAAAAMAAJCSAgADAAAAMDAAAJGSAgADAAAAMDAAAJKSAgADAAAAMDAAAACgBwAEAAAAMDEwMAGgAwABAAAAAQAAAAKgBAABAAAAAAQAAAOgBAABAAAAAAQAAA6iBQABAAAACAMAAA+iBQABAAAAEAMAABCiAwABAAAAAgAAAAGkAwABAAAAAAAAAAKkAwABAAAAAAAAAAOkAwABAAAAAAAAAAakAwABAAAAAAAAADCkAgAMAAAAGAMAADGkAgANAAAAJAMAADKkBQAEAAAAMgMAADSkAgAcAAAAUgMAADWkAgALAAAAbgMAAAAAAAABAAAAPAAAAAQAAAABAAAAMjAxOTowMzoyMCAxNDoyMTo1MgAyMDE5OjAzOjIwIDE0OjIxOjUyAAYAAAABAAAABAAAAAEAAAAAAAAAAQAAAEcSAAAhBQAAFQAAAAEAAABjfAEAEQAAAEcNBQA5AAAASnVsaWVuIFJpb3UAMTMzMDYzMDc4NDQ2AAASAAAAAQAAADcAAAABAAAAAAAAAAEAAAAAAAAAAQAAAEVGLVMxOC01NW1tIGYvMy41LTUuNiBJUyBJSQAwMDAwNDEwNWJjAAAJAP4ABAABAAAAAQAAAAABBAABAAAAAAEAAAEBBAABAAAAAAEAAAIBAwADAAAA7AMAAAMBAwABAAAABgAAAAYBAwABAAAABgAAABUBAwABAAAAAwAAAAECBAABAAAA8gMAAAICBAABAAAABRgAAAAAAAAIAAgACAD/2P/gABBKRklGAAEBAAABAAEAAP/bAEMACAYGBwYFCAcHBwkJCAoMFA0MCwsMGRITDxQdGh8eHRocHCAkLicgIiwjHBwoNyksMDE0NDQfJzk9ODI8LjM0Mv/bAEMBCQkJDAsMGA0NGDIhHCEyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMv/AABEIAQABAAMBIgACEQEDEQH/xAAfAAABBQEBAQEBAQAAAAAAAAAAAQIDBAUGBwgJCgv/xAC1EAACAQMDAgQDBQUEBAAAAX0BAgMABBEFEiExQQYTUWEHInEUMoGRoQgjQrHBFVLR8CQzYnKCCQoWFxgZGiUmJygpKjQ1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4eLj5OXm5+jp6vHy8/T19vf4+fr/xAAfAQADAQEBAQEBAQEBAAAAAAAAAQIDBAUGBwgJCgv/xAC1EQACAQIEBAMEBwUEBAABAncAAQIDEQQFITEGEkFRB2FxEyIygQgUQpGhscEJIzNS8BVictEKFiQ04SXxFxgZGiYnKCkqNTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqCg4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2dri4+Tl5ufo6ery8/T19vf4+fr/2gAMAwEAAhEDEQA/APYxRRS0AFFFLTAKKKKAClpKKAFooooAWlpKKAFpaSlpAHQZpksoiUE8ljhR6mo7idUilOQAiElieBxXDap4xi/tkRJLtWNMKxPAz1P14/zmmB3iyh5Cic7fvN2FSZB4zXl938UtHsrhbS3kmkjXq8a9fzxmtjSfiRoV4VRrtYfTzAxY/oaAO5xSYrNt/EGk3TbYb+F2/wB7FaSsrjKkEetIAopaKAEopaKAEopaSgBKKWkoAKSlooASkpaSgBlLSUtMBaKKKACiiigAozSUmaAHZozTM0ZpAPzS5qPdRupgS5qjqeq2ml2zy3NxHGAP4mANRaxq0OjaZNfTn5Ixn6184eKfGN54i1GR2bbHvOxVJGFzxQB2Xif4lfbLWSysNyIcq7Hjdya8wutQnlJklnYlieCetVnLlWbOTk0ySLc8fOVxz7GkMcC+3zJPz6Cp4pdo3tkE9MDpToTCwLTt8o+6McmpJEQsGztToBQMltLxreVXjkZSDnO7Br0HQPitqem7Ibpo7mDOPnA3AexHWvMpAgJCNuIqIFl6rj3pXBo+tNB8Sab4hs1nsrhGOMsmcMv1HWtevkvRtbvtIukns7ho5AemflNfR3gjxKviTRI5nIFygCyr/td8e1Mm1jpqKWigBKKWkoAKSlpKACkpaKAEpKWigCKloooAWiiimAUhpaaaAENITQaaaADNITSGkNIBd1IXwMk00msTxZfjT/DV7NuKnymCkHBzimB5P8VPGjapfLpFjJ/okJzIwP32/wA/zridN0W5v2BRCR6mmabYyalqqp97JyTXsGk6RDZQIoRcgAdKzlKxtTp825x9h8PpJlBmlCg84xW6fh3C0KqkgGfvcV2MKAAcVoQrxyKy52dSpROPt/h1pccf7xAzeuKST4eaczEqoA9MV3QxjpRgelS5MpRj2PNLz4c24ibynIPUVwWs+HrjS7nEvK9sV9CtGGHSuS8V6Qt5bM20ZANOMmiKkItHiZXDAYOPeuw8Aa7c6P4ntkWbbDOwjcHoc9P1xXL3kXkzlDlSDSwsY5VlU4ZCCfrW9zka6H1wh3IrccjPFOxWV4buDd+HbGYkktEM5OfatWqMxKKWigBKSlooASkxTsUlACUlOpKAIqWkpaYBS0UUAIaaaeaaaAGGmmnGmmgBpphNONMNACE1xfxNdx4TlC9Cw3fSuyNc343tPtvhi6i77SR9RQCPKfA9iru9wVyema9HiXFcZ4dnt9L0GOVzy/OPWraeMLZMl/l9s1g02zrg0kdtABxmrTNtHFclpfi22u51QArk4Ga6tGWVQQeMVm4s6IyTLMWWGealC471kXOsrZqURNz471zM/j2WKfyja7ucZFPkJdRI73FUL+DzYmXHUVycfijUJZhJHEQn90qa3rDXItQbyyu2QdRmly2FzXPFvFlrJaamwddvPBqpbR+cQvUsB0r0b4kaZE+lrdBMOrdcVwehxB9UswRlZHQfmRWsXdHNUVmfSHg6zax8LWMDEkhM8nPUk1u1XsYxFYwIBgKgH6VYrU5wpKWigBKSlooGJRS0lACUUtJQBFS0lLTAKXFJS0AIaaacaYaBDTTTTjUZoAQmmGlNNJpjGms/VofOsXX2q+TUUy74mHqKQHkVjpi3FhDFKPlTPH5VMbXQ7RcTxpkdSVrfhswICgGOT0qknh5PMlZX5l4ctzn86we51RWhhyWentul059rDkrXXeFJnurciTPy8VlzaQljZmGDOG5Iz1ra8NQeTbsuOS3NTI1iiXUbOHcRt3+1YMl2unx+bFYq0YfaSMACutli8xzjr71WXSonLLMh2k8r2NRHfUprTQyLXVorjZ5lsY9
|
|||
|
|
<ul>
|
|||
|
|
<li>Julien RIOU</li>
|
|||
|
|
<li>Open Source DBA</li>
|
|||
|
|
<li><a href="https://julien.riou.xyz" class="uri">https://julien.riou.xyz</a></li>
|
|||
|
|
<li><a href="https://hachyderm.io/@jriou"><span class="citation" data-cites="jriou">@jriou</span><span class="citation" data-cites="hachyderm.io">@hachyderm.io</span></a></li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h1 id="summary">Summary</h1>
|
|||
|
|
<ul>
|
|||
|
|
<li>Who are we?</li>
|
|||
|
|
<li>Internal databases</li>
|
|||
|
|
<li>Data Lake</li>
|
|||
|
|
<li>ETL</li>
|
|||
|
|
<li>CDC</li>
|
|||
|
|
<li>Other uses</li>
|
|||
|
|
<li>The future</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h1 id="who-are-we">Who are we?</h1>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAoIAAABkCAYAAAASclGjAAAACXBIWXMAABcRAAAXEQHKJvM/AAAdTElEQVR4nO2d+1EbyRbGRy7/P9wIYCMAR4AcgbkRIBcBrDYCQwQrB6ACIjBEYBSBUQSLMvBEoFvNfuM7yFKf0zP9ON1zflXUem1A8+jH1+c52W63lWJnUi/Pqqo6yvgxvWybqxcB16EoiqIoiiDe68uwM6mXRgA+VVVVS75Ogv8aMSj6ChVFURRFic47feQki8xF4GrbXD0IuA5FURRFUYShQtACXMKXYi+QxzyHi1QURVEUJT4qBO0sJF8cg/ttc/Us/ioVRVEURUmCCsEDTOrlRVVV5yIvjkej1kBFURRFUWw4CcFJvczdQuZC7vd6vW2ufqa+iEm9PIGLXVEURVEUYbCF4KReXldV9eekXk5Lf4m412MBl9KXzba5kiJkFwWIakVRFEUpEpYQRAmV1s14V/JQ2LnXXJlJuG4cGj4ZFztc7YqiKIqiCIJrEZx3Sqgcw2JWKiWUi3kScB2G7jhRq6CiKGJBGMsUxgBFEQHGZFBPLNlZBJPiZUccmUSEEwkxaD5BLNuPzG/jDwldRCb10lglb3f++vO2uSraoqwoSl5gj3vYSQ78um2uNNlOSQa8aHcd7fWaABpiD+VYBOd7LGR1oRae3O/pqxAReLRjDWwR4bJWFEXpsCsCK8TDl+z5UgRjrNNVVX3b0V7mz7chrIOcFnOHTkWXJou4lDp1hZSLkbJwzQ8k25hYwakg13VUdnpWH+pfbebTq6V9rM9JUWKBOXlo3Z8LWlOVcWEzmszQ9tYbViEI954tXs5Y0LLPIoYFK3dr4FxIuRgq2WbuexBLBKe2KQTfWZ8s9Em9bP+4gkA0X0+xrb7YLM17O8Ff3eXs4h96Pzg0ziDkzbvI4kCc63UHxhYPmHOsuFIuJ5w7g1Vxhv3nBAaGJ+whb/Zga4zgpF6+MDaw/+beyxYugC8CLqUv621zJaJW36Re3jHa8omIY/QJBPAFvj4F/rg1JvRd6I0c4uHbnn8yXWuyc/UPvR/UUv1zzz+JXgdzve7Q4MD2/dDHbJurSTE3q2QDoUlMQqjVADepl601+2fHiHACUXhq1ruu8ehgjCAmCMeKkbUlDao596BgEdfv0Ju5mCBsZBrewcJyG0EEVpjIZlP/MamXz7Dch+LQ/L7MtKboIcsfeT8Y3/vElO33JifX61YUxR2IyL8R+3q2ba4uts2VaTAxg8HoL+zTv6yCtmQR7uaSezmZ68xdAI+CYsm4h4Lsk0Y6AvAfTKpUY+gUAcQvvgUhBITtMJiVoIc10PaeKGFrq4VZCxbGuV63oigOYM3+gsRRsx/MjT7rfE3RbOKj2Tta7bZXCMLNxbHstMxhWcsKLIAu9ykRKdbAqUOyTR3YihUMMzfgZvtH2Ng5hiB89tjSj4o5/ZRZzTXNWlcUpWQWO4mjX7DuTfHn7+ZADOPRI7Tb0SGLoGsXiDrT7KrcM8JuBMXaubqYsus0AoH1bHGzSeAULuPBYxtjqyG+LQtxBcFKue01S1tRlCzBGneOJLDuIf4OMYX/wf+3e2/bPOPikBDsY2XKKmYIFqncy8WIiM9EYKprVmxW1iSMlx+eelCvkQl86IsSXxy+TOrlk4dnTCUS5GJlo66zgchXFEXJkdYTdGgda/eC3fJkJ7+Vj4GL97TnQ7jOoZyMpeBxTkgqF9P3WV7kEKzOzIQ+xBpiyky6FxcLLg5WbfkZKr5tH+agY8TgbEB2MXXvJs7kLIMyJNTh9qG0TkmKooyS3XVshvhoo+s2OwakV6PDvjqCQ1x259h0pG/ufSxYklgJesb7Os9wES8Ee4rANSbbIHGBE9svdyUm84Xj9ZxCDE77iDVzDZN6uSHmy0xy4ggj6aXSXtiKohTCrhfoBfvIKdzEXWPE6969zzU81NVzLdnlV0i5GBHWTDzLIfUXRbuHkRTiIrqMW/ejSdE3Qt23hcnUe0MmmIn1uHH4UTPZHwY8a0qsS3cPU/N9o4WVFUXJnFbg7SYLmgLSRjN8RcjQa2JvJ5Tv5Y0QxEbR1y3ccixcaOVeLuZeULkYH4JUZCgBYgK5SSHGvP7ZBOTGeDdGYGJi/wHxyeEYlsE+YpASgjWslVKhrk2tgYqiZA0sfas2E3jPvVzv5Ba0Ou1p1yLoazEXWU6mgHIxYvoJe3yW4gQEXIm3zG83E+8khaveTHxkg/3F/JHTPqKns8DYEGkVZLTJrBgJMYqiKDkw36ni8rE9yMNDZfaLBfbvT23lkV0h6Ms6Uws9ZeeeILIQVC7G17OUaBHkirp7WAGTJhmgQOgHZrbxZU/rHfVMpLr5qXt9LK3doaIo4wQhLiZs6M9JvTQH3Ofu+oZ/b9sqruFZ+i1G0Oem/ElSOZkCysXsZvskw/OzPJZkPUb9PU54hKheu50JzhGDd66iDRbPrGoKYlxRtQPVGqgoSjFA3P2F/eAFZcRM7oaJE39pO4909d4vIYhF03cmrRThclRAHNC1oPIWvi2rIg4MDskvokRgi4MYrHvG8eZWU5CyBjYZVDhQFEVxAl6iM+ie1iV8BM+OSWh8U36uWz4mxGZ8KqSczJASJxIQUy4GFjPfB4apkDIyHIG7ligCW4wYRIFvKsbRZI/tlhKgoLKopdUUpMSuikBFUYoEazvLaNN1DYeyyixSxg55KHEiASn9hI8CXUtyiyDGCZX80uTQGg+HhkfGtzpZdiHwNsS3SRmrnNqBKgQVRRk9XSHoq1H9Ln3dUL7I3SV8L8jCsghkWT0WkGjAEUXXGSUWzBgu4ssez52aT1KEMrXmrLV2oKIoylshOLR+oI0vKRICOinSudIIsrBwLGZDCHUQIYEYou5tg7iLLED8B2fsuI4vKk6wRjJRaihBqtZARVFGT9UKQbhRQpNiE83dGrgQlCASeuNMJgSZSQ7ZlR6Ci5hy5TqJNlhEKbdzUqsgs3agCkFFUUZP1bEIxtiEo5aTwWYQ0soZmk1b4yc1eG+hS++kLCFDiaFNxtml1Bg67nEQ5NQUlPw+HwUdsBRFUZLSCsFYi3YUC10h5WIktemL8SyTWAQhWKgDQ86FyB981/8zPY8ZvzOJVRDvkzq0qDVQURQFxLQIVigvEUPglFAuRkSh24iW1VSuYY5gybboMCxf1PWH6DSS6iDDse5qEWlFURTQ1hGMmbF5jfplQVwzWi7GH5Etq6mEOxWuUIIb8YFIhnnt7uKYEW2E4J/E70xRU5ASgioChYF1Zt9B8Fld+P+C8I3dffqnZr7L4sBYFv+eWiEY0xrTNkQOJXZydwl/HTpoJvXyOcf4yETCgRKC2QsHYwGb1MuGENtORb1RuHpNjLN5zG4jiGWlagfmvj5kDQTNFF9kWMakXpr/rIworKrqyXyVLA4hJC46z8ca5oDnU3Wekfl6iPGMcK2zjkB1LVDv81pmnRC3hxj7yM5YttYtxXvatGM41jvi0grB2NYY0xB54XvQFFIuZlA8WuZJMlFrCWIiU2P/KdLlhOaJmBt9uruY7//b8u+x4wQp0bnOqA5kMXQEw7xnV6JzfL1aoCf18h6io5S52a7bQ3q4n3d+9nZSLx8hNoLEw+KdPu+8T1Mm7nPMxDpcx9POnhfsOjpNFWY9xvIxPDOXaLTxIKU27buE2X0hBkvuQeCD+gkXkCQTu6g0ZQnfFCQcqE2zzzpAzbdoNQU7lhQbag2MiHknaEn5ggODr9aUZiP9jmb6IvqU98XMj0m9fEFLSJ+VGT5BEL4EmoOLA+8zdiexxQHDx61PbbMzlr94GMs1xvE/xiiWuqHCu4RlO859TmIkofjugRsTHwWLc0+SiZ0wQo39YiwOjHtx3oRwaLknvi2WVfCCMfY1PjASWNufsWmGWpPOIQiTb6SuGG8EQnhuA+9bxxBFT56NPodijuvI67hNQ3jRFxHGsrF0P6c81LxjfE9IvFjwsAjkXOKjGhpLVUiSTGyoiVdMIDYnZqZnYXlKXMWqKUjNn3tNPIgDLFDfIx7MzUb6FKkxwmDwfHbdmaE5h9gQ
|
|||
|
|
<hr />
|
|||
|
|
<h1 id="internal-databases">Internal databases</h1>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA+4AAAGDCAYAAAC1EUoTAAAgAElEQVR4Xuy9B5hkx3UeeibHzXmxOWAXwCIscg4EM8UgkZQpUDStbMmyniVb7z3Zkv29T37+JD+9z5L1ydITZVsSBUqiRNECMwIJEiByBjbnhM07O7MTunume95/Ktx7bnXV7Z60M7tbTS5m5nbdunVPnfPXSXWqgeInUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSYMZSoGHGjiwOLFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUoCi4R6ZIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFJgBlMgGu4zeHLi0CIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFouEeeSBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSYAZTIBruM3hy4tAiBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBaLhHnkgUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUmAGUyAa7jN4cuLQIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgWi4R55IFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFJgBlMgGu4zeHLi0CIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFJtVwHx0dfRQkfSSSNVIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUqAmBT7b0NDwpVqtouFei0Lx+0iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBqaFANNynhq6x10iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSaFANNwnhYyxk0iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBqaFANNynhq6x10iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSaFANNwnhYyxk0iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBqaFANNynhq6x10iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSaFANNwnhYyxk0iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBqaFANNynhq6x10iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSaFANNwnhYyxk0iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBqaFANNynhq6x10iBSIFIgUiBSIFIgUiBSIFIgUiBSIFIgUiBSaHA9Bjuo6OjjzQ0NBB+Jm/Bf/PHvWb/tt/bNrK97Uv+9JGH+3Lb+MYQIq29X36fdy30nb3ffSd5Xd4rxy2f7dKQvwvRJY9d6h2npJVvHD762jHxd/I5vvn2vb+c/7HSIcRLtZ4zKaIVO4kUiBSIFIgUiBSIFIgUiBSIFIgUiBSYHApcfMO9XC4/irE/4hqvrvHoM9jsPa4xHzJgbbs8wzTkMAg5CNwxuEagz3gOGYrus+V71OMkcOng3iP7k04Cl3dCDo1adHVpZP+WxnrI4eKjm9uffX7IYVCPDPjewZ0j2Q+3z6NVPc+MbaaeAnlOP5fvQ3ISHYfaWSrl1ie7Y3WYSTn24VgIf+T1WnPjyqx0DIZwMDoOp14uL9UnRDxJccBd/0IOeR9uRDxJg1Ehvc/FWFe/qUfvymtTr953qcrqdIw74kPEB1eepytQWalUPtvc3PylWnKQane1WtbxPV72UfxTEXep4PmUOUkoV6nzGYR5Cmbo/vEsSiHFMy/CO9XGoFRK61GW65gqbxPXCBpvP/G+SIHxUEAaaD4l0SpFElvyHD8hHJJ87sqTKwO1FK2QE84dq4t3eTKdp+yF3imUmeS+awiXa9EhZDDL9/TNuQ+D63UW+MaUN6cuDrtzZ/uTfeTxTy0ertdxKOdmqteKWmO+kr6PeJIq5HLefcZfSL4l71rssDJvZSeEga4e58NviYv16B8RT3TmpYtlV5JcT9a7RnyI+CDX/5mgb2AM02u4u4auC/qhRSCknF9KCo+rlLsLXT0GSC3nR0gBz1OepTHkKvq+Mcr3kOPxGSHy/tBc5S3wtZT5UKTVN27fWHzPnqwFIPYz9RRwlbrQfOYZxJL/XUXWlQ2frPjeMhTBkvLpU0h9+Jgnu277eu+fjJmph6aT9RyJjZPRZ+wjUiAkV7512G1bD++7zh9f4CHiSUrZemg6GVxbjyNgMp4T+7j8KBD1jexW56hvXDQev/ip8pjcRxHqTyLu0tgKeXRDC6XvXjfCETI+fX2GFPN6+vQp+SHHhC/y5S7s0lvqfucqE3ns4vbjM65tf9Iwdp0CvoWUr7mGuw/MfP3bMVta+J7tfiffM8/IzutLGkuhSOlFE7/4oHFRwJ1fn1IcuubDA588uXLiwwDJS3n8OK6XnOKbXHmWMipl033HEM6NxREXMpJ8xoyca98YXfyx4/W9g4tD9YzDN8e1eCtESx+Ou7x3KTmfp5hFL1r3EU8mTuqIJ/7EVN+6UC+/jTUQEcLtic/uld1DvfPlro159/n0C7l2RX0jWwOt3nWyXj3MbReaK3ceZoK+MS2p8naPuy9FyjXmQnDB97qM7zMs7f2hSXKVvDyFNS9qJp/jMwLsNZ/BLscgjV53zD4GcunlM0RDCqwcp6RzyBAPzY2PtnkKcp7ymmeAjcUw8CkRtZYe2X9UnmtRa/q+d2XEt9iFHG0+vpwOx6FvEZK8X0v2Q86HemS9lhLiYmYIU0IOQJcz7PPkmH24m4eNPgXW59TLwz8ffvnWDB/+5SnfPklwaSz5zr67XcN8mDd90nXlPTniSTat2rc++2RdXquFR7V4PqQz5clWxJPUsHExVtIt6jITw7SIDxEf5Hpeyz5yuS2kX+bpCLX0jWkx3DGoTHG6esSqXi+Kq3T6/ua+JNC5ir9cZEIGrRTmUPt63msibeqliTvWiTxzIvfWM97xjLVWvz7lPMQXUoGYyLvGe6eWAiEj16d0ukqMj8d8xlNeXz7HYQgrfPggQdtHqZBSnPeMEI7Ze6ShWEvp9Y15pjsOay12UuZDNM/jg+g4nFqZns7eI57o03x8wQm5Jko6RTzRHBtaJ3xORNcB6eP5WjhWa42TeO/7fTrl7FJ9dsSHiA+uzVBLTqdS38CzZ84ed7tAuJ4IqWiHlO4QILp9hQw2nyLr8yL7nlMLSEN9h0DMN+a8a1LJzjNSxwqaIdq5z5MLex5z12L0PAM89F4hA8ingISMFff6WOkU208PBXxY4DM43dGNR0byeLMe5ct9pv3bjWS5BrvsOw8Hxzq+yZixep8ZmqfJGMNY+qhnvNxGtqsnWlWr3zx+c7/zYelY3jG2HT8FIp5UHyU7fmqO/c5acuQqwvXI5thHUf8d9Yw34kn99JzpLSM+RHwYC49OJT5Ma8TdglpIWfEpta4xKY1ml1D1GN6yP5+hKhVs+axQSqar9Ml7fE4DXwqa68DwGcq2r1qRs1rGqo8RQ2Nyn+l7Hzl22bfr8HANbt9c1brf933IsZL3fB/fjEVAY9uLTwGfXLlyE8KKPF5zv/PxjYtXLi/L713M8MlMHu6E5CT0jJA85xmP7j31jEdikrx/LM+pxTW+cbj0k8qU+3s9DtN6xxtq55t733oWoqkP62vRJX4/+RSIeJJVyvMoXI9c+TDBJ0O15K/W92PhhIgnY6FWbOvycK21NeobmkIRH8KnpdWLZ7X0DZB5+ovTWSVQKjx28t0F1ReR8rVxjTHJUHlKZ4jxrNCGImRSoXSVaukZ9gm3ZHTXIeC+r3zXvL7c9w0ZtC4Y+aLPPoU4pKy7z3VpEZqHWoxayziq5WwILUM+Bwxfm25vflw2wxRgXpA8GeI5n6y4/Oj2I59aC2tc3nExZ7yOQ/k+Phzzva/EHzvuep2LEn8lzoUUbZdGPuM5r5+88fvwwtdXaG5DfYccMLX6dr+vp3/fWhEdhzMX0SKepMq2O0sSAyKepNSJeDJz5XmyRxbxIeLDTNM3pjXiPtkCNpP7CyngM3nMYx3blfCOY6VJbD/5FPAZinnOJR9f5nnHXcPZNcSsoRtSZH1Re9fgC41JtvM5o0KOPWm4u8bvRByHISPdpUG9Too8w9z37vK9QnQN
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="statistics">Statistics</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li><strong>3</strong> DBMS (MySQL, MongoDB, PostgreSQL)</li>
|
|||
|
|
<li><strong>7</strong> autonomous infrastructures worldwide</li>
|
|||
|
|
<li><strong>500+</strong> servers</li>
|
|||
|
|
<li><strong>2000+</strong> databases</li>
|
|||
|
|
<li><strong>100+</strong> clusters</li>
|
|||
|
|
<li>Highly secure environments</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="cluster-example">Cluster example</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAmQAAAGmCAYAAADS2ZieAAAgAElEQVR4XuydBbgcRdaGK3Jv3N2VCIEACW4JLosFWdzdF1ts8Swui8vi7izuEFwDBAgQiLsAIcT9f79m5v59+87c8ZmeO6eep56Z6emuOvVVVddXp06dquUsGAKGgCFgCBgChoAhYAgUFIFaBc3dMjcEDAFDwBAwBAwBQ8AQcEbIrBEYAoaAIWAIGAKGgCFQYASMkBW4Aix7Q8AQMAQMAUPAEDAEjJBZGzAEDAFDwBAwBAwBQ6DACBghK3AFWPaGgCFgCBgChoAhYAgYIbM2YAgYAoaAIWAIGAKGQIERCBUh69q16zarVq3qASYb1CbwfQofa/G7G3FOrVq1Xps0adItBcbMsjcEQoNAt27ddl69evVN9I3v6C/j6S/jli9f/sn06dO/Do2QJoghYAgYAoZAQgQKQsgYRHqsXLmyL4NHXwaSPgwoGyHpusQ6CSV2bjH3vMFzN0HO3kni/oLdAsE8hsxPJa4iziR+gNwfLFmy5PNZs2YtLJhglnFRIdC5c+dO9JU1Ebo/cT1iX2IX+k172lPdOIVZyv8/8f/rdevWvXX8+PGTQ1roWpRvLcp3EfJ1JS4hzkXub/l8lD7+Y0jlNrEMAUPAEMgqAnkjZJCTQ5F8O+LWxA5ZKsVEXty38dK+JkvppZJMLYilSKQGynUZ/PoSuyFPO343JTZIkJgGmufRajw2depUDT4WDAHXpk2bxg0bNtyBtrQ9cAwgDiQ2yQI040jjlsmTJ/8nC2mllQT9RYRyq8gETGWT5rsVMe57iP6kicxzfD4wceLEz9LK2B4yBAwBQ6AIEMgpIevdu3e9ZcuWXQoORxBb5xCPRaR9D4PNKbnKozuBgWQX4pbksX5kMKmdpfxmMeBcVyBimaUiWDKZIMCEZXPawAm0r/0zSSeJZ9VXHqSvnMDn6iTuT/sWCNgBTDh2o1wbkIi0X/G0ecnmMY8bXyTNq5nEfJfsQ3afIWAIGALFgEDOCBkv4z0ZXKS56pkBECt5VjImS3x+Y+njLGbS92WQZ6VHGSgP5sI/ibJly3VYDGb3sMQ0fMKECbNynZmlX3gE2rVr16i8vPy/kJZcE7FgYf+krZ0zZcqU27OJAv3+TNI7mrTXiPTdbCbvT+tdiNk5ELPPc5WBpWsIGAKGQD4RyDohwx6kJYPLNURpxZIN83iBf80zz61YseK5evXqyaZ/8bhx42YrAdJswH9rEWU/cxJRyx1xSRppjYLU7JAJqdEGA9K5lzw1s89WkEYiGcxFRM9Ai3FjtjK2dMKHAORlU9rYA0jWOwXppCXS8uP3tM0v6C+yS5wCMfmdNrsLv0fRdzoSh3F9J9LXkmd14WuIzbZ6PgUZqtxK3jdw8Thi/UzS4dlk+4iXDeX7tE6dOsczCfsmw3ztcUPAEDAECopAMuQgaQEhTgMZCF7hgU5JPPQx98g4/02W6vQ96TBgwIDy+fPnn8PL+Cie7xLnwXkakEj7k6QTjtyoWX5Eu5fsoxpENEhOIs8xDHBaThlJGt/z2ZkBowubGMYw6E0j7fX4/0Du243/pEWIGzTYgOcBlGFCsoLYfcWBQJcuXY6X/WMiablnDu3gVRnnEz+GeExM9Iz/f5kNLF269HyePVxtMc6zC2ln26RjowUR0+ToDWLHZOSiLCuQRdrfyXyXHeWXfP5QVlY2kQmUNh54hEyG/vSbPfhvX34rj2oDaV5APxme6D773xAwBAyBsCKQNUIG0ZCx7ksUtGE1hdVW/MfRXj2erV1fslMhX2mSYtmoreZFfRAv6keTrQAGmOe5V2QpZiAv7ZgU4fqAQWwEg8YX6ZaFvFogn8jfiaTXLE6Wy7l+INqyp5Itg90XbgQgYydQ77dWI+UY9RORsHQmFPHSxQxyXyYD1/N/LPKkdnYU7ezBZNEjvcNJ707uL6vmmaX8N4I2/ib9/iVIl8qWUoBUNoVU7slD54JJn+reL9is7jBz5sw5KWVgNxsChoAhEAIEskLIIBZ7U5Z4hGEVL9Eb0RA9nkt7Dwa5u8nnyDiY7sZA82IivEnjG9JYJ8590lD8t3Xr1i+MHDlSg1dWQ48ePY5huek28ojp+oMBbUfsfV7PaqaWWN4RoK/sQ6ZPxslY2tw3aKsX51IwZLiS9E8jlgfziUxgHkmUPxOhf9Mmz4tz3zQmKx9DwI4aO3bsn4nSSuV/8t2afLVRaLM4z82lH62DH7YpqaRr9xoChoAhUGgEMiZkzJJ3ZJb8apyCfMmLeVeWQrR1Pech8rJ+loyC2qblyLgmhHBsPCEYpN7iv21iDFA/ce08NBXP5bwAZIAc0jjIf1mloKUesNwymxqTfJTH8vh/BFiGGyKtahxMzoOIXZEvvCK+zUQAg0v+yyFlfatbJqeNns1zInXBsJJnT+XZ6rR/WSmi7O9I6D76RSyN2cwmTZp0Gz169LKsZGaJGAKGgCGQBwQyImRolHrxApYxbWO/rFyTBmkYL+aX81CGSlnI6Swv6fdiDDSzGfDkI6xK4JlH47gbkHsA+U/La2DA2xZ5pJELblxYiNZhzXSXSPNaCMusEgIsu7Vh2W08dVqpr3CTSMNBhVqSjqMVno48Me1Aq9GG/4hmaod8a6aq0dRNpAw69cOCIWAIGAJFgUCmhCzWEp9m2DtAxt4tFAI9e/ZsxuAwkvx7BWT4iJf05v5rDDCX8/vcoKwQottZIpSvpoIEEUu0ep+DZdA27hPKIO2AhSJCgHb2EeIG620eGrOhhd4hSFv7mvYuJ8cVgd+v0f538l9r3759G1x0aCmwXgB6LbPuUKjqQP5NkFendgR3eL6JXHKwa8EQMAQMgdAjkDYhY4CRr6/DAi9x7aDakZfg22EoOTLKuDdIaE5Fvpskn7yiN2jQQNv9KxklU4YLIZSXFboMLAf3g5R9ihyVlmAZfA5nsLy/0PJZ/skhAGE4kToLnsGqvjIk1R3GyeWY+l30ld94qqX/SdrezizzV5gjcM/P/B/cGfwh/WmL1HPM7hP0lXWRV0uwlUgZ1/aiDDJjsGAIGAKGQKgRSIuQRWy1YpGuQ3g5PxSWEuuoFgZC7Yj0G8rPRUZv4GGAkRZtkF/eQmvGgtjFWSKaRxmahwVnkyM+AoMHDy6bM2fOXO5oFLhrV+pQu5JDEWhnMpL/MCDMBGT0HDvTly6jb/wr8P9o/s+Hw+SkMMIu7iA0jsH3z6/I2CapBOwmQ8AQMAQKiEBahIyX9/vIHJwVP8yLT17tQxWwkbkWTcQZAdJ1Cy/uiQww1waE/Zwy6KDzUAUGw9uQ9fiAUA8h6yGhEtSEqYJALDcqtEcdkyWP9qEKyPowAh0Y6CvHo429g/906Ld/qXIly5eN2EUptxahCbE09wj3H/qKdpVaMAQMAUMgtAikTMggB3tBDp4OlEhbzftg0PtrGEvKS3oqcvmNlBfyWxsP/FqmpezMahrWnVmUQbY7fseeCxhksnHodBirrEbIpGOROHVCbh/8mzPGUW+peObPKxa0M8nrb1eTIJByciw/YBWBCc3p2L7JO3/oAmWYjlAdfIL9Cebx/PyFTn4TyBAwBEoTgZQJGRqnT3lBV9Ii8ftkZvxBG5nQIIp9yTnYkiRyKXBJrv0/ZQIIg8yxPH9HII3TkPk/maRrz+YOAersflKvtEuXvjKYvvJV7nLNLOVYS5MR7/oVB4PL0z5laJ9ZTrl7OuIsupIvNcpwQrbP7cxdCSxlQ8AQKEUEUiJkvOg68GLT7NMfZM8kw/kVYQaQwVG+0GK6vWCAmcMA0zbM8ku2GJq+8WAf3Eka9mKUjHz0Fx3gXaFt4vtnkIKNww4A7WwRMjaIJyfa8EFow3XqRmgDE8fZ9OsK2zGw/xns+4ZWYBPMEDAESh6BVAmZjimqNPPkpfdvyEzQ2Dd0wEa8k8uhZZUgWzJe1meFTuiAQJThQi5d4r8M/j3trMvw1RyEYA/qppIz4WI5bQEi+TGybhIH1bFMAqo9gzUMtUEZrqYM/j69GrmDfv3CIKrJYAgYAoaAh0CqhOweXnJH+LFj
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="mutualized-environments">Mutualized environments</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAfMAAAEYCAYAAABIjB8KAAAgAElEQVR4XuydBdhcxdXHC/FAgEJCQgIhBoEGD9oWbYsWt6KlQHGKU7xocZfi7k5xDy4lRQMEYkgoIVAkQIzA9/vvd+92dvbq7qy978zz3Gflzpw58x85I2fOme1nPngEPAIeAY+AR8Aj0NIIzNbS3HvmPQIeAY+AR8Aj4BH4mRfmvhF4BDwCHgGPgEegxRHwwrzFK9Cz7xHwCHgEPAIeAS/MfRvwCHgEPAIeAY9AiyPghXmLV6Bn3yPgEfAIeAQ8Al6Y+zbgEfAIeAQ8Ah6BFkfAC/MWr0DPvkegVRAYNGjQ3LNmzRr2008/rcqz0myzzbYgz498n0UZfuLR95/4rwffF+Z5m+drfs/g75l8n6HvP/7448zgv+l8Fv7nmc4zi9+zAnrF7/qPNIXfcd9nn332H4I4hU/9htfCZ4cOHX4gzAo+9XvWzJkzf+jUqdMPHTt2nDVt2rRZXbp0+WHKlCn6nDX33HPPGjVq1A8qT6vUTR357DBs2LAOX3/9dYfp06d36NGjR4cZM2Z0APPZwU5AdwD2Dp07d9b3jnzv1LVr105870xddNZvfVKPnUjTkc/O1Hcn/dbDd/2nd4rXkc9CXJ6OvNN3/WfG1f+F9zwd9B0sOvJ04JmP3wP5fxTfv+SZFjxT+W8qdKby+7/wdNsnn3zyYR0xjMzKC/NG14DPv2kQGDJkyFwMLD3ppD0ZsHvSkfXMx6AgQVHoyLz7eTA4vE8n/pxBZtynn346uWkK0USMILz7I/QOAa9fw9YgnrmaiL16sKIJiuYnP4JBYbKi38ETTlzM/8RT+FvfNbEJaYT86neWUBjbSV/4KBBjnmQkDL/rM3zCuIXfShukSY0b5pHhMwvvrRhH48N4MHsBzO5iMvLomDFjNG7ULXhhXjeofUbNgsDAgQOHIoR/BT+/oPMtzqeevjxdKuTxe+iMoRM/T/rneJ7/8MMPx1VIq+WTLbzwwrtSiEPAY9GWL4wvgEegcgT+Q9LzGAtOqZxE9pQNFeZst3Rma2onBsL16Pjz8zkHn3PAfjeerjydefR79mB2qy21cIYbzmC1RafvesLv+iw+0A238vSpWXLhnfnd+K+wTWfGEe3gtz4LNMI8w+/GzLsQh1CYUet7wHNhRm3E009zll38rjikt2uxWFfGDLtkps0Kcjbr3exBnubsO4xTeBdgq/fF34UpOb+DWb3+F/5hHKUv/EfQzF3fS96RrvhfSEtxwnQmPfN9+L+Z/v8h+396Jg/W/2F+ZjnFf5gm/L876dSmwrLaGDv7Da8jyP4KOvL1zog2N6EO/fv3vwAWt+GZu7lZ9dx5BOqKgI6KLvzggw+OssZ8p0zUVZgvuOCCa1CozXm0KhrCo7MxHzwCbRkBzc4vZSfgCs7VPmqLBUWIn0y59ufRBNwHj4BLBMJFTng8Idrh9+KRRfhfsNgyjzQKCzFzAUbcqEVfYcFH3B/CxRzfu/O9P/HHBwXSmXvhjJ7f4aOJa9Z2rzP2y9mCP4ot+G9cgiRaNRfmffv27YmSyMXk9XueSrcxXZfb0/MINAKBWziT37ctnbEjyHWs8MtGgFlBntMZiP/EgPyZBuVAOaqgIMX/UoTqwPfCJ7tOUoYKv2tnKtyh0o5UYSfK2Gkyz6MLu0DatdKnsXOljSJt2RVeBDTMnaMsxSmeoQc7d+HunyncCsLO2B0sntcb/xV3DCXotI1ImoLQC+KYO5GhQmG44xgqEf6g+FIQ1KcUAiUI9ZtHuoNSDpwZKg7yeyYKgzNRevtRSoN6pk6dGn7/UcqDvP+xe/fuUiD8ceTIkaHAzYJLQ+MMIFC+HSj/b2FkCZ55UxhSO/zDRx99dLdLxmsmzCnf6hTwNAq4okuGPS2PQIsjIE3Yo9lyO7OVy4EQHwb/T/DM77AcGsClof4dg52O2sbwTOH5mkf/fcc7/ZaOgxSNJECk9T5AD78HSBin8UOc14m7D0cgz6bF9e89AnkRQPbNw/xoT9IdxDNfXHomPcdNmDDh2Lz04+I7F+Z08h3oKCfSYbQ9kSXoHPwTnvdJo62HrwHiSwr6Xz6/gNZkNIv78Dmad9rO6Eo8rfD1dCVOl+C3tj+66NqCPnlX3BIJZtu6eqCZduHhezgLD68hFD71v97rO09hNh48+l/nsOHv8Fy5ZPYdxC05o+a/SG1RA5woLVNb+zQRS/gyt6MUN+o8Pmmryt7OMmf75rl/cQsLPGydgHCGr1dFvQLxot+a9fM9jGPqHoT6C0W9hyC+rdtgbpkpjQZzpZFAWZpHA3uuEOTzFYm+41HHmyAe+b+wSrGI6VhIdaWz90L7C55KzuDH0nZ3/fjjj0fkYrgJItPHtaV+Ok+q4IxhV21Ixw+6evYcOIwA7zcnTpz4RbXFgzdpzq/Ps19QT0kkn2HLc3O2PP1thGqB9+kjEaA97suLI3j6REVgjLmDif0WLuBzKsxh/BaY2iqFsa/ouO9SiKd4bqEgr7ooiKfR/hAIjnCkdLV1jtJL8/wW2uBdrMweIJ0mDFUFdEGGMIlcBSLrBILk5zkI3gcfG+aI39Co9HEJyXPyMgHmk8H8QT6voc+PIH3N72AvtNBCR5LfAeQVuzoK+LiBOtgxb5l8fI9AVgRoi1L0vpz4WiSWBO0UMX6sM378+ElZ6UXFcyLMg/ukj8HUIjHMaCb+HJ35AM4JXqmGYZ/WIyAEpExJB7iCr7q/nCU8R/u8CkFyNZGrFuBJGWp1SFtfl/z+RDxdeUsLbyJMlkqL1Oj3waTl3agBKYY37XY8yM7aiQxUrzeKf7Y916U+juVZKYGHb3l/COOT9Ht88Ag4R4B22JU2pl2oqHb4MQZ0BmNsSAaQKgpVC3PulG4Lc1eSe5Rymxi7CyWIfdDk/bwiDn0ij4CFAG1uH9rc+RmAkdC+HKF/NWdTL2aI7zwKvG4Or2dAeEAK8U/pJys2s8Y7kxRp4y+YAaTPKfOhCMarMsStWxTsC/RGj+dBMtSRTNzRSEvtlNQNPJ+RMwToR7quup1NkD5zN31m00ozqkqYM1CdDgMHR2XOquQlVkHScq35dlqlhffpWg8BOsI1cJ22JTpaW1q0Td3zlhnGhgf43g0mLuIp22YzmJOW60Z06EcazrDFAPw/xF86RkgK6uvHg/nxfGa1VFb3ogZC/S4y1tFIVBjLBHB5JoDSpfDBI+AcAfrTYRDVlU477Eb/uaySDCsW5jCjs3GdkdtBJux2bUfGMirB3aepAAHa3Jsk09WPuPAsg/BFDMI3VUC+5kmCbbYrEdgyrBIbmIisz0RYK8imCJz3HQFPJyUxo4kT141OYDv9g6ZgOgMT1Mc6rNQvjVHW/ZqdkrXYKfl3BlI+ikcgNwKylEi/sQW3bgoug2KsxrpcoSJhztlZPwZNXRuxL8uPo2P8loEovGSfixkf2SMQhwCCXO1tcMz7J+gAp9MBtHps+kAnPijYeo/jdRqT4Tl5WdOz/axAgb3OvqXBHxkoyy7sJuiorSUD9XERZdBVIjtMYzz7FeOZF+gtWbPNzzR96x643Mji9F36v65f5goVCXMYkLAeYOX0KAysnSt3H9kjkIJAr1695uzWrZuUrvpFRH2LQfgoBIk6REsF+tBvA+3uyOtdvPuQcuW+aucaBATd8fBydAxdXS/bkn4vwzEtHYKdxqsphO63m+FbVu/DuTb3XksX0DPflAhot46FyASY620xeAr96vA8TOcW5lFnZ8xe32P2OjRPxj6uRyALArQ3uR/8hRVX57HH0diPy0KjWeOwfb0EfUe7CVETFbHd8Aky+Ou6TJRhmNFsQ/+6LSm2BjuOsmchfxBm+BzLfb9oS5b7mrVPtEe+mDCvxYT5cavs3zC+5fJxkEuYx+zxf09DH+AbentshrUtM8LuToSdrd0pV4O/p6Hbjb+2zNSIOsJyOKQl0HtGZUH5ZS3uxBpln0iW/r4dg0yUo5j/gH+WK3eNYLvqPKmTbyME+iuUeYWqiXsCHoEIBGhzsqa4pvmKvr89ff+GrIDlEuZkKAfs
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="analytics-needs">Analytics needs</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>Billing</li>
|
|||
|
|
<li>Revenue</li>
|
|||
|
|
<li>Enterprise strategy</li>
|
|||
|
|
<li>KPIs</li>
|
|||
|
|
<li>Fraud detection</li>
|
|||
|
|
<li>Electrical consumption</li>
|
|||
|
|
<li>Metadata analysis (from JIRA)</li>
|
|||
|
|
<li>Work time detection of support teams</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="mix-of-workloads">Mix of workloads</h3>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><img src="data:image/gif;base64,R0lGODlh4AEOAff/AAwSBhEYBTlALyUyEUpSOlBZRVNeJGVtVXB7SAoEBRcLBRkKBRkSBhoTByoXCCoYDTEKBj4KBE4MBlkHBkElB1EmBTAlESckEkEnD0A4E2M3CVM2DlAnCD02JTsyFFocBE8NBWYuBnUnBF4fGGlFB3pbCXNXDWtkE3hEC3tODIhXDZFIDUxMIWVEF09NLVhLOk5CM1hXMGVgMnZTLVpTPmhbTGVXRl9aRmZFNGU4CnY8DoJHJ4pmCopvDHpnEZ6DDZR7KX53H5doCZBgDZlwC597CrB6Eq9sD6JvMbNqL6xhBXRxTW1tN4ZiP21jU2lnR21jVXFuVXVoWW9nW3+FRoJ/W3x6TpVxTnd0Unt2WX58WoR/Xn98Y4F5a4xwWX1cS59pPJRcPQABAiIkJ2tyc01KZSkNIzkmRndzemtobF1OdoSOfJKZipOeYLjGdbrKf7G8gKqzj8DTgMjgjczhjcHThb3RhMXXjcjXk8vWoc/imNjko8/TzOfv4rOXCrOjDrSWCsKNC72ZBrWXCcKUCc2WDsSkFM6RC9KYGN/KINXJIMy2IeGdHuWiJeG1MO2tKqqOEbKbH6KdS6SRPbGJELSSE8KMD7uJIrKBNcp1Ncp1PsKWK8uWKNuQKseSTtGERtaLTN6JR8B8GY2JZYqOVK6GYqOBXYiGZI2MbomGf52dabKQbKKVhLF7VcKMX9KGYMemVcLFdcezUtujKdOmO+GdJ+SjJ+alLuqfStaiUOCZS+egUuO1PPGvYq6wd8iie72/hNSqgtS1h7msncyXZdSjedWWcd2ofdmPa+6PMuPYH+PRJOPTJerXMenTQPrgQ/TcPPfdQvvgSO3eRP3iTP3hUPzqU/PoSPTaNuzNNPbbPPbaQPndR+3SH/3uVOrSUsfRgdDRf+jMgvjZZvzkZvLdavXdguXhdPTLV/rYYvfHeMXNic7YlcvLl+fImtvZpOHZy93MuPrrfe7k1Ozk1vTp4Pz7+vry5vLkvvO+k+fu79PV2pCJj8G8wf77/fTr7yH5BAAJAAAAIf8LTkVUU0NBUEUyLjADAQAAACwAAAAA4AEOAQAI/gCdCBw4sIaTGlKkONmCRaETKVgePlRIUaBDhxZrvAiwIMOLgyAN1hgJo4OFkzBeGExIseXEhA0hypTZEGFFJ1hqmLTwUSZIi0AlCr0YlKjAGlsUwrCwIACFlyxjzpwqlaXVq1izSq2aVSLRhBNfQmU51mpYsGhdpj2rVm3QowdFyo0bd6Tdu3jz6t2LV6XGkX4DAx6s8kVhw4Y1Fv7r9y/fx5Dxhpw8NzLfxnYPIz4oBSFCryMpvtjZoSdar0Uf2jz4gunJ17AXyLZQeuVVtgq3RsUacSDYFzBmv4B52qjx1F6j1lgqfOVE3cSjT81K/WUaslcbap869C1B79+h/oYfT768+e4ORVKuLJcv3ZGU4+Ml4Fiw4Pr4ETNevJjxYMjvySegenCpB9+AdRGWWX6MCXWUFI1N1Jpss9lU0Veo5SZVDbC9tkAHtSEUU0zXJffQddHphsWKCn0GVEI6yQbDZzddmOGNF7bYmmswmJXdbtJBh1F52K1FnVUo3mhRE6qxZGFCTN5kUWcwVilFlDZlaaWWVHbpYkHrwWdge3sZwJeZZN6Fpl6Y8edfDfQlphmcciqmYH+O5fVemnqKKR97Cd4V6Ej0LajYYZwZFKAUT3R2EoULnIQdauO1aBZcnO3G1W05Ujfibzz5dmFOwX04XKU2DhmeQsC5ZoGF/lNKcZ2QtF5q5ZE/0qRrQihK6RuTDzVxK5USRSnsVcbidmWwzC7rbLLOkmfggHYNumYNZmZb7UhmxnUtoXTC+RcBbt65H7nh1plnoY0VKtm2le1ZF4LTvsunf3WqFCe65AHbGXNNWZDBbB10hpOqq5bVVkvVpZohiltJxFRp3zGcEMABhPqcrw+x6JkTozFF4atllVUrrs5SeWyUp+GIJJAmXlQcS8dGixGT/iYnbJNFnsVWWE1+7ASTZAb6U7yWDVpmnuUKhu5++Lqpr33jYqaXvABuux7WCe6ZF9WIIUr00GBJZBKFGdSABXMWwEDQkA772B3QDPsc91gReSby/qtlG+WbRo9GSkHbbofXBGCGmRQBhQFkMKOiUyoL5HQprsXyixZH1Ommdwdr8c6T3voSy2n5iJWFXD4JrGp/+smeZXppG+iahcZpLrqL2S61fkyLi2d/VouJ9NaACu91e4PafqhGuifmG1ydhSycFEnpxOOByL39opJz3925UmdHqhJZqTFckxOPunrS4B7uHakFAdSmUsJtccXVdb0m6eKwpSucP69yw1CVZlC6GuyMgKKTCEIIiBsCekYKCOwSAyf4k5AYDHnF+5PS8PItbu1FeY0BG778A8IRSq13fSre1bT2uhZ2jU2+w51+DJMoYkmvKRQYDpRy0oHFLeAj/kWhUY0cRimZ4eiILVmKDz+Ele4lxyyA6QD7IpCx2ICoNPNLClkU9ZXNZaVXpaOZRkrDwC6ZjDcNq1sYZUY+sNiEgaDrVEIQOBaB0BFuL2Hgx+olqDHdC1uA1BYHPUhIA7DLhOlCpL4QWcI84Qdce8FaBjE4Jkm2cD+akVNhTrMcCkCKNjbZTmfO5pQO2KZYnHFQpZ6XMPQkR205mdDIhhOTobhElWUbC+QIJKquBA0jaoSYdGaWKLW1IFIzeNIXWZIGqUClIQfDTR1ZUkYnUTOXD6kmAun4Swc5kEoOfCAfX2jBA5kzJGeCHQx9Z64TulNBCnIXPN11tDAJr4/o/lxhXGjYmT8ijj9Te0FNOskRgo0vVyPpAIVgUMa30ehLQbRRxWxplVjKskKRUyXPlFQj8BTEiC3LyHEs9r+4XWVCr9ISA3+WK87NzEoRhEoEIdi3OYKlodSZoBmzmR4qKWp48aIL1x6zpqLek3bhYucJ0WU73TFoQSGs2l3uA9Q+ofOqfazWjiL1GMTEKWx2UgkMYJABT3LkJLX5lCidxDbajM+JEjWjT0OKKpApdGQ/7BjCQFMcimJzPDSypYOIJdLnqbGvEmWNa/zCqZIBcDpYmJWsTMdSYg6xoUVp4h0pMgM7lu2N2XyQ8e5pV78cT02FQmo6P4hIpR5mhCjE/szT2vRIe53WaJN0XfjEt0vFkGuGmEQMpGRDgdoQZ627gtBdI1WbBzUpoheBVWIH0oQ0JC5SeLUAFrYgnr+5Ja5PLJ8CiVjE1QATiRmK7kgosDgYHA51nRKLVsBouiPptCXVNO9EHqhHjbQARoV1EH+/hLR+vmc0kZoR8fIiz7wIUk3iKmG7oJavpjmmqYbqD30OmcLbAnWDI0lfU7iamRdwWJP5qsFs3GqbHyHXLABj7g4nJUSRDglYNmkCFt67FJHNJm2yNCWvtNfRVDUxQ4ENndyeG2C7HfEsgKvQsnjqhGqm1ytg3Fh9rwmW6tqUrxekKU0VOAML5NBgPAVn/qYcSJAadLaG9wxNyETmtn7G2S4NhnDsvvYfvzhVJW07IST/02dCv8lpo83n6wIkEgM/ZEJULM3eOpASQ5VYqmFz3q60vEzlLHcBFKh0Q2pGkSRjLj2saZWPX0PDhrxgZM4RLUhd1tjvxqyIdjM1ejGEviXyDZuqMqmSo8LpncoxORQcL0ukJ2S7ySxLVa6yTyljMDu/wJOQ2sBH7jxo1Oq5g+LqS4brg9ZKMxKeVI0qVTu8wkS7Li7hC1UTLsriQjMv04hh0XaS49LJTk4KMMD2iimtzLL5S5kkGdxwmdsjroCsoMbdHrGMzCnEqrHgL/pMjQn7ZKF45q60YUoO/skm7cA6Ob5oOdkayXLA6PaTLMmEweIs8N8rlVohne1MzmeQhp9eEM5XBXRTIpC+beP2LnkO956RntRHAs81pgQePO3dLnWDK3jzghefJqPAeAMRMD4O2IzaBNzD2KrYKMvKcgIHqbGadrAOAtFJlvi+QDdRLkGeX6ZO/sQ02hphOYJyR4nU5tYsrpQjm9GxnHteYSvssSvfqeC75KSdcDWXYi6sz7OUpcOVMyE3BGWpHvcYd5l+tXzm
|
|||
|
|
<div class="notes">
|
|||
|
|
<ul>
|
|||
|
|
<li>Online Transactional Processing (OLTP)</li>
|
|||
|
|
<li>Online Analytical Processing (OLAP)</li>
|
|||
|
|
</ul>
|
|||
|
|
<p><a href="https://giphy.com/gifs/trump-consequences-NTur7XlVDUdqM">Link to image</a></p>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h1 id="data-lake">Data Lake</h1>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<ul>
|
|||
|
|
<li>3 environnements (production, integration, sandbox)</li>
|
|||
|
|
<li>2 geographical regions (OVH Group, OVH US)</li>
|
|||
|
|
<li>900 TB</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="data-lake-1">Data Lake</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABE4AAAFNCAYAAAAekCqFAAAgAElEQVR4XuydB3wU1drGz+xuEhIIkNCTAElAEbFR9IoVe1dE7L171c92LdeOXntXLFevBbFTFHtFUVFEBWxYEEgCJBA6JKTu7nzPO9lZZje72dlks9lsnvkxzOzMqf+ZzDnzzHveoykuJEACJEACJEACJEACJEACJEACJEACJEACIQlo5EICJEACJEACJEACJEACJEACJEACJEACJBCaAIUT3hkkQAIkQAIkQAIkQAIkQAIkQAIkQAIkEIYAhRPeGiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRA4YT3AAmQAAmQAAmQAAmQAAmQAAmQAAmQAAlER4AWJ9HxYmgSIAESIAESIAESIAESIAESIAESIIEORIDCSQe62KwqCZAACZAACZAACZAACZAACZAACZBAdAQonETHi6FJgARIgARIgARIgARIgARIgARIgAQ6EAEKJx3oYrOqJEACJEACJEACJEACJEACJEACJEAC0RGgcBIdL4YmARIgARIgARIgARIgARIgARIgARLoQAQonHSgi82qkgAJkAAJkAAJkAAJkAAJkAAJkAAJREeAwkl0vBiaBEiABEiABEiABEiABEiABEiABEigAxGgcNKBLnZbVbV///5jNU17y5L/vGXLlo1qbnkGDhw4S9f1HZBGz+am0dx4AwYMuAJxb0N9TiopKfmwuelIPNRjBOpxD3ZHY5W/xQ9QpxNakibjkgAJkAAJkAAJkAAJkAAJkAAJxJYAhZPY8mzV1HyCwb6+TLZg+yfWyXjZfhJbd7SZQwSYgTiHIn6naONGEz4vL2+w0+kcK3EgFPwbm+J2LJxMQ/mPw3ob6jAhGg7WsPlYvF7vrziWDiafYFvmcDj+hBjzQHPTbGm8eN0P0ZYT5dqION+B96HRxmV4EiABEiABEiABEiABEiABEmgpAQonLSUYx/g+4WRHvGi/BouHrsh6f6y5+P3m8uXL5WU+qqUtXpSR52IUcmN7FU4gAuWC/dE1NTUvrVmzpjIq4JbAsMK5H+lcjWt3Eq7dG81NJ5bx2uJ+sFN+Cid2KDEMCZBAohJIJqvLeDAGr5/QPu5syeso9BneaypvtBPyUWM3fJAYvWLFitJ4lDNSHuiz7YE2/j2sj6OdvyVSeJ4nARIgARJIbAIUThL7+gSUziec9EQHYgc5AaOFTh6P51V0MI7FzxNxfEo01WmLF+X2LpxEwzdCJ+9rnN/eN9xIj1W6LUmnLe4HO+WlcGKHEsOQQPIToNVlwzVuy+Gq8bjL8Mw/A/2a3shLhrOejK0d4WQtwvWA5eZ+xcXFs1qrnOh37QJxZgHSvx7ttwy1DbvgOl2K8k9EXb6ENemY1ioT0yUBEiABEogPAQon8eEck1yChRNJNCcnp6fL5ZKvK5+hET9CjqHT8SI2Z2Ctw1qC9dXMzMy7Fi5cKL+l0/UdGvN/hCpUcAOPLz8lODYAYauw/ob9x9ABeKW5FYoknEQqu6/8jXycoJwXoWxPYR2H8hn+VAoLCwfU19c/jGMH4qcL69cYMnR5UVHRX9GUXwQqdJSqrXGkM4cvSK+bxyydqQuR32E4fzDOrcH2XoR7Kjg/1HMhjnlxzXYMce4RHLtcjiOt82SDdP6DbSq2ryK9y7Cv4zoW4PdS2ce6EftyXSfga9v31jTBZm+kcw2OiV8Z8Qsjw7rWok77I+xiu/eD9Ssg4m6DDurVSEc6tZux3oS6vAiLnENx/EPkdz6uw7NSDrDZHeHnYPdKhJG6GQuObwfh706EHYOfGVj/xv4jiPd8KOZBnIyhUpZwr+D3aWYY8K3B/kc4ZgwRs1N2S1y5hleBZQGOlWH/CZTp/qD8+ZMESCDOBGh12QA82YUTy7NYnukvYY0onOAZvyvanm3wrH61NW/LaISTwYMHp9XV1Z2JtmQm2u0lrVkupk0CJEACJND6BCictD7jmOUQSjiRxPGS+CVe7oaiwyBfaKRTdQkaankhF98l8rI8DOsjeIm8Us6jg3EiwvfD7rlYt8MqL9XmsgLhxOTVWJD2o9ikydAgpClDg/pgOxadgLebU7FIwkmksvvqF0o4eQ5lPAdrITgUIZwTef2G7WCsb2KVF+njsa6CiLSdKSLZrIMT5TLY+bie0YRwshDn/kIHToSC07G/E0SDf4iYASGnG/azJB1sZ2JTi3CHm2WAALZy8eLFtRAf9kU9RmJ9EOcew3oU9l9FWvtgf2+sB+MafVpQUDAQwsP1vvh9sT0Ea53b7d6hrKxsuRwX/zLIQ0Qa+VsXK5dSpFOH9DKR3wVLly7dZPd+AINTEX9PxP8n1geQhvjbeR/7wl0EvBx0EkfbEU58os98xO+G9VOsK5HG4UirF/bPRP1ewfX7P1/d7sa2GOvTJivk8R2+Kn5nVzixU3ZhIeFQhpeRj/gP+hLrrlhHWIUgswzckgAJxJcArS4beFM4ie99Z80tGuGk7UrJnEmABEiABFqDAIWT1qDaSmk2IZyIhcnpeNkUqwpvUPYaXkBlbPBBOC9f9f1OZKMdmoEOQ3e88P8uL5VIS0SUqJdIwomdsgd3GkWQgFgglherUC4RiaRjabwAYz0FIs9rvmNiCfKB9Vi0FUD5xyPO1CaEk9kog4gbIlrshBf8n7F7K47djrjiGFdEgJALwo4WMcB30oXw9djfAOb7Qnj5FXUajnxFbLgO6d0XnAjy2w1pzDXzk/MQRc7CS/8LiHcVODzcVH3t3A8II9Y7nyK9nyHa7AGBpgp5XIk8HhKBCOey7QgnSOcZhD1fLGogdD0n5erbt2+vtLS0CbAKukFEDLOsTQ3VsSucSFqRyi7iFsL8jaDVqampu4qIhX0R4L4SYQjlHBLt/cLwJEACsSNAq8sGlqGEk9a0uvS1JU1an5qCAp6VYiUivj1Ox1YsVZ/2OVI3hqT62plh0oYi7GQcEsu+efh9Edoo+djhXxC2SYsT08LRGgftTz+0o6usx3x9BLHalGHN8vFHhPpX8Jy/zfecl7bybBx7Huc9WMuxfoK0rjfTCpVXUL7+IUIhrDiFwUXBfwnI82DkKX0CGX69DuvLvXr1unHevHnS9otVpjksqElLVpQtHencgPUYH890bMUK9TUwNT9ABGfP3yRAAiRAAlESoHASJbC2DB5OOEHja1hbwJIizWpJkZub2yM9Pb0eVgBnodyPyvAKGZph1sHOi7IvrAPWDb3wQruxtrZWHNPuhk5AXnNY2BVOmiq7pdPYF3XfToZ3oCwHoJNwNjoJk6RcyEdEggtTUlLyq6urDTEJ9dfQUVqG3ftQ/lubWf5Iwsk9SNuwAvEJTRtQvqfw0n0xOjfbYN8QdrB9HJt6lNmwZPEd+xJxN/h+GsIJwv2CuIaTvJEjR6asW7dOOpzlOLbSjDds2LDULVu2ZKODtwZx1uC4TGtsDFtBnv8Q6wzsLkFa/8X2c8SV8dmN/KrYuR9M8QHxH0MexnAiHMtCHgMheixBOfa0KZwsQtRspCEWJk36eIm1cBKu7BjW5YAAJ53NOyAKiaWVscjwLrC7Hh3adLNDa57jlgRIIH4EaHXZwDqMcNKaVpcRrU8twsnreIa6sX6Oooq15LFipYi2WdofUzjZU9oprGZbJO1xJZ6xg6zP2EjCicxOh7SlTRZrUBmWKe1PgHAi7SYcuX+Dc2I9uADl+Qlhd8f+UPmogPbwHF+5DsJWnOw7cX4wzo3B/o9ooySsx2fhKXURB/EyTFU+SInlqLGgzZhWWlq6wnd9TsE2B2HlY9VtWBsJJ7iGQ3H+F5yrwvY9pClWwuL8/wGwMqyALcJJWEtWX37PS/8H+8VYf8B+BdJzYH8Oyi8fKbiQAAmQAAnEgACFkxhAjFcSTQgnH6CR3BsNZCbKIl/I5QvGBVhlGIQs4ttE/GPsaP2iE+lFGY12X4gNExFXvmKkYJUXXLFY2exzahp11SMIJ7bK7uNgTstslAH1/wgdoMPMAiEfcZQrQ3MaLeAgHu6b9RUmksUJynEzynGHZGoKJ9ht1GlCOn+iHDUoxy5h
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h1 id="extract-transform-load-etl">Extract Transform Load (ETL)</h1>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><strong>Extract</strong> data from a source</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><strong>Transform</strong> data to an expected format</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><strong>Load</strong> data into a destination</p>
|
|||
|
|
<hr />
|
|||
|
|
<h2 id="sqoop">Sqoop</h2>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA6cAAAFPCAYAAACxlLi2AAAgAElEQVR4XuydB3wUxfvGjwRC7x0SSADpShOwIcWKKBasiIgKFkDsIoiIWLEXxAaKoigqKIoISrXQIyBNIIRAQgkBAgQICSn/573f7f03x5Xdu72ye89+PpPd2536ncnuPjsz75SycSMBEiABEiABEiABEiABEiABEiCBMBMoFeb0mTwJkAAJkAAJkAAJkAAJkAAJkAAJ2ChO2QhIgARIgARIgARIgARIgARIgATCToDiNOxVwAyQAAmQAAmQAAmQAAmQAAmQAAlQnLINkAAJkAAJkAAJkAAJkAAJkAAJhJ0AxWnYq4AZIAESIAESIAESIAESIAESIAESoDhlGyABEiABEiABEiABEiABEiABEgg7AYrTsFcBM0ACJEACJEACJEACJEACJEACJEBxyjZAAiRAAiRAAiRAAiRAAiRAAiQQdgIUp2GvAmaABEiABEiABEiABEiABEiABEiA4pRtgARIgARIgARIgARIgARIgARIIOwEKE7DXgXMAAmQAAmQAAmQAAmQAAmQAAmQAMUp2wAJkAAJkAAJkAAJkAAJkAAJOAgkJib2KCoqWlyqVKkHd+3aNZFgQkeA4jR0rJkSCZAACZAACZAACZAACUQsAUWUOTJYiP2B4uJiEWkv7t69e7PejCO+9hB5axFuFMK/oje8Hv+qvN+EtL7XE9bVL8VpIPQCC0txGhg/hiYBEiABEiABEiABEiABSxBQibJ5EKVpKFQzuF5wuRCol6EXcbmeglKc6qFFv0KA4pTtgARIgARIgARIgARIgARIwOau9zEhIeFiCNP5wJOKHsk2ejBRnOqhRb8Up2wDJEACJEACJEACJEACJEACdgKehsY2btz4TfSkPgLXJT09fTV+J+E4FUGK4Y7geAXcuIyMjFUST3x8/JUxMTG/esKKaz3T0tKWyHWI37sgfj/FoQwjzoT7DddH4fp+PdWidVivr7yrObjOOW3UqNF+nMtED3I7JW84dx/OPYryJ+HcXhy/j+uv6ck7/f4/AfacsjWQAAmQAAmQAAmQAAmQAAl4E6c9Ib4WwQ2DOJ2UlJTUuLCwcJQDWT3sr4DLLygoaLt37950x/Xr4b8hxNrjuDYHbqGCGGG/37NnT4b8hri7DLt+cLHw3wz+e+B4DXppz8NeBKumTas49ZV3SczdnNMGDRoklC5dejcuf4G83Sn+IHRvR56/xOF/cEvhOsN1RBmGQKBO1pRxeipBgOKUDYIESIAESIAESIAESIAESMCbOFV6SsdDmD3rigo9pV3Q27kS55/F9fHKdX+G9UKsPonwEyD6ukMI/6G1WrSKUy15dydOka+xCPsc3M0o43cSD85txy43Li6uc0pKSh6OY3HuD4jTWhCnLbTmnf7+nwDFKVsDCZAACZAACZAACZAACZCAR3Gq6jV8GcJstIKqTZs2cSdOnKiBIbhZEGVZOD8X1wf4I05r165dqUyZMuXKli3bED2r6/T2PuoVp97yrhanEMlfoTw3wr0Ll4HytcK+oEmTJlXRUyxDml9Aft9RyhwbG/sQ8j4K5SmfnJx8ms1KHwGKU3286JsESIAESIAESIAESIAELEnAk8BTekZliC56BN+Av/MgyN7C764AIXqiAC4WbhbEmwg5+6al5xSiVobISm+szNmULR8uTu8ao1rFqZa8uyyp46xr9A6fDyG+Qk4g362x2+SpIYBPLQxdPmTJhhLEQlGcBhEuoyYBEiABEiABEiABEiABsxDwJPBUQ1ovRW/hcsy9TEeZZA3UdyHYxHBRBRx/gP1vanGKcGLddyOE5liI2uddOYhQxDqoy8QaMMJ/DXcMflrLuqrBEKfoAa6gJe8qDn8hP8nIVwLycwOOR6N8L0s5MN+0Ps6LAaQfsX/ftWzwtxjnNM+ZNUsbCXY+KU6DTZjxkwAJkAAJkAAJkAAJkIAJCLgTpxCYnZD1RXDHILiSIMrOgRhLhii7B4JTrOxKL2IT7FLgSvScNmvWrEp+fv5RnJ+PsFe6IoCl3kcRzxs43xTXxfqv03pvMMQp8t5RS95d55y2aNGicm5urpSvPIYeN9uxY8cBR7m3YF8LcZ6H+bE7TFDFEZ9FitOIryJmkARIgARIgARIgARIgASCT0AlyuZBcKUhRRlqewkcOjiLrsVSMfMaNmxYE/MqxdLuHjixSBsHdzf8b4agbI/j3/Ly8oZlZmaecAi437G/FG4Nrm+AP+ll/RpibjbEYh8cz4Fbgus/oxc2HscyVLgx/B7C8Q8QreO0lFydd+T1DKGIuMbCldKS9/Lly3dGHIvVAhl5HY7w7yEvHyFP9zvKdjX2s+FO4dqP8C9cquJ4Bco3VUu+6ackAYpTtggSIAESIAESIAESIAESIAG1QSShUQQnRo7+hHsFgixZQYQez74QYi/g91lwMqxXLNleBHev+MHQ39pYUuagHEPMxkMQvo3DXiLc4HLEP+ITA0PSUyoGlmSt0DrYyxxOOZ4OgddcfsNfWy1V42meqBIWwjcJ80XTtOQd1nfbuorTTp06lcnKypKe0kS4dsiXfb6pI10pg4jqKnB5yPtEiFNZQoebTgIUpzqB0TsJkAAJkAAJkAAJkAAJkAAJkIDxBChOjWfKGEmABEiABEiABEiABEiABEiABHQSoDjVCYzeSYAESIAESIAESIAESIAESIAEjCdAcWo8U8ZIAiRAAiRAAiRAAiRAAiRAAiSgkwDFqU5g9E4CJEACJEACJEACJEACJEACJGA8AYpT45kyRhIgARIgARIgARIgARIgARIgAZ0EKE51AqN3EiABEiABEiABEiABEiABEiAB4wlQnBrPlDGSAAmQAAmQAAmQAAmQAAmQAAnoJEBxqhMYvZMACZAACZAACZAACZAACZAACRhPgOLUeKaMkQRIgARIgARIgARIgARIgARIQCcBilOdwOidBEiABEiABEiABEiABEiABEjAeAIUp8YzDSjGRo0aPYwI3oLrtnv37r8CioyBSYAESIAESIAESIAESIAESMAkBKJWnCYmJvYoKipa7KinQuwPFBcXLy5VqtSLEIWb9dYf4muP+NYi3CiEf0VveMU/xam/5BiOBEiABEiABEiABEiABEjAzASiXpxCjM6DKE1DJTaD6wWXi3OX7dq1a7meiqU41UOLfkmABEiABEiABEiABEiABEigJIGoF6fAcRN6Or8XLAkJCRdDmM7HYSrOtdHTWChO9dCiXxIgARIgARIgARIgARIgARKgOLUTUA3rdYpTOd+4ceM30ZP6CFyX9PT01fidhONUXCqGO4LjFXDjMjIyVon/+Pj4K2NiYn711LBwrWdaWtoSh/i9C+L3UxzLMOJMuN9wfRSu71fCq4b1DsI5mX/aCi4FaY5Gfn5S+fscxwPh8uF2wU2vXLnyS5s2bZLfzg35vxth73bEUxX7E3DbIb7PVTyBRTn4eQnuNpyrAbcZ+XwavcdzFT8oZ3mcGw13Lc4lwZV38Pga+XrQU/l5ngRIgARIgARIgARIgARIgAS0EGDPqarnVIBBzPWESFsENwyia1JSUlLjwsLCUQ6Y9bC/Ai6/oKCg7d69e9Md16+H/4YQbo/j2hy4hQp8hP1+z549GfIbwvMy7PrBxcJ/M/jvgeM1EIrnYS+CVfzYDSLh+npcT8PxTrib4eriXHvkaaMjn8Pw+2wcl4MToSk9vW8jrkeUtBGXiFcRsUfg/oQ7hDBFiDcH/iQd+wZ/n2A3GNeW4No2HF8FVx/CuS2E83+O9D7F9btwLHlajeMc+I3B8XLE9bESF/ckQAIkQAIkQAIkQAIkQAIk4A8BitMzxanSUzoeoutZV6joQewC0bYS55/F9fHKdX+G9UIUPonwEyD0ukN0/uEQioo4naj0SMJfa1zbCDE4Fb2Z0gvqupWCHxHFlyFPFbAvcMQ1Ffs7Ea4jwomxpjM2h7jegQtTEPY+8YBzdSGqRaR+o5xD/Gn4Xal27dr1k5OTT/vT2BiGBEiABEiABEiABEiABEiABDwRoDh1EacNGjRI
|
|||
|
|
<div class="notes">
|
|||
|
|
<p>“mózg” = “brain” in polish</p>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="sqoop-1">Sqoop</h3>
|
|||
|
|
<div class="r-stack">
|
|||
|
|
<div class="fragment">
|
|||
|
|
<ul>
|
|||
|
|
<li><strong>Extract</strong> with <code>SELECT * FROM... WHERE id BETWEEN...</code></li>
|
|||
|
|
<li><strong>Load</strong> into Hive</li>
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
<div class="fragment">
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABDwAAAGbCAYAAAAoS4ASAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAASegAAEnoBeu9ThgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N3Xc1x5eub57++cdCe9gydBEvRlWLa7qr1Tl6TRzM6MVrOKudjL/b/2Yi82NmJDuzdqadU9O9sm1C2pu9oWXdGT8EhvTrpz9uI9h0hmASQAAiTIej8RCJAsIJEJJBl1nnx/z2t830cppZRSSimllFKvJ2OMASzADt7HgIQDCR+MDb02jAAPcIFe8Gvff4VDA/MK33ellFJKKaWUUkrtYCzkiALZGEzHYMpAKgEpB7IO5A04LvRG0BxA24VmD1Y82OpBG6gDXcB71cIPDTyUUkoppZRSSqljIggqwrfxC/Y9T1sYYyJAMQknIzCThXNpeCMHc1HI2ZBNQDwNdhzsHlgdGA6g3YdWA3otWOvDn9pwvQmf9+E+0PR9f3T4j/poaOChlFJKKaWUUkq9BBNTGDEgDsRjkIlC3MDQDqYr6vK+hRw58Xzf93a4rRiQT8FcHt4twMdpuFiEhTIUc5BKgZ2UYywGMBaPkxW/JzfuuzBqgrsOWxW4X4OrG/DPbfh9Gx4Cdd/3hy/q+3RQGngopZRSSimllFIvSDB9EUWOlqQtKERgJgrlOBTikE9CPg6ZAaT64Hqw0oeVFqwM4X4f1rpQBTq+73vBbZbTcDELH6fg63OwdBJm5yBTgEhCpjmIjN+X4P1kKjAEBuB3wduC4SrU78LqGlyvw9/X4dd9uBN+/SP+lh2YBh5KKaWUUkoppdQRMcZYSM4QBXI5mLNgLgknHTgfl+Mm6Qw4ccgnIJ2ESBSiEYgMwG9DqwOVBrQ3odaE33Thl0242oNKHKbK8PUp+MEcvH8STk5DKgNWiu2m0tB4CjB5bmYyBBkhRR41GC1D6w5cXYafVuAft+B3QPW4hh4aeCillFJKKaWUUodkrIPDAlJxmDYwn4TpNLyRgbfzUCxAvgDTOcg7EIuBccBEwRiwrOAGfGAox0y8Hvgt6D+ArRX44xb88xasZuDtE/C1czLVkS3JRMeORSD7fjzBew8JPlagdxeWb8MvH8L/vQH/HVg/joWmkWd/iFJKKaWUUkoppXYThBwgUxz5GEzbkEvDiQK8k4NzKZgvwMI0lLIQd8BOgxUHE2M73PD5YlBh5Le2DwwgMgeJNSjchbOb0J2GmXNQLEMsze7THAcRfr4FpIFTUnZ6KgVZH6YGkKjBPxljVo/bpIcGHkoppZRSSiml1D6MBRw24DiQH0EmDvk8XMjCOwm4kISTM1CagmwO4nmwHLCjwSeOBRpPvIeJ9SxshyBRIA9WEtIFODcALw1WGqzoDp97WMLbjAGz8jiKUfjYh3gcIg0JPVZ83x8cwZc/EA08lFJKKaWUUkqpHYxtUQnFAAdIxiFhQzYNi1m4GIW3bFiagvw8FMqQL0LUATsBJjJ2Q2bi6+wloJjs2YgjwYMX3KzF0QQdO7GAPJgLkDbw3lUYPAB7HX5sjHlwXEIPDTyUUkoppZRSSikeF4yGwxTxJOQMlD0JOYhD0oH5FJyw4ZyBpQJk5iBThKkSZJIQccAkpI/j8Y3BdiBxWMGEz5NHYQ7iIB0f4cRJFjgPaRs+sIE+RFfhR8aYe77vjw54lw6NBh5KKaWUUkoppb6UxgIOC0g4UDAwZUMyArkMLOXgfBTmB/LnTgFMAZIFKAaFo5GEFI5aDnJU5agCjp3s97Z3WkX7vKHHWcja8MEQYgPwXPg7Y8zayy4y1cBDKaWUUkoppdRrb2J7igUkE1A0ULZlJWwpD+cycN6CBVtWveZKkMtCOiurYu1gxatJgomPTXFMdm4cF+PHZ0ZAV948D/zgjWjQARLnyfM7e739DJglyI3g3ToM1uB+G35hjKm/zNBDAw+llFJKKaWUUq+diYDDRlbETllQsMFJwEweLmTgrAWzUZiaglwZ8llI5mRyw4qCFUPKQsNxEDheoca4ne5fGHTUYbgK7U1Y24JOF0YOjILHPTcnkyt27Is1I8/8mmlgEdKX4c0R/Ic2bHbht0DvUB7YAWjgoZRSSimllFLqlTdWMBouM8nGZHojZ4GTgRN5uJiEMwZmHShOQa4I+RwksnKhb8WDSYfwaMpOx1MOUjp6FHYLXzygD/TAb4HXBm8DunfhYQ2uufCvLbjXgmoc0nm46MB7ZXj7bZhfgFQazH6mPQyQB85Cvg5frcHNLjwyxjx6WVMeGngopZRSSimllHrlTBaMOpC3pGA0HZUjKKdycDEBix7MZaAwA7kCZPMQT8vkhh0DE+PpAcek4xRw+GwHHF0JN/we+F3o16BWgY06VDdheQt+1oY/+PDIhYZ8CvYG/DoBP1+Db7Thz9+Gt8/Kil2z1+0vPvI9nIHYeTi/CX8xhJubUANah/dd2DsNPJRSSimllFJKHXuTBaMJKFpQ8iDlQD4sGI3BCQtmsjLBkS1AJgex9PbxFBPjyeMp4x0cx+moym4THCOeCDi8HvhN6NdhqwbrdejWoNmHW1243oEHfXjQgYdAHRj4vu8bY0zwvu3Cigv3PVix4G8t+NoFWbu7Lw5wElIr8HYdPnHhtjHmhu/7/ef7buyfBh5KKaWUUkoppY6Vsf4NgwwOJONQsiXkSMWgVICzQcHoiQhMFSA/JdMb6Zyshg2PqHyhfwOOZ8noTttSfGCIHE/pyOSG1wW/DZ2qBBwbNei2oNqHmy05SrJsYLUHFVemKzpIRjIaP14S/jp47xpjHmzAT2IQsyCfhiunwYnt4zFYQBbMOShW4es9+FMLai/jaIsGHkoppZRSSimlXqqJ/g0bSMehbEHehlQCpsOCURsWIrJBJVOGXA5SWbATMr1hxZEL3fAGQz7HJ9gI7XZEZQD0JeDwOuC74DWgXYWNBlS2oNuBjQFcb8PnPVgewroHte52wDH0fX+4n/vj+/7QGLO8CT8BFvJQzsOpMkT22ufhI9//GYiehdM1+O6W9IZs8IILTDXwUEoppZRSSin1Qk0EHBGkYHTKlh4OJwPzebiUhDM+zCagNC2bRLJ5SGaCctHYHgpGj4vdpktGSMARTHB4XXk/bEKzAus1qNUk4FgZwLU23HVhGVhvSQ9HGHCMfN8fPe/9DEKPh3X4fx/A5VkoJeXI0J6/nz5ytGUBMo/gzTW4sgJ/REKPF0YDD6WUUkoppZRSR2qyYDQJOQNTI8jakCrAYlYKRk/7MDNWMJoL+jfsuHRwmATbScmrEHBMbncZ69/wO0H/RlsKRus1CTjqDVkZ+7ALV7twfwArfdjoQRNoAy5y0sU/omMivRZcXYN/ugNLU3AlCRGbvX+PbSAPkQU4cRc+bsJPjDHV/U6dPA8NPJRSSimllFJKHaog4AhziYQDBSMbVFJRyJfgTA4u2LCIHFfJT0MuLxMc0dT29IaJ8uTxlFepYDTs3xjfoNKX970qVKtSMNpqQsuF21243oX7I1jvwWZPpjfayFGQ4YvqwAiKTCsu/GYZrq/A2WnIpfZxGwZIAKUg0NqA8y24j0ylvBAaeCillFJKKaWUOrDdCkYtKPngpKCUh3MZOG/DPDBTgtwMZHOQyUIkKdMbdgIpGN2pf2OnX79Me+jfeDzB0YJuBTZrsiLWbUB1ADdcKRl9NIQ1D6pBwWgruJnRiy75nDCow704/PQ+vHNGulT23OUBj6c87JMwtwEftuD3xpjmi3pcGngopZRSSimllNqXsQkOG0iH/Ru2rIudzcPFDJy1YDYK5TLky5DLQjIHkbB/I4FclI5vUDnO0xuwc8DhBttTwoLRJrS3YL0KW03oteU4yrUm3BrCqgWrrhSMtpG3IS8/4HhCMOXR6MBnVbi5DidykE2yt59NeIQnBWYOZlLwQQz+vidTHoMjvfMBDTyUUkoppZRSSu2ZMSaRgPkRTMcgk5FtHhccOAPMOVIwmi1JwJEINqjYUTDjG1RehYBj8j55yPGUiYLRUQMawfGUahV6Ljxy
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="etls-limitations">ETL’s limitations</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>Mix of workloads</li>
|
|||
|
|
<li>Direct acces to the databases</li>
|
|||
|
|
<li>Lagging data</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h1 id="change-data-capture-cdc">Change Data Capture (CDC)</h1>
|
|||
|
|
<p>Detect changes happening in databases</p>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="triggers">Triggers</h3>
|
|||
|
|
<p>Execute <strong>actions</strong> on <strong>events</strong></p>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<ul>
|
|||
|
|
<li>Capture/log/audit tables</li>
|
|||
|
|
<li>Fonction inserting <code>OLD</code> and <code>NEW</code> values</li>
|
|||
|
|
<li><code>AFTER INSERT OR UPDATE OR DELETE</code> trigger</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="triggers-1">Triggers</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>Impact on write performance</li>
|
|||
|
|
<li>Relations visible to the users</li>
|
|||
|
|
<li>Retention policy managed by users themselves</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="listennotify">LISTEN/NOTIFY</h3>
|
|||
|
|
<p><strong>Publication</strong> and <strong>subscription</strong> system (pub/sub)</p>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<pre><code>NOTIFY channel [ , payload ]
|
|||
|
|
LISTEN channel</code></pre>
|
|||
|
|
<div class="notes">
|
|||
|
|
<ul>
|
|||
|
|
<li>https://www.postgresql.org/docs/current/sql-notify.html</li>
|
|||
|
|
<li>https://www.postgresql.org/docs/current/sql-listen.html</li>
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="listennotify-1">LISTEN/NOTIFY</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>Impact on write performance</li>
|
|||
|
|
<li>Lost events when disconnected</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="logical-replication">Logical replication</h3>
|
|||
|
|
<p>Decode events based on Write-Ahead Logs (WAL)</p>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<ul>
|
|||
|
|
<li><code>CREATE PUBLICATION</code></li>
|
|||
|
|
<li><code>CREATE SUBSCRIPTION</code></li>
|
|||
|
|
<li>Replication slot</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="debezium">Debezium</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABLwAAAGqCAYAAAABaF0bAAAgAElEQVR4XuydB3gc1dWGV9WWjRu2sXGTZMvYYFpwANNN7xBIgEAKpiSElgQIgZDQe4cAIRAgOPTee+/FYEpsjG3ZktxwkXuRrfp/R/+uslptmd2dXe3OvvM895ndmTv3nvPO3dmZb+49N8/HAgEIQAACEIAABCAAAQhAAAIQgAAEIAABDxHI85AvuAIBCEAAAhCAAAQgAAEIQAACEIAABCAAAR+CF40AAhCAAAQgAAEIQAACEIAABCAAAQhAwFMEELw8dTpxBgIQgAAEIAABCEAAAhCAAAQgAAEIQADBizYAAQhAAAIQgAAEIAABCEAAAhCAAAQg4CkCCF6eOp04AwEIQAACEIAABCAAAQhAAAIQgAAEIIDgRRuAAAQgAAEIQAACEIAABCAAAQhAAAIQ8BQBBC9PnU6cgQAEIAABCEAAAhCAAAQgAAEIQAACEEDwog1AAAIQgAAEIAABCEAAAhCAAAQgAAEIeIoAgpenTifOQAACEIAABCAAAQhAAAIQgAAEIAABCCB40QYgAAEIQAACEIAABCAAAQhAAAIQgAAEPEUAwctTpxNnIAABCEAAAhCAAAQgAAEIQAACEIAABBC8aAMQgAAEIAABCEAAAhCAAAQgAAEIQAACniKA4OWp04kzEIAABCAAAQhAAAIQgAAEIAABCEAAAghetAEIQAACEIAABCAAAQhAAAIQgAAEIAABTxFA8PLU6cQZCEAAAhCAAAQgAAEIQAACEIAABCAAAQQv2gAEIAABCEAAAp4lMGzYsL/IuataWlo+nTt37k76fqm+X6T05pw5c/YN5/iQIUP2yM/PP037dlHqr7RG6au8vLw7a2pqnrJjysrKtm1ubv4q6Ph6fV6s9Knqukd1vRYBaoFs+KX2WdpWqZdSrdL7Kv9ald9WZmlp6U4q62Mrp7i4uGtDQ8PO+v62vjbK9iLPnjQcgwAEIAABCEAAAi4QQPByASJFQAACEIAABCCQmQSGDh16loSkm5Tek5g0PiCAydqXJBodEmq19l+nbef6tzdpvUSpt1JX/7bzddy1QYKX5TFBrFipJFCehKn7JHr9Rt+bA9tGjRrVo66u7ll938u/bYPWy5Q2USpQalTaU+V/aPsleP1I5Uy2z9qWr+/j/ALYWn3fKDOJYxUEIAABCEAAAhDIDAIIXplxHrACAhCAAAQgAIEUEJDgdarErn+o6NckEh2g72fr+40Sjp6WIPXT4ColdplAdbdSi/Jc1dTUdNO8efOWVVRUdKmvrz9V28/R9v0knE0LCF76/pm+j7Ny1DNssHqG/VYf/6pUoH1/0r4bA3Wo7oe07Th9X6l0pnptPV5ZWblh0KBB/QoLC2/QtuElJSUHT58+fbUdI4Frc9n5nT5ukO1d9X07ff9S35fpe98U4KJICEAAAhCAAAQg4BkCCF6eOZU4AgEIQAACEIBAKAGJTCdIZLpP25+TSPQTfT9N3++QcPSwBK9fBPKPHTu2aMmSJTX6vqn2/VX7rgotS8JUtwULFqyz7eEEr0B+CVOnq4zb9b1WotYQE7Ukhm0lMexbbWvWvvEq/4OQ8vNUZpfq6ur1QeWUK+9sfV8p23tLkNtCn6cqLdD3wZxtCEAAAhCAAAQgAIHIBBC8aB0QgAAEIAABCHiWgASuYyVwPSwHH5NI9HOJUSdZjC1t+7d6X50YJC7tpe1vmUjVo0ePwVOnTrWYXBGXaIKXDsqXODVHaxOl9lG9b8mOy1Tnhfr+gr4f5gS4bDXxbYHyLtIxA1XGCJVRqe9V+j7cSRnkgQAEIAABCEAAArlKAMErV888fkMAAhCAAARygIBEoyNs+KJcnSiRaII/YPwD+v5Pfbdhiq2L8v1J+a4PN9QxHKYYgpdP9Tyo434hgepcCWs36PuL+n6w0u9V721O0OuYPspnMb5qdEzZ4MGDhxQUFMzV9+/1fXMnZZAHAhCAAAQgAAEI5CoBBK9cPfP4DQEIQAACEMgBAhKJNlN8rF9rRsVJGkb4nHpJbamhhT+X659IiHopgEDi0jX6fJ7SbRKTfh8LjQPBy4ZE2gyR16q88yWo2eyNO0oA+6nqNQEu5mLDLGtray/WcbUq45YBAwZ079q1q5W5UGXYkEkWCEAAAhCAAAQgAIEIBBC8aBoQgAAEIAABCOQ8gYDgJXHpDgljZ8QC4kDwulRlXCSB62aJU2cHBC9tO0ri1ZOxymc/BCAAAQhAAAIQgEByBBC8kuPH0RCAAAQgAAEIeICABKlzJXZdJ1ccxdhyIHjdpbJ+K8HrIglel0tQs95kB6mOcySo3eQBZLgAAQhAAAIQgAAEMpoAgldGnx6MgwAEIAABCEAgHQQ01HE/iVOvqa4VGv44aN68eXXR6nUgeH2n4zcPDGGU4BUY4viaengdkA6fqAMCEIAABCAAAQjkMgEEr1w++/gOAQhAAAIQgEArgSFDhpQottd8feyjXliXqxfWRYkKXhK3jtaxjynVSTwbIvFsmQSycfr8iba1KO0n0etN0EMAAhCAAAQgAAEIpI4Aglfq2FIyBCAAAQhAAAJZREBClQWtt+D1PovlpRkRb6jWYt/9AeR3Kyoq+qKysnJVuB5e5eXlA5qamo5X9suUuqiMGyScnRtAoGGTr2ib9e5ap3S+BLYHVPwK2z98+PBe9fX1O0kcezWLkGEqBCAAAQhAAAIQyFgCCF4Ze2owDAIQgAAEIACBNBPI19DGuzUM8aSgepfr8wal/koF2nekYnI9ExC8tK1JaY1SsVJJ0HHPFRcXHyNxzI5tXQYOHNhf2yyW1/b+Tc1a11q5Sn2VWiSY9Z8/f/7SNPtNdRCAAAQgAAEIQMBzBBC8PHdKcQgCEIAABCAAgWQI+ON5naTeWOMkcA1QWQ1aL9D6I6XbJXhNDhK8AlU1+MWrL7T+j4YsPmUCVqgd/p5ix6tsG/a4jVIfpdVKc7TtTQli18+aNWtxMvZzLAQgAAEIQAACEICAz4fgRSuAAAQgAAEIQAACEIAABCAAAQhAAAIQ8BQBBC9PnU6cgQAEIAABCEAAAhCAAAQgAAEIQAACEEDwog1AAAIQgAAEIAABCEAAAhCAAAQgAAEIeIoAgpenTifOQAACEIAABCAAAQhAAAIQgAAEIAABCCB40QYgAAEIQAACEIAABCAAAQhAAAIQgAAEPEUAwctTpxNnIAABCEAAAhCAAAQgAAEIQAACEIAABBC8aAMQgAAEIAABCEAAAhCAAAQgAAEIQAACniKA4OWp04kzEIAABCAAAQhAAAIQgAAEIAABCEAAAghetAEIQAACEIAABCAAAQhAAAIQgAAEIAABTxFA8PLU6cQZCEAAAhCAAAQgAAEIQAACEIAABCAAAQQv2gAEIAABCEAAAhCAAAQgAAEIQAACEICApwggeHnqdOIMBCAAAQhAAAIQgAAEIAABCEAAAhCAAIIXbQACEIAABCAAAQhAAAIQgAAEIAABCEDAUwQQvDx1OnEGAhCAAAQgAAEIQAACEIAABCAAAQhAAMGLNgABCEAAAhCAAAQgAAEIQAACEIAABCDgKQIIXp46nTgDAQhAAAIQgAAEIAABCEAAAhCAAAQggOBFG4AABCAAAQhAAAIQgAAEIAABCEAAAhDwFAEEL0+dTpyBAAQgAAEIQAACEIAABCAAAQhAAAIQQPCiDUAAAhCAAAQgAAEIQAACEIAABCAAAQh4igCCl6dOJ85AAAIQgAAEIAABCEAAAhCAAAQgAAEIIHjRBiAAAQhAAAIQgAAEIAABCEAAAhCAAAQ8RQDBy1OnE2cgAAEIQAACEIAABCAAAQhAAAIQgAAEELxoAxCAAAQgAAEIQAACEIAABCAAAQhAAAKeIoDg5anTiTMQgAAEIAABCEAAAhCAAAQgAAEIQAACCF60AQhAAAIQgAAEIAABCEAAAhCAAAQgAAFPEUDw8tTpxBkIQAACEIAABCAAAQhAAAIQgAAEIAABBC/aAAQgAAEIQAACEIAABCAAAQhAAAIQgICnCCB4eep04gwEIAAB
|
|||
|
|
<hr />
|
|||
|
|
<h2 id="debezium-1">Debezium</h2>
|
|||
|
|
<ul>
|
|||
|
|
<li>Source <strong>Kafka Connect</strong> connector</li>
|
|||
|
|
<li>Compatible with <strong>PostgreSQL</strong> and <strong>many RDBMS</strong></li>
|
|||
|
|
<li><strong>Real-time</strong> thanks to logical replication</li>
|
|||
|
|
<li><strong>Standard</strong> message format with Apache Avro</li>
|
|||
|
|
<li><a href="https://github.com/debezium/debezium">Open source</a> (Apache License, Version 2.0)</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA5kAAAHGCAYAAAD+EU8tAAAgAElEQVR4XuydB7gVxfnG76UXsQIiWKg2sGJPUKyJPdEYe4zdqNG/JYk1Ytdo7DXGklijURN7F3sLKgoqCgIiSLEjncv9/97Nzsmcc0/Zc+/uuXvu/eZ55jl7dmenvDM7M+/MN99XW2POEDAEDAFDwBAwBAwBQ8AQMAQMAUPAEIgJgdqY4rFoDAFDwBAwBAwBQ8AQMAQMAUPAEDAEDIEaI5nWCAwBQ8AQMAQMAUPAEDAEDAFDwBAwBGJDwEhmbFBaRIaAIWAIGAKGgCFgCBgChoAhYAgYAkYyrQ0YAoaAIWAIGAKGgCFgCBgChoAhYAjEhoCRzNigtIgMAUPAEDAEDAFDwBAwBAwBQ8AQMASMZFobMAQMAUPAEDAEDAFDwBAwBAwBQ8AQiA0BI5mxQWkRGQKGgCFgCBgChoAhYAgYAoaAIWAIGMm0NmAIGAKGgCFgCBgChoAhYAgYAoaAIRAbAkYyY4PSIjIEDAFDwBAwBAwBQ8AQMAQMAUPAEDCSaW3AEDAEDAFDwBAwBAwBQ8AQMAQMAUMgNgSMZMYGpUVkCBgChoAhYAgYAoaAIWAIGAKGgCFgJNPagCFgCBgChoAhYAgYAoaAIWAIGAKGQGwIGMmMDUqLyBAwBAwBQ8AQMAQMAUPAEDAEDAFDwEimtQFDwBAwBAwBQ8AQMAQMAUPAEDAEDIHYEDCSGRuUFpEhYAjEjcCqq676CHHuXF9ff/CUKVNuizv+psbXp0+fldu2bTuFeOo+++yzdk2Nz943BAwBQ8AQMAQMAUOgJSBgJLMl1KKVwRCoAAIrr7zyoDZt2uxPUvID8QsgVp2STDouktmjR4+lOnfuPFt5pQydJ02aND+OfBvJjAPF/8XRHG0s3hJUb2x8a/8h90OXLFmy4+eff/5E9ZbEcm4IGAKGgCGQBgSMZKahFiwPhkDKEVhllVX2ra2tvSsnm0YybScztpbbXG0stgJUeURGMqu8Ai37hoAhYAikDAEjmSmrEMuOIZBGBFZbbbXDEFk9mbzdg38J/wzeSKaRzNiaa3O1sdgKUOURGcms8gq07BsChoAhkDIEjGSmrEIsO4ZAGhHo27dvJydi6omIxkYyEZPszE7pOfh9KX93fsdBak/k+gR8gzOZ7HrdSZj9eFaP/w4/lvB3cG7zL1wvcRgycRYp3rsYpsTzwuTJk4e7MKHI5sfh/wX8TiPMSMRsL544ceI4Py5fXJb7q+NvxW+Mn4N/oq6u7tSpU6d+7r8TNe/unYEDB3ZcuHDhb/i/G35N/PJ49d1f4ycgsvxjP/6hQ4e2nzVrlrD7FX4Afi7+RfA5A3zGFMOi2DOwfJnnPyr1/rx587qR/g+lwuU+T7qNKb1+/fqtSJ2cwuXO+FVDbN7n91pwvDc3T5T5AO4djx+MX4R/A382YV/xw4bhbufe1bSVJ8D6Yq4H4b/i+hZwH8F1nXuHNvBXwh2K35PnEj0/Fr8i/lP+n014tdssN2DAgJ6LFy8ewfNdwrCzeP/flOdMxFvVFrJclLKSj6OI4/rcd3P/U96suYLqCrFa4SjReeH4Lf7Jdu3anfHpp59+5t4Hl19wfZ/7T9734QzzaPJ8PenqO1EZbub7u4DrzHdbKj/23BAwBAwBQyD9CBjJTH8dWQ4NgVQhkATJZDL6EIXcNSyoSNkK+DlMQP/DxPSnuYp/CP8vng/Hqw/rFv7q9X8yId7Lm+SO4Ho7fBv85uH9V/kVOQ0ccb/DpP637j8T6M2YQLszaR25786dziXscMK+5cLmkMwHuL8HXvlfGd+W8J/hN/BJQNS8h2nUEv5Zrrd2aYZ5X6j4wecDJujrec8U/kH+7x7eE9nrrLB4nUndDHw+8MJHvoSQPEB6m5R6ASK0+rRp00RsG+2SaGMsHgxkoeAFMtU7zJhIo5Q1uXFQ5HGEyzQ4nsq1yI+cCKIwlFskokc7eMoLKzJ6O/j8S22E66/w3+CHhvGfQtwinoFzJJPL2/AH8s4Y3u3CtYhpDe1vM9qMCG3gwGMFyJnaXb/wlhZWlgmvxxB+E8LPc+GjlpV87E66vwvfUztaCj8WL9KYcTkLGbXsOj+m7zIMsJhfp/TqC+5vBDbT9IxvaTh5O4/LXngteAjTPfGrku50wq4bxnEQafzdT9OuDQFDwBAwBKobASOZ1V1/lntDoOIIxE0AmMxvTyE0YZ8HCdiJHdORmrQykf0Tk1DtZIoIFtQuq527mTNnbs27dxCuB5PXbSBez/vANEXxDxPx3sR5NvEdhn+RyfBWLm4PC916tn379vtNmDBhZriLJJHiIeTpUibdbiKfVV+l8k7aG5G2yMUSfg9np+gR4p8lSBSRdoB9cgGWItjakRPB2Zu8PrPiiit26dix43X8187mI9xzZL7ibSdqgnG3MaULNiKYW4Ljx9TJIeDwBiRoKUjQ77l/arjjeKjC9u7dexWw/lTtEH/hggULzhcJBMe/h+RKO8gihEE9eDuZNYQ7mfb355z7MwgvohU4j2QuIvxWhH+N21og0E682tk/CL+PC8+3cC3pHs3/d8nvL6nzT2hjq7EjqIWNDf00yy2rSyOquCx534f07uY97fIfSD4fpB2uzfenhZ9+PLuV8hzi17W3Y/oY9+d169Ztv7Fjxy4kTRFv4X8/8WjX05whYAgYAoZAC0HASGYLqUgrhiFQKQTiJgDeBPomJppHuHKEYp8yD7JiFBMmTGRPZ4J7HmHPg9Sd6ePRFJIZxtOOCbF2AtszQe6iCbLu+ySTdDfxdznJz24SZyTYp5RLuzgFXaG8k+a2vCSy+hU7hKuW2iEMdxt/TvjjSPNqlyCkqQukSVguRf67ufxXqs2Um07cbQxCtgaE7CPyUU+dbAQJetvPE21wC+5phztw4P5//FwuQsp9iSgHZDJccJjMZTvqe2PqWxpZfZIpMqmdUif6qXajHUbgb7csoqTagfRJ5n2E/6VLN9z50wLJu9zfILwv8vkl18tDMNeFYEq8N3CQu00hd6+Tz4zId7ll9cocSbtsuBOvnfKs79Vb4JhN3pf1MFB5M2K5lGF1keQQN7XvO/FvVcPih8PKfg2BtCPgLXw9y7claZ6CLjwPf5MfwO9T0l5Wy180BMKjQE8Seh6SMdtw/GdGtDcbH8pIZuOxszcNgVaJQNwEgAno8wxow/H7M6HP0mDrnuWSzDXWWKMbZ/+06yQRP+00SpRVon7b6YwZ8WjXJ+PKJZkiHcS3F3414uuKn8q1dlq6sVu5onYrFblPMnNNo/Tq1atHhw4dgnCk32HUqFESz6wpJ++QjmVJVztvPXhVRENiiiP5fdKJJPrlZGKhybvO+A1gYqGduIyjTCN5dysm+YOY5I9Pc+NNoI057ciTwMWJnBaEABw14dKO4l8If2QOxiJ5Q/C/5tnf9Myb0L2SI1qqZxJZ7kr76Mcu/SSF985knk9bPcPF7+1cTyaevrrPxKAP70oEewr3dP7RdyKxMscznWcS0VbcZZXVRRZ1J9O1MdrSfrRB7WgGLhTpVRutYQLTlwmMyHjgPJI5PtwBzimG/TUE0olA0maV+J50xvphxpg36As2iwuFckhmmAedsZbTUZU1WzrJTLpe6fMuBcOTwPJi+jyHbVzV26h4mAOcTL99SfiypFDuaFREZbxkJLMMsCyoIWAIZBGrWBT/uMkt2O5Ap/e0jzHPHuF/luKfcPdGYndSgJPP3Ug8R/kPyiCZ2jXS2TCdscvrmPCvBFeYrocl7GSKAATE0u1iNSbvpLE670vhixT/dA0zpZ21ZxnEjmNi8qHLKOl9wXVGLDNfAYhnncYoAKrmM5ngIqJ4Q9SJHOGdwqgLaUun+Th6iyK/Bftr9KzYhK4EyTyTOHRmMXD5SKa3M1moSer+V+Sze5iXssrqtZ2oO5lq+5Iu+Il/LpV7+nZ0PrNNbhvzSOaL5DMjbl6sQPbMEGhuBCphVikNJNPHmfwEyrpaMsmsRL2mkWSGY8nT1K2ka4Yz9mi+kKgzkpkovBa5IdDyEIh7l4lBTSZR
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="demo">Demo</h3>
|
|||
|
|
<hr />
|
|||
|
|
<video data-autoplay src="data:video/webm;base64,GkXfowEAAAAAAAAQQoKFd2VibQBCh4ECQoWBAhhTgGcBAAAAAAIJLxFNm3QBAAAAAAAAjE27AQAAAAAAABJTq4QVSalmU6yIAAAAAAAAAJhNuwEAAAAAAAASU6uEFlSua1OsiAAAAAAAAAEF7JoBAAAAAAAAElOrhBBDp3BTrIj//////////027AQAAAAAAABJTq4QcU7trU6yIAAAAAAACCQfsmgEAAAAAAAASU6uEElTDZ1OsiP//////////FUmpZgEAAAAAAABhKtexgw9CQESJiEDIcxjQmeDnTYClR1N0cmVhbWVyIG1hdHJvc2thbXV4IHZlcnNpb24gMS4yMC4zAFdBmUdTdHJlYW1lciBNYXRyb3NrYSBtdXhlcgBEYYgKH3uGKucUuBZUrmsBAAAAAAAAWa4BAAAAAAAAUNeBAYOBAXPFiP8C9cmm3Bq6U26GVmlkZW8A4AEAAAAAAAAlsIIHMLqCA9OagQJVsAEAAAAAAAAQVbmBAVWxgQFVuoEBVbuBAYaGVl9WUDgAH0O2dQEAAAAAAgeR54ELo1g8gQABgPC9AZ0BKjAH0wMRxwiFhYiZhIgKgiMIlfw+HGWHWleQT0D3aPY8fcYm3n/q/UR+Xf+37gPOY/bH8QPgD+zXqtf4f9u/cD6AP9y/xPqtf632Cf5t/qfYB/kn839MP9of/r8h39r/237Z//X5BP2R///uA/wH0AeEo/mHmt7vPwHgf3+/IHtT6tv6b4duWv7t6A/yj63fcP6r+6vqH/vPAP4IfrvqC+lP7b+Y/9u9KvZdAB/MP5x/vP774y/8v6A+ID/MP6N/uPtV+Zv7Z4O3jfsEfob1V/5D/k/3j/KfsH7XPoj/uf4T4Cf1W/2/50/2DwB+id+0D57Q7jilf377OzQZJzpAB3NE93Bojt12nRYCBJc3q8/k7cGHyduDD5O3Bh8nbgw+TtwYfJ24MPk7b6Bt6JOYpTKSOzctJADmdqfjYRJdu6pNzJA7as959FO4QPsiOdsGhxAM7hP0chVt29xcijNh3I73SAoNDs9+IdTaMDuRVartI7tqe8/NYdtzoQDt3ZuVIB38DERyfsMk1b1efyduDD5O3Bh8nbgw+TtwYfJ24MMQbtvTInQJOCVPjOKVL4Ark8wDRlO7kPfbJyHvtk5D32ych77ZOQ99snIe9IvPN0uCtySADEE8Zo4waX1A3LCLAchBJm7lMMRqi+//CqbfQsgTrOZAldWDuQfwWW5ANZ0GwniafbYM853ewDtnwQBhqo0qBZLteLk5D32ych77ZOQ99scKD995o8l9/9vfeaPJff/b33mjyX3/2995o8l9/9vfeaPJeUzdkCwawaQJAoPpLBrBrBrXHgJ7kPfbJyHvtk5D32ych77ZOQ+A/S7Xi5OQ99snIe+2TkPfbJyHvtk5D320Ilych77ZOQ99snIe+2TkPfbJyHvtk5D32zGWntk5D32ych77ZOQ99snIe+2TkPfbJydFgHvtk5D32ych77ZOQ99snIe+2TkPfbJyP+eTkPfbJyHvtk5D32ych77ZOQ99snIfDFae2TkPfbJyHvtk5D32ych77ZOQ99snIfAfpdrxcnIe+2TkPfbJyHvtk5D32ych77aES5OQ99snIe+2TkPfbJyHvtk5D32ych77ZjLT2ych77ZOQ99snIe+2TkPfbJyHvtk5OiwD32ych77ZOQ99snIe+2TkPfbJyHvtk5H/PJyHvtk5D32ych77ZOQ99snIe+2TkPhitPbJyHvtk5D32ych77ZOQ99snIe+2TkPgP0u14uTkPfbJyHvtk5D32ych77ZOQ99tCJcnIe+2TkPfbJyHvtk5D32ych77ZOQ99sxlp7ZOQ99snIe+2TkPfbJyHvtk5D32ycnRYB77ZOQ99snIe+2TkPfbJyHvtk5D32ycj/nk5D32ych77ZOQ99snIe+2TkPfbJyHwxWntk5D32ych77ZOQ99snIe+2TkPfbJyHwH6Xa8XJyHvtk5D32ych77ZOQ99snIe+2hEuTkPfbJyHvtk5D32ych77ZOQ99snIe+2Yy09snIe+2TkPfbJyHvtk5D32ych77ZOTosA99snIe+2TkPfbJyHvtk5D32ych77ZOR/zych77ZOQ99snIe+2TkPfbJyHvtk5D4YrT2ych77ZOQ99snIe+2TkPfbJyHvtk5D4D9LteLk5D32ych77ZOQ99snIe+2TkPfbQiXJyHvtk5D32ych77ZOQ99snIe+2TkPfbMZae2TkPfbJyHvtk5D32ych77ZOQ99snJ0WAe+2TkPfbJyHvtk5D32ych77ZOQ99snI/55OQ99snIe+2TkPfbJyHvtk5D32ych8MVp7ZOQ99snIe+2TkPfbJyHvtk5D32ych8B+l2vFych77ZOQ99snIe+2TkPfbJyHvtoRLk5D32ych77ZOQ99snIe+2TkPfbJyHvtmMtPbJyHvtk5D32ych77ZOQ99snIe+2Tk6LAPfbJyHvtk5D32ych77ZOQ99snIe+2Tkf88nIe+2TkPfbJyHvtk5D32ych77ZOQ+GK09snIe+2TkPfbJyHvtk5D32ych77ZOQ+A/S7Xi5OQ99snIe+2TkPfbJyHvtk5D320Ilych77ZOQ99snIe+2TkPfbJyHvtk5D32zGWntk5D32ych77ZOQ99snIe+2TkPfbJydFgHvtk5D32ych77ZOQ99snIe+2TkPfbJyP+eTkPfbJyHvtk5D32ych77ZOQ99snIfDFae2TkPfbJyHvtk5D32ych77ZOQ99snIfAfpdrxcnIe+2TkPfbJyHvtk5D32ych77aES5OQ99snIe+2TkPfbJyHvtk5D32ych77ZjLT2ych77ZOQ99snIe+2TkPfbJyHvtk5OiwD32ych77ZOQ99snIe+2TkPfbJyHvtk5H/PJyHvtk5D32ych77ZOQ99snIe+2TkPhitPbJyHvtk5D32ych77ZOQ99snIe+2TkPgP0u14uTkPfbJyHvtk5D32ych77ZOQ99tCJcnIe+2TkPfbJyHvtk5D32ych77ZOQ99sxlp7ZOQ99snIe+2TkPfbJyHvtk5D32ycnRYB77ZOQ99snIe+2TkPfbJyHvtk5D32ycj/nk5D32ych77ZOQ99snIe+2TkPfbJyHwxWntk5D32ych77ZOQ99snIe+2TkPfbJyHwH6Xa8XJyHvtk5D32ych77ZOQ99snIe+2hEuTkPfbJyHvtk5D32ych77ZOQ99snIe+2Yy09snIe+2TkPfbJyHvtk5D32ych77ZOFD83QsCEF2lQLJdrxcnIe+2TkPfbJyHvtk5D32ygZeLk5D32ych77ZOQ99snIe+2TkPfbJx/GX88nIe+2TkPfbJyHvtk5D32ych77ZOQ99sxlp7ZOQ99snIe+2TkPfbJyHvtk5D32ycod5teLk5D32ych77ZOQ99snIe+2TkPfbJyIgeTkPfbJyHvtk5D32ych77ZOQ99snIe/VAi5OQ99snIe+2TkPfbJyHvtk5D32ych77cy/T2ych77ZOQ99snIe+2TkPfbJyHvtk7Stza8XJyHvtk5D32ych77ZOQ99snIe+2Tkf88nIe+2TkPfbJyHvtk5D32ych77ZOQ+IDKgWS7Xi5OQ99snIe+2TkPfbJyHvtk5D34AgHvtk5D32ych77ZOQ99snIe+2TkPfbKijbm14uTkPfbJyHvtk5D32ych77ZOQ99snOIi5OQ99snIe+2TkPfbJyHvtk5D32ych/caVAsl2vFych77ZOQ99snIe+2TkPfbJyHwH6Xa8XJyHvtk5D32ych77ZOQ99snIe+2kEvpdrxcnIe+2TkPfbJyHvtk5D32ych77ZQMvFych77ZOQ99snIe+2TkPfbJyHvtk5FEi0qBZLteLk5D32ych77ZOQ99snIe+2TkPoX0u14uTkPfbJyHvtk5D32ych77ZOQ99yPzych77ZOQ99snIe+2TkPfbJyHvtk5D32zGWntk5D32ych77ZOQ99snIe+2TkPfbJyh3m14uTkPfbJyHvtk
|
|||
|
|
</video>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="snapshot-and-streaming">Snapshot and streaming</h3>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="snapshots">Snapshots</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li><strong>Initial</strong> copy (once)</li>
|
|||
|
|
<li><strong>On-demand</strong> snapshots with signal tables</li>
|
|||
|
|
<li><strong>Partial</strong> snapshots
|
|||
|
|
<ul>
|
|||
|
|
<li><code>*.include.list</code></li>
|
|||
|
|
<li><code>*.exclude.list</code></li>
|
|||
|
|
</ul></li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA3wAAAFWCAYAAAAyvJ9bAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7N13fFvlvcfxz3OOZFteWU6c4TiDQHZCFitA2KtAGG2BFkopUEp7oYW27I7bxeguFFp2KaO0pUBve9swwuYCIcwMZnac5QwvWeuc5/4h2YmT0MiJYyk633deHZHOkR4plqWvfs/z/Iy11iIiIiIiIiIFx8n1AERERERERGT3UOATEREREREpUAp8IiIiIiIiBUqBT0REREREpEAp8ImIiIiIiBQoBT4REREREZECFcr1AIJg3bp66tevp76+nqqqKqr69KFv36pcD0tERERERAqcAt9utGDhezz6+N9Z+N7721xXWzuY4485mkMOnp6DkYmIiIiISBCYHTVe9z68ndNO/y0LUtu71mXU1/7EY1/bB3f3jG+P9cijj/Po43+ntLSUQ6YfxOhRIyktLQVg7htvMvfNN6mvX88hB0/n7M+dSVnmOhERERERka6SdYXPHXoY5x41fKsTHPpN7o3p8mF1RpT/ueQwLn9+PN+ddQfn9M/9ssS2sDd50r5cdOH524S5MaNHcdqpM/nbo48z68mniEajXHbpf+VotCIiIiIiUqiyDnyhvY7l65efgOpQ/1nbNM7Jk/bl8q9f8onHlZWWcs7nzwJg1pNP8a9ZT3L8sUd31zBFRERERCQAuqgcZmmYdQVTx03iyBvfIrnlNRse58tT92XUzN+ywEtfllr9Knd/98scP+Mgxk2azozPXsZP//djou1npXjzpycxasIZ/OalF7jl0s9y0JSpjNv/eD579f28sSkzC7Xpn3xl0oFc/nQckq/zgyMmsc/+1zI7kbmVVS/zu6u+xDGHHMjYCfsx7dhz+fbdc1jndc2j3p62aZwXXXh+h8uXLl3Gl7/6X7w+940Ol5/z+bOoqurDo48/vvsGJSIiIiIigdRFgc/Q45ATObIXrJz9FG+3Jz7Lhuee4tVWlzGfOp6RLvjrnubaL3yVG/+5ikFHn8s3vnYG03iTu644j4sfWEyHLOav5pEf/IxX+xzNVy7/GmeMs8z7+8/56g+eoN4CkfGc892r+MzIELgDOPq/vsOPrz2VUSEg9hY/v/BSfvHEOoaceBHXXncJn95rA//+xSVcfPeH7I7Mt25dPQvfe59Dph/0CWvytj/59bSZJxONtm4TBkVERERERHZF1lM6Ux/P4te/+LDjCe4gjvrS6UyqMFC6PzOP7sfjf5nNrHe+ztQpYbAbee7J12kNj+ek44fiEue123/O46sG8Nnb/sgPDu6ZjkDnHMOQs8/hltvu5JmZP+ao8szt22Z6nPR77vqvURQBnL4voVPP557nnuCVlmM5sbyW6adWsWH2L/nLokEceNppfCazhs9b9iovLktRcdxl3HzlEZQAnDaDaQ//H837lJKCLt9oZl19PQCjR43c5rohQ2q5/dabt3ve6FGjAFi6bDlTp0zu4lGJiIiIiEhQZR34vCXPcvedz3a8MDyV/meclg58FDPtxKMZ9JcHeOqJd7liymRCG1/g36+1UjLlUxw/yAHvQ557eTUMPJNPTSyiNdo2ibOGww8bzq2/m8v/LUxx1LTMxaYXk6btlQ57ACX7MGavELZuE5uafSj/5AKlO2AUY3oZ3p99M9/6zQZmTp/EvmOHccTnz8j2IXdafSbwlW6nutcSjXL7nXdz9llnbtODTz35RERERERkd8g68BUfeT2v3fyfN20JT/wUJw57kN/PfpK3vjmJ4c89xauxUg448QiqHSC5kfoNFq/hAc7Z/4Ftb8BE2NDg0z7T1IQIuVtOgzQ4mb/+x14SABWHcu2t1xH55X38464fMet3FlPUi5EHn86lV17E0YOLdnQLnVZV9cnBbenSZcx9401qBw/m9FNndriuJRr9hLNERERERER2Xtc2XndHctKnRnL7Lc/y77e+wJgnXidWfjAzj8i0bjA96N0DTGo6377lQiaFt74Bhz7Dw9AlK+wMlWNP5/t3ns53Y/UsWvAuc568n5vvv4tvrI3w2EMXsHcXz+nsmwl8c994kzGjR2V93sKF72XO79O1AxIRERERkUDr4qZ1DsNPOIGJ7mqefOgWHn8tRq/DTmBGj0xZLrQPBx/QFye2mEWtw5gyeRJTJ09i6uSx9NqwgAVLm/G3CYE7YjAGsAni8c2Xxt7/Fzff+BPufKURU1LFiMmHc9blF3N8tSG5dDHLd8OuLX37VlFV1Ye5b7657XWZMLi9UDfryacAmKL1eyIiIiIi0oV2bdMWDJWTTuOCw2vaN0BxBh3DzCm38L1Z/2CV048zTzqQ8vbjSzjwK9/gmOev45HLz2HNqSdwUG0JDe/N5pF/zCM6/hL2O2l6Jx9CEUOGD8J9ej53X3MddQcezhcuPprBpY28/re/8Or/zOfDM45nYlWKZS8+yl9WQfXJR26nutg1Tpt5MrffdQ9/fOCh9j57kA6Dv7/15m1273z+hRdZ+N77nDrz5E/Y2VNERERERGTn7NqmLTgM+PxBnH94zRYX9eOYk/bnxleeIzbwaE6eVtLxjP4n8LM/VjDqN3fz6L/v4WeNUN5vOFM+/z0u+erMdEuFTnEZd+FP+N666/ndk0/w0OIkk885iqGDP8tv7nT51c0P8+Qff8Xj8TC9akZz9Ne+waXnH06v7XdI2GWHHnIwr7/xJrOefApjDGd/7sz267YX9m6/6x6G1A7mODVdFxERERGRLmastTvc/0Q6pyUa5fd33MUbb75FVVUfTj9lJqNGjqRv3ypaolHee+99/v3Ekyx8732G1A7mmquuUHVPRERERES6nALfbvSvWU/y6ON/J/oJu3CeOvNkjjv2aIU9ERERERHZLRT4usHrc99g6bLl7X/vW9WHKVMmK+iJiIiIiMhupcAnIiIiIiJSoLq4LYOIiIiIiIjkCwU+ERERERGRAqXAJyIiIiIiUqAU+ERERERERAqUAp+IiIiIiEiBUuATEREREREpUAp8IiIiIiIiBUqBT0REREREpEAp8ImIiIiIiBQoBT4REREREZECpcAnIiIiIiJSoBT4RERERERECpQCn4iIiIiISIFS4BMRERERESlQCnwiIiIiIiIFSoFPRERERESkQCnwiYiIiIiIFCgFPhERERERkQKlwCciIiIiIlKgFPhEREREREQKlAKfiIiIiIhIgVLgExERERERKVCdCnzLlq/g/C9/lQmT92f6YUdx6+/uwPf99uuttfx71pMce+IpTJx6IF88/yKWLFna4TZSnsevb76Vg2YcybQDD+Wa7/43LS0tHY5Zv2EDl33rKibvP50ZRx3H3ffe1+F63/eZN38BZ579RSZOOYBTTj+DF158ubOPXUREREREpKBlHfhSKY/zv3wx5eVlPHT/vfz3d6/lvgce5L77HwTSYW/O3De48trvcNEF5/HIww9QUzOI8y78CtFoFEgHtdvvuJvH/v4Pfn7T9dx1+60s+ngRV1/3/fb7SSaTfOvKa2hsbOSBP9zNd6+9invve4AHHnq4/X7q16/nK1/7OpP2ncgjf36QU0+ZyWXfvpJ5CxZ24VMjIiIiIiKyZzPWWpvNgW+/O48zPvcF5r7yAmVlZVhrueue+/jH//6Lx/76J3zf56KvXsrQIbVce/UVQDq8nfqZszjzjM9w9llnEI22cthRx/GTH36fo448HGstK1fWccLM0/nrn+5n7xF78cZbb/OlCy9m9hP/pE/v3niezz/+91/88tc38/Ss/8Van7vu+QNPzX6Gvzx0PwCJRIIfXX8TLS0t/Pym63ffsyUiIiIiIrIHybrC5xiDMYZQKNR+WVFRGGMMAJ7nMef1uRx33DHt17uuy5FHHMYrr7yGtZb5CxbgeR6HHjIdAGMM1f2rGT9uLK++Ngff93n11TnsN3UyfXr3ztyGw+EzDmFTQwMfffwxnufx2utzOebIIzrczxGHz+C1OXPJMr+KiIiIiIgUvKwD35jRoxg2
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="streaming">Streaming</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>Based on <strong>logical replication</strong></li>
|
|||
|
|
<li>Detect <strong>DML</strong> statements (<code>INSERT</code>, <code>UPDATE</code>, <code>DELETE</code>)</li>
|
|||
|
|
<li>Primary key highly recommended</li>
|
|||
|
|
<li><code>pgoutput</code> plugin</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA30AAAFZCAYAAAAsKEawAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7N13nGRVnf//1zn3VlVXdZqZngBMIkhOkkSSJBExwAJfBXfB7LpmQWBRdPHnriuLaVdd1wSoK8vuuoK4QRGVBRNhCEvOeRiGmemZThXvPef3x63uYehQoXuo6er38/Hg8WC6qm6d6q7uup/7+ZzPx3jvPSIiIiIiItKWbKsXICIiIiIiIluPgj4REREREZE2pqBPRERERESkjSnoExERERERaWMK+kRERERERNqYgj4REREREZE2pqBPRERERESkjYWtXsBcsG7detZv2MD69etZuHAhC/v6WLRoYauXJSIiIiIic4CCvq3o/gce5Jprf8YDDz407rYVK5Zz0utO4Kgjj2jBykREREREZK4w3ns/1R3iR77Daaf/I/dHE90asMeH/pWffmg3gq2zvlnrJ9dcyzXX/oxcLsdRRxzOnnvsTi6XA+D2O+7k9jvvZP36DRx15BGc9adn0lm9TUREREREZCbVnekLdjyGd7x255c8wLL4wAWYGV9WI/L850eO4dyb9uWvrvsuZ2/X+m2KowHfgQe8kve/7z3jArq99tyD0049hauvuZbrrv8V+Xyecz764RatVkRERERE2lndQV+4y4l87Nw3oHzU1EZLOg884JWc+7GPTHq/zlyOs//sbQBcd/2v+Pl113PSiSe8XMsUEREREZE5YobSYp6B6y7g4H0O4Pi/u4vKi2/pv5Y/P/iV7HHKP3J/nHwtev4WLv+rP+ekow9nnwOO4Oi3nsMX/+cx8mOPirjzi29mj/3O4Gu//y3f+OhbOfygg9nn0JN46yd/xB2bqhWpQ//NXxxwGOf+ugSVVXzuuAPY7dCL+E25epQ1f+BbF76b1x11GHvv9yoOOfEdnH/5bayLZ+ZVT2S0pPP973vPFl9/6qmn+fMPfphVt9+xxdfP/rO3sXBhH9dce+3WW5SIiIiIiMxZMxT0GXqPehPHz4fVv/kV/zcW9Xn6b/wVtxQC9nrjSewegFv3ay56+wf5u/9ew9IT3sHHP3QGh3Anl13wLj5w5RNsEY+55/nJ577ELX0n8Bfnfogz9vHc+7Mv88HP/ZL1Hsjuy9l/dSFv2T2EYHtO+PBn+PxFp7JHCBTv4svv+yhf+eU6Vr7p/Vz06Y/w/3bp5xdf+QgfuPwRtkbct27deh548CGOOuLwSfboTVwIe9opJ5PPF8YFhCIiIiIiItNVd3ln9Nh1/MNXHtnyAcFSXvvu0zmg20DuUE45YTHX/vg3XHf3xzj4oBT4jdx4/SoKqX1580k7ElDi1u98mWvXbM9b/+mf+dyR85Iw6OzXsfKss/nGP32PG075PK/tqh7fD9P75m9z2Yf3IA1w+isJT30PV9z4S24eOZE3da3giFMX0v+br/Ljx5dy2Gmn8Zbqnr746Vv43dMR3a8/h6//5XF0AJx2NIf82x8Z3i1HBDPefGbd+vUA7LnH7uNuW7lyBd/55tcnfNyee+wBwFNPP8PBBx04w6sSEREREZG5rO6gL37yf7n8e/+75RdTB7PdGaclQR8ZDnnTCSz98ZX86pf3cMFBBxJu/C2/uLVAx0Fv5KSlFuJHuPEPz8MOZ/LG/dMU8qMFncs49pid+ea3buePD0S89pDql818DjhklyTgA+jYjb12CfHPbWLTsIOuyROVwfZ7sNd8w0O/+Trnfa2fU444gFfuvRPH/dkZ9b7khq2vBn25CbJ8I/k83/ne5Zz1tjPHzejTzD4REREREdla6g76Msd/gVu/PnUjl9T+b+RNO/0L3/7N9dz1iQPY+cZfcUsxx6vfdBxLLFDZyPp+TzxwJWcfeuX4A5gs/QOOsapTExIGLy6JNNjqP6ecMwHQ/Rou+uanyX71h/zXZX/Ddd/ymPR8dj/ydD76l+/nhOXpWkdo2MKFkwdvTz31NLffcScrli/n9FNP2eK2kXx+kkeJiIiIiIhMz8wOZw92581v3J3vfON/+cVdb2evX66i2HUkpxxXHetgelnQCyY6gvO/8T4OSL30AJa+nVMwIzvuDD17n85nv3c6f1Vcz+P338Nt1/+Ir//oMj7+QpafXvVedp3h+s5F1aDv9jvuZK8996j7cQ888GD18X0zuyAREREREZnzZnionWXnN7yB/YPnuf6qb3DtrUXmH/MGju6tpufC3Tjy1YuwxSd4vLATBx14AAcfeAAHH7g38/vv5/6nhnHjAsFaDMYAvkyptPmrxYd+ztf/7m/53s2DmI6FvOLAY3nbuR/gpCWGylNP8MxW6OSyaNFCFi7s4/Y77xx/WzUgnCiwu+76XwFwkPbziYiIiIjIDJteIxcMPQecxnuPXTbWFMUufR2nHPQNLr7uv1hjF3Pmmw+ja+z+HRz2Fx/ndTd9mp+cezZrT30Dh6/oYODB3/CT/7qX/L4f4VVvPqLBl5Bm5c5LCX59H5d/6tM8d9ixvP0DJ7A8N8iqq3/MLf95H4+ccRL7L4x4+nfX8OM1sOTk4yfIMs6M0045me9cdgX/fOVVY3P4IAkIv/3Nr4/r6nnTb3/HAw8+xKmnnDxJx08REREREZHmTa+RC5bt/+xw3nPsshd9aTGve/Oh/N3NN1Lc4QROPqRjy0ds9wa+9M/d7PG1y7nmF1fwpUHoWrwzB/3ZxXzkg6ck4xYaErDP+/6Wi9d9gW9d/0uueqLCgWe/lh2Xv5WvfS/g77/+b1z/z3/PtaUU85ftyQkf+jgffc+xzJ94esK0veaoI1l1x51cd/2vMMZw1p+eOXbbRAHfdy67gpUrlvN6DWYXEREREZGtwHjva/ZEkcaM5PN8+7uXccedd7FwYR+n/8kp7LH77ixatJCRfJ4HH3yIX/zyeh548CFWrljOpy68QFk+ERERERHZKhT0bUU/v+56rrn2Z+Qn6c556ikn8/oTT1DAJyIiIiIiW42CvpfBqtvv4Kmnnxn796KFfRx00IEK9kREREREZKtT0CciIiIiItLGZnhkg4iIiIiIiGxLFPSJiIiIiIi0MQV9IiIiIiIibUxBn4iIiIiISBtT0CciIiIiItLGFPSJiIiIiIi0MQV9IiIiIiIibUxBn4iIiIiISBtT0CciIiIiItLGFPSJiIiIiIi0MQV9IiIiIiIibUxBn4iIiIiISBtT0CciIiIiItLGFPSJiIiIiIi0MQV9IiIiIiIibUxBn4iIiIiISBtT0CciIiIiItLGFPSJiIiIiIi0MQV9IiIiIiIibUxBn4iIiIiISBtT0CciIiIiItLGFPSJiIiIiIi0MQV9IiIiIiIibUxBn4iIiIiISBtT0CciIiIiItLGFPSJiIiIiIi0MQV9IiIiIiIibUxBn4iIiIiISBtT0Pcy8t4zMpJv9TJkBsWxY3h4hEKh2OqlyAyJoojh4RGKxVKrlyIzqFyuMDw8QrlcbvVSZAYViyWGh0eIoqjVS5EZUigUGR4eIY7jVi9FZtDISJ7h4RG8961eypyloE9ERERERKSNKegTERERERFpYwr6RERERERE2piCPhERERERkTamoE9ERERERKSNKegTERERERFpYwr6RERERERE2piCPhERERERkTamoE9ERERERKSNKegTERERERFpYwr6RERERERE2piCPhERERERkTamoE9ERERERKSNKegTERERERFpYw0FfY8+9jhnv+u97Hvgqzj2hJP4l3/99y1ud87xtW/8E4cffTwHv/ooPvmZzzIyMrLFffo3buSc8z/JgYcewdHHn8hlV/xgi9u999x3/wOcedY72f+gV3PyaW/lpt/+vsmXJyIiIiIiMrfVHfTl83ne8Z4/Z+edduJnP/l3PnHOx/jy33+NX/zyeiAJ1r793cu5+qc/48uXfoErvvctHn/8CS686OKxY0RRxCcu+CQDmzZx5Q8u5+LPfIrv//BKfvQvV43dZ0N/P+//4Ed55f77cfWPr+Itp5/Kx8/7S+69/4EZ
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="how-to-configure-debezium">How to configure Debezium</h3>
|
|||
|
|
<p>Kafka Connect REST API:</p>
|
|||
|
|
<pre><code>curl -XPOST -d @payload.json -s localhost:8083/connectors</code></pre>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="payload-example">Payload example</h3>
|
|||
|
|
<pre><code>{
|
|||
|
|
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
|
|||
|
|
"database.ApplicationName": "debezium",
|
|||
|
|
"database.dbname": "test",
|
|||
|
|
"database.hostname": "${file:/path/to/secrets/test:hostname}",
|
|||
|
|
"database.password": "${file:/path/to/secrets/test:password}",
|
|||
|
|
"database.port": "${file:/path/to/secrets/test:port}",
|
|||
|
|
"database.ssl.mode": "required",
|
|||
|
|
"database.user": "${file:/path/to/secrets/test:user}",
|
|||
|
|
"decimal.handling.mode": "double",
|
|||
|
|
"field.name.adjustment.mode": "avro",
|
|||
|
|
"heartbeat.action.query": "update ovh_debezium_heartbeat set ts=now();",
|
|||
|
|
"heartbeat.interval.ms": "1000",
|
|||
|
|
"incremental.snapshot.chunk.size": "100000",
|
|||
|
|
"name": "test",
|
|||
|
|
"plugin.name": "pgoutput",
|
|||
|
|
"producer.override.compression.type": "lz4",
|
|||
|
|
"publication.name": "debezium_test",
|
|||
|
|
"sanitize.field.names": "true",
|
|||
|
|
"schema.name.adjustment.mode": "avro",
|
|||
|
|
"signal.data.collection": "public.ovh_debezium_signal",
|
|||
|
|
"slot.name": "debezium_test",
|
|||
|
|
"status.update.interval.ms": "1000",
|
|||
|
|
"topic.creation.default.cleanup.policy": "compact",
|
|||
|
|
"topic.creation.default.delete.retention.ms": "604800000",
|
|||
|
|
"topic.creation.default.partitions": "6",
|
|||
|
|
"topic.creation.default.replication.factor": "3",
|
|||
|
|
"topic.heartbeat.prefix": "namespace.debezium.heartbeat",
|
|||
|
|
"topic.prefix": "namespace.test",
|
|||
|
|
"transforms": "schema,public",
|
|||
|
|
"transforms.public.regex": "namespace\\.test\\.public\\.(.+)",
|
|||
|
|
"transforms.public.replacement": "namespace.test.$1",
|
|||
|
|
"transforms.public.type": "org.apache.kafka.connect.transforms.RegexRouter",
|
|||
|
|
"transforms.schema.regex": "namespace\\.test\\.((?!public).+)\\.(.+)",
|
|||
|
|
"transforms.schema.replacement": "namespace.test.$1.$2",
|
|||
|
|
"transforms.schema.type": "org.apache.kafka.connect.transforms.RegexRouter"
|
|||
|
|
}</code></pre>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="automation">Automation</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQAAAAE7CAYAAAAyz18IAAABhWlDQ1BJQ0MgcHJvZmlsZQAAKJF9kT1Iw0AcxV9TtUVaHOwg4pChOlkQFT82rUIRKoRaoVUHk0s/hCYNSYuLo+BacPBjserg4qyrg6sgCH6AuLo4KbpIif9LCi1iPDjux7t7j7t3gFAvMc3qGAE0vWKmEnExk10RA6/oQhBhTGNSZpYxK0lJeI6ve/j4ehfjWd7n/hxhNWcxwCcSzzDDrBCvE09sVgzO+8QRVpRV4nPiYZMuSPzIdcXlN84FhwWeGTHTqTniCLFYaGOljVnR1IjHiaOqplO+kHFZ5bzFWStVWfOe/IWhnL68xHWaA0hgAYuQIEJBFRsooYIYrTopFlK0H/fw9zt+iVwKuTbAyDGPMjTIjh/8D353a+XHRt2kUBzofLHtj0EgsAs0arb9fWzbjRPA/wxc6S1/uQ5MfZJea2nRI6BnG7i4bmnKHnC5A/Q9GbIpO5KfppDPA+9n9E1ZoPcW6F51e2vu4/QBSFNXyRvg4BAYKlD2mse7g+29/Xum2d8PzJlyy/XQ/bUAAAAGYktHRAD/AP8A/6C9p5MAAAAJcEhZcwAADdcAAA3XAUIom3gAAAAHdElNRQfnBggNOg8wrwxuAAAgAElEQVR42u2deXxTZfb/P+cm3ejC1jbJTYEiCGrBDXFAQNkUkE2QfSnbqKM44+yLzqpfZ77qqL/RcWZUdFTKLoIsiht8nXEDRHGUTRko0NwkLXtb6JLc8/uDgmkodMlNcpOc9+vF60XT5rnPPc95Pvc85z4LQYgb2rVrl9WqlSUXtUp7ELX3E7UH0B7Q2xNTEhG3BQAmSgaQHvR1JubjAMBM1QBOEVDJwBECDgMoI6t+BEg+kpqaenjv3r3VYvHYh8QEsUMvIKnEZutusVi6MnM+AZ3B3BkK8sHIB5AZoaowGG4Q7wdQzEAxEYoBLrZY9J2HDh3WpLVEAIQQsNlsuRYLXwVduYoJPQnoCaAAQHIMVP8ICF8Q05dM+pfMyhfp6elfStQgAiBcgNzc3EuSFKU/iPsxqD+Ay+OsfXwAvgDoQ2L+wMf8vtfrLZWWFwFISPLyci7VdWUEQLeAuR9AbRLQDF8D/D5D2VBdXf3u0aNHT4pniADEaYfPS9P1mn7QlaEgjAZwhVilHn4A20FYx0xr3W73ZwBYzCICEOOdvnYoMU1kYDzOz7wLF+YQQKtAtELTtA9FDEQAYoL8/PzUmprTN9d1+tsQucx8PHMQoNUiBiIApsVutxcowEwQ7gDQTiwSNvaA8E9dx4sej6dMzCECEDXatWuXlZKSMoXAdwLoJRaJKNUgrAH05zSt9D2JCkQAIvm0v0Ih+hHAU2VcbwIYe5nomdra2hcOHz5cLgYRAQgLqqr2B/RfgDFSbGdKTgL0Eoj+rGnaITGHCIARWFXVNglMPwFwrZgjJqhhYKnFoj9eUlL6HzGHCEBLUJx2++1M+B8A3cQcMTo4IKwnsvzG5XJtF3OIADTJJqpqGwVWHgL4KjFHXKCDsBJQfqNp2h4xhwjABcb4tjHQ6SEQrhRrxCU+EBYx0+/cbvcBMYcIQF3HVy8j1p9gYIRYIzFyBAD9o6a29teJ/tYgoQUgLy+vne73/w7gewBYpV8kXIZAY6I/uN3uBQB0EYDEwepwOO4l8G8BtJWekPBx8GZFxw9KPJ4tIgBx/9TPvZL9ygIGeovnC4HxAIGfr671/zSRhgWUOB0/L03Xfb8A41eIjV11hCgNC6Ao8zVNWy0CECc4HI4BBH4OwGXi4UITe8YKXcf8eF9wFNcCkJeXl6b7fI+CMB/yxkNoPmUgnqdp3rUiADGG3W4vUIgWyWQewYBOstCn891er7cy3u5Nicf2cjhs9ymEbdL5BWPSAphpUehTVVXjbi1IXEUAOTk59iSr9RWAbxa3FcJADRgPaB7P44iT/QfiRgAcDsd1BF4JoKP4qRDmkGBtcmpqYXFx8XERAHN0/jsJ/DTk9Z4QOb7WGeM9Hs8OEYAokZ+fn1pTXf1XgOeJPwpRoIIYc10ez4pYvQFLrFY8Ly/P6auteRfASPFDIUokgzAhKzMzubyiYpNEAJHq/DZbT12hdTLeF0zUkVaSxTqzpKTktAhAGFHV3JsBZQUYrcXtBJP1pk/8fh4bS2ceKrHV+W3zAGW9dH7BlDD6WBX6WFXV7iIABuN0OB4G0wIwksTTBPNqAC4B6x/k2WzfkSGAQXV0OmxPMOiH4l5CDFEJ0m/TtNJ3RQBajkW1258DYa74kxCDVIN4kqZ515i2g5m58zsc9heJMFv8SIhRrADdnpWRsbu8omKnCEATKSgoSE6yWl8lYLL4kBDjWEAYn5mV/t/y8sovRQCaUKcki6UIhNvFd4Q4QQHotsys9D3l5ZU7RAAuDKkOx3MgzBSfEeJOBIjGZrVK/7y8svJr03Q4M3V+h8P+NwK+J74ixDGniTHS5fGYYuqwaeYBqKr9Een8QgKQxoR1DodjgEQAdTgctl8R6I/iG0ICcQyk3KBp2u6EFgCn3T6JCUsQn9uTCcLF2O/z631LS0u9CSkAqqr2B+vvAEgVXxASlE/9Og+M1oajUXvqdsjN7QLWX5POLyQ411kstBRReiMXlYuqqprNwP+B0EHaXxDQLSMzI72iouLtRBAAJTMjfSUI10u7C8K5sfgNWRkZB8orKrbH9RBAtdv/CGCYNLkg1IcJf4/02QMRTQKqqm0smFZBjukShAtRTIrlOpfLdSSuIgBVVbuD6RXp/OYjMzMTNpsdGRkZYozok8+6viRSw/OIdEabzZZuUehTyOm8puHqa67BrMJZuGngTbDZ7Oc+P3ToEN555228sOAFFBfvF0NFKyfA+K3L43koLgRAdTieB/i70qzRJz09HQ8//EdMmDgRRBdu/traWjzx+ON4+umnwMxiuMij64zBHo/n/XBeJOxhhqqqtwH8qLRn9MnIyMCyZcsxZOjQi3Z+ALBYLOjfvz/atG2DTRs3ivGiEAQQ4aak5OR/nj59ujomBaBDh2yVdXoDQCtpz+jzxJNPYtDgwc36zjXXXIvi/fuxa9cuMWDkaZtkteSVV1SsisUhADkd9vUMjJB2jD59+/bFqytfa9F3y8rK0LfPd3D69GkxZHQSAtM1zbs4HEWH7S2Aw+G4Tzq/eZh/7/db/N2cnBxMmiy7s0VRAf6mqmqHmBEAu92eT+CHpOHMQaf8fNx0000hlTFnztxG8wZCmGC0JtafjRkBUIifBSAvlU3C7FmzoSihNfWll16KPn37ijGjpgEYoaq2KaYXAKfdPhugW6TJzEFqaqph4fuswlli0KiqAD1tt9tzTCsAqqpmM+ExaSnzMG7ceLRp08aQskbceivsdrsYNXpkK4qx/cvYCEDX/wogW9rJPMwsLDSsLKvVimnTp4tRozsWmHXmhGyTCYDTbh8EkoM8zMS1vXrhqquuMlZQZsyENUnOZ42uCChP9YIxh+QaJQCKhP7mIxxj9lybDcOGyWruKHOZ5rDdYxoBcDgcdwDoJe1iHtq1a4fRY8bEjLAIzYNAv1NVNTvqAtCuXbssAv9emsRcTJ06DSkpKWEpu1///rjsMlnYGWXaAvofoi4AKSnJvwMgqWEToSgKZswM7+lq02fMEENHPReAu/Jstp5REwCHw9GJgHulJczF4CFD0LFjx7BeY9KkycjMzBRjRxeLbqE/RU0ACPg1gGRpB3Mxa9bssF8jIyMDt40bJ8aOfhQw0mmz9Ym4ANjt9nyAC6UFzEWnTp0wcODAiFxrzpy5YnAzaICiPBhxAbAQHpSnv/mYObMw5Hn/TaV79+64/nrZ3d0EEnCz3W6/KWIC4HQ6uzEwVQxvLpKTkyO+bHfW7NlieBOgEP4nYgLAuv57AFYxu7m4bdw4tG/fPqLXHDVyVL1NRYWo0d9psw0OuwDUjf0nir3NRySSf8FYk5IweYrMADdHLoB+HnYBsBD9UJ7+5qNHjx64+uqro5Z3sFrFJUzAMKcz96qwCUC7du2yGDxH7Gw+5s2L3q7rqqpi6NCh
|
|||
|
|
<pre><code>ansible-playbook -e '{"database_name":"test"}' database-connector-create.yml
|
|||
|
|
ansible-playbook -e '{"database_name":"test"}' database-connector-delete.yml</code></pre>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="debezium-endpoint">Debezium endpoint</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>Replication slot on the <strong>primary</strong></li>
|
|||
|
|
<li><a href="https://bdrouvot.github.io/2023/04/19/postgres-16-highlight-logical-decoding-on-standby/">Postgres 16 highlight: Logical decoding on standby</a> (<a href="https://www.youtube.com/watch?v=RQucDdeg0Ac">video</a>)</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="replica-promotion">Replica promotion</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li><strong>Patroni</strong> advances replication slot position on all replicas</li>
|
|||
|
|
<li><strong>Transparent</strong> for Debezium</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="passwords-security">Passwords security</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>Custom vault microservice (HTTPS)</li>
|
|||
|
|
<li>Copy secrets to <strong>local files</strong></li>
|
|||
|
|
<li><strong>Reference</strong> to local files using <a href="https://docs.confluent.io/platform/current/connect/userguide.html#fileconfigprovider">FileConfigProvider</a></li>
|
|||
|
|
</ul>
|
|||
|
|
<pre><code>{
|
|||
|
|
"database.password": "${file:/path/to/secrets/test:password}"
|
|||
|
|
}</code></pre>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="database-connection-lost">Database connection lost</h3>
|
|||
|
|
<pre><code>io.debezium.DebeziumException: Couldn't obtain encoding for database test</code></pre>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p>Manual restart of the connector</p>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="table-inheritance">Table inheritance</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li><code>events</code>
|
|||
|
|
<ul>
|
|||
|
|
<li><code>events_2023_10</code></li>
|
|||
|
|
<li><code>events_2023_11</code></li>
|
|||
|
|
<li><code>events_2023_12</code></li>
|
|||
|
|
<li><code>events_2024_01</code></li>
|
|||
|
|
</ul></li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="table-inheritance-1">Table inheritance</h3>
|
|||
|
|
<pre><code>test=> select count(*) from events;
|
|||
|
|
count
|
|||
|
|
------------
|
|||
|
|
1314817483
|
|||
|
|
(1 row)</code></pre>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<pre><code>test=> select count(*) from only events;
|
|||
|
|
count
|
|||
|
|
-------
|
|||
|
|
0
|
|||
|
|
(1 row)</code></pre>
|
|||
|
|
</div>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><a href="https://github.com/debezium/debezium/pull/5159" class="uri">https://github.com/debezium/debezium/pull/5159</a></p>
|
|||
|
|
<div class="notes">
|
|||
|
|
<ul>
|
|||
|
|
<li>Snapshots selecting data from primary table</li>
|
|||
|
|
<li>Those rows are not included in the logical replication</li>
|
|||
|
|
<li>Rows never deleted</li>
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="topic-routing-smt">Topic routing (SMT)</h3>
|
|||
|
|
<p>Table name + regex = topic name</p>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<ul>
|
|||
|
|
<li>Tables with the same structure</li>
|
|||
|
|
<li>Standardization of table names on all DBMS</li>
|
|||
|
|
<li>Reduce number of partitions on Kafka</li>
|
|||
|
|
</ul>
|
|||
|
|
<div class="notes">
|
|||
|
|
<ul>
|
|||
|
|
<li>SMT = Single Message Transformation</li>
|
|||
|
|
<li>Remove “public” schema on PostgreSQL</li>
|
|||
|
|
<li>Remove double schema names on MySQL</li>
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="tables-with-the-same-structure">Tables with the same structure</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA4MAAAFACAYAAAAGdKnJAAAgAElEQVR4XuydB7xcRfn+0wtJqAkJKaRCAqGHrkhQmiBFehFEQRBQBKlSJCJIFUGKolTp0nuH0GuoRgykQhJIoZeQRv7f5/zO7P/cc3f3brt39977zOczn909Z8o7z8yeM8+877zTto2DETACRsAIGAEjYASMgBEwAkbACLQ6BNq2uha7wUbACBgBI2AEjIARMAJGwAgYASPQxmTQg8AIGAEjYASMgBEwAkbACBgBI9AKETAZbIWd7iYbASNgBIyAETACRsAIGAEjYARMBj0GjIARMAJGwAgYASNgBIyAETACrRABk8FW2OlushEwAkbACBgBI2AEjIARMAJGwGTQY8AIGAEjYASMgBEwAkbACBgBI9AKETAZbIWd7iYbASNgBIyAETACRsAIGAEjYARMBj0GjIARMAJGwAgYASNgBIyAETACrRABk8FW2OlushEwAkbACBgBI2AEjIARMAJGwGTQY8AIGAEjYASMgBEwAkbACBgBI9AKETAZbIWd7iYbASNgBIyAETACRsAIGAEjYARMBj0GjIARMAJGwAgYASNgBIyAETACrRABk8FW2OlushEwAkbACBgBI2AEjIARMAJGwGTQY8AIGAEjYASMgBEwAkbACBgBI9AKEahpMjhgwIDL27ZteyDxlGnTpp2er39WXnnl67i/byrNH957770xhfTrwIEDd1myZMkVxMfff//93cizpJB8TmMEjIARMAJGwAgYASNgBIyAEWiOCLQYMgiZOwUit03cCUP4XIlYMBmETN5L+u2Vf/HixQNmzJgxvTl2aLVk7tWrV/euXbt+ofrbtWvXderUqd9USxbXawSMgBEwAkbACBgBI2AEjEDDCLQYMphsKhrFi9EmHl4kGdyd9FcSn0CbuBOf1gw2PH4yKUwGiwDLSY2AETACRsAIGAEjYASMQA0gYDJYA53QEkQwGWwJveg2GAEjYASMgBEwAkbACLQmBMoig/369evfvn379wHs9U6dOm28YMGCi/m+C7E98VHMBY/BXHBqABRTzBP4fqb25rEv76BwfdCgQSO+/fbbt/n9EVq5nuF6cs8g1+aT7yg+l+NzAp8XUMbV2TqrUM0g8vyd/IekypiEDMNyDQJkXYf6jyd+jzSS9WPis/w+C3leKWfwjBo1quOcOXN+Sxn7E4cSvyY+RdknU/Z/QtmYxP6Za79F+3kTeyn3TtdJu97j2gDS/Jh8dyZwluynEjfjWg/iZOJVtPc8Pr9N9NNP+H4t8SLqeJD0Z/N9FfUP36+kzDF8X6z01HUTH3vmazdlPImco5NpaMN69Pnh3NuE6/2JSxG/jOs4kTpuLgdL5zUCRsAIGAEjYASMgBEwAkYgPwKVIoNTqUbESgTvdeLqcZzO/rt12H/3UUwcSiWDN0FC9qKMTyAPi/jeS+Xx/VhIhohMnVAEGfwdGXeNM3fjcwQxJxmk3J2pUySlU5xnQeL7QuT6ESTm4RIHXVuI1R3klYmqgohRV6KItfbibQxp+69uIMcGyPESXz/v0aNHr/Hjx0uOKAwePHhtMFcffAkZ7xX27lH2d7j2SFymTGBVvghhmzQ5J21EBqnjTu6N5rv67xPiKKLGzAnIIoIoMjiGjy2J7YgidgrPqdggE2W8Bi6/Dr/Jo/QPEDuEa3wuJIqQdqbeXehXYeFgBIyAETACRsAIGAEjYASMQCMhUCky+Bny/R6CcCmfizTJZ8L/IJ8/IJ7NdZFAEYeSyCBZ50EQDoIg3KByIEOH8fsSvn4D8ekfyGbAqFAymMQUjd/GaKqe51pWMsj9Zbk/lfvLEO+H4BwBwZlEm+Ss5q/ErZHpUGS8opS+ohztWfw3UaRrTzB7tHfv3kt17txZmEpTeC/Xdghlk34i34cix7bI8VC4jsbtZK79EVluRJZ9EumlWRzJvRs6dux42OTJkz/r37//5hDG27m+PHF9yh8X91PQDIpwH0M5f05dn0XaPsl2FmMmGpz1SGNIPLJDhw5vT5w4cb7KGzlyZCfIrUihxpGDETACRsAIGAEjYASMgBEwAo2EQKXIoDxIroYW6n9BTojGtlx7gMn+O5CJ4TGZKJUM3g/5iDx9JsjNq3xfl7g/92TSmAmNQQYhMDInlfbzg0WLFg2bOXOmTDijMGzYsM6YyK4RyFQpfYXMt4PVj8l7BOVcFMro27fvUpAlmeJ2RwvYI2gBkedPXJNm8++kPzSBy4t83zBpIkpfrEVfvMH1mXwOTXr6pJzjuX4WMeN5NWgGuSbS15fPYEIqkj+P34jUYVkRylBvkWTwafJ9l3g85Z9TCl7OYwSMgBEwAkbACBgBI2AEjEB5CFSKDC6BDHQeN26cTP2i0KdPn17sI5wtIsGEvzOfi8rQDJ5FGSI+mYAG7BIIjzSEZ0A2T07eayQyeBl1HEy8Gll+Vh7s9XODzbtc1V7FoZSvvXzJto6lrZujmVxl+vTp0gi2SRI80mvP3RJMRHujKf2A7zIRXTGQPrDal/w6h/Fa0krLmAloPEdT7hNcuJ570ghKgxs0g89yTaQtE7gn89JulD84uR+0GDJI/5xEv+ncSJmSShv7KDI8Ttue4Xu0F9HBCBgBI2AEjIARMAJGwAgYgcZFoFJkcDGkIbn/S1K3hzhEpn4Qh+UgDp+WSgazHToPwTkdgnMSxV9A3XIskwmNQQap78Z432LG7LWSXQM2InF1TC/T5VP/mklHMuTRHsLVIFIbQaReQsafa/+fTEFJt2/In9Bq5hP5NnDcTQkSZPAxrml/XyZUggxqbNBHh9Cvh/F9ZCic33OQ/by0Q5tK4uyyjIARMAJGwAgYASNgBIyAEfg/BBqNDCY0g22wbuws88ZcZBASswkkQE5HcnoTRfsnTVKSlFzDj/2zEUWIhrxg/or7BR8639CeweB5NL0Xr1IDifJlYjs89t6ZMb9Mlo/n1t9MmTJlVrhGnlP5PoZ4JgTqRHC8Axx3TnsRBY+9JDfpphOfzSYz958GY+3DLIkMsr+xG/sbpTVsgxltt6QZbT6MkHkw8opwbkXcmdgRWX6NLPJM62AEjIARMAJGwAgYASNgBIxAIyFQKTK4BO3fUsm9aEzyt2OSfx9yT4OoDJL8Cccvt3Btj9AmSI3ML2WGWRAZjImHzCWlSduSsh5L4kM950EojuZaPa1hLhwLIINhz+AnaOKGoYnTkRIVC2CgfY8yzzyR9pxZSMGYhQ7HLFQk8m1Mctdl3+JcvqsvMiaiKgeT0mFckxnq3K5duw6ZMGGCvJPmDKVoBimsHflkJtwO7FeHzOmokKIC+bV3UXsY7wGDHYvK7MRGwAgYASNgBIyAETACRsAIFIVApcigKj2NfWOna98g5w+ugBZLRyysByG8BJNFaemkcZL2R9e/gjBsBWF4HqKyJkTlKq7p2AJ50tSeOX2KPF5OugOT2j959YQAXaHjB0jyFmnX4TNzRp7yQUQPp15plqYgxxZo06bFBHIg6aPjGdKhADK4HHmmEXUcwzPUf3CS8NCO5XMRxNh8U05h7gj78tL1g4325smxikxrj50/f/4/Z82a9VWcri3ydU6S7ZCffK/xXRj8hnhh2kQ0kU7HSkgD91yseZMDnigge1dkl2OYKJRIBpUvyHI/fXQIXl6liWyj8ZD0+MrvVbt06fLppEmTtKc0CmC0GrJfzdcNG0v7GurypxEwAkbACBgBI2AEjIARMAIVNBMFzKeIGxBlxjiQqD2Ec5ngrw0ZnBmDrX2E8moZ9olJkySzwGO0Vyx0CN/3Js9NgQxyXaRBHjV1FqAOP9fZezMgkVsmPZiG/JCN/pBAacK6xNfk+VOHmr8KGRPpDNqyq0MeZOhBvWvx+xtidMSCwrx587blIPjI/BHCsgtpdMh6x/j253zq3tLE7mFvZMgbPmmzHKPonD+V1yOUl05H+adQ/mnxdTlS0T5CkcMVieci+5gsZUceWok6mqGzzETB7q50uiFDhqyM+eaTXB8U35MpqjSJwnQFyhY+0R7PUslgwlFNqD7yPEpsR/nq
|
|||
|
|
<div class="incremental">
|
|||
|
|
<pre><code>{
|
|||
|
|
"transforms": "events",
|
|||
|
|
"transforms.events.type": "org.apache.kafka.connect.transforms.RegexRouter"
|
|||
|
|
"transforms.events.regex": "namespace\\.database\\.public\\.events_[0-9]{4}_[0-9]{2}",
|
|||
|
|
"transforms.events.replacement": "namespace.database.public.events",
|
|||
|
|
}</code></pre>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="remove-public-schema">Remove “public” schema</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyIAAACICAYAAAAbIl9LAAAgAElEQVR4Xu2dB7gcVfn/Ifem0gMhIY0khBp6V6rSQQRRpAsKKqLwowgoggRB6YgCCihKFQUhgBSVIghSRAQiRUqSGxJCGhAMIbm5N8n/893/zD6ze3f3zu7dOvs9z3Oe3Z095X0/c2bmvHPOec/yyzmYgAmYgAmYgAmYgAmYgAmYQJUJLF/l+lydCZiACZiACZiACZiACZiACSxnQ8SNwARMwARMwARMwARMwARMoOoEbIhUHbkrNAETMAETMAETMAETMAETsCHiNmACJmACJmACJmACJmACJlB1AjZEqo7cFZqACZiACZiACZiACZiACdgQcRswARMwARMwARMwARMwAROoOgEbIlVH7gpNwARMwATKRWDs2LF9P/7445V79+69cq9evVZesmTJyssvv/wqxJWpY8CyZctW4PgK+uR36pP/NuR7f2ILsRexNfJdxxQVOontpJ/H53vkfYP4t2nTpv2lXPK7HBMwARNoZgI2RJr57Ft3EzABE6gvAr3WXnvtwXT8B3d2do5raWnZiI7/WEQcTlyN4zImZED043gfPnsHhkS1tXgJWc5GhraFCxdOnTNnzscjR478HELswPG3li5d+hayv9XW1jaz2oK5PhMwARNoJAI2RBrpbFlWEzABE2hgAoxeDFq0aNEYRihG0mEfSYd9Kz7XQaVhxIGBkdGIGi5AaBlJ2UEjKa9isDyDzk+j6zM2Thrx9FpmEzCBShGwIVIpsi7XBEzABJqUgAyO9vb2Lel4bwCCDfWJ0bExn6s3KZKo2v+Fw2MYJxPeeeedR8zDBEzABJqZgA2RZj771t0ETMAEykBgxIgRG/PGf3uMjU/Ryd6eIjcqQ7HNUMR78Jpgo6QZTrV1NAETyEXAhojbhQmYgAmYQGwCQ4cOHcDC8KjR8SkyV2Kko4Ny59NRn8vne8QZxA/otL/PsTl8zuK31o28ggHUiiHUymdvfWpBOnF1vkuugfoeyDhQozLBb00Fq5tnIDLNQZ47kO8ej5TEbo5OaAIm0OAE6uYm3OAcLb4JmIAJJJbAqFGjNsAb1b50kvdDyc+WUdH/UdYk4mQ64q9hOLys79QzhbUUWl9R0TB8+PCB1Kl1Kp3Ur3Uqw/jUwvhhHBvB50jimhUVInfhH3D4buK1GCUv1KB+V2kCJmACVSFgQ6QqmF2JCZiACTQWAbxA7YDE+9Eh35fO+WZlkH4hZT1HWc9T1vMYAM9jbLSVodyKFqERoD59+ozEENMC+7UV0WFtKlWUoSLDJXT3my3LLfKiRfqt+GNLooybogJ5XyaeN3369AlFZXRiEzABE2gAAjZEGuAkWUQTMAETqAYB1np8Phj10MiHRgh6El6SwaEYGB36ncjAiJGmqt2OcjJOtPfIQ8RHGM24KqowIzDDYCEDb/cgji4CyAecm5uIZ2G/LSoin5OagAmYQN0SsCFSt6fGgpmACZhA5Qmwb8eWvHE/ipq+QtS6iVLDVMp5VB1wRhAenTRp0uxSC2qWfFrkj2ERGiV7orf2RekuLCHPU7A+G0Pnqe4S+38TMAETqGcCNkTq+exYNhMwAROoAAHtRt7R0XFUYIDsXGIVi2V0UIbio0wdmlhiOc72/wm0YJikjBIMjS/xOSoGmHdJ+7OpU6deGiOtk5iACZhA3RGwIVJ3p8QCmYAJmEBlCDD6IQ9XoQGyYgm1vEaeB2R8TJs2TaMfS0oow1liEGAa17ZM4zqVpAcQ+3WTZS7n5Keck5/EKNpJTMAETKBuCNgQqZtTYUFMwARMoDIEMEAOp+ST6KxuV0INWttxL2/e7+XN+4sl5HeWHhLAccA34H8K508bRBYK8rb1c6ZsndfDKp3dBEzABKpCwIZIVTC7EhMwAROoPgE6sFr3cTJxi2Jqp8P7nAwPGSB0ajUK4lAHBDAo1+LcXIkociawQgGRPuL8XY3heHYdiG0RTMAETCAvARsibhwmYAImkDACrDX4qt6go9YmcVWjg/ssae/k8z7We7wdN5/T1YYARuYZnKvvcp4HFZDgY/77BcbkmbWR0rWagAmYQGECNkTcQkzABEwgIQTonH4dVWSAbBhTpXdJ9zs6s7d72lVMYnWWDKPzFM6fDI3BBUTT7vSnYpDcUmfiWxwTMIEmJ2BDpMkbgNU3ARNofAJ0Rg9kYfPFvCFfL442dFx/r30vWNx8X5z0TlP/BDBCT0TKs4hD8klL+5hIPJIRr//Uv0aW0ARMoBkI2BBphrNsHU3ABBJJgF2/R7S2tl6DcvvHUPAZOqG3avSDN+MfxkjvJA1IAKP0eM7xOYg+tIBBMoG2cAQGycIGVNEim4AJJIiADZEEnUyrYgIm0DwE6HCeQ4fzh2jc2o3Wd5DuOqZePdY8dKxpYJD8GBL5NqlchDFyKaNiakMOJmACJlATAjZEaoLdlZqACZhAaQTYX2JXDIubiCMLlLCATuYvidfz1vut0mpyriQQwNPW1bSD49GlJZc+tKOpHD8MQ/WZJOhrHUzABBqLgA2RxjpfltYETKCJCbAOQKManymAYAr/Xcl6kevb2toWNTEqqx4hMGrUqCGsCbqVQ7vlAyPDldGREwzOBEzABKpJwIZINWm7LhMwARMogQDTbE7gzfUlZM23d8Rs/judtR83l1C8szQJAQyS7TFI5DlrbB6VJ3H8ANrRq02CxGqagAnUmIANkRqfAFdvAiZgAoUIMAoiz1b5FqO3899FdBzHm6IJxCVAm9Iml1o/MiBHnmUcu5g29f245TmdCZiACZRKwIZIqeSczwRMwAQqSIC315vz9vp+qhiWp5o729vbvzpr1qwFFRTDRSeUAO1rVdrXvai3cx4VX8Ej236TJ09+J6EIrJYJmEAdELAhUgcnwSKYgAmYQJQAb6zlyei8XFSYovUG8/n35Y31ZFMzgZ4SYDH7t2lPl1NO3xxldXDscNraH3taj/ObgAmYQM5nmrGYgAmYgAnUB4EhQ4YM6tOnjzp9+d5SX0enUB6QHEygbASCxeyaArhNnkI1Vet7ZavQBZmACZhAQMAjIm4KJmACJlAHBHDLuwnerh5GlMHZ4jAKMn/JkiWH4or3wToQ1SIklABOEc6irWkkLtfeNPdjjMTZODOhdKyWCZhAJQjYEKkEVZdpAiZgAkUQwAjZDiPkz2RZNUe2F/hvd9zxziuiSCc1gZIIYIzshDFyD5lzbYT4Km1xR7fFktA6kwmYQA4CNkTcLEzABEyghgSYFrMri4YfQIRcHoz+wVvoHWsonqtuUgKsHXmMtSO59qz5AENldzZAfLFJ0VhtEzCBMhKwIVJGmC7KBEzABIohQGdvHzp7mpufayrMKRghVxZTntOaQDkJ0D5/Qfv8VnaZHOtkZGRn78ZeTtouywSak4ANkeY879baBEygxgTo5B1Eh+6uPGIcjxFyXY1FdPUmsBzt9DjaqdpirywcixgZ+bRHRtxITMAEekLAhkhP6DmvCZiACZRAAPe8XyLbnXmyHu0d0kuA6iwVI8C6kW0wOv5CBatlVbKAkZGtWTPy34pV7oJNwAQSTcCGSKJPr5UzAROoNwKjR4/eDA9YzyJXvxyyHYIRcke9yWx5TACHCmMxOp6ExJAsGh+1tLRsNmXKlKmmZAImYALFErAhUiwxpzcBEzCBEgkMHTp0ALtV/5Ps43IU8XmMkD+VWLSzmUDFCTAyMpSRkZepaI2syuZijGyMMTKr4kK4AhMwgUQRsCGSqNNpZUzABOqZAFOyNNpxcLaMdO6+zlz7X9ez7JbNBEQAL2+j8PImj1nZrqanY0iPMCUTMAETKIaADZFiaDmtCZiACZRIgEW/P2bR71k5snvX6hKZOlttCDC9cH2mF/6L2lfMkuABjJHP1UYq12oCJtCIBGyINOJZs8wmYAINRYApLccw6vHbHELfRcdNC9cdTKChCLBmZBPWjDyH0P2zBP8mbfr6hlLGwpqACdSMgA2RmqF3xSZgAs1AgJGQLRkJeSGHrq92dnZuO2PGjE+agYN1TB4Bphp+E62uzdJs
|
|||
|
|
<div class="incremental">
|
|||
|
|
<pre><code>{
|
|||
|
|
"transforms": "public",
|
|||
|
|
"transforms.public.type": "org.apache.kafka.connect.transforms.RegexRouter"
|
|||
|
|
"transforms.public.regex": "namespace\\.test\\.public\\.(.+)",
|
|||
|
|
"transforms.public.replacement": "namespace.test.$1",
|
|||
|
|
}</code></pre>
|
|||
|
|
<div class="notes">
|
|||
|
|
<p>Multiple transformations can be defined using a comma-separated list of transformation names:</p>
|
|||
|
|
<pre><code>{
|
|||
|
|
"transforms": "events,public"
|
|||
|
|
}</code></pre>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="toast">TOAST</h3>
|
|||
|
|
<p>The Oversized-Attribute Storage Technique</p>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><code>UPDATE</code> without modifying the attribute</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<pre><code>{
|
|||
|
|
"op": "u",
|
|||
|
|
"before": null,
|
|||
|
|
"after": {
|
|||
|
|
"namespace.database.public.toasty.Value": {
|
|||
|
|
"id": "2e142822-cc92-4c4f-9fdc-ec45504753a9",
|
|||
|
|
"updated_at": {
|
|||
|
|
"string": "2024-01-17T10:53:34.929630Z"
|
|||
|
|
},
|
|||
|
|
"clob": {
|
|||
|
|
"string": "__debezium_unavailable_value"
|
|||
|
|
}
|
|||
|
|
...</code></pre>
|
|||
|
|
<div class="notes">
|
|||
|
|
<ul>
|
|||
|
|
<li>https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-toasted-values</li>
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="toast-1">TOAST</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li><code>ALTER TABLE... REPLICA IDENTITY DEFAULT</code>
|
|||
|
|
<ul>
|
|||
|
|
<li>Value replaced by <code>unavailable.value.placeholder</code></li>
|
|||
|
|
</ul></li>
|
|||
|
|
<li><code>ALTER TABLE... REPLICA IDENTITY FULL</code>
|
|||
|
|
<ul>
|
|||
|
|
<li>Value included</li>
|
|||
|
|
</ul></li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="databases-with-low-writes">Databases with low writes</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABEwAAAJiCAYAAADdd1fGAAAgAElEQVR4XuzdCXxU1fn/8cxMFvZNEIRACARRQMjihtUW97VqLdraWm2rf3dtXeteta7Vav2pdV9ata1LtS6tay1WrQKGBBBXIJCwqKwBZEkyM//vmc7gZJjlzmSWO3c+83pdksyce+5z3ucwmfvk3HNdRTwQQAABBBBAAAEEEEAAAQQQQAABBDoJuPBAAAEEEEAAAQQQQAABBBBAAAEEEOgsQMKEEYEAAggggAACCCCAAAIIIIAAAghECJAwYUgggAACCCCAAAIIIIAAAggggAACJEwYAwgggAACCCCAAAIIIIAAAggggEB8AWaYMEIQQAABBBBAAAEEEEAAAQQQQACBCAESJgwJBBBAAAEEEEAAAQQQQAABBBBAgIQJYwABBBBAAAEEEEAAAQQQQAABBBCIL8AME0YIAggggAACCCCAAAIIIIAAAgggECFAwoQhgQACCCCAAAIIIIAAAggggAACCJAwYQwggAACCCCAAAIIIIAAAggggAAC8QWYYcIIQQABBBBAAAEEEEAAAQQQQAABBCIESJgwJBBAAAEEEEAAAQQQQAABBBBAAAESJowBBBBAAAEEEEAAAQQQQAABBBBAIL4AM0wYIQgggAACCCCAAAIIIIAAAggggECEAAkThgQCCCCAAAIIIIAAAggggAACCCBAwoQxgIDzBUaMGLFWrewbbKlXX9dom63tqUGDBj1SX1/fHlIYOXJktc/na4hQWetyuZr13NMej+fOhQsXtoa/rvr7+/3+81TmGD0/SluHtjl67q6Wlpa/Ol+YFiKAAAIIIIAAAggggIDTBZhh4vQepn0FKRCWMDGJDo+2XmEQM9va2g7/4osvVpjnwhImJrGyIbK8kiCz+/Tps/u8efPaTPnKyspJXq/3VX07OFjnSn3to600+PO5zc3NdxYkPI1GAAEEEEAAAQQQQAABxwiQMHFMV9IQBL4RCCVM3G5390WLFm2uqqrqs2XLlmM0I+QWlRqo7SUlNb4bnjDRa9MXL168p3muvLy8u/adqm/v09ZdM1AOXbJkySvmtYqKirPMTBJ9+6LKXKz6P6mrqytZuXLlTXr+fNWzQvWYZIqfPkEAAQQQQAABBBBAAAEE8lWAhEm+9hxxIxBHIDJhEioanE0yQz+XaNtVSZP60AyT8IRJqLzqeVPf76vXpioJ8rew5w/Qvv8KT4qMHz++dP369WaGSoku4xnZ1NS0mE5CAAEEEEAAAQQQQAABBPJVgIRJvvYccSOQQsLE7KIZIn/RTJAfartO641cGS1hohkpZcEZKQ9pl7W6BGeXpUuXrkqErgSLKTNAdVep7gWJyvM6AggggAACCCCAAAIIIGBXARImdu0Z4kKgCwKxZpgEEyanKKHxgL7/h2aJHBFlDRPzvtBbW5PKTdNlN9dpdklTonC0tslgJVa+ULktpaWlfefPn78l0T68jgACCCCAAAIIIIAAAgjYVYCEiV17hrgQ6IJAvITJ8OHDD9IlNq+GLsGJcZccc3QzQ+QTJU3u12yRFxKFo5krF6qsWSPlRSVijkxUntcRQAABBBBAAAEEEEAAATsLkDCxc+8QGwIpCiSYYbKvEhtmbZIGJTZqo12SYxZxXbFixY4qc722o4K3Cz4nVjjDhg0r17olc/R6P81I2UsLwb6fYujshgACCCCAAAIIIIAAAgjYQoCEiS26gSAQSK9Aghkmx2t2yZ91xDeVMNk/3qKveq2b7pCzUGWH6OtY3Snn88hIR40a1bejo8MsAFun7beq81fpbQ21IYAAAggggAACCCCAAALZFyBhkn1zjohAxgXiJUz02j0K4HRtdyq5cW68hIkJVOVf15cDzEKxujTnyfDglSwZ0d7e/oISMJP0+p/1+ol63ZvxBnIABBBAAAEEEEAAAQQQQCDDAiRMMgxM9QjkQiBWwkTPj1I8s7X10oyRQzVj5JUECROP9pmv8iOVFPm+Fn99NtQerYXyAz33B/1s7orzOyVLLtb3vly0l2MigAACCCCAAAIIIIAAAukWIGGSblHqQ8AGApEJk8GDB/fUnWuOUILjVoVXrm2mZpfsoa/+WAmToUOHDiwuLjZrmJxqEiG67GbksmXLWsrLy4dpnZKH9dxB2tYqWXKSlUVhbcBCCAgggAACCCCAAAIIIICAZQESJpapKIhA/giEEiaKuFWbW5u5TXDo8YkWaD2kqalpsXkiym2FzdOd9lGi5R7NLjnTvKCZJUfr5+eClbXp64ooMu8pIXNs/ogRKQIIIIAAAggggAACCCDQWYCECSMCAQcKhCVMTOvMZTJrtH2k2SDPeb3e+zRTZGOo2XFuK2ySIQuUHHlIyZLbg/VEJkyi6mmft7TPFAfS0iQEEEAAAQQQQAABBBAoEAESJgXS0TQTAQQQQAABBBBAAAEEEEAAAQSsC5AwsW5FSQQQQAABBBBAAAEEEEAAAQQQKBABEiYF0tE0EwEEEEAAAQQQQAABBBBIRmBxdfWBRS7XBf6iot20Xy9tK3Tp9bUVs2bdH1nPotraY4r8/of0/JsVDQ1TdaKp3fLvsai6+kK1+RbF/0e146fpaEFTdXU/1XmJ7I6WUYXq3KKfW+Q4UcYn6/sHIo7z1siGhinpOHaydSyurT1Bl/E/pv3+pRgOSHb/eOVle4qd2mqlbSRMrChRBgEEEEAAAQQQQAABBBAoIIFFNTU/V3Mf1BY6ZzTr2xX7Xa5TKmfNeiSSQgmTl5QMONw8X+z1Di+fM2dJPnKlO2Hy1fjxvTaWlr4vi/FBD7O+YIe2L5SQqFCC4gglKC4Jvradvu6kzZEJE7u11cr4JGFiRYkyCCCAAAIIIIAAAggggECBCHy+xx59Stralqq5vTQj4Amvy3XZ6Pr6Zv+xx3rqFy5071pf3x5JoZkSxyqZ8rCmlfxbiYCjmGHyPyElnq7Vlyu1rdQMk5+7y8reGP7ee5s+r6oqGzN//pZwx8U1NVPl97RTEyZ2a6uV/84kTKwoUQYBBBBAAAEEEEAAAQQQKBCBYPLjKZ0stqxwu0dHS5A4lSLdM0yUMJknq3FKJp2umTn3xXMjYWK/UUXCxH59QkQIIIAAAggggAACCCBQQAILJ0wY7CkuvsTnch2uE7QRavpGfZ2r2R13a52LpyIpgutM/ELPm8s82jUrYbrb77+morHx3fCyOlk3l9SYNTK+r/qqdMnM2So7WM8t1NdrKhsa/hqNuam29mqX3/9rvfaYZoucGKsrdIJ/r+o5LeL1BdqnKnIfxfJhMF7z0icqs7OSE5cptjO09VVs9fp65shZsz42BVT+C30ZYGZl6JKV3wfa6fcf73a5fqpjHqufP+ooLj6maubMltCxPqirKxnk852v103Mo4OO/3F5vVeMmDPHHL/TQ46nqk7jaMp+qTbfpn1L0rWGiepyy8jMxnG7iotHV8ycuTDesLaaMJHNZtVTFnnpk/b/q475A8V/jhzv6jQWamu/p59PlHO1vpoxUKKtVdsqt9t92Ij6+gWmfPgaJqrrN/K/XfvsrJdWa3u4oqrqatfTT3vjtcPKa1bbGhgLdXW1RT7fWfp2stpWrnh66PsN+rrK73ZfpkTUk6bckokTyzs8nhbFPbfY7/++1+2+T2XM+jsm3n+Wut3nDq2vX2klvlAZEibJaFEWAQQQQAABBBBAAAEEEEijQEttbZXX739LVQ4NVmtOsIu1Bc7VNDPhGp0QXh06pBYQvVQnsTcEfzYngp7Qfj6f74hRs2e/FiobSpio/KNKDPxEz3+ok8keqnhMoG63e8/K+vrpkc1RIuM2nZieZxIHOvG+OFZzg7F8P/h6T30162/ESpj8Sa+N0ra7Nl3l4zpOMf1d38/VVq7NrN+xQAutjjGX8wQTJubE/m8qt0rlT9X3yxT/Y3r9YH1fra83qfylgbbIq7mm5jl9PSoYzwZ97R7wcbnWKxmyp8p+FGZzgb6/NfizOYl2axugE+zHVf6EdCz6Gry0ySQlinSy3lMn6xtjWZrnrSYRkk2YaMbQrzWOro44
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="databases-with-low-writes-1">Databases with low writes</h3>
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABEwAAAJiCAYAAADdd1fGAAAgAElEQVR4XuzdCXwU9f3/8ezm4iZcghBIuKsgJMELf1JBpSpaqxWtWutRqfWu2tZ6V229rda/Wo961fuoWrXeWrVqFTCEQzwQCAmXyBlucuz+P9+4C8tmj5nZmZ3vzrz28VhCdr/zne/3+f1msvPOHIE8HggggAACCCCAAAIIIIAAAggggAACOwkE8EAAAQQQQAABBBBAAAEEEEAAAQQQ2FmAwIQZgQACCCCAAAIIIIAAAggggAACCMQJEJgwJRBAAAEEEEAAAQQQQAABBBBAAAECE+YAAggggAACCCCAAAIIIIAAAgggkFqAI0yYIQgggAACCCCAAAIIIIAAAggggECcAIEJUwIBBBBAAAEEEEAAAQQQQAABBBAgMGEOIIAAAggggAACCCCAAAIIIIAAAqkFOMKEGYIAAggggAACCCCAAAIIIIAAAgjECRCYMCUQQAABBBBAAAEEEEAAAQQQQAABAhPmAAIIIIAAAggggAACCCCAAAIIIJBagCNMmCEIIIAAAggggAACCCCAAAIIIIBAnACBCVMCAQQQQAABBBBAAAEEEEAAAQQQIDBhDiCAAAIIIIAAAggggAACCCCAAAKpBTjChBmCAAIIIIAAAggggAACCCCAAAIIxAkQmDAlEEAAAQQQQAABBBBAAAEEEEAAAQIT5gAC3hcYMGDAOull10hPW+TrWnnOkuezvXr1eri6uropqlBeXl4RCoVq4lTWBQKBenntufz8/DsXLlzYEPu+1N8tHA5fKGV+Kq8PkmezPGfLa3ctXrz4ae8L00MEEEAAAQQQQAABBBDwugBHmHh9hOmfLwViAhMVdOTLs1MMxPTGxsbDv/3225XqtZjARAUrG+PLSwgyq0uXLnvPnTu3UZUfOHDg6JaWljflv70jda6Sr13kWRT5/vz6+vo7fQlPpxFAAAEEEEAAAQQQQMAzAgQmnhlKOoLADoFoYBIMBtsvWrRo65AhQ7ps27btp3JEyC1Sqqc8/y2hxo9jAxN5b2pdXd2+6rXS0tL2suxk+e998mwvR6ActmTJkjfUe2VlZeeoI0nkv69ImYul/q/GjBlTuGrVqhvl9YuknpVSjwpTwowJAggggAACCCCAAAIIIJCrAgQmuTpytBuBFALxgUm0aORokmnyfaE895TQpDp6hElsYBItL/X8R/4/Qd6bLCHI8zGvHyzLvhsbiowYMaJow4YN6giVQjmNp7y2traOQUIAAQQQQAABBBBAAAEEclWAwCRXR452I2AhMFGLyBEiT8mRIMfL889yvZErEwUmckRKceSIlAdlkXVyCs4eS5cuXZ0OXQIWVaa71D1E6l6QrjzvI4AAAggggAACCCCAAAK6ChCY6DoytAuBDASSHWESCUymSKDxd/n/q3KUyBEJrmGitgud5Vkr5d6X027+LEeX1KZrjlzbpLcEK99KuW1FRUVd58+fvy3dMryPAAIIIIAAAggggAACCOgqQGCi68jQLgQyEEgVmPTv3/9HcorNm9FTcJLcJUetXR0h8pWEJvfL0SIvp2uOHLnyOymrrpHyigQxR6Yrz/sIIIAAAggggAACCCCAgM4CBCY6jw5tQ8CiQJojTCZIsKGuTVIjwUZVolNy1EVcV65cOUzKXCfPn0RuF3xesub069evVK5bMlveL5EjUvaTC8F+arHpLIYAAggggAACCCCAAAIIaCFAYKLFMNAIBOwVSHOEyQlydMmTssb/SGByUKqLvsp77eQOOQulbB/5OlzulPNNfEsHDRrUtbm5WV0Adow8b5Y6/2Bvb6gNAQQQQAABBBBAAAEEEMi+AIFJ9s1ZIwKOC6QKTOS9e6QBZ8rzTgk3zk8VmKiGSvm35cvB6kKxcmrOM7GNl7BkQFNT08sSwIyW95+U90+W91sc7yArQAABBBBAAAEEEEAAAQQcFiAwcRiY6hFwQyBZYCKvD5L2zJJnJzli5DA5YuSNNIFJviwzX8qXSyhyjFz89YVof+RaKD+T1/4m36u74vxFwpKL5f8hN/rLOhFAAAEEEEAAAQQQQAABuwUITOwWpT4ENBCID0x69+7dUe5cc4QEHLdK80rlOV2OLtlHvoaTBSZ9+/btWVBQoK5hcoYKQuS0m/Jly5YtLi0t7SfXKXlIXvuRPNdJWHKKkYvCasBCExBAAAEEEEAAAQQQQAABwwIEJoapKIhA7ghEAxNpcYM8g/JUtwmOPr6SC7QeWltbW6deSHBbYfXyTstI0HKPHF1ytnpDjiw5Sr5/MVJZo3xdmUDmEwlkjs0dMVqKAAIIIIAAAggggAACCOwsQGDCjEDAgwIxgYnqnTpNZq08v5CjQV5saWm5T44U2RztdorbCqswZIGEIw9KWHJ7pJ74wCShnizzgSwz3oO0dAkBBBBAAAEEEEAAAQR8IkBg4pOBppsIIIAAAggggAACCCCAAAIIIGBcgMDEuBUlEUAAAQQQQAABBBBAAAEEEEDAJwIEJj4ZaLqJAAIIIIAAAggggAACCJgRqKuomJgXCPw2nJe3lyzXSZ4r5dTra8tmzLg/vp5FVVU/zQuHH5TX/1NWUzNZdjRlsdx7LKqo+J30+RZp/z+kH6fa0YPaiooSqfMSsTtKjMqkzm3y/WJxHCXGp8v//x63ng/Ka2rG27Fus3XUVVWdJKfxPybLvSttONjs8qnKi+0UnfpqpG8EJkaUKIMAAggggAACCCCAAAII+EhgUWXlL6W7D8gzus+orm9XEA4EpgycMePheAoJTP4tYcDh6vWClpb+pbNnL8lFLrsDk+9GjOi0uajoU7EYEfFQ1xdslue3EkiUSUBxhAQUl0Te6yFffyBPTwYmuvXVyPwkMDGiRBkEEEAAAQQQQAABBBBAwCcC3+yzT5fCxsal0t1OckTAEy2BwGWDq6vrw8cem1+9cGFwz+rqpngKOVLiWAlTHpLDSt6TIOAnHGHyvZAET9fKlyvluUqOMPllsLj4nf6ffLLlmyFDiofOn78t1rGusnKy+D3n1cBEt74a+XEmMDGiRBkEEEAAAQQQQAABBBBAwCcCkfDjWdlZXLwyGBycKCDxKoXdR5hIYDJXrHaXMOlMOTLnvlRuBCb6zSoCE/3GhBYhgAACCCCAAAIIIICAjwQWjhzZO7+g4JJQIHC47KANkK5vlq9z5OiOu+U6F8/GU0SuM/EbeV2d5tEkRyVMDYbD15TNnPlxbFnZWVen1KhrZBwj9Q2RU2bOlbK95bWF8vWagTU1Tydirq2qujoQDv9R3ntMjhY5OdlQyA7+vVLPr+PeXyDLDIlfRtryeaS96q2vpMxuEk5cJm07S55dpW3V8vXs8hkzvlQFpPy38qW7OipDTln5a2s/w+ETgoHAqbLOY+X7L5oLCn46ZPr0xdF1fTZmTGGvUOgieV+1eXDE8b+BlpYrBsyerda/00Mcz5A6laMqu0L6fJssW2jXNUykrqAYqaNxgoGCgsFl06cvTDWtjQYmYrNV6imOP/VJln9a1vkzaf954njXTnOhqupo+f5kca6Qr2oOFMqzQZ6rg8HgpAHV1QtU+dhrmEhdfxL/22WZ3eStNfJ8qGzIkKsDzz3XkqofRt4z2tfWuTBmTFVeKHSO/Hes9K1U2tNB/r9Rvq4OB4OXSRD1jCq3ZNSo0ub8/MXS7jkF4fAxLcHgfVJGXX9Htfe1omDw/L7V1auMtC9ahsDEjBZlEUAAAQQQQAABBBBAAAEbBRZXVQ1pCYc/kCr7RqpVO9gF8mzdV5MjE66RHcKro6uUC4heKjux10e+VzuC+dHlQqHQEYNmzXorWjYamEj5RyQY+IW8/rnsTHaQioe21h0M7juwunpqfHckyLhNdkwvVMGB7HhfnKy7kbYcE3m/o3xV199IFpg8Ku8Nkufe8pSzfALHSZv+Jf+fI89SearrdyyQC60OVafzRAITtWP/vJRbLeXPkP8vk/Y/Ju8fIv+vkK83SvlLW/siXvWVlS/K159E2rNRvrZv9QkENkgYsq+U/SLG5rfy/1sj36ud6KA8u8sO
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="heartbeats">Heartbeats</h3>
|
|||
|
|
<pre><code>{
|
|||
|
|
"heartbeat.interval.ms": "1000",
|
|||
|
|
"heartbeat.action.query": "update ovh_debezium_heartbeat set ts=now();"
|
|||
|
|
}</code></pre>
|
|||
|
|
<hr />
|
|||
|
|
<h2 id="kafka">Kafka</h2>
|
|||
|
|
<ul>
|
|||
|
|
<li>Queue system</li>
|
|||
|
|
<li>3 clusters dedicated to CDC
|
|||
|
|
<ul>
|
|||
|
|
<li>Europe (5,000 to 15,000 events per second)</li>
|
|||
|
|
<li>Canada (2,000 events per second)</li>
|
|||
|
|
<li>USA</li>
|
|||
|
|
</ul></li>
|
|||
|
|
<li>20 brokers</li>
|
|||
|
|
<li>PCI DSS</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="features">Features</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>Partitioning</li>
|
|||
|
|
<li>Replication</li>
|
|||
|
|
<li>Compaction</li>
|
|||
|
|
<li>Compression</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="partitioning-and-replication">Partitioning and replication</h3>
|
|||
|
|
<pre><code>{
|
|||
|
|
"topic.creation.default.partitions": "6",
|
|||
|
|
"topic.creation.default.replication.factor": "3"
|
|||
|
|
}</code></pre>
|
|||
|
|
<div class="notes">
|
|||
|
|
<ul>
|
|||
|
|
<li>Partitioning: slice large piece into smaller ones</li>
|
|||
|
|
<li>Replication: maintain copies of a partition</li>
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="number-of-partitions">Number of partitions</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li>1 group of 80 databases
|
|||
|
|
<ul>
|
|||
|
|
<li>Country + <em>Universe</em></li>
|
|||
|
|
</ul></li>
|
|||
|
|
<li>140 tables per database</li>
|
|||
|
|
<li>1 table = 18 partitions (<code>partitions</code> (6) * <code>replication factor</code> (3))</li>
|
|||
|
|
<li>Estimation of <strong>201,600 partitions</strong></li>
|
|||
|
|
</ul>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><a href="https://www.confluent.io/blog/apache-kafka-supports-200k-partitions-per-cluster/">Maximum 200,000 partitions</a></p>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="number-of-partitions-1">Number of partitions</h3>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><strong>Topic routing</strong></p>
|
|||
|
|
</div>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p>14 topics</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p>252 partitions</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<p><strong>-99.88 %</strong></p>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="compaction">Compaction</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li><strong>Retention</strong> policy</li>
|
|||
|
|
<li>Keep only the <strong>last value</strong> of each key identifying a message</li>
|
|||
|
|
<li><strong>Full copy</strong> of the database <strong>on Kafka</strong></li>
|
|||
|
|
<li>History of changes kept for 7 days (tombstones)</li>
|
|||
|
|
</ul>
|
|||
|
|
<div class="incremental">
|
|||
|
|
<pre><code>{
|
|||
|
|
"topic.creation.default.cleanup.policy": "compact",
|
|||
|
|
"topic.creation.default.delete.retention.ms": "604800000"
|
|||
|
|
}</code></pre>
|
|||
|
|
<hr />
|
|||
|
|
</div>
|
|||
|
|
<h3 id="compression">Compression</h3>
|
|||
|
|
<pre><code>{
|
|||
|
|
"producer.override.compression.type": "lz4"
|
|||
|
|
}</code></pre>
|
|||
|
|
<div class="notes">
|
|||
|
|
<p><code>zstd</code> algorithm for the future but it’s not a priority</p>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
<h3 id="from-kafka-to-the-data-lake">From Kafka to the Data Lake</h3>
|
|||
|
|
<hr />
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABKEAAANTCAYAAAB2Og67AAAgAElEQVR4XuydB3hUxRbH594t2ZKy6aEGQkmjhxSKNFGx0gQLvSRBSGgioIIEEEVAKQklAakiSlEQEVR6TwKhpVGSAOl1N2V3s+3ed2aTDSsPBVEU4cz39rE7OzN35ncvH+v/O+d/GIINCSABJIAEkAASQAJIAAkgASSABJAAEkACSAAJPGICzCNeH5dHAkgACSABJIAEkAASQAJIAAkgASSABJAAEkACBEUofAiQABJAAkgACSABJIAEkAASQAJIAAkgASSABB45ARShHjlivAASQAJIAAkgASSABJAAEkACSAAJIAEkgASQAIpQ+AwgASSABJAAEkACSAAJIAEkgASQABJAAkgACTxyAihCPXLEeAEkgASQABJAAkgACSABJIAEkAASQAJIAAkgARSh8BlAAkgACSABJIAEkAASQAJIAAkgASSABJAAEnjkBFCEeuSI8QJIAAkgASSABJAAEkACSAAJIAEkgASQABJAAihC4TOABJAAEkACSAAJIAEkgASQABJAAkgACSABJPDICaAI9cgR4wWQABJAAkgACdyHwK5k/olgNLAV/q54Im4kHgIJIAEkgASQABJAAo+GAP5YfDRccVUkgASQABJAAg9OgIpQVgJOmkoby/J8urejbKllkTSlZirheW9fJ3m4pS+9tKoPzzCTCRG+5+tkc4X2p1XwzgynXcnx3E9+jrab78xXRxHC2Po6yqbVzVfpXie8aYiYSCd7OTK3aH96cYU3JxB8LhIKYlvYS/ZaxqaWaRbB9XV+zvLZlr6rKvVYnme7h5y6eVClM64mA1vLHvzQOBIJIAEkgASQABJAAkjgaSOAItTTdsfxvEgACSABJPD4EbASoUDs+YwwjN7PUVon9qQptSMY3vQir5FP8G3IlNIDXFeq2xsJ+Yxn+EV+CtuDdWJTmToW5l8FsekLS981pSaSI0xbH0fp2DuiVGV3QgQfcSx5399BlkD7j2RlKTwc3FdxPH/Q31m+/s6a2pmE4RvAmpF1ApRS3Q/WHC1i2anNf74eQARkA4pQj9+jhTtCAkgACSABJIAEkMDjRABFqMfpbuBekAASQAJI4OkkUCtCpao0MxieNAKxJ8ICIrWs+mVCTBN4Hfeefz27FNqfWszXY4TaaMKSXb4Osm13xmoWsixj9FFIZ90RkKreBlFqAC8mkX5yeT7tv5JX6SuQsEvgR8BqiKz6sW6+SruK5fgsHyfZ4rr5Km0ow/PdTHlZ7/j7+1fR/uQyTRcBQz4G4elDX3vZabL9yhsoQj2djy6eGgkgASSABJAAEkACf4YAilB/hhaORQJIAAkgASTwKAiACJX6bLOxIPb0dFZIx70yad16nucHNm/orBn8XHvJD8eTq1MyCuiVhfDShg7oIssuVJoOnErVER6kKIaYXu3eWuKisGU37DmrJoSDzDuGb9O8gejFLr6y7b9eqMrKLaW+U5xMKiajXwuxTcnI0x85d0MLA3nCc+StFzraMgxDtv18rgIGiuAHgiGotaeka9tmsq37z6kKSisg8Irh67vaC4e82FFx/EJGVfyVm9BHBCUerky2bwuX9kdO3zTjMTtcwbpwIAsueNMQll+esCJsxqNAiGsiASSABJAAEkACSAAJPP4EUIR6/O8R7hAJIAEkgASedAIgQqX18vqRGI3vDp+77VMQg/rVd1McjRj8jOvFa9lJ23+5SBWoV0FYOv3esJ4t9CZOt2zrsV8YwveDvqqhfQKqmjd2ax7zzbHdpeVVbXiGdQnwaXjp9efa9zmSeP3Mr6fTdDxDesMa+2aM6h1SqlQr1+6Oj6faFYhFJ0b36+zrYi91Xrzl4AGOZ0NAq3Lo0q55+mvd/LvsOXL51JnLN6UgdLW1l9rsjxzSvdf1W8W523+9mAVqUw9YM7Nln+Cm3+qEzdofORlLCBsEqlY1/MBI5sGwimUgYZDwzRnCBIIQJYcP1+Njwryf9FuK50MCSAAJIAEkgASQABL4fwIoQuFTgQSQABJAAkjg3yYAItTV55p187aXngiKXHtaLGIEcbPfSmcZwZlRc7/abjJyMaDl/Lpx3tv1QDTy8HWSTQyMXDOXJQL5xrlvHyMMG04EkqkjP4xtAyLSsN4dmn8yvH/IuzDnh5HzNp40mQTLQVjatHn+sI70qL4K+QdBEbHgGcWWb5o/9BZELL0gc5COf31S3CD4+plZo1/5oqWn4zyO8OtGz9mUQ3jhQpOAWbBlzpDXeZ7L9nOy/Qz2uRaitZI3fzxcu/G2cshn18sCg46c/BAENN/46LAwC9LgyNjnISTqXY6wMwWEaw+xUesSYsLYfxs5Xh8JIAEkgASQABJAAkjgnyeAItQ/zxyviASQABJAAkjgtwSsjMmDIuMOhw/o0rJLW6/VIz7a0hES6/pCZl117yAfvoWnq/DL3Wer9HqjBOKLBC0bu2kH9W4n3330sjols9AA/6g7wMJVkK4nzS1SGn86maaHaCVIz+M/2TR/mN3RxOsLN/wQXwRjbOHFBPo10vcI9JZ9c+CcKrtQBal+vFxqY6Ma27+TXVpmvvZgwjUNjGvEMWTI5nlDWu0/kfb+N78kVcB1pJByx0O6nqlLu6bixaevGy82bCTtePysAdbQgugE24YLQKoefIYoKoZAGp4oeFJsV55jjiREh4nwEUACSAAJIAEkgASQABJ4+gigCPX03XM8MRJAAkgACTxuBKxEqOgDFzJtxEJRWK/WjUCQuglbtR/4XPvdAd4NgzbvS9yTmpWvgpS2rs0audwe069zr8SUW5e/P3wpFaKmIPDJxE4Z0us5g8FkivnmxHGIcBoEKXCSjVHDogrKysd+vPZAkKZat4TjmZeD/BpdHti7fdd9J9POnEi6ngfXaQWvineH9fIuUanVm/bGXwIVyR/Epq6b5g19l/C894g5W8eA9rQSop3Ez3ZsaejTxffFH44l79tabnAsbNhgaNCJ04tMJlJowcuyrAyipTqABtWXCk+1ItQx8KCanBAzLvpxuw24HySABJAAEkACSAAJIIFHSwBFqEfLF1dHAkgACSABJHB/ArUiVLpSM/FcWvaMFduOZRCW/x6ihqa1a9mgaOrQnkohSz5o4SA7Sxc7ciRL4t7OLZblBWd8nCRrLBdIK9POBOGpAVTXi6R9IGLtDmndtNW41ztfjd118uiZy7fmg2d5m40fD9NBhNIyljN97eNst+POfPUCiGIS+DnJZlr6pn55kA8b2PlbhjVGDPvwGxCY2LWbPxq6lBOYPhcJBbEt7CV7/6g6nnX0E+yHCk/jQZCCCClsSAAJIAEkgASQABJAAk8bARShnrY7judFAkgACSCBx48AiFApPZsMZRm237Slu92KlVUuEHF0pbG7Y4vhLwd5ejfxGO3jKPnBsvH0cs0yzsQU+zlJF9QJSCptKBSj66bTVYxr6+EBKXiEjF2x93jfHm3aNW/k9urw2VtegIik6S62ki6fTx/0Dng7xfs5yVdb5l9VVk/lGL6lL1Tnq+srq37p4o2cfa2aNWo1au76loQX7Groqhi9IOKV5zmeP+jvLF9vHrv9yhuMgN3ID/SX3g3XIkIRjh9LBOwwiKjqjul4j98jiDtCAkgACSABJIAEkMA/QQBFqH+CMl4DCSABJIAEkMAfEaDV8Xo2/YUVi6YP/2DjYkihs986J/T58zduXEvLKiif++YzddXk0krVUUTA2PoqZNPuCEia/mDCNNLIsFNbKyQZtP+CUtnk9q3yM8eTMjQnzme8SFiyA1LrWm6ePzQWPJqKfRR3BKyUkqqRrJB9gVQZI3wb2pfS+ekqdUdI21v42YaDz6bezO5CONECELGeWTy57w/ujnYQgSVbbLn+2HPZWzdmVww29vf/P6+nWhHqKPzg+IXwzNfgZbUBRSj864AEkAASQAJIAAkggaeTAIpQT+d9x1MjASSABJDA40SAilDPNn/BVyH5BVLWfoat2W6cO+TST6fSnt/+8/nchJjw7nS7aUpNJERItduukIVGMTXm3+kV
|
|||
|
|
<hr />
|
|||
|
|
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABVgAAAKiCAYAAAAjVLcgAAAgAElEQVR4XuydBXxVx9bFNwGCuzspUtyLu7s7FEopUKBA0eLFXSu4tLhLcXd3d4IUSnEIloSEN2voyUtCQu5NbnJt7f54QHLOzJ7/nJPvY909a0f6oEIYJEACJEACJEACJEACJEACJEACJEACJEACJEACJEACZhOIRIHVbGa8gQRIgARIgARIgARIgARIgARIgARIgARIgARIgAQ0AQqsfBBIgARIgARIgARIgARIgARIgARIgARIgARIgARIIJQEKLCGEhxvIwESIAESIAESIAESIAESIAESIAESIAESIAESIAEKrHwGSIAESIAESIAESIAESIAESIAESIAESIAESIAESCCUBCiwhhIcbyMBEiABEiABEiABEiABEiABEiABEiABEiABEiABCqx8BkiABEiABEiABEiABEiABEiABEiABEiABEiABEgglAQosIYSHG8jARIgARIgARIgARIgARIgARIgARIgARIgARIgAQqsfAZIgARIgARIwEEIuLu7y7Vr1+TBgwfy8OFDefLkidy+fVvOnz8vGTJkEFdXVwdZKZdBAmEj4OXlJTdu3JAcOXJIunTpJFGiRJI0aVJJnjy5ZMqUSdzc3MI2Ae8mARIgARIgARIgARJwKgIUWJ1qu7lYEiABEiABRyKwZ88e2bBhg5w4cULu3r0rnp6ejrQ8roUErEYgWrRokiZNGsmfP79Uq1ZNSpUqZbVcODEJkAAJkAAJkAAJkIDtE6DAavt7xAxJgARIgARIwI8AKu+6dOki27dvF/w5cKBKFb+iR48uMWPGlKhRo8qjR48kY8aMrGDlc0QC/xHAu3P9+nVJkiSJeHt7y5s3b+Tdu3f6nQruvSpfvrxMnjyZ7xGfIhIgARIgARIgARIggU8IUGDlQ0ECJEACJEACdkDg8uXLMnDgQDl69Kh8+PBBZ+zi4qKPNeOYc7ly5aRp06Z2sBKmSAK2T2DRokWyY8cOba8Buw1fX1+ddKRIkaRgwYIyZMgQyZIli+0vhBmSAAmQAAmQAAmQAAlECAEKrBGCmZOQAAmQAAmQQOgJ9OvXTxYsWOA3QJw4caR+/foyaNCg0A/KO0mABEwmgHdtxYoV4uHh4XdP8+bNZfjw4SaPwQtJgARIgARIgARIgAQclwAFVsfdW66MBEiABEjAAQjUqlVLTp8+rVeCBjytW7eWtm3bOsDKuAQSsD8CM2bMkNmzZ+tGcog8efLI2rVr7W8hzJgESIAESIAESIAESMCiBCiwWhQnByMBEiABEiAByxBwd3eXevXqyZMnT/SA8H+EsMMgARKwPgF80AEfZESiRIlk5cqV4ubmZv3EmAEJkAAJkAAJkAAJkIBVCFBgtQp2TkoCJEACJEACnyeQL18+La5GjhxZevToIR06dCAyEiABGyIwZcoUGTdunPj4+GiR9eTJkzaUHVMhARIgARIgARIgARKISAIUWCOSNuciARIgARIgARMIGLYAEFd/+eUXqV69ugl38RISIIGIJrB+/Xrp3LmzFllpFxDR9DkfCZAACZAACZAACdgOAQqstrMXzIQESIAESIAExH9Dq59++omVq3wmSMDGCaCSdfTo0TpLNr6y8c1ieiRAAiRAAiRAAiQQTgQosIYTWA5LAiRAAiRAAuYSuHz5slSqVEnfRs9Vc+nxehKwHgH/nqxbtmyRLFmyWC8ZzkwCJEACJEACJEACJBDhBCiwRjhyTkgCJEACJEACQRNo2LChHDlyRJInT65/Z5AACdgPgUKFCsmDBw8Evy9btsx+EmemJEACJEACJEACJEACYSZAgTXMCDkACZAACZAACYSdgJeXl2TOnFk+fPigbQLatm0b9kE5AgmQQIQRmDFjhgwfPlwiRYokV69eFVdX1wibmxORAAmQAAmQAAmQAAlYlwAFVuvy5+wkQAIkQAIkoAm0b99eNm7cKHHixJHz58+TCgmQgB0SyJEjh3h4eEjVqlVl6tSpdrgCpkwCJEACJEACJEACJBAaAhRYQ0ON95AACZAACZCAhQlkypRJUMXaqlUrGTRokIVH53AkQAIRQQDv7ty5c3X16rVr1yJiSs5BAiRAAiRAAiRAAiRgAwQosNrAJjAFEiABEiAB5yawZ88eadGihbi4uIi7u7tzw+DqScDOCbi5uYmvr6/MmzdPSpUqZeerYfokQAIkQAIkQAIkQAKmEKDAagolXkMCJEACJEAC4UigV69esnTpUja3CkfGHJoEIoqA0eyqUaNGMmbMmIialvOQAAmQAAmQAAmQAAlYkQAFVivC59QkQAIkQAIkAALlypWT69evS/ny5WX27NmEQgIkYMcEWrduLdu3b5eMGTPKjh077HglTJ0ESIAESIAESIAESMBUAhRYTSXF60iABEiABEggnAhkzpxZPD09ZeTIkdK0adNwmoXDkgAJRASBRYsWSZ8+fSRatGhy9erViJiSc5AACZAACZAACZAACViZAAVWK28ApycBEiABEnBuAvBcLV26tIZw+/Zt54bB1ZOAgxBIly6dXsnu3bsFnqwMEiABEiABEiABEiABxyZAgdWx95erIwESIAESsHECW7dulTZt2rDruI3vE9MjAXMIZMqUSby8vGTmzJlSsWJFc27ltSRAAiRAAiRAAiRAAnZIgAKrHW4aUyYBEiABEnAcAug0PmDAAIkdO7ZcuHDBcRbGlZCAExPInj27vHr1SoYOHSotWrRwYhJcOgmQAAmQAAmQAAk4BwEKrM6xz1wlCZAACZCAjRIYN26c/Prrr5I4cWI5ceKEjWbJtEiABMwhkD9/fnn8+LF06tRJevToYc6tvJYESIAESIAESIAESMAOCVBgtcNNY8okQAIkQAKOQwDNcNAUJ23atLJv3z7HWRhXQgJOTKBEiRJy584d3bQOzesYJEACJEACJEACJEACjk2AAqtj7y9XRwIkQAIkYOMEIMAcOHBAMmTIIDt37rTxbJkeCZCAKQTKli0rN27ckOLFi8vChQtNuYXXkAAJkAAJkAAJkAAJ2DEBCqx2vHlMnQRIgARIwP4J5MqVS168eCFx48aVc+fO2f+CuAISIAHJmTOnvHz5UuLFiydnz54lERIgARIgARIgARIgAQcnQIHVwTeYyyMBEiABErBtAnXq1JGTJ09Kvnz5ZPXq1badLLMjARIwiQDfa5Mw8SISIAESIAESIAEScBgCFFgdZiu5EBIgARIgAXsk0KhRIzl8+LAULlxYli5dao9LYM4kQAKBCPC95iNBAiRAAiRAAiRAAs5FgAKrc+03V0sCJEACJGBjBCjE2NiGMB0SsAABvtcWgMghSIAESIAESIAESMCOCFBgtaPNYqokQAIkQAKOR4BCjOPtKVdEAnyv+QyQAAmQAAmQAAmQgHMRoMDqXPvN1ZIACZAACdgYAQoxNrYhTIcELECA77UFIHIIEiABEiABEiABErAjAhRY7WizmCoJkAAJkIDjEaAQ43h7yhWRAN9rPgMkQAIkQAIkQAIk4FwEKLA6135ztSRAAiRAAjZGgEKMjW0I0yEBCxDge20BiByCBEiABEiABEiABOyIAAVWO9ospkoCJEACJOB4BCjEON6eckUkYI/v9evXr+XcuXPy7Nkzefr0qf49ZsyYkiRJEkmaNKmkTp1aUqVKxc0lARIgARIgARIgARIIggAFVj4WJEACJEACJGBFAvYoxFgRF6cmAbsgYC/v9b59+2Tjxo1y+vRpuXjxYohsM2XKJIULF5YiRYpItWrVQryeF5AACZAACZAACZCAsxCgwOosO811kgAJkAAJ2CQBexFibBIekyIBGyVg6+/1woULZfHixbpi1YhIkSJJnjx5JFmyZJIgQQL96+3bt/Lw4UN59OiRFmBfvXrld33GjBmlRYsW8vXXX4uLi4uN7gTTIgESIAESIAESIIGIIUCBNWI4cxYSIAESIAESCJKArQsx3DYSIAHzCdjqe713714ZNWqUXLhwQS8qRYoU0rhxYylWrJjkzZtXokSJ8tnFHj9+XA4fPixr
|
|||
|
|
<hr />
|
|||
|
|
<h1 id="other-uses">Other uses</h1>
|
|||
|
|
<ul>
|
|||
|
|
<li>Caching system</li>
|
|||
|
|
<li>Asynchronous tasks</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h1 id="the-future">The future</h1>
|
|||
|
|
<ul>
|
|||
|
|
<li>Hortonworks Data Platform (HDP) Open Source is deprecated</li>
|
|||
|
|
</ul>
|
|||
|
|
<hr />
|
|||
|
|
<h1 id="do-you-really-need-cdc">Do you really need CDC?</h1>
|
|||
|
|
<div class="notes">
|
|||
|
|
<p>Before opening up your laptop and install everything to run Debezium on your infrastructure, you should ensure that you really need change data capture. It can be expensive, hard to setup and complex to maintain. You should go through easier steps first, like creating an asynchronous replica to perform reporting queries. Then, when simpler solutions have been ruled out, you should consider setting up CDC.</p>
|
|||
|
|
</div>
|
|||
|
|
<hr />
|
|||
|
|
<h1 id="questions">Questions</h1>
|
|||
|
|
</article>
|
|||
|
|
</body>
|
|||
|
|
</html>
|