New ask Hacker News story: Change Data Capture (CDC) Tools should be database specialized not generalized

Change Data Capture (CDC) Tools should be database specialized not generalized
3 by saisrirampur | 0 comments on Hacker News.
Reference: https://ift.tt/KqfAXvi Parent Tweet: https://ift.tt/d3g8psL Change Data Capture(CDC) is hard, it has 100s of edge cases / failure points. At PeerDB (https://www.peerdb.io/) , instead of focusing on multiple sources we just focused on Postgres. This helped us ensure that we gave enough care to iron out as many edge cases as possible. We were also able to implement a bunch of Postgres native performance and reliability optimizations. Our engineering blog https://blog.peerdb.io/ more on the optimizations and how we ironed out edge cases. Pipeline failures have been rare these days, and so far none of the source databases were affected due to load. Also, most of our customers are in the shorter tail, i.e., data sizes over 300-400GB to 15-20TB. This helped battle test the product and make seamless for the long tail. However, I don’t think CDC is a solved problem, as Postgres is full of mysteries and it keeps evolving. We need to continue polishing the experience and evolve along with Postgres! TL;DR specialized CDC tools that focus on a single (or limited) database are reliable ways to provide a solid CDC experience.

Comments

Post a Comment

Popular posts from this blog

New ask Hacker News story: Brother Printers Sending Ink Data to Amazon?

Brother Printers Sending Ink Data to Amazon? 29 by Ajay-p | 13 comments on Hacker News. A most unusual thing. Every once in a while I get an email from Amazon that it's time to re-order Brother ink. I always delete these because I rarely print, but also figure it's just Amazon reminding me to buy something. Today I decided to opt out/unsubscribe once and for all. Instead I see this at the bottom of the email: "Click here to view or manage settings, including the option to opt out if you are already using another replenishment service. This took me to https://ift.tt/nMGIXiy "The data shown is based on estimated consumption reported by smart devices and orders you place through Amazon." Here it had a link to "Consumption history" which upon clicking showed me the ink levels of my Brother printer for the past two weeks . Date and time. WTF?! It is not apparent that I can disable this function. Can anyone else duplicate? Update : This is part of Alexa it
Read more

New ask Hacker News story: Tell HN: Equifax free credit report dark patterns

Tell HN: Equifax free credit report dark patterns 4 by PopAlongKid | 0 comments on Hacker News. For years, I have been obtaining free annual credit reports (annualcreditreport.com) which must be provided by law. Recently, for the first time, when I tried to obtain my Equifax report, I was prompted for an email address and a mobile phone number, a new requirement that apparently cannot be bypassed. The other two bureaus, and Equifax previously, confirmed identity by asking knowledge based questions. They do not need my phone or email for anything. Next, trying to obtain the report by phone instead of web site per instructions, I got to the point where I entered my ZIP code on the phone keypad, the voice menu system correctly repeated the number back to me, but every time I press 1 to indicate it is correct, the system acts like it got an invalid response and only gives me the option to enter further information by voice, not by the phone keypad. Just as with my phone number and email
Read more