WEBVTT 00:00.000 --> 00:10.000 Okay, everybody, we're ready to do it. 00:10.000 --> 00:15.000 We're ready to do it. 00:15.000 --> 00:17.000 And stuff, things. 00:17.000 --> 00:18.000 Yes, and things. 00:18.000 --> 00:22.000 Hello again, much longer talk this time, and only me, so I don't 00:22.000 --> 00:23.000 doesn't get to help this time. 00:23.000 --> 00:28.000 So I'm going to talk about a new piece of open source software. 00:28.000 --> 00:31.000 It was published by the power DNS team last year. 00:31.000 --> 00:32.000 I think you came out in 24, right? 00:32.000 --> 00:33.000 Or did it come out in 23? 00:33.000 --> 00:37.000 Anyway, very recently, which is another way to do 00:37.000 --> 00:41.000 zone replication between authoritative servers, which we all need 00:41.000 --> 00:45.000 to do for various reasons, and we'll talk about that. 00:45.000 --> 00:48.000 So first thing is, we're going to do a little bit of primer here. 00:48.000 --> 00:51.000 I know it's late in the day and you're all falling asleep, so a lot of this 00:51.000 --> 00:55.000 will be old news for many of you, but there's three things that 00:55.000 --> 00:57.000 authoritative server manages. 00:57.000 --> 01:00.000 Data wise, there's the actual content of the zones. 01:00.000 --> 01:02.000 The record sets that it needs to manage. 01:02.000 --> 01:05.000 There's the list of zones for which it's willing to provide 01:05.000 --> 01:06.000 authoritative answers. 01:06.000 --> 01:10.000 And then, at least in modern authoritative servers, there's 01:10.000 --> 01:11.000 metadata about the zones. 01:11.000 --> 01:14.000 How the authoritative server should treat that zone? 01:14.000 --> 01:17.000 So, for example, if it's doing online DNS sex signing, or if it 01:17.000 --> 01:20.000 needs to know where to send notifies two or other things, 01:20.000 --> 01:23.000 those are things which are not stored in the zone itself. 01:24.000 --> 01:26.000 They're stored in metadata alongside the zone. 01:26.000 --> 01:29.000 So there's three sets of things that your authoritative server typically 01:29.000 --> 01:33.000 needs to have to be able to provide the services to provide. 01:33.000 --> 01:37.000 And then, this seems like a silly question, but it's true. 01:37.000 --> 01:39.000 Why do we need to replicate this stuff? 01:39.000 --> 01:41.000 Why do we need to have multiple servers? 01:41.000 --> 01:44.000 Well, there's obvious things, like fault tolerance, right? 01:44.000 --> 01:48.000 If you had just one server, providing answers for your zone, 01:48.000 --> 01:51.000 and that server becomes unavailable for a variety of reasons 01:51.000 --> 01:54.000 and anyone trying to get it results for any of your zones is going to fail, 01:54.000 --> 01:55.000 because your server is not available. 01:55.000 --> 01:58.000 So you obviously want to have more than one for that purpose. 01:58.000 --> 02:01.000 And I have three BGP in there because we all know that when it's not DNS 02:01.000 --> 02:05.000 to each P, so the two reasons why things are broken. 02:05.000 --> 02:08.000 Then, there's the other situation, which is, you know, 02:08.000 --> 02:11.000 the internet's big goes around the whole world and, I guess, 02:11.000 --> 02:13.000 out into space a little bit. 02:13.000 --> 02:16.000 If you just have one server located in, I don't know, 02:16.000 --> 02:20.000 let's say, Western United States, and you've got users in South Africa, 02:20.000 --> 02:23.000 looking for answers to DNS queries for your zone. 02:23.000 --> 02:25.000 That's going to be very slow, because they're around trip time 02:25.000 --> 02:29.000 to your server's long, because of speed of light and other things like that. 02:29.000 --> 02:31.000 So you want to have more servers spread around the world, 02:31.000 --> 02:34.000 so they can get answers more quickly. 02:34.000 --> 02:38.000 And the last thing is, if you happen to be the owner of an extremely popular zone 02:38.000 --> 02:43.000 that's getting thousands and tens of thousands and hundreds of thousands of queries all the time, 02:43.000 --> 02:45.000 having just one server of course would be terrible, 02:45.000 --> 02:49.000 because the server would get overloaded and possibly crash and all those things. 02:49.000 --> 02:53.000 So these are all reasons why people have needed to be able to do replication. 02:53.000 --> 02:57.000 Now, going back to the beginning, and by the beginning, I mean, 02:57.000 --> 03:00.000 way back in the past, and there's lots of people in the room who've never had to do this stuff, 03:00.000 --> 03:02.000 because they're not old enough. 03:02.000 --> 03:07.000 The original authoritative servers, literally, you put a zone file, 03:07.000 --> 03:10.000 an actual zone file, which lots of people still use with bind, 03:10.000 --> 03:13.000 but this is existed forever. 03:13.000 --> 03:16.000 Into a directory, and you pointed the authoritative server 03:16.000 --> 03:20.000 that directory in an answered queries with data from that file, and that was it. 03:20.000 --> 03:21.000 That's all it did. There was nothing else. 03:21.000 --> 03:25.000 There was no metadata. There was no replication. 03:25.000 --> 03:27.000 There was nothing. It was just, hey, here's this data, 03:27.000 --> 03:31.000 and I need you to answer queries for it. 03:31.000 --> 03:34.000 That's way back when. 03:34.000 --> 03:39.000 And the way that you replicated changes to that was you're 03:39.000 --> 03:44.000 self, FTP, or SCP, if you actually had were an early adopter of 03:44.000 --> 03:49.000 SCP back then, or any of these other ways, that's how you would move those things around. 03:49.000 --> 03:52.000 So you'd copy the files to your servers, and then go restart the DNS 03:52.000 --> 03:56.000 demon on all of your machines, and now suddenly they had the current copy of your zone. 03:56.000 --> 04:01.000 We wouldn't want to do that now, because that's a very error prone and slow and lots of other things. 04:01.000 --> 04:08.000 So some wonderful people published some RFCs back in, where did I say that was? 04:08.000 --> 04:12.000 1987. So still approaching 40 years ago. 04:12.000 --> 04:14.000 It's hard to believe that. 04:14.000 --> 04:18.000 So what I said, hey, we should just be able to do this with DNS, right? 04:18.000 --> 04:20.000 The DNS serve the heart to your server itself. 04:20.000 --> 04:25.000 Should be able to send the data that's needed to the other servers without us having to copy 04:25.000 --> 04:28.000 the zone files around and do all of this sorts of stuff. 04:28.000 --> 04:32.000 So they introduced the concept of doing transfer X for initially. 04:32.000 --> 04:34.000 Now we have X for as well. 04:34.000 --> 04:37.000 Using the DNS protocol itself. 04:37.000 --> 04:41.000 And just like copying the files around this copy is the data itself. 04:41.000 --> 04:46.000 There are sets from the zone, not all of the other stuff, just the RR sets. 04:46.000 --> 04:50.000 And so that now, of course, means that you've got to configure all the servers. 04:50.000 --> 04:54.000 You've got to configure the secondary servers to know whether where the primary server is. 04:54.000 --> 04:57.000 You might have to configure the primary server, which I don't know where the secondary server is, 04:57.000 --> 05:00.000 for access control purposes and things like that. 05:00.000 --> 05:05.000 And lots of other access control things like T-SIG and other stuff came along later. 05:05.000 --> 05:08.000 And so that became zone metadata. 05:08.000 --> 05:11.000 Now the servers had to be able to manage this kind of stuff. 05:11.000 --> 05:15.000 And so once you have this set up, then the secondary server would say, 05:15.000 --> 05:18.000 OK, I've been told this zone exists over here. 05:18.000 --> 05:23.000 And I'm going to periodically go doing SLA query for that zone and find out if the serial number is changed 05:23.000 --> 05:25.000 from the one I have. 05:25.000 --> 05:28.000 If it's changing gotten lower, something's broken. 05:28.000 --> 05:32.000 And if that number has gone up, then I'll do an AX for an old pull copy of the zone down 05:32.000 --> 05:34.000 and replace the copy that I have. 05:34.000 --> 05:35.000 And that's that. 05:35.000 --> 05:39.000 And those SLA queries would be controlled by the refresh interval in the SLA record itself. 05:39.000 --> 05:41.000 How often do you want to do this? 05:41.000 --> 05:45.000 One today, once an hour, once a week, whatever is appropriate for your zone. 05:45.000 --> 05:46.000 So great. 05:46.000 --> 05:47.000 We have an improvement now. 05:47.000 --> 05:50.000 You don't have to copy the zone files around yourself. 05:50.000 --> 05:52.000 But that's polling. 05:52.000 --> 05:56.000 We'll know polling has problems. 05:56.000 --> 06:02.000 One of those is, if you change the zone content frequently, more frequently than the polling interval, 06:02.000 --> 06:05.000 then you're going to have to go poke the servers to update anyway. 06:05.000 --> 06:08.000 Because they're not going to pull off enough to pull this stuff down. 06:08.000 --> 06:09.000 So that's not great. 06:09.000 --> 06:14.000 And second, I'm excuse me. 06:14.000 --> 06:17.000 Oh, yes, this is another RFCS gift. 06:17.000 --> 06:19.000 I forgot I took a slide out. 06:19.000 --> 06:24.000 So after that point, so that's as though it was the motivation then for RFC1996, 06:24.000 --> 06:30.000 which is for the server that hosts the primary server, which actually introduced the term primary server by the way, 06:30.000 --> 06:33.000 before that there was no such term in the DNS terminology. 06:33.000 --> 06:36.000 To say, hey, I've just become aware of a change in this zone. 06:36.000 --> 06:39.000 And I know that there are all of these secondaries out there who care about it. 06:39.000 --> 06:43.000 I'm going to notify all of them that the zone contents have changed. 06:43.000 --> 06:48.000 And now they can come back to me and get a copy of the zone changes if they would like to have them. 06:48.000 --> 06:51.000 That's what introduced all of these terms. 06:51.000 --> 06:53.000 Some of which are not in common use. 06:53.000 --> 06:58.000 Like I've literally never heard someone refer to a server as a stealth server in regular conversation. 06:58.000 --> 07:00.000 I think that term is gone by. 07:00.000 --> 07:04.000 Also, I don't think I've ever heard anybody use the term primary master, 07:04.000 --> 07:08.000 even though it kind of does make sense because it's the root of the graph that holds all of your zone. 07:08.000 --> 07:10.000 If you have multiple levels. 07:10.000 --> 07:15.000 And then of course, for various reasons, we don't generally use the words master. 07:15.000 --> 07:20.000 So even more we use primary and secondary, which makes primary and master extremely cumbersome as a term, 07:20.000 --> 07:22.000 because we replaced master with primary. 07:22.000 --> 07:24.000 So now it's primary primary. 07:24.000 --> 07:26.000 So that makes things even more confusing. 07:26.000 --> 07:31.000 So that was really useful because now it makes you could get stuff out to your secondary servers more quickly. 07:31.000 --> 07:35.000 You could just push a notification up to them that they need to update. 07:35.000 --> 07:38.000 This means now there's more zone metadata, right? 07:38.000 --> 07:39.000 It's not just the zone file anymore. 07:39.000 --> 07:42.000 Now you have to know all the secondaries are which ones want to notify. 07:42.000 --> 07:46.000 Maybe there are some that are not listed in the NS record set of the zone. 07:46.000 --> 07:48.000 And you want to send notifications to them anyway. 07:48.000 --> 07:50.000 So you've got to have a place to do all of that. 07:50.000 --> 07:54.000 And what that means now is you've now gone from this. 07:54.000 --> 07:59.000 Just here's these bunch of servers that happen to provide answers to the same questions. 07:59.000 --> 08:03.000 To here are a bunch of servers that actually have to know about each other's existence. 08:03.000 --> 08:07.000 They have to know what their IP addresses are and how what they should communicate with them. 08:07.000 --> 08:11.000 The configuration data for the servers actually becomes important. 08:11.000 --> 08:14.000 And it's sort of tightly coupled with them. 08:14.000 --> 08:17.000 Not super tight, but before they were not coupled at all. 08:17.000 --> 08:19.000 So this is obviously an increase there. 08:19.000 --> 08:24.000 It also introduced eventual consistency, not strong consistency, right? 08:24.000 --> 08:28.000 If you send if you make a change in the zone and notify goes out, 08:28.000 --> 08:31.000 you don't actually know for sure when all the secondaries are going to be updated. 08:31.000 --> 08:35.000 It will happen sometimes soon you hope maybe in the next couple of minutes. 08:35.000 --> 08:39.000 But it's certainly not instantaneous and it's definitely not synchronized. 08:39.000 --> 08:43.000 You're not going to be able to say, I have six servers all around the world. 08:43.000 --> 08:45.000 And they're all going to appear at this exact same moment. 08:45.000 --> 08:48.000 So the next query that any of them gets will have the new data. 08:48.000 --> 08:50.000 That was definitely not part of the design. 08:50.000 --> 08:54.000 So now you had to come account for that. 08:54.000 --> 08:56.000 All right. 08:56.000 --> 09:00.000 So for replication, this is technically all you really need to have. 09:00.000 --> 09:03.000 Right? This works. People do this all the time. 09:03.000 --> 09:05.000 My own personal stuff works this way. 09:05.000 --> 09:07.000 I don't use anything anymore complex in that. 09:07.000 --> 09:11.000 Even though I'm going to tell you about something more complex in that than I used for a while. 09:11.000 --> 09:16.000 If the set of servers that you're managing is relatively static, 09:16.000 --> 09:18.000 having tight coupling is not really a problem. 09:18.000 --> 09:20.000 Because you're not adding in or moving servers all the time. 09:20.000 --> 09:21.000 We're changing their addresses. 09:21.000 --> 09:23.000 So you don't have to update the configuration. 09:23.000 --> 09:25.000 So that's really no big deal. 09:25.000 --> 09:27.000 If the records in your zones aren't changing frequently, 09:27.000 --> 09:29.000 and they have relatively long TTLs, 09:29.000 --> 09:32.000 then eventual consistency is not really a problem. 09:33.000 --> 09:36.000 All the TTLs and all the zones are an hour, 09:36.000 --> 09:39.000 and it takes two seconds for the update to make its way around. 09:39.000 --> 09:40.000 That's fine. 09:40.000 --> 09:44.000 It's a tiny fraction of how long it might have been cached in a result of our anyway. 09:44.000 --> 09:47.000 So it carries if it took two seconds for it to get out there. 09:47.000 --> 09:52.000 And then other good thing about this is two things about this. 09:52.000 --> 09:56.000 Because of the way this works, it doesn't have to be just one level. 09:56.000 --> 09:58.000 You can have multiple levels of replication. 09:58.000 --> 10:00.000 And there are people to do that now. 10:00.000 --> 10:03.000 Hidden primary server somewhere where they manage all of the content. 10:03.000 --> 10:06.000 They push it out to their own secondary servers that they run, 10:06.000 --> 10:09.000 and then they push it further to secondary servers that they rent 10:09.000 --> 10:11.000 from DNS service providers for example, 10:11.000 --> 10:14.000 to provide more redundancy around the world and things like that. 10:14.000 --> 10:17.000 Since this is all done just using the DNS protocol, 10:17.000 --> 10:18.000 that's very simple. 10:18.000 --> 10:19.000 There's no special. 10:19.000 --> 10:22.000 You don't have to have certain specific software or any of that kind of stuff. 10:22.000 --> 10:24.000 So this is all great. 10:24.000 --> 10:25.000 Sounds awesome. 10:25.000 --> 10:29.000 But there are reasons you might want to do something else. 10:30.000 --> 10:33.000 Zones can be really, really big. 10:33.000 --> 10:34.000 Really big. 10:34.000 --> 10:37.000 Like tens or hundreds of thousands of records in them easily. 10:37.000 --> 10:40.000 And if you're using X for every single time, 10:40.000 --> 10:42.000 there's a change in one of those. 10:42.000 --> 10:46.000 That's a bunch of data that the hidden primary or whatever it is. 10:46.000 --> 10:48.000 Has to organize to send. 10:48.000 --> 10:51.000 And then it all has to be streamed to every secondary server. 10:51.000 --> 10:53.000 And they're all going to have to get their own copy of it. 10:53.000 --> 10:56.000 And so that's not particularly fast. 10:56.000 --> 11:01.000 And depending on how you're paying for bandwidth on the ends, 11:01.000 --> 11:05.000 it could be expensive for you to have to do that as well. 11:05.000 --> 11:11.000 And we now we have this wonderful stuff called cloud native deployments, 11:11.000 --> 11:14.000 which have changed the picture a lot. 11:14.000 --> 11:19.000 You might have, first of all, you might have records being created 11:19.000 --> 11:22.000 when a virtual machine gets launched. 11:22.000 --> 11:24.000 And then destroyed when that virtual machine is turned down. 11:24.000 --> 11:29.000 And they need to be visible to all of the consumers of that almost instantly. 11:29.000 --> 11:35.000 You might also be deploying DNS servers in a cloud native type of thing, 11:35.000 --> 11:39.000 like to have something very local next to all of the VMs, 11:39.000 --> 11:44.000 which means if all of the configuration of the DNS servers all has to be updated every time you deploy one, 11:44.000 --> 11:47.000 that's very frustrating or very difficult I should say. 11:47.000 --> 11:53.000 So a solution, which I think all of the certainly all of the open source DNS authority 11:53.000 --> 11:57.000 of server support is to push all the stuff into a database. 11:57.000 --> 11:59.000 So okay, we're not going to do this with own files anymore. 11:59.000 --> 12:01.000 Yeah, we don't have to. 12:01.000 --> 12:06.000 We're going to let you use a database, some common off-the-shelf database to do this sort of thing. 12:06.000 --> 12:10.000 And if you really want to, you can use a super fancy database that has clustering. 12:10.000 --> 12:13.000 You can have nodes all around the world and it will replicate all of the data for you. 12:13.000 --> 12:16.000 And the DNS servers don't have to do any of that kind of stuff. 12:16.000 --> 12:19.000 They'll just say, I've deferred this responsibility. 12:19.000 --> 12:21.000 The database is going to do it for me. 12:21.000 --> 12:22.000 Great. 12:22.000 --> 12:26.000 That might actually give you some things you wouldn't get otherwise, for example, 12:26.000 --> 12:29.000 depending on the database you choose and how you choose to configure it, 12:29.000 --> 12:31.000 you might be able to enforce strong consistency. 12:31.000 --> 12:35.000 And if that's actually important to you, you could choose to do that. 12:35.000 --> 12:39.000 It will have its costs to do that, but you could certainly choose to do that. 12:39.000 --> 12:44.000 It means that the servers, the DNS servers no longer have to know about each other, 12:44.000 --> 12:47.000 because the database is handling all of replication between them. 12:47.000 --> 12:52.000 So the fact that there are one or five or 20 doesn't matter because the database is handling that 12:52.000 --> 12:54.000 and not the DNS server level. 12:54.000 --> 13:01.000 But obviously the database cluster nodes will all have to be aware of each other and configured to be that way. 13:01.000 --> 13:03.000 So that could be the answer for you. 13:03.000 --> 13:07.000 I'm sure there's lots of people here who use databases underneath their authority 13:07.000 --> 13:09.000 of servers to distribute data around. 13:09.000 --> 13:15.000 I would say that if you are the DNS team and not the database team, 13:16.000 --> 13:20.000 you probably don't want to go down this road unless you also want to become the database team, 13:20.000 --> 13:24.000 because managing a global cluster databases and non-trivial thing. 13:24.000 --> 13:28.000 And I put a tiny joke in here, which is DNS itself, 13:28.000 --> 13:32.000 is supposed to be geographically distributed and redundant database, 13:32.000 --> 13:34.000 and you didn't want to have to do the work to do that, 13:34.000 --> 13:37.000 so you picked another geographically distributed and redundant database, 13:37.000 --> 13:42.000 underneath it to solve the problem for you just pushing the problem off to somebody else. 13:42.000 --> 13:47.000 So in addition, all of the databases that are generally used for this, 13:47.000 --> 13:50.000 they're not designed just for DNS, right? 13:50.000 --> 13:54.000 There are relational databases, my SQL and MariaDB and Postgres and lots of other things. 13:54.000 --> 13:58.000 They have thousands and thousands and thousands of features that are in authority 13:58.000 --> 14:01.000 to server with never need to use, so that makes them big and cumbersome. 14:01.000 --> 14:05.000 So I'm going to skip this one because we don't really need to do that. 14:05.000 --> 14:09.000 So there is another option, the thing that I mentioned that's relatively new. 14:10.000 --> 14:15.000 The first piece of it is two pieces of this puzzle, is a very lightweight database. 14:15.000 --> 14:19.000 Some want similar to SQLite, but not exactly the same thing as SQLite. 14:19.000 --> 14:24.000 Called LMDB, which is a relatively single file database 14:24.000 --> 14:27.000 where everything is loaded into memory by the application that uses it. 14:27.000 --> 14:32.000 It's very, very fast, very simple to make schema changes and all of those kinds of things. 14:32.000 --> 14:34.000 And when you use it to back up an authoritative server, 14:34.000 --> 14:37.000 then it holds all of the data that the authoritative server needs 14:37.000 --> 14:39.000 outside of what's in the configuration file. 14:39.000 --> 14:41.000 That's pretty simple. 14:41.000 --> 14:43.000 It's actually like the most trivial part of this. 14:43.000 --> 14:45.000 It just turned it on and it works. 14:45.000 --> 14:48.000 The second part is the thing that does the replication between them. 14:48.000 --> 14:50.000 So there's this tool now called lightning stream, 14:50.000 --> 14:53.000 which is a second process that you're on on your machine, 14:53.000 --> 14:55.000 and you'll see a picture of that in the moment, 14:55.000 --> 14:58.000 which watches for changes in the database, 14:58.000 --> 15:01.000 packages them up and makes them available in a way 15:01.000 --> 15:04.000 that other nodes can get copies of those changes. 15:04.000 --> 15:07.000 So you run one instance on every machine. 15:07.000 --> 15:09.000 It creates these things called snapshot files. 15:09.000 --> 15:11.000 Stores them into a, I've storage bucket, 15:11.000 --> 15:16.000 anything that uses the SREAPI is a suitable location to put it. 15:16.000 --> 15:19.000 And then all of the instances watch that bucket for changes. 15:19.000 --> 15:21.000 I want to see that there are changes there, 15:21.000 --> 15:24.000 download them, merge them into the local copy of LMDB, 15:24.000 --> 15:26.000 and you move on. 15:26.000 --> 15:29.000 So this is, just like X for a notify, 15:29.000 --> 15:31.000 this is only going to give you eventual consistency. 15:31.000 --> 15:33.000 You don't have any way to know exactly when all of those 15:34.000 --> 15:36.000 changes are going to show up, 15:36.000 --> 15:39.000 but the servers don't have to know about each other. 15:39.000 --> 15:42.000 Because we've used a database layer below, 15:42.000 --> 15:45.000 that does know about all of the nodes in a way. 15:45.000 --> 15:47.000 So this is what that looks like. 15:47.000 --> 15:49.000 It's a really trivial thing, 15:49.000 --> 15:53.000 which is application sits on top of a database file. 15:53.000 --> 15:56.000 Next to that, the application in lightning stream 15:56.000 --> 15:58.000 don't know about each other really at all. 15:58.000 --> 16:01.000 There's independent things that are sharing the same database file. 16:01.000 --> 16:04.000 And all the stuff back and forth to the object storage bucket. 16:04.000 --> 16:07.000 It works extremely well. 16:07.000 --> 16:09.000 So there are some benefits to doing this. 16:09.000 --> 16:13.000 The first is, none of those servers have to know about each other at all. 16:13.000 --> 16:17.000 The only thing is that none of the pieces of software 16:17.000 --> 16:20.000 in this equation have to know about the other instances at all. 16:20.000 --> 16:23.000 The only thing they have to know about is the object storage bucket. 16:23.000 --> 16:26.000 You can go from one server to 20 to 50 back to 3, 16:26.000 --> 16:29.000 and nothing has to change on any of the existing servers. 16:29.000 --> 16:32.000 The only thing that changes is the new servers come and go. 16:32.000 --> 16:34.000 That's it. 16:34.000 --> 16:39.000 So that means you can obviously add and remove things at the any time you like. 16:39.000 --> 16:42.000 The bandwidth requirements replication are much lower than the extra, 16:42.000 --> 16:47.000 because the data is compact, very compact format in these snapshots 16:47.000 --> 16:48.000 and center round. 16:48.000 --> 16:52.000 But it does use polling of the object storage bucket. 16:52.000 --> 16:54.000 So if you're paying for bandwidth, 16:54.000 --> 16:56.000 which is the reason I stopped using this, 16:57.000 --> 16:59.000 you will end up seeing that. 16:59.000 --> 17:01.000 And the polling frequency, obviously, 17:01.000 --> 17:04.000 how often you want your dynamic record changes to show up, 17:04.000 --> 17:06.000 is going to drive that bandwidth cost. 17:06.000 --> 17:09.000 If you have them polling every 10 seconds, for example, 17:09.000 --> 17:11.000 because you want stuff to show quickly, 17:11.000 --> 17:15.000 then suddenly you're going to have lots of network bandwidth to deal with. 17:15.000 --> 17:19.000 One other thing that is useful here, 17:19.000 --> 17:21.000 which X for a notify do not support, 17:21.000 --> 17:25.000 is that you can actually make changes to the zones at any server. 17:25.000 --> 17:29.000 If you wish to, if you want to allow dynamic update, 17:29.000 --> 17:32.000 DNS update, for example, on any of the servers in the graph, 17:32.000 --> 17:35.000 you can do that. 17:35.000 --> 17:37.000 There are some risks to using this. 17:37.000 --> 17:41.000 A really, really big one is the object storage bucket is your single point of failure. 17:41.000 --> 17:43.000 If something goes wrong there, 17:43.000 --> 17:45.000 you don't have replication anymore. 17:45.000 --> 17:47.000 The servers are going to keep answering queries, 17:47.000 --> 17:48.000 so you don't have to worry about that. 17:48.000 --> 17:52.000 You just can't get changes out to any of them anymore. 17:52.000 --> 17:55.000 The second one is, and this is something that I ran into, 17:55.000 --> 17:58.000 when replication isn't working properly, 17:58.000 --> 18:00.000 it's not just a regular database. 18:00.000 --> 18:03.000 You can open up a console for and do SQL queries against it 18:03.000 --> 18:04.000 to see what the data looks like. 18:04.000 --> 18:07.000 These bizarre compressed snapshots of data, 18:07.000 --> 18:09.000 and the only tool that exists, 18:09.000 --> 18:11.000 look at them is the lightning stream tool. 18:11.000 --> 18:13.000 So you have to learn a little bit about that. 18:13.000 --> 18:18.000 The final thing is, for now at least, 18:18.000 --> 18:20.000 while this is all open source, 18:20.000 --> 18:24.000 it's also only implemented in the PowerDNS authority of server. 18:24.000 --> 18:28.000 So using this is really simple. 18:28.000 --> 18:30.000 You already have PowerDNS off server installed, 18:30.000 --> 18:32.000 using 4.8, which is the latest. 18:32.000 --> 18:34.000 I think if that's the current. 18:34.000 --> 18:36.000 Okay, so 4.8 or 4.9. 18:36.000 --> 18:39.000 Version, that's all fine. 18:39.000 --> 18:41.000 You configure it to use an LMDB back in, 18:41.000 --> 18:43.000 which is really a really simple thing to do. 18:43.000 --> 18:45.000 You have to build lightning stream from source. 18:45.000 --> 18:46.000 It's very small. 18:46.000 --> 18:49.000 It's really easy to build. 18:49.000 --> 18:50.000 Get that all set up. 18:50.000 --> 18:52.000 On the server server, you have a service, 18:52.000 --> 18:54.000 where you use system D or unit scripts, 18:54.000 --> 18:56.000 or whatever you use on your server. 18:56.000 --> 18:58.000 It's a run lightning stream in sync mode, 18:58.000 --> 19:00.000 and it watches for changes in the LMDB, 19:00.000 --> 19:02.000 pushes them out to the bucket. 19:02.000 --> 19:03.000 On all the other machines, 19:03.000 --> 19:04.000 your run lightning stream receive, 19:04.000 --> 19:06.000 which watches for changes in the bucket, 19:06.000 --> 19:07.000 pulls everything down. 19:07.000 --> 19:08.000 It's really quite simple. 19:08.000 --> 19:11.000 You can run sync on all of the machines, 19:11.000 --> 19:12.000 not just receive, 19:12.000 --> 19:14.000 if you want to do multiple, 19:14.000 --> 19:16.000 multiple bi-directional replication on all of them. 19:17.000 --> 19:19.000 Be prepared for screwing that out, 19:19.000 --> 19:20.000 because I did twice, 19:20.000 --> 19:22.000 before I realized what I was doing wrong, 19:22.000 --> 19:23.000 which was, 19:23.000 --> 19:25.000 if you have two servers, 19:25.000 --> 19:27.000 both of which currently have a copy of the zone contents, 19:27.000 --> 19:29.000 and you set up bi-directional replication 19:29.000 --> 19:31.000 on both of them at the same time, 19:31.000 --> 19:32.000 both of them will say, 19:32.000 --> 19:33.000 aha! 19:33.000 --> 19:34.000 There's a new copy of the zone, 19:34.000 --> 19:36.000 and they will just keep fighting each other 19:36.000 --> 19:37.000 over and over again, 19:37.000 --> 19:38.000 because they, 19:38.000 --> 19:39.000 anyway, it's fun. 19:39.000 --> 19:40.000 Yeah. 19:40.000 --> 19:42.000 So, it's quick summary, 19:42.000 --> 19:44.000 and almost out of time, 19:44.000 --> 19:46.000 lightning stream and LMDB together, 19:46.000 --> 19:47.000 give you a lot of the benefits, 19:47.000 --> 19:49.000 you've used a new cluster database 19:49.000 --> 19:52.000 for replicating data between your authoritative servers. 19:52.000 --> 19:56.000 The resource consumption is extremely low, 19:56.000 --> 19:57.000 by comparison, 19:57.000 --> 19:58.000 for example, 19:58.000 --> 20:01.000 to running a MariaDB or Postgres instance on every machine, 20:01.000 --> 20:04.000 with their associated clustering software. 20:04.000 --> 20:07.000 It easily supports a femoral server deployments, 20:07.000 --> 20:10.000 which is fantastic when you're actually working on 20:10.000 --> 20:11.000 troubleshooting a problem, 20:11.000 --> 20:12.000 you can just say, 20:12.000 --> 20:15.000 it's going to launch a copy of the author's server right here on my laptop, 20:15.000 --> 20:17.000 and go get a copy of the data from the bucket 20:17.000 --> 20:18.000 and see what the data looks like. 20:18.000 --> 20:21.000 That's really, really nice to be able to do that. 20:21.000 --> 20:22.000 And then, of course, 20:22.000 --> 20:24.000 because it's not a complex database system, 20:24.000 --> 20:26.000 you don't have to administer one. 20:26.000 --> 20:28.000 So, and that is that. 20:28.000 --> 20:29.000 Thank you all. 20:29.000 --> 20:30.000 Thank you. 20:30.000 --> 20:33.000 I made one minute to stare. 20:33.000 --> 20:34.000 Yes. 20:34.000 --> 20:35.000 One minute to stare. 20:35.000 --> 20:37.000 It means we have room for one question, 20:37.000 --> 20:38.000 maybe two. 20:38.000 --> 20:40.000 How far can it scale? 20:41.000 --> 20:46.000 So, the question is, 20:46.000 --> 20:47.000 what is a scalability? 20:47.000 --> 20:49.000 Could it handle 100 billion records on? 20:49.000 --> 20:53.000 Personally, I have no experimental data to answer that question with. 20:53.000 --> 20:56.000 I don't know if anybody has actually done that yet. 20:56.000 --> 20:58.000 Okay, the answer from the team is, 20:58.000 --> 20:59.000 yes, it can do that. 20:59.000 --> 21:00.000 Yes. 21:00.000 --> 21:01.000 So, yeah. 21:01.000 --> 21:03.000 What do you mean? 21:03.000 --> 21:04.000 What do you mean? 21:04.000 --> 21:07.000 I've got a lot of the full cover of the open source project 21:07.000 --> 21:08.000 for redesses. 21:08.000 --> 21:10.000 But, you know, when you're talking about the failure, 21:10.000 --> 21:11.000 we'll book it. 21:11.000 --> 21:13.000 I'll think that with my soul. 21:13.000 --> 21:15.000 So, your question was about, 21:15.000 --> 21:18.000 having a single point of failure for the storage bucket. 21:18.000 --> 21:19.000 Yes. 21:19.000 --> 21:20.000 Right. 21:20.000 --> 21:21.000 Yeah. 21:21.000 --> 21:22.000 So, for example, 21:22.000 --> 21:23.000 if you really want to do this to, 21:23.000 --> 21:25.000 well, certainly you can get 21:25.000 --> 21:28.000 as three-style storage buckets from lots of cloud providers. 21:28.000 --> 21:29.000 And depending on which one you choose, 21:29.000 --> 21:33.000 they may have lots and lots of redundancy already available in their network. 21:33.000 --> 21:34.000 So, that helps you. 21:34.000 --> 21:36.000 If you want to do this yourself, 21:36.000 --> 21:38.000 this works very well with Minayo, 21:38.000 --> 21:41.000 which is the open source S3-style thing, 21:41.000 --> 21:44.000 which itself supports clustering amongst multiple nodes, 21:44.000 --> 21:46.000 and all of that kind of stuff. 21:46.000 --> 21:48.000 So, there are multiple ways you can approach that. 21:48.000 --> 21:49.000 Yeah. 21:49.000 --> 21:50.000 All right. 21:54.000 --> 21:55.000 Yes.