[Zonemaster-devel] Submitting bulk checks

Anand Buddhdev anandb at ripe.net
Wed Oct 5 14:14:22 UTC 2016


On 05/10/16 09:23, Michal Toma wrote:

Hi Michal,

> Well, my first advise would be to use a "real" database server like
> PostgreSQL that doesn't die because you are sending queries at it...
> I admit I might be "slightly" sarcastic here, lol. I abandoned MySQL
> more than 10 years ago and never looked back, now I'm a PostgreSQL DBA
> here at Afnic administering the Afnic Labs BigData related PostgreSQL
> server.

Alright, so I have switched to PostgreSQL 9.6 now :)

Also, in order to better understand how long things will take, I
submitted my bulk checks in increasingly larger batches.

1. Submitting 100 domains took 1.7s
2. Submitting 1000 domains took 16s
3. Submitting 10000 domains took 383s

I expected that submitting 10000 domains would take 10x as long, ie.
160s, but it took longer.

Now, if I wanted to submit all my domains (721150), it will take at
least 7 hours (based on simple linear extrapolation). If the system
becomes busy, then I expect it will take even longer to submit.

It seems like the batch submit method, using the API, is quite slow.

With the old DNScheck code, I could add all the domains to be checked by
writing them directly into the "queue" table of the database. Can I do
the same here? If so, what are the fields of the table that I should
update myself? If I understand it correctly, should I just queue new
jobs by adding entries to the "params" column?

And what is the purpose of the "priority" column?

Regards,
Anand


More information about the Zonemaster-devel mailing list