[Zonemaster-devel] Submitting bulk checks
anandb at ripe.net
Wed Oct 5 14:14:22 UTC 2016
On 05/10/16 09:23, Michal Toma wrote:
> Well, my first advise would be to use a "real" database server like
> PostgreSQL that doesn't die because you are sending queries at it...
> I admit I might be "slightly" sarcastic here, lol. I abandoned MySQL
> more than 10 years ago and never looked back, now I'm a PostgreSQL DBA
> here at Afnic administering the Afnic Labs BigData related PostgreSQL
Alright, so I have switched to PostgreSQL 9.6 now :)
Also, in order to better understand how long things will take, I
submitted my bulk checks in increasingly larger batches.
1. Submitting 100 domains took 1.7s
2. Submitting 1000 domains took 16s
3. Submitting 10000 domains took 383s
I expected that submitting 10000 domains would take 10x as long, ie.
160s, but it took longer.
Now, if I wanted to submit all my domains (721150), it will take at
least 7 hours (based on simple linear extrapolation). If the system
becomes busy, then I expect it will take even longer to submit.
It seems like the batch submit method, using the API, is quite slow.
With the old DNScheck code, I could add all the domains to be checked by
writing them directly into the "queue" table of the database. Can I do
the same here? If so, what are the fields of the table that I should
update myself? If I understand it correctly, should I just queue new
jobs by adding entries to the "params" column?
And what is the purpose of the "priority" column?
More information about the Zonemaster-devel