[Zonemaster-devel] Submitting bulk checks

Anand Buddhdev anandb at ripe.net
Tue Oct 4 17:19:30 UTC 2016


On 03/10/16 16:28, Michal Toma wrote:

Hi Michal,

> Attached you will find the batch_test_FR_zone.pl script that we use
> to run the zonemaster backend in batch mode.

I read the documentation of the API call "add_batch_job", and created a
job for this purpose. My JSON looks like this:

{
"method":"add_batch_job",
"params":{
	"username":"bulk",
	"api_key":"bulk",
	"test_params":{
		"ipv4":true,
		"ipv6":true
	},
	"domains":[
"200.193.193.in-addr.arpa",
"17.45.212.in-addr.arpa",
...
...
"176.188.89.in-addr.arpa",
"e.5.4.c.2.1.0.2.8.b.7.0.1.0.a.2.ip6.arpa"
]
}
}

The total number of domains to test is 721150. I submitted it like this:

curl http://localhost:5000 -d @zones.json

This ran for several seconds, and I saw results starting to appear in
the "test_results" table of the database, but then it died:

{"jsonrpc":"2.0","error":{"message":"DBD::mysql::db do failed: MySQL
server has gone away at
/opt/zonemaster/lib/perl5/Zonemaster/WebBackend/DB/MySQL.pm line
151.\n","code":-32603},"id":null}

It looks like the mysql server was killed, because it was overloaded. So
my batch job was not submitted properly, and I will have to retry. My
question is: when submitting the batch job, where does zonemaster keep a
journal of which domains are to be tested? How can it resume the testing
after the database has been restarted? The old DNScheck had a "queue"
table where all domains to be tested were queued, but zonemasters
doesn't appear to have anything like this.

Also, should I be doing anything differently when submitting a large
number of domains for testing in bulk?

Regards,
Anand


More information about the Zonemaster-devel mailing list