By “flaky” you mean slower, I assume.
Improved non-IPv6 in non-forwarding mode isn’t in HBT yet, so I suspect in your case you’d still benefit from setting net_ipv6 explicitly. (There is some auto-detection in init script, firing two minutes after (re)start of DNS, but I suspect it’s not perfectly reliable.)