In this commit we add a new helper method in the sqldb package:
ExecuteCollectAndBatchWithSharedDataQuery. This can be used to paginate
through items in the database while at the same time performing batch
data collection for those items.
In this commit, we add a makefile helper that can be used to add the
"-short" flag to the "go test" command when running unit tests via "make
unit ...".
Tests can then be expanded with a `testing.Short()` check to potentially
skip the test in short mode. This is useful for if a dev wants to
quickly run most of the tests in a package but would like to opt out of
running the longer form tests such as stress tests or rapid generation
tests.
This commit adds this check to two graph/db tests.
When users set `gossip.ban-threshold` to 0, it's now treated as setting
the ban score to max uint64, which effectively disables the banning. We
still want to record the peer's ban score in case we need it for future
debugging.
We move all kv particular code which was in the payments.go file
to the kv related file. Code which will be backend independant
will remain in the payments.go file although only the kv backend
is currently supported.
We also addd this new db on the server level to use it in the
following commit to do all the payment related queries of the
rpcserver.
We add a new payment db instance on the server level. Which we
will you for the payment related queries in a following commit.
In the following commits we will gradually unify the current
payment db operations into an interface to later down the road
support both backends (sql+kv).
In this commit, we add a new atomic bool to only permit a single gossip
backlog goroutine per peer. If we get a new reuqest that needs a backlog
while we're still processing the other, then we'll drop that request.
In this commit, we add detailed documentation to help node operators
understand and configure the gossip rate limiting system effectively.
The new guide addresses a critical knowledge gap that has led to
misconfigured nodes experiencing synchronization failures.
The documentation covers the token bucket algorithm used for rate
limiting, providing clear formulas and examples for calculating
appropriate values based on node size and network position. We include
specific recommendations ranging from 50 KB/s for small nodes to
1 MB/s for large routing nodes, with detailed calculations showing
how these values are derived.
The guide explains the relationship between rate limiting and other
configuration options like num-restricted-slots and the new
filter-concurrency setting. We provide troubleshooting steps for
common issues like slow initial sync and peer disconnections, along
with debug commands and log patterns to identify problems.
Configuration examples are provided for conservative, balanced, and
performance-oriented setups, giving operators concrete starting points
they can adapt to their specific needs. The documentation emphasizes
the importance of not setting rate limits too low, warning that values
below 50 KB/s can cause synchronization to fail entirely.
In this commit, we complete the integration of the configurable gossip
filter concurrency by wiring the new FilterConcurrency configuration
through all layers of the application.
The changes connect the gossip.filter-concurrency configuration option
from the command-line interface through the server initialization code
to the gossiper and sync manager. This ensures that operators can
actually use the new configuration option to tune their node's
concurrent gossip filter processing capacity based on their specific
requirements and available resources.
In this commit, we complete the integration of the asynchronous
timestamp range queue by modifying ProcessRemoteAnnouncement to use
the new queuing mechanism instead of calling ApplyGossipFilter
synchronously.
This change ensures that when a peer sends a GossipTimestampRange
message, it is queued for asynchronous processing rather than
blocking the gossiper's main message processing loop. The modification
prevents the peer's readHandler from blocking on potentially slow
gossip filter operations, maintaining connection stability during
periods of high synchronization activity.
If the queue is full when attempting to enqueue a message, we log
a warning but return success to prevent peer disconnection. This
design choice prioritizes connection stability over guaranteed
delivery of every gossip filter request, which is acceptable since
peers can always resend timestamp range messages if needed.
Elle Mouton