In this commit, we add a new atomic bool to only permit a single gossip
backlog goroutine per peer. If we get a new reuqest that needs a backlog
while we're still processing the other, then we'll drop that request.
In this commit, we add detailed documentation to help node operators
understand and configure the gossip rate limiting system effectively.
The new guide addresses a critical knowledge gap that has led to
misconfigured nodes experiencing synchronization failures.
The documentation covers the token bucket algorithm used for rate
limiting, providing clear formulas and examples for calculating
appropriate values based on node size and network position. We include
specific recommendations ranging from 50 KB/s for small nodes to
1 MB/s for large routing nodes, with detailed calculations showing
how these values are derived.
The guide explains the relationship between rate limiting and other
configuration options like num-restricted-slots and the new
filter-concurrency setting. We provide troubleshooting steps for
common issues like slow initial sync and peer disconnections, along
with debug commands and log patterns to identify problems.
Configuration examples are provided for conservative, balanced, and
performance-oriented setups, giving operators concrete starting points
they can adapt to their specific needs. The documentation emphasizes
the importance of not setting rate limits too low, warning that values
below 50 KB/s can cause synchronization to fail entirely.
In this commit, we complete the integration of the configurable gossip
filter concurrency by wiring the new FilterConcurrency configuration
through all layers of the application.
The changes connect the gossip.filter-concurrency configuration option
from the command-line interface through the server initialization code
to the gossiper and sync manager. This ensures that operators can
actually use the new configuration option to tune their node's
concurrent gossip filter processing capacity based on their specific
requirements and available resources.
In this commit, we complete the integration of the asynchronous
timestamp range queue by modifying ProcessRemoteAnnouncement to use
the new queuing mechanism instead of calling ApplyGossipFilter
synchronously.
This change ensures that when a peer sends a GossipTimestampRange
message, it is queued for asynchronous processing rather than
blocking the gossiper's main message processing loop. The modification
prevents the peer's readHandler from blocking on potentially slow
gossip filter operations, maintaining connection stability during
periods of high synchronization activity.
If the queue is full when attempting to enqueue a message, we log
a warning but return success to prevent peer disconnection. This
design choice prioritizes connection stability over guaranteed
delivery of every gossip filter request, which is acceptable since
peers can always resend timestamp range messages if needed.
In this commit, we introduce an asynchronous processing queue for
GossipTimestampRange messages in the GossipSyncer. This change addresses
a critical issue where the gossiper could block indefinitely when
processing timestamp range messages during periods of high load.
Previously, when a peer sent a GossipTimestampRange message, the
gossiper would synchronously call ApplyGossipFilter, which could block
on semaphore acquisition, database queries, and rate limiting. This
synchronous processing created a bottleneck where the entire peer
message processing pipeline would stall, potentially causing timeouts
and disconnections.
The new design adds a timestampRangeQueue channel with a capacity of 1
message and a dedicated goroutine for processing these messages
asynchronously. This follows the established pattern used for other
message types in the syncer. When the queue is full, we drop messages
and log a warning rather than blocking indefinitely, providing graceful
degradation under extreme load conditions.
In this commit, we make the gossip filter semaphore capacity configurable
through a new FilterConcurrency field. This change allows node operators
to tune the number of concurrent gossip filter applications based on
their node's resources and network position.
The previous hard-coded limit of 5 concurrent filter applications could
become a bottleneck when multiple peers attempt to synchronize
simultaneously. By making this value configurable via the new
gossip.filter-concurrency option, operators can increase this limit
for better performance on well-resourced nodes or maintain conservative
values on resource-constrained systems.
We keep the default value at 5 to maintain backward compatibility and
avoid unexpected resource usage increases for existing deployments. The
sample configuration file is updated to document this new option.
So far, most of the AI code review tools I've seen in the wild have been
pretty lackluster. I figure we should give the new Gemini Code Assist
variant a try: https://github.com/marketplace/gemini-code-assist/.
This PR adds an initial yaml config for it.
Relevant commands (as comments on the OP PR):
* `/gemini review` to perform a code review
* `/gemini summary` provides a code summary
* comment like normal tagging `@gemini-code-assist`
* `/gemini help`
This commit introduces the ResultOpt type, which represents an operation
that can either succeed with an optional final value or fail with an
error.
The .gitignore file is also updated to exclude specific files related to
the `aider` tool.
In this commit, we fix a flake in the
`TestRbfCloseClosingNegotiationLocal/send_offer_rbf_wrong_local_script`
test.
This flake can happen if the test shuts down _before_ the state machine
is actually able to process the sent event. In this case, the
expectations are triggered, and we find that the error isn't sent.
To resolve this, we create a new wrapper function that'll use a sync
channel send to assert that the error has been sent before we exit the
test.
In this commit, we add an option to allow a conf req caller to receive
the full block. This is useful if the caller wants to be able to create
an SPV proof.
This commit introduces a new generic type assertion function
`assertState` to the state machine tests. This function asserts that the
state machine is currently in the expected state type and returns the
state cast to that type. This allows us to directly access the fields of
the state without having to perform a type assertion manually.
In this commit, we add a new ConfMapper which is useful for state
machines that want to project some of the conf attributes into a new
event to be sent post conf.
In this commit, we start to signal the prod bit for the rbf coop close
feature. We keep our signaling of the staging bit in place to ensure
the protocol continues to work between those nodes in the wild that are
still signaling the bit.
Fixes https://github.com/lightningnetwork/lnd/issues/9852
In this commit, we add a new CLI option to control if we D/C on slow
pongs or not. Due to the existence of head-of-the-line blocking at
various levels of abstraction (app buffer, slow processing, TCP kernel
buffers, etc), if there's a flurry of gossip messages (eg: 1K channel
updates), then even with a reasonable processing latency, a peer may
still not read our ping in time.
To give users another option, we add a flag that allows users to disable
this behavior. The default remains.
In this commit, we make a change that enables the `auto_vacuum =
incremental` pragma for SQLite databases, but only when the database
file is first created. Incremental auto-vacuum allows SQLite to reclaim
unused space within the database file over time, preventing indefinite
growth.
In this commit, we test all the combinations of rbf close and taproot
chans. This ensures that the downgrade logic works properly.
Along the way we refactor the tests slightly, and also split them up, as
running all the combos back to back mines more than 50 blocks in a test,
which triggers an error in the itest sanity checks.
In this commit, we fix a flake in the rbf loop sub-test for the
TestRbfCloseClosingNegotiationLocal test case.
The fix here is that when we go from ClosePending for an RBF iteration
loop, we first transition to LocalCloseStart. However we only do this
extra transition if we're doing an iteration (starting from ClosePending).
To fix this, we add a new bool that tracks if this is an iteration or
not. We can then also eliminate the extra assertion at the end, as we'll
terminate in `ClosePending` which is checked by
`assertLocalClosePending()` in `assertSingleRbfIteration`.
Fixes https://github.com/lightningnetwork/lnd/issues/9526.
In this commit, we implement logic to downgrade to the legacy coop close
for taproot channels. Before this commit, we wouldn't allow nodes to
start up with both the taproot flag and the rbf flag activated.
In the future, once we implement the spec updates, we'll add support for
this combo, and can revert parts of this commit.
In this commit, we fix a flake in the
`TestRbfChannelFlushingTransitions/early_offer` test. The fix is simple:
this is actually an "iteration", as we have a self transition to the
ChannelNegotiation state first. We also don't need to send the
remoteOffer, so we set `sendInit` to false. The offer still needs to be
passed in to ensure that the assertions work however.
In this commit, we reduce log spam a bit during state transitions. We
only log the type of the event when sending the event, as we'll print
the same event when creating the queue "applying", and later when
processing". Next we remove the "applying" log as that first event will
always be logged twice.
The combo of these to changes makes the logs much easier to follow.
In this commit, we revamp the old message based rate limiting. First, we
move to meter by bytes/s instead of messages/s. The old logic had an
error in that it limited groups of message replies, instead of each
message. With this new approach, we'll use the newly added
SerializedSize method to implement fine grained bandwidth metering.
We need to pick two values, the burst rate, and the msg bytes rate. The
burst rate is the max amt that can be sent in a given period of time. We
need to set this above 65 KB, or the max msg limit, otherwise no
messages can be sent. The bucket starts with this many tokens (bytes).
As those are depleted, the amount of tokens is refilled at the msg
bytes rate.
As conservative values, we've chosen 200 KB as the burst rate, and 100
KB/s as the limit.
With what we added in the prior commit, we can significantly shrink the
size of this test. We also make it easier to extend in the future, as
this will fail if a new message is added, that doesn't have the needed
methods, as long as MsgEnd is updated.
In this commit, we add a new `TestMessage` interface for use in property
tests. With this, we'll be able to generate a random instance of a given
message, using the rapid byte stream. This can also eventually be useful
for fuzzing.
This fixes an issue in the itests in the restart case. We'd see an error
like:
```
2025-03-12 23:41:10.754 [ERR] PFSM state_machine.go:661: FSM(rbf_chan_closer(2f20725d9004f7fda7ef280f77dd8d419fd6669bda1a5231dd58d6f6597066e0:0)): Unable to apply event err="invalid state transition: received *chancloser.SendOfferEvent while in ClosingNegotiation(local=LocalOfferSent(proposed_fee=0.00000193 BTC), remote=ClosePending(txid=07229915459cb439bdb8ad4f5bf112dc6f42fca0192ea16a7d6dd05e607b92ae, party=Remote, fee_rate=1 sat/vb))"
```
We resolve this by waiting to send in the new request unil the old one
has been completed.
The itest has both sides try to close multiple times, each time with
increasing fee rates. We also test the reconnection case, bad RBF
updates, and instances where the local party can't actually pay for
fees.
In this commit, we alter the existing co-op close flow to enable RBF
bumps after re connection. With the new RBF close flow, it's possible
that after a success round _and_ a re connection, either side wants to do
another fee bump. Typically we route these requests through the switch,
but in this case, the link no longer exists in the switch, so any
requests to fee bump again would find that the link doesn't exist.
In this commit, we implement a work around wherein if we have an RBF
chan closer active, and the link isn't in the switch, then we just route
the request directly to the chan closer via the peer. Once we have the
chan closer, we can use the exact same flow as prior.
We'll properly handle a protocol error due to user input by halting, and
sending the error back to the user.
When a user goes to issue a new update, based on which state we're in,
we'll either kick off the shutdown, or attempt a new offer. This matches
the new spec update where we'll only send `Shutdown` once per
connection.
Olaoluwa Osuntokun