In this commit, we update the mockChannelGraphTimeSeries to implement
the new iterator-based UpdatesInHorizon interface. The mock maintains
its existing behavior of receiving messages through a channel and
returning them to the caller, but now wraps this in an iterator
function.
The implementation creates an iterator that pulls the entire message
slice from the mock's response channel, then yields each message
individually. This preserves the test semantics while conforming to the
new interface, ensuring all existing tests continue to pass without
modification.
In this commit, we update ApplyGossipFilter to leverage the new
iterator-based UpdatesInHorizon method. The key innovation here is using
iter.Pull2 to create a pull-based iterator that allows us to check if
any updates exist before launching the background goroutine.
This approach provides several benefits over the previous implementation.
First, we avoid the overhead of launching a goroutine when there are no
updates to send, which was previously unavoidable without materializing
the entire result set. Second, we maintain lazy loading throughout the
sending process, only pulling messages from the database as they're
needed for transmission.
The implementation uses Pull2 to peek at the first message, determining
whether to proceed with sending updates. If updates exist, ownership of
the iterator is transferred to the goroutine, which continues pulling
and sending messages until exhausted. This design ensures memory usage
remains bounded regardless of the number of updates being synchronized.
In this commit, we complete the iterator conversion work started in PR
10128 by threading the iterator pattern through to the higher-level
UpdatesInHorizon method. This change converts the method from returning
a fully materialized slice of messages to returning a lazy iterator that
yields messages on demand.
The new signature uses iter.Seq2 to allow error propagation during
iteration, eliminating the need for a separate error return value. This
approach enables callers to handle errors as they occur during iteration
rather than failing upfront.
The implementation now lazily processes channel and node updates,
yielding them as they're generated rather than accumulating them in
memory. This maintains the same ordering guarantees (channels before
nodes) while significantly reducing memory pressure when dealing with
large update sets during gossip synchronization.
In this commit, we update all callers of NodeUpdatesInHorizon and
ChanUpdatesInHorizon to use the new iterator-based APIs. The changes
use fn.Collect to maintain existing behavior while benefiting from the
memory efficiency of iterators when possible.
In this commit, we update the SQL store implementation to support the
new iterator-based API for ChanUpdatesInHorizon. This includes adding
SQL query pagination support and helper functions for efficient batch
processing.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetChannelsByPolicyLastUpdateRange is updated to support:
- Cursor-based pagination using (max_update_time, id) compound cursor
- Configurable batch sizes via MaxResults parameter
- Efficient batch caching with updateChanCacheBatch helper
In this commit, we update the SQL store implementation to support the
new iterator-based API for NodeUpdatesInHorizon. This includes adding a
new SQL query that supports efficient pagination through result sets.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetNodesByLastUpdateRange is updated to support:
* Cursor-based pagination using (last_update, pub_key) compound cursor
* Optional filtering for public nodes only
* Configurable batch sizes via MaxResults parameter
In this commit, we refactor the ChanUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
In this commit, we refactor the NodeUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
The new implementation uses Go 1.23's iter.Seq type to provide a
standard iterator interface. The method now supports configurable batch
sizes through functional options, allowing fine-tuned control over
memory usage and performance characteristics.
Rather than reading all the entries from disk into memory (before this
commit, we did consult the cache for most entries, skipping the disk
hits), we now expose a chunked iterator instead.
We also make the process of filtering out public nodes first class. This
saves many newly created db transactions later.
In this commit, we introduce a new options pattern for configuring
iterator behavior in the graph database. This includes configuration
for batch sizes when iterating over channel and node updates, as well
as an option to filter for public nodes only.
The new functional options pattern allows callers to customize iterator
behavior without breaking existing APIs. Default batch sizes are set to
1000 entries for both channel and node updates, which provides a good
balance between memory usage and performance.
In this commit, we introduce a new utility function `Collect` to the fn
package. This function drains all elements from an iterator and returns
them as a slice. This is particularly useful when transitioning from
iterator-based APIs to code that expects slices, allowing for gradual
migration to the new iterator patterns.
The fn module's go.mod is also updated to require Go 1.23, which is
necessary for the built-in iter.Seq type support.
The replace directive will be removed once the fn package changes are
merged and a new version is tagged.
In this commit, we add some docs that explain how to use tools like
heap escape analysis and memory profiling to fully eliminate allocations
in a sample program.
This guide is meant to help devs/contributors use Go's excellent perf
tools to zoom in on an optimization problem.
In this commit, we eliminate the final allocation that takes place when
we write out messages. Once again this was escaping to the heap, so we
make it an attribute on the Machine struct, which allows pure static
allocation.
```
goos: darwin
goarch: arm64
pkg: github.com/lightningnetwork/lnd/brontide
cpu: Apple M4 Max
BenchmarkWriteMessage-16 25840 46376 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25646 46672 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25874 46391 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25512 46427 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25760 46309 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25789 46520 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25602 46619 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25766 46464 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25820 46487 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25634 46553 ns/op 2 B/op 0 allocs/op
PASS
ok github.com/lightningnetwork/lnd/brontide 16.907s
```
In this commit, we complete the migration to lnutils.SpewLogClosure for
the remaining application-layer components. This includes the autopilot
channel management system, the funding manager, the RPC server, and the
invoice storage subsystem.
In this commit, we update the contractcourt, sweep, and watchtower
packages to use lnutils.SpewLogClosure instead of directly calling
spew.Sdump in logging statements.
In this commit, we replace all instances of spew.Sdump in the lnwallet
package and its subpackages with lnutils.SpewLogClosure. This change
ensures that expensive debug dump operations are only performed when the
log level actually requires them.
In this commit, we revamp the way our skip-labels worked to instead mark
certain stages as green, instead of skipping them. Skipped changes
didn't count towards the set of required checks, which slowed down PR
velocity.
Previously, when PRs were labeled with 'no-itest' or 'no-changelog',
the corresponding CI jobs would be completely skipped. This caused the
GitHub checks to show as skipped rather than successful, which could
be confusing and prevented certain merge rules from working properly.
This commit changes the behavior so that these jobs still run but
immediately report success when the special labels are detected. Each
affected job now starts with a label check step that sets a skip
flag, and all subsequent steps are conditionally executed based on
this flag. When skipped, the jobs add a notice to the GitHub step
summary explaining that tests were auto-passed due to the label.
The change affects five jobs: basic-integration-test, integration-test,
windows-integration-test, macos-integration-test, and milestone-check.
In this commit, we add a new atomic bool to only permit a single gossip
backlog goroutine per peer. If we get a new reuqest that needs a backlog
while we're still processing the other, then we'll drop that request.
In this commit, we add detailed documentation to help node operators
understand and configure the gossip rate limiting system effectively.
The new guide addresses a critical knowledge gap that has led to
misconfigured nodes experiencing synchronization failures.
The documentation covers the token bucket algorithm used for rate
limiting, providing clear formulas and examples for calculating
appropriate values based on node size and network position. We include
specific recommendations ranging from 50 KB/s for small nodes to
1 MB/s for large routing nodes, with detailed calculations showing
how these values are derived.
The guide explains the relationship between rate limiting and other
configuration options like num-restricted-slots and the new
filter-concurrency setting. We provide troubleshooting steps for
common issues like slow initial sync and peer disconnections, along
with debug commands and log patterns to identify problems.
Configuration examples are provided for conservative, balanced, and
performance-oriented setups, giving operators concrete starting points
they can adapt to their specific needs. The documentation emphasizes
the importance of not setting rate limits too low, warning that values
below 50 KB/s can cause synchronization to fail entirely.
In this commit, we complete the integration of the configurable gossip
filter concurrency by wiring the new FilterConcurrency configuration
through all layers of the application.
The changes connect the gossip.filter-concurrency configuration option
from the command-line interface through the server initialization code
to the gossiper and sync manager. This ensures that operators can
actually use the new configuration option to tune their node's
concurrent gossip filter processing capacity based on their specific
requirements and available resources.
In this commit, we complete the integration of the asynchronous
timestamp range queue by modifying ProcessRemoteAnnouncement to use
the new queuing mechanism instead of calling ApplyGossipFilter
synchronously.
This change ensures that when a peer sends a GossipTimestampRange
message, it is queued for asynchronous processing rather than
blocking the gossiper's main message processing loop. The modification
prevents the peer's readHandler from blocking on potentially slow
gossip filter operations, maintaining connection stability during
periods of high synchronization activity.
If the queue is full when attempting to enqueue a message, we log
a warning but return success to prevent peer disconnection. This
design choice prioritizes connection stability over guaranteed
delivery of every gossip filter request, which is acceptable since
peers can always resend timestamp range messages if needed.
In this commit, we introduce an asynchronous processing queue for
GossipTimestampRange messages in the GossipSyncer. This change addresses
a critical issue where the gossiper could block indefinitely when
processing timestamp range messages during periods of high load.
Previously, when a peer sent a GossipTimestampRange message, the
gossiper would synchronously call ApplyGossipFilter, which could block
on semaphore acquisition, database queries, and rate limiting. This
synchronous processing created a bottleneck where the entire peer
message processing pipeline would stall, potentially causing timeouts
and disconnections.
The new design adds a timestampRangeQueue channel with a capacity of 1
message and a dedicated goroutine for processing these messages
asynchronously. This follows the established pattern used for other
message types in the syncer. When the queue is full, we drop messages
and log a warning rather than blocking indefinitely, providing graceful
degradation under extreme load conditions.
In this commit, we make the gossip filter semaphore capacity configurable
through a new FilterConcurrency field. This change allows node operators
to tune the number of concurrent gossip filter applications based on
their node's resources and network position.
The previous hard-coded limit of 5 concurrent filter applications could
become a bottleneck when multiple peers attempt to synchronize
simultaneously. By making this value configurable via the new
gossip.filter-concurrency option, operators can increase this limit
for better performance on well-resourced nodes or maintain conservative
values on resource-constrained systems.
We keep the default value at 5 to maintain backward compatibility and
avoid unexpected resource usage increases for existing deployments. The
sample configuration file is updated to document this new option.
So far, most of the AI code review tools I've seen in the wild have been
pretty lackluster. I figure we should give the new Gemini Code Assist
variant a try: https://github.com/marketplace/gemini-code-assist/.
This PR adds an initial yaml config for it.
Relevant commands (as comments on the OP PR):
* `/gemini review` to perform a code review
* `/gemini summary` provides a code summary
* comment like normal tagging `@gemini-code-assist`
* `/gemini help`
This commit introduces the ResultOpt type, which represents an operation
that can either succeed with an optional final value or fail with an
error.
The .gitignore file is also updated to exclude specific files related to
the `aider` tool.
In this commit, we fix a flake in the
`TestRbfCloseClosingNegotiationLocal/send_offer_rbf_wrong_local_script`
test.
This flake can happen if the test shuts down _before_ the state machine
is actually able to process the sent event. In this case, the
expectations are triggered, and we find that the error isn't sent.
To resolve this, we create a new wrapper function that'll use a sync
channel send to assert that the error has been sent before we exit the
test.
In this commit, we add an option to allow a conf req caller to receive
the full block. This is useful if the caller wants to be able to create
an SPV proof.
This commit introduces a new generic type assertion function
`assertState` to the state machine tests. This function asserts that the
state machine is currently in the expected state type and returns the
state cast to that type. This allows us to directly access the fields of
the state without having to perform a type assertion manually.
In this commit, we add a new ConfMapper which is useful for state
machines that want to project some of the conf attributes into a new
event to be sent post conf.
In this commit, we start to signal the prod bit for the rbf coop close
feature. We keep our signaling of the staging bit in place to ensure
the protocol continues to work between those nodes in the wild that are
still signaling the bit.
Fixes https://github.com/lightningnetwork/lnd/issues/9852
In this commit, we add a new CLI option to control if we D/C on slow
pongs or not. Due to the existence of head-of-the-line blocking at
various levels of abstraction (app buffer, slow processing, TCP kernel
buffers, etc), if there's a flurry of gossip messages (eg: 1K channel
updates), then even with a reasonable processing latency, a peer may
still not read our ping in time.
To give users another option, we add a flag that allows users to disable
this behavior. The default remains.
Olaoluwa Osuntokun