Add comprehensive test coverage to verify that state handler errors cause
the channelGraphSyncer goroutine to exit cleanly without entering endless
retry loops. These tests use mutation testing principles to ensure they
would fail if the fixes were removed.
TestGossipSyncerStateHandlerErrors is a table-driven test covering four
scenarios: context cancellation and peer disconnect during syncingChans
state, and context cancellation and network errors during queryNewChannels
state. Each test case verifies both attempt count (no endless loop) and
clean shutdown (no deadlock).
TestGossipSyncerProcessChanRangeReplyError verifies that errors from
processChanRangeReply in the waitingQueryRangeReply state cause clean
exit. This test sends multiple malformed messages and checks that only
the first is processed before the goroutine exits, using channel queue
depth to detect if the goroutine is still running.
All tests are race-detector clean and use mutation testing validation:
removing any of the error return statements causes the corresponding
tests to fail, confirming the tests properly verify the fixes.
This commit fixes a critical bug where the channelGraphSyncer goroutine
would enter an endless loop when context cancellation or peer disconnect
errors occurred during the syncingChans or queryNewChannels states.
The root cause was that state handler functions (handleSyncingChans and
synchronizeChanIDs) did not return errors to the main goroutine loop.
When these functions encountered fatal errors like context cancellation,
they would log the error and return early without changing the syncer's
state. This caused the main loop to immediately re-enter the same state
handler, encounter the same error, and loop indefinitely while spamming
error logs.
The fix makes error handling explicit by having state handlers return
errors. The main channelGraphSyncer loop now checks these errors and
exits cleanly when fatal errors occur. We return any error (not just
context cancellation) because fatal errors can manifest in multiple
forms: context.Canceled, ErrGossipSyncerExiting from the rate limiter,
lnpeer.ErrPeerExiting from Brontide, or network errors like connection
closed. This approach matches the error handling pattern already used in
other goroutines like replyHandler.
My key recently expired, in this commit, we update the keys to the new
refreshed version. These are the same keys, but with an expiry further
out.
Here's a clear sign of the latest Bitcoin block hash:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
000000000000000000013215ef7c32bc0427f388fc83623affe712f388
-----BEGIN PGP SIGNATURE-----
iHUEARYKAB0WIQQpYhJoGq3wVlaize6QUl997uCthgUCaPi6xwAKCRCQUl997uCt
hpqNAQC5VnnbO6h/PjywGhU4LLRvH8SdgdDEMSc7xrtWd1vgPgD+IDrHqiAb+h38
ORBnUVJCVuZrPebtdnYXVQhII91eaw4=
=WRbl
-----END PGP SIGNATURE-----
In this commit, we add a call to "go clean -cache" after each platform
build in the release script to prevent the Go build cache from accumulating
unbounded disk space during the sequential 15-platform build process.
When building for multiple platforms in sequence with "go build -v", Go
creates intermediate build artifacts and caches compiled packages for each
target platform. While this caching improves build performance within a
single platform build, it causes the cache to grow substantially when
building for many platforms sequentially. With 15 different platform/
architecture combinations, each with their own cached artifacts, this
accumulation was contributing to the disk space exhaustion.
By clearing the build cache after each platform completes, we prevent this
unbounded growth while still allowing each individual platform build to
benefit from caching during its own compilation. The module cache is
preserved (we only clear the build cache), so dependencies don't need to be
re-downloaded between platforms.
In this commit, we replace the basic inline cleanup command in the release
workflow with the comprehensive cleanup-space action that was previously
only used in the main CI workflow. The previous release workflow cleanup
simply removed the hostedtoolcache directory, which freed only a few
gigabytes and proved insufficient for multi-platform release builds.
By switching to the cleanup-space action (now enhanced to free 20-25GB),
the release workflow will have substantially more disk space available
before beginning the build process. This should resolve the disk space
exhaustion issues that were occurring during the Windows ARM build phase,
which is one of the final platforms in the 15-platform build sequence.
In this commit, we significantly expand the cleanup-space GitHub Actions
workflow to free up substantially more disk space on GitHub runners. The
previous cleanup only removed three large toolsets (dotnet, android,
ghc), which should free ~14GB. This enhancement adds removal of several
additional large packages and caches, bringing the total freed space to
approximately 20-25GB.
The specific additions include removing Swift and Julia language runtimes,
the hosted toolcache directory, all Docker images, numerous large apt
packages (aspnetcore, llvm, php, mongodb, mysql, azure-cli, browsers, and
development tools), and various cache directories. We also add disk space
reporting before and after cleanup to provide visibility into how much
space is actually being freed during workflow runs.
This enhancement was motivated by release builds running out of disk space
when building for all 15 supported platforms (darwin, freebsd, linux,
netbsd, openbsd, windows across multiple architectures). The sequential
builds with verbose output were consuming more space than the basic cleanup
could provide.
In this commit, we update the mockChannelGraphTimeSeries to implement
the new iterator-based UpdatesInHorizon interface. The mock maintains
its existing behavior of receiving messages through a channel and
returning them to the caller, but now wraps this in an iterator
function.
The implementation creates an iterator that pulls the entire message
slice from the mock's response channel, then yields each message
individually. This preserves the test semantics while conforming to the
new interface, ensuring all existing tests continue to pass without
modification.
In this commit, we update ApplyGossipFilter to leverage the new
iterator-based UpdatesInHorizon method. The key innovation here is using
iter.Pull2 to create a pull-based iterator that allows us to check if
any updates exist before launching the background goroutine.
This approach provides several benefits over the previous implementation.
First, we avoid the overhead of launching a goroutine when there are no
updates to send, which was previously unavoidable without materializing
the entire result set. Second, we maintain lazy loading throughout the
sending process, only pulling messages from the database as they're
needed for transmission.
The implementation uses Pull2 to peek at the first message, determining
whether to proceed with sending updates. If updates exist, ownership of
the iterator is transferred to the goroutine, which continues pulling
and sending messages until exhausted. This design ensures memory usage
remains bounded regardless of the number of updates being synchronized.
In this commit, we complete the iterator conversion work started in PR
10128 by threading the iterator pattern through to the higher-level
UpdatesInHorizon method. This change converts the method from returning
a fully materialized slice of messages to returning a lazy iterator that
yields messages on demand.
The new signature uses iter.Seq2 to allow error propagation during
iteration, eliminating the need for a separate error return value. This
approach enables callers to handle errors as they occur during iteration
rather than failing upfront.
The implementation now lazily processes channel and node updates,
yielding them as they're generated rather than accumulating them in
memory. This maintains the same ordering guarantees (channels before
nodes) while significantly reducing memory pressure when dealing with
large update sets during gossip synchronization.
In this commit, we update all callers of NodeUpdatesInHorizon and
ChanUpdatesInHorizon to use the new iterator-based APIs. The changes
use fn.Collect to maintain existing behavior while benefiting from the
memory efficiency of iterators when possible.
In this commit, we update the SQL store implementation to support the
new iterator-based API for ChanUpdatesInHorizon. This includes adding
SQL query pagination support and helper functions for efficient batch
processing.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetChannelsByPolicyLastUpdateRange is updated to support:
- Cursor-based pagination using (max_update_time, id) compound cursor
- Configurable batch sizes via MaxResults parameter
- Efficient batch caching with updateChanCacheBatch helper
In this commit, we update the SQL store implementation to support the
new iterator-based API for NodeUpdatesInHorizon. This includes adding a
new SQL query that supports efficient pagination through result sets.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetNodesByLastUpdateRange is updated to support:
* Cursor-based pagination using (last_update, pub_key) compound cursor
* Optional filtering for public nodes only
* Configurable batch sizes via MaxResults parameter
In this commit, we refactor the ChanUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
In this commit, we refactor the NodeUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
The new implementation uses Go 1.23's iter.Seq type to provide a
standard iterator interface. The method now supports configurable batch
sizes through functional options, allowing fine-tuned control over
memory usage and performance characteristics.
Rather than reading all the entries from disk into memory (before this
commit, we did consult the cache for most entries, skipping the disk
hits), we now expose a chunked iterator instead.
We also make the process of filtering out public nodes first class. This
saves many newly created db transactions later.
In this commit, we introduce a new options pattern for configuring
iterator behavior in the graph database. This includes configuration
for batch sizes when iterating over channel and node updates, as well
as an option to filter for public nodes only.
The new functional options pattern allows callers to customize iterator
behavior without breaking existing APIs. Default batch sizes are set to
1000 entries for both channel and node updates, which provides a good
balance between memory usage and performance.
In this commit, we introduce a new utility function `Collect` to the fn
package. This function drains all elements from an iterator and returns
them as a slice. This is particularly useful when transitioning from
iterator-based APIs to code that expects slices, allowing for gradual
migration to the new iterator patterns.
The fn module's go.mod is also updated to require Go 1.23, which is
necessary for the built-in iter.Seq type support.
The replace directive will be removed once the fn package changes are
merged and a new version is tagged.
In this commit, we add some docs that explain how to use tools like
heap escape analysis and memory profiling to fully eliminate allocations
in a sample program.
This guide is meant to help devs/contributors use Go's excellent perf
tools to zoom in on an optimization problem.
In this commit, we eliminate the final allocation that takes place when
we write out messages. Once again this was escaping to the heap, so we
make it an attribute on the Machine struct, which allows pure static
allocation.
```
goos: darwin
goarch: arm64
pkg: github.com/lightningnetwork/lnd/brontide
cpu: Apple M4 Max
BenchmarkWriteMessage-16 25840 46376 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25646 46672 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25874 46391 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25512 46427 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25760 46309 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25789 46520 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25602 46619 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25766 46464 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25820 46487 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25634 46553 ns/op 2 B/op 0 allocs/op
PASS
ok github.com/lightningnetwork/lnd/brontide 16.907s
```
In this commit, we complete the migration to lnutils.SpewLogClosure for
the remaining application-layer components. This includes the autopilot
channel management system, the funding manager, the RPC server, and the
invoice storage subsystem.
In this commit, we update the contractcourt, sweep, and watchtower
packages to use lnutils.SpewLogClosure instead of directly calling
spew.Sdump in logging statements.
In this commit, we replace all instances of spew.Sdump in the lnwallet
package and its subpackages with lnutils.SpewLogClosure. This change
ensures that expensive debug dump operations are only performed when the
log level actually requires them.
In this commit, we revamp the way our skip-labels worked to instead mark
certain stages as green, instead of skipping them. Skipped changes
didn't count towards the set of required checks, which slowed down PR
velocity.
Previously, when PRs were labeled with 'no-itest' or 'no-changelog',
the corresponding CI jobs would be completely skipped. This caused the
GitHub checks to show as skipped rather than successful, which could
be confusing and prevented certain merge rules from working properly.
This commit changes the behavior so that these jobs still run but
immediately report success when the special labels are detected. Each
affected job now starts with a label check step that sets a skip
flag, and all subsequent steps are conditionally executed based on
this flag. When skipped, the jobs add a notice to the GitHub step
summary explaining that tests were auto-passed due to the label.
The change affects five jobs: basic-integration-test, integration-test,
windows-integration-test, macos-integration-test, and milestone-check.
In this commit, we add a new atomic bool to only permit a single gossip
backlog goroutine per peer. If we get a new reuqest that needs a backlog
while we're still processing the other, then we'll drop that request.
In this commit, we add detailed documentation to help node operators
understand and configure the gossip rate limiting system effectively.
The new guide addresses a critical knowledge gap that has led to
misconfigured nodes experiencing synchronization failures.
The documentation covers the token bucket algorithm used for rate
limiting, providing clear formulas and examples for calculating
appropriate values based on node size and network position. We include
specific recommendations ranging from 50 KB/s for small nodes to
1 MB/s for large routing nodes, with detailed calculations showing
how these values are derived.
The guide explains the relationship between rate limiting and other
configuration options like num-restricted-slots and the new
filter-concurrency setting. We provide troubleshooting steps for
common issues like slow initial sync and peer disconnections, along
with debug commands and log patterns to identify problems.
Configuration examples are provided for conservative, balanced, and
performance-oriented setups, giving operators concrete starting points
they can adapt to their specific needs. The documentation emphasizes
the importance of not setting rate limits too low, warning that values
below 50 KB/s can cause synchronization to fail entirely.
In this commit, we complete the integration of the configurable gossip
filter concurrency by wiring the new FilterConcurrency configuration
through all layers of the application.
The changes connect the gossip.filter-concurrency configuration option
from the command-line interface through the server initialization code
to the gossiper and sync manager. This ensures that operators can
actually use the new configuration option to tune their node's
concurrent gossip filter processing capacity based on their specific
requirements and available resources.
In this commit, we complete the integration of the asynchronous
timestamp range queue by modifying ProcessRemoteAnnouncement to use
the new queuing mechanism instead of calling ApplyGossipFilter
synchronously.
This change ensures that when a peer sends a GossipTimestampRange
message, it is queued for asynchronous processing rather than
blocking the gossiper's main message processing loop. The modification
prevents the peer's readHandler from blocking on potentially slow
gossip filter operations, maintaining connection stability during
periods of high synchronization activity.
If the queue is full when attempting to enqueue a message, we log
a warning but return success to prevent peer disconnection. This
design choice prioritizes connection stability over guaranteed
delivery of every gossip filter request, which is acceptable since
peers can always resend timestamp range messages if needed.
In this commit, we introduce an asynchronous processing queue for
GossipTimestampRange messages in the GossipSyncer. This change addresses
a critical issue where the gossiper could block indefinitely when
processing timestamp range messages during periods of high load.
Previously, when a peer sent a GossipTimestampRange message, the
gossiper would synchronously call ApplyGossipFilter, which could block
on semaphore acquisition, database queries, and rate limiting. This
synchronous processing created a bottleneck where the entire peer
message processing pipeline would stall, potentially causing timeouts
and disconnections.
The new design adds a timestampRangeQueue channel with a capacity of 1
message and a dedicated goroutine for processing these messages
asynchronously. This follows the established pattern used for other
message types in the syncer. When the queue is full, we drop messages
and log a warning rather than blocking indefinitely, providing graceful
degradation under extreme load conditions.
In this commit, we make the gossip filter semaphore capacity configurable
through a new FilterConcurrency field. This change allows node operators
to tune the number of concurrent gossip filter applications based on
their node's resources and network position.
The previous hard-coded limit of 5 concurrent filter applications could
become a bottleneck when multiple peers attempt to synchronize
simultaneously. By making this value configurable via the new
gossip.filter-concurrency option, operators can increase this limit
for better performance on well-resourced nodes or maintain conservative
values on resource-constrained systems.
We keep the default value at 5 to maintain backward compatibility and
avoid unexpected resource usage increases for existing deployments. The
sample configuration file is updated to document this new option.
So far, most of the AI code review tools I've seen in the wild have been
pretty lackluster. I figure we should give the new Gemini Code Assist
variant a try: https://github.com/marketplace/gemini-code-assist/.
This PR adds an initial yaml config for it.
Relevant commands (as comments on the OP PR):
* `/gemini review` to perform a code review
* `/gemini summary` provides a code summary
* comment like normal tagging `@gemini-code-assist`
* `/gemini help`
This commit introduces the ResultOpt type, which represents an operation
that can either succeed with an optional final value or fail with an
error.
The .gitignore file is also updated to exclude specific files related to
the `aider` tool.
In this commit, we fix a flake in the
`TestRbfCloseClosingNegotiationLocal/send_offer_rbf_wrong_local_script`
test.
This flake can happen if the test shuts down _before_ the state machine
is actually able to process the sent event. In this case, the
expectations are triggered, and we find that the error isn't sent.
To resolve this, we create a new wrapper function that'll use a sync
channel send to assert that the error has been sent before we exit the
test.
In this commit, we add an option to allow a conf req caller to receive
the full block. This is useful if the caller wants to be able to create
an SPV proof.
This commit introduces a new generic type assertion function
`assertState` to the state machine tests. This function asserts that the
state machine is currently in the expected state type and returns the
state cast to that type. This allows us to directly access the fields of
the state without having to perform a type assertion manually.
In this commit, we add a new ConfMapper which is useful for state
machines that want to project some of the conf attributes into a new
event to be sent post conf.
In this commit, we start to signal the prod bit for the rbf coop close
feature. We keep our signaling of the staging bit in place to ensure
the protocol continues to work between those nodes in the wild that are
still signaling the bit.
Fixes https://github.com/lightningnetwork/lnd/issues/9852
Olaoluwa Osuntokun