Update some of the node related graph CRUD methods to take a version
rather than hardcoding them in the SQLStore layer. Move the version up
one layer instead. This will make it easier to make it configurable
later on.
HasNode currently is very v1 specific since it returns a time.Time
timestamp which is specific to V1 node announcements. However, it is
mostly only used for the "exists" return value. So here we split it up
into HasNode which just checks existence and HasV1Node which retains the
same behavaiour as before.
Update the SQLStore node writer and readers to handle V2 ndoes.
Currently no logic will actually add such nodes. The following commits
will update what is needed so that CRUD for v2 nodes can be tested.
Instead of embedding Store in ChannelGraph so that any methods of the
Store interface not implemented by the ChannelGraph are redirected to
the underlying Store, we update things in this commit to instead
make the "redirection" explicit. This is in preparation for changes we
will make soon where some underlying store methods will take an explicit
"version" parameter but then we will keep the ChannelGraph methods as is
so that existing call-sites dont all need to be updated. We will then
add "Versioned" ChannelGraph wrapper which decides the version use.
Initially, most call-sites will just create a wrapped V1 ChannelGraph so
that the logic remains as it is today.
We add a new NewV2Node constructor which takes a new NodeV2Fields as a
parameter. This NodeV2Fields struct defines the fields that can be set
in a models.Node if the version is V2.
The underling store will store gossip messages across gossip versions
and we will instead expose version parameters on many of the methods. So
this interface really just abstracts the underlying store/schema type.
This commit fixes a backwards compatibility issue that prevented nodes
from upgrading from v0.19.x to v0.20.x.
In v0.19.x, channel edge features were serialized as raw feature bytes
without a length prefix. In v0.20.x (commit 2f2845dfc), the serialization
changed to use Features.Encode() which adds a 2-byte big-endian length
prefix before the feature bits. The deserialization code was updated to
use Features.Decode() which expects this length prefix.
When v0.20.x reads a database created by v0.19.x, Decode() tries to read
a length prefix that doesn't exist, causing an EOF error:
unable to decode features: EOF
The fix adds a deserializeChanEdgeFeatures() helper that detects which
format is being read and decodes accordingly:
- New format (v0.20+): First 2 bytes encode the length of the remaining
bytes. Detected when uint16(bytes[0:2]) == len(bytes)-2.
- Legacy format (pre-v0.20): Raw feature bits without length prefix.
Uses DecodeBase256 with the known length.
The format detection is safe because in the legacy format, the first byte
always has at least one bit set (the serialization uses minimum bytes),
so the first two bytes can never encode a value equal to len-2.
Fixes#10528.
We should avoid taking the lock of a mutex inside transaction.
Currently we also take this lock in other places and there is a
chance that in case the application lock aquires the lock but
all transactions are already blocked waiting for the mutex to
unlock, we end up in a deadlock.
The DisconnectBlockAtHeight method was modifying the rejectCache and
chanCache without holding the cacheMu lock. This caused races with
other operations that properly held the lock, such as AddChannelEdge
which modifies the caches in its OnCommit callback while the batch
scheduler holds cacheMu.
Fix by acquiring cacheMu before removing channels from the caches.
Both NodeKey1 and NodeKey2 methods had the same race condition as the
Node.PubKey method, where concurrent calls could race to write to the
cached fields.
Remove the caching for the same reasons: parsing overhead is minimal
and doesn't justify the complexity and race risk.
The PubKey method had a race condition where concurrent calls could
all pass the nil check and race to write to the cached pubKey field.
This is a classic check-then-act race.
Remove the caching entirely to fix the race. The overhead of parsing
a public key is minimal and doesn't justify the added complexity and
race risk of caching.
This commit fixes a race condition where multiple goroutines call
SetSourceNode concurrently during startup, causing sql.ErrNoRows
errors. The race occurs when multiple code paths (setSelfNode,
createNewHiddenService, RPC updates) read the same old timestamp,
independently increment it to the same new value (T+1), and race to
write.
The fix uses the new UpsertSourceNode SQL query (without strict
timestamp constraint) instead of UpsertNode. This allows
last-write-wins semantics for our own node, ensuring all parameter
changes persist even when timestamps collide.
Refactored sql_store.go for reusability:
- upsertNodeAncillaryData: common logic for features/addresses/extras
- populateNodeParams: common parameter building with callback pattern
- buildNodeUpsertParams: builds params for strict UpsertNode
- buildSourceNodeUpsertParams: builds params for lenient UpsertSourceNode
- upsertSourceNode: new function using lenient query
Updated TestSetSourceNodeSameTimestamp to verify that concurrent
updates with the same timestamp now succeed and parameter changes
persist.
Fixes the itest error:
"unable to upsert source node: upserting node(...): sql: no rows in
result set"
This commit adds TestSetSourceNodeSameTimestamp to demonstrate the
current behavior when SetSourceNode is called with the same last update
timestamp. The test reveals a difference between the SQL and bbolt
implementations:
- SQL store returns sql.ErrNoRows when attempting to update with the
same timestamp, as the upsert query's UPDATE clause requires the new
timestamp to be strictly greater than the existing one
- bbolt store silently ignores stale updates and returns no error
This behavior is important to document because our own node
announcements may change quickly with the same timestamp, unlike
announcements from other nodes where same timestamp typically means
identical parameters.
Fix a bug where channels with both policies disabled were not added to
the graph cache during startup. When a policy update later re-enabled
one of the directions, the update would succeed in the database but fail
to update the graph cache (since the channel structure was never added),
preventing the channel from being used for routing.
Copy over all the code that the graph SQL migration needs to a
separate folder. This will let us advance the main graph SQL CRUD code
without worrying about changing the sql migration code. It will also let
us change the SQL queries without changing the migration. In this
commit, only the migration logic is "frozen" but in an upcoming commit,
the sqlc queries & models will be frozen too.
This tests was a temporary helper to let devs test the graph SQL
migration before it was plugged in to LND. But that migration has now
shipped and so we can remove this.
It can happen that we are handling 2 of the same node announcements in
the same batch transaction. In that case, our `UpsertNode` conflict
assertion may fail. We need to handle this gracefully.
In preparation for adding a NodeAnnouncement2 struct along with a
NodeAnnouncement interface, this commit renames the existing
NodeAnnouncment struct to NodeAnnouncement1.
In this commit, we update the SQL store implementation to support the
new iterator-based API for ChanUpdatesInHorizon. This includes adding
SQL query pagination support and helper functions for efficient batch
processing.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetChannelsByPolicyLastUpdateRange is updated to support:
- Cursor-based pagination using (max_update_time, id) compound cursor
- Configurable batch sizes via MaxResults parameter
- Efficient batch caching with updateChanCacheBatch helper
In this commit, we update the SQL store implementation to support the
new iterator-based API for NodeUpdatesInHorizon. This includes adding a
new SQL query that supports efficient pagination through result sets.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetNodesByLastUpdateRange is updated to support:
* Cursor-based pagination using (last_update, pub_key) compound cursor
* Optional filtering for public nodes only
* Configurable batch sizes via MaxResults parameter
In this commit, we refactor the ChanUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
In this commit, we refactor the NodeUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
The new implementation uses Go 1.23's iter.Seq type to provide a
standard iterator interface. The method now supports configurable batch
sizes through functional options, allowing fine-tuned control over
memory usage and performance characteristics.
Rather than reading all the entries from disk into memory (before this
commit, we did consult the cache for most entries, skipping the disk
hits), we now expose a chunked iterator instead.
We also make the process of filtering out public nodes first class. This
saves many newly created db transactions later.
In this commit, we introduce a new options pattern for configuring
iterator behavior in the graph database. This includes configuration
for batch sizes when iterating over channel and node updates, as well
as an option to filter for public nodes only.
The new functional options pattern allows callers to customize iterator
behavior without breaking existing APIs. Default batch sizes are set to
1000 entries for both channel and node updates, which provides a good
balance between memory usage and performance.
The SQL* helpers are meant to always set the `Valid` field of the
sql.Null* type to true. Otherwise they cannot be used to set a valid,
empty field. However, we dont want to break the behaviour of the
existing SQLStr helper and so this commit adds a new helper with the
desired functionality.
Elle Mouton