Commit Graph
239 Commits
Author SHA1 Message Date
Elle Mouton 5c09652e1a graph/db: add dev migration test helper
This commit was inspired by the invoices package
TestMigrationWithChannelDB test helper.

This test is purely for running locally to test the graph SQL migration.
It allows a dev to run the migration against a local `channel.sqlite` or
even `channel.db` file.
2025-07-07 13:35:30 +02:00
Elle Mouton e3572e77c5 graph/db: add SQL migration function and test framework
In this commit, the basic framework for the graph SQL migration is
added. This sets us up for the commits to follow which will add
migration logic for each table in the graph commit by commit.
2025-07-07 13:34:19 +02:00
Elle Mouton 5d7abcdf03 graph/db: refactor SQL DB creation files
Factor out the transaction executor construction so that we can have
access to the raw BatchedSQLQueries type from within tests.
2025-07-07 08:36:12 +02:00
Elle Mouton c9a775e7fb graph/db: fix build flag directive
Fix a typo in the build directive.
2025-07-07 08:36:12 +02:00
Elle Mouton 2f2845dfc0 refactor+multi: use *lnwire.FeatureVector for ChannelEdgeInfo features
In this commit, we move the serialisation details of a channel's
features to the DB layer and change the `models` field to instead use a
more useful `*lnwire.FeatureVector` type.

This makes the features easier to work with and moves the serialisation
to where it is actually used.
2025-07-01 17:02:07 +02:00
Elle Mouton 92849388b8 graph/db: return channel DB info from insertChannel
In preparation for the kvdb->migration code, this commit updates
`insertChannel` to return the ID of the newly inserted channel along
with the IDs of the nodes that the channel links to.
2025-07-01 13:37:09 +02:00
Elle Mouton 4bde8e2d04 graph/db: refactor and clean-up
Refactor channelIDToBytes to return a slice instead of an 8 byte array
so that we dont need to use `[:]` everywhere.

Also make sure we are using this helper everywhere.
2025-07-01 13:37:09 +02:00
Elle Mouton 2310756307 graph/db: refactor source node fetching logic
Small refactor to introduce a re-usable getSourceNode function that we
can then make use of in the upcoming kvdb->SQL migration logic.
2025-07-01 13:37:07 +02:00
Elle Mouton 016fd9bb4d graph/db: create a re-usable forEachChannel
Create a forEachChannel function that does not rely on a KVStore being
instantiated. We will use this in our upcoming kvdb->SQL migration.
2025-07-01 13:31:48 +02:00
Elle Mouton 9452c0b950 graph/db: detach getChannelMap from KVStore 2025-07-01 13:31:48 +02:00
Elle Mouton 8660f76b20 graph/db: detach forEachNode from KVStore
So that we can use it in a context that only has access to a
kvdb.Backend. We'll use this in our kvdb->SQL migration.
2025-07-01 13:31:48 +02:00
Oliver Gugger 500808fadd Merge pull request #10010 from ellemouton/sqlGraphUpdates
graph/db: various misc updates
2025-07-01 12:40:50 +02:00
Elle Mouton c5b2e4e50a graph/db: update outdated comments 2025-07-01 10:12:54 +02:00
Elle Mouton e4137a3f14 graph/db: fix ChanUpdate message/channel flag bug
Here we start using the newly added message_flags and channel_flags
columns of the channel_policies table. The test added previoulsy to
demonstrate the bug is now updated to show that the bug has been fixed.
2025-07-01 10:12:54 +02:00
Elle Mouton fa32f97c05 graph/db: unit test to demonstrate SQL chan policies bug
This commit adds a new graph store unit test that demonstrates that
there is currently a bug in the SQL version of the graph store where
unknown message or channel flags in a channel_update are not persisted
correctly. This will be fixed in an upcoming commit.
2025-07-01 10:12:53 +02:00
Elle Mouton 64fc92abe7 graph/db: update nextUpdateTime in tests
Let it return time.Time instead. Also, make use of the helper in a few
more places.
2025-07-01 10:04:03 +02:00
Elle c7aa17971b Merge pull request #10006 from ellemouton/fixSQLFetchChannelEdgesByID
graph/db: let FetchChannelEdgesByID behave as promised
2025-07-01 10:01:36 +02:00
Elle Mouton f1b7ccc6b2 sqldb+graph/db: prune graph nodes in a single query
Update the pruneGraphNodes routine to prune the graph nodes in a single
query instead of two.
2025-06-30 18:41:28 +02:00
Elle Mouton 714e528a3a graph/db: fix address fetching error 2025-06-30 18:40:25 +02:00
Elle Mouton 272a2db16c graph/db: demonstrate SQL node addresses fetch bug
This commit demonstrates that currently the Addresses field of a
LightningNode is not populated correctly if the node has no addresses if
the SQL graph store is being used. This will be fixed in an upcoming
commit.
2025-06-30 18:40:21 +02:00
Elle Mouton 73eca1bf3d graph/db: update outdated comment 2025-06-30 18:40:13 +02:00
Elle Mouton 64110cf459 graph/db: let FetchChannelEdgesByID behave as promised
The comment of FetchChannelEdgesByID says that if the ErrZombieEdge
error is returned, then the ChannelEdgeInfo return parameter will also
still be populated and returned (ie, wont be nil). This commit updates
the SQLStore implementation of FetchChannelEdgesByID to do this. This is
needed to prevent nil dereference panics at any call-sites that depend
on the method working as it describes.
2025-06-30 11:04:35 +02:00
Elle Mouton 8cf567b948 multi: use the "errors" package everywhere
Replace all usages of the "github.com/go-errors/errors" and
"github.com/pkg/errors" packages with the standard lib's "errors"
package. This ensures that error wrapping and `errors.Is` checks will
work as expected.
2025-06-30 09:46:55 +02:00
Elle Mouton cc0be0bca2 graph/db: let IsZombieEdge return an error 2025-06-26 10:19:51 +02:00
Elle Mouton b4b1c4bd8d graph/db: remove KVStore fall-back
Now that SQLStore fully implements V1Store, we no longer need the
KVStore fall-back.
2025-06-26 10:12:46 +02:00
Elle Mouton 3ccaacd16f multi: rename MakeTestGraphNew to MakeTestGraph 2025-06-26 10:12:30 +02:00
Elle Mouton c06036d2b4 multi: use MakeTestGraphNew for all remaining unit tests
And delete the old MakeTestGraph function.

Here we need to update some of our unit tests to ensure that any updates
to existing nodes or channel policies have newer timestamps. This is so
that we don't violate our SQL DB constraints that prevent updates to a
node or channel policy record if the new update is not newer (has a
newer LastUpdate time) than the currently presisted record).
2025-06-26 10:12:27 +02:00
Elle Mouton daf6a08747 graph/db: fix log precision 2025-06-25 13:26:53 +02:00
Elle Mouton 81ce05fced graph/db: start lastID for pagination at -1
Just to be very sure that we don't miss a valid ID.
2025-06-25 13:26:53 +02:00
Elle Mouton 23c8f7099c graph/db: impl ForEachChannelCacheable 2025-06-25 13:26:52 +02:00
Elle Mouton 34cb6a9f27 graph/db: impl GraphSession 2025-06-25 13:26:52 +02:00
Elle Mouton 933ab3c6b7 graph/db+sqldb: impl PutClosedScid and IsClosedScid 2025-06-25 13:26:52 +02:00
Elle Mouton d46552f5ad graph/db+sqldb: implement AddEdgeProof
And run `TestAddEdgeProof` against the SQL backends.
2025-06-25 13:26:51 +02:00
Elle Mouton e875183c4f sqldb+graph/db: impl DisconnectBlockAtHeight
Which lets us run `TestDisconnectBlockAtHeight` and
`TestStressTestChannelGraphAPI` against our SQL backends.
2025-06-25 11:22:03 +02:00
Elle Mouton 9dd0361ed0 graph/db+sqldb: impl PruneGraph, PruneTip, ChannelView
Which lets us run `TestGraphPruning` and `TestBatchedAddChannelEdge`
against our SQL backends.
2025-06-25 11:22:00 +02:00
Elle Mouton 2da701cc4f graph/db+sqldb: impl PruneGraphNodes
Which lets us run `TestChannelEdgePruningUpdateIndexDeletion` against
our SQL backends.
2025-06-25 10:48:35 +02:00
Elle Mouton eec89362a5 graph/db+sqldb: impl DisabledChannelIDs
Which lets us run `TestDisabledChannelIDs` against our SQL DB backends.
2025-06-24 21:04:36 +02:00
Elle Mouton f1da3812de graph/db+sqldb: impl IsPublicNode
Which lets us run `TestNodeIsPublic` against our SQL DB backends.
Note that we need to tweak the tests a little bit so that
`AddLightningNode` for the same node is always called with a newer
LastUpdate time else it will fail the SQL constraint that only allows
the upsert if the update is newer than the persisted one.
2025-06-24 21:04:36 +02:00
Elle Mouton acae165f0a graph/db: implement FetchChanInfos
Which lets us run `TestFilterChannelRange` against our SQL backends.
2025-06-24 21:04:36 +02:00
Elle Mouton ba701de88a graph/db: impl FilterKnownChanIDs
Which lets us then run `TestFilterKnownChanIDsZombieRevival` and
`TestFilterKnownChanIDs` against our SQL backends.
2025-06-24 21:04:36 +02:00
Elle Mouton 13bf6a549f graph/db+sqldb: implement HasChannelEdge and ChannelID
And run `TestEdgeInfoUpdates` against our SQL backends.
2025-06-24 21:04:35 +02:00
Elle Mouton 4fad4a7023 graph/db+sqldb: implement FetchChannelEdgesByOutpoint/SCID
And run `TestEdgeInsertionDeletion` against our SQL backends.
2025-06-24 21:04:35 +02:00
Elle Mouton 2a6e6683eb graph/db+sqldb: implement DeleteChannelEdges
This lets us run TestGraphZombieIndex against the SQL backends.
2025-06-24 19:02:45 +02:00
Elle Mouton 00b6e0204c graph/db+sqldb: implement various zombie index methods
Here we implement the SQLStore methods:
- MarkEdgeZombie
- MarkEdgeLive
- IsZombieEdge
- NumZombies

These will be tested in the next commit as one more method
implementation is required.
2025-06-24 19:02:14 +02:00
Elle Mouton 137fc09230 graph/db: add channelIDToBytes helper 2025-06-24 18:58:52 +02:00
Elle Mouton cee872ad0e graph/db: make makeZombiePubkeys reusable
So that we can use it in our SQLStore implementation too without needing
to fetch the entire ChannelEdgePolicy.
2025-06-24 18:35:18 +02:00
Elle Mouton 3687171cd5 sqldb+graph/db: implement FilterChannelRange
This lets us run `TestFilterChannelRange` against the SQL backends.
2025-06-24 12:17:52 +02:00
Elle Mouton ff84fa1cb2 graph/db+sqldb: impl ForEachNodeCached and ForEachChannel
Which let's us run `TestGraphTraversal` against our SQL backends.
2025-06-24 12:17:52 +02:00
Elle Mouton 5bb5b6a260 graph/db: fix log line in KVStore
Ensure that the log line in `ChanUpdatesInHorizon` cannot cause a
divide-by-zero panic.
2025-06-24 12:17:52 +02:00
Elle Mouton 39e521e12b graph/db+sqldb: implement ChanUpdatesInHorizon
Add `ChanUpdatesInHorizon` method to the SQLStore. This lets us run
`TestChanUpdatesInHorizon` against our SQL backends.
2025-06-24 12:17:52 +02:00