The GetInvoice query used an OR IS NULL pattern for each filter
parameter:
WHERE (i.hash = $1 OR $1 IS NULL)
AND (i.payment_addr = $2 OR $2 IS NULL)
SQLite's query planner decides on an execution plan at prepare time,
before seeing any parameter values. Because either condition can be
trivially true when its parameter is NULL, the planner conservatively
falls back to a full table scan rather than using the unique indexes on
hash and payment_addr. This caused every invoice lookup and update on
the hot path (HTLC settlement) to scan the entire invoices table.
Replace the single catch-all query with three dedicated queries, each
using a direct equality on a uniquely constrained column:
- GetInvoiceByHash: WHERE hash = $1
- GetInvoiceByAddr: WHERE payment_addr = $1 (new, covers AMP path)
- GetInvoiceBySetID: existing, unchanged
Update getInvoiceByRef to route to the appropriate query based on which
fields are present in the InvoiceRef. When both hash and payment address
are provided, we look up by hash and then verify the returned invoice's
payment address matches.
GetInvoice is now unused and removed from the SQLInvoiceQueries
interface.
Remove four indexes from the invoices table that either duplicate existing
UNIQUE constraint indexes or are never used in WHERE clauses:
- invoices_hash_idx: redundant, UNIQUE constraint on hash already creates
an implicit index
- invoices_payment_addr_idx: redundant, UNIQUE constraint on payment_addr
already creates an implicit index
- invoices_preimage_idx: unused, preimage is NULL on all new invoices and
is never used as a query filter
- invoices_settled_at_idx: unused, settled_at is NULL on all pending
invoices and is never used as a query filter (settle_index is used
instead)
Dropping these reduces B-tree working set size, which improves page cache
utilization as the invoice table grows.
Older LND versions could create multiple payments for the same hash.
We need to preserve those historical records during KV→SQL migration,
but they don’t fit the normal payment schema because we enforce a
unique payment hash constraint. Introduce a lean payment_duplicates
table to store only the essential fields (identifier, amount,
timestamps, settle/fail data).
This keeps the primary payment records stable and makes the migration
deterministic even when duplicate records lack attempt info. The table
is intentionally minimal and can be dropped after migration if no
duplicate payments exist.
For now there is no logic in place which allows the noderunner to
fetch duplicate payments after the migration.
- Remove duplicate compile-time interface assertion for SQLStore.
- Fix the down migration to drop payment_intents before payments to
respect the foreign key dependency order. This was not a bug in the
first place bc we have the CASCADE when deleting payments.
The 000009 schema version slot is now taken by 000009_graph_v2_columns
which was merged ahead of the payments schema. Bump the payments schema
file number to 000010 to avoid the collision.
Moreover update the migration_dev.go file to reflect this change and
update the order of migration.
Previously a one(intent)-to-many(payment) relationship it is now
changed to a one-to-one relationship because a payment request
only can have 1 payment related to it. Looking into the future
with BOLT12 offers, the fetched invoice from the offer could be
stored here as well and the relationship would still hold.
All call sites have been migrated to the targeted replacement queries
in the previous commit. Remove FilterInvoices and its associated
params struct from the SQL source and regenerate.
The existing FilterInvoices query uses optional parameters via the
pattern `(col >= param OR param IS NULL)` for every filter. SQLite
cannot use indexes with this pattern because the OR prevents the query
planner from determining at plan time which rows satisfy the condition,
resulting in a full table scan regardless of the available indexes
(invoices_state_idx, invoices_settle_index_idx, and the primary-key
clustered index on id).
Additionally, the conditional ORDER BY:
ORDER BY CASE WHEN reverse = FALSE ... THEN id ELSE NULL END ASC,
CASE WHEN reverse = TRUE ... THEN id ELSE NULL END DESC
prevents the planner from using the index ordering and forces an
explicit sort.
Add five focused replacements, each with a plain sargable predicate and
a direct ORDER BY so the planner can always choose an index scan:
- FetchPendingInvoices: WHERE state IN (0, 3)
- FilterInvoicesBySettleIndex: WHERE settle_index >= $1
- FilterInvoicesByAddIndex: WHERE id >= $1
- FilterInvoicesForward: WHERE id >= $1 ... ORDER BY id ASC
- FilterInvoicesReverse: WHERE id <= $1 ... ORDER BY id DESC
FilterInvoicesForward and FilterInvoicesReverse accept non-nullable
timestamp parameters (created_after, created_before) so the planner
always sees plain range predicates on created_at. Callers supply
Go-side defaults when no date filter is needed, following the same
convention already used by the payments query.
FilterInvoices is kept in this commit so all existing call sites
continue to compile. It will be removed once all callers have been
migrated.
The ListChannelsWithPoliciesForCachePaginated query was missing the
policy version column, causing extractChannelPolicies to hardcode
lnwire.GossipVersion1 for that row type. Add cp1.version and
cp2.version to the query and use the fetched values instead.
Add a gossip version parameter to DisabledChannelIDs in the Store
interface and both implementations. Add a new version-filtered SQL
query and update the builder caller.
Update the UpsertChannelPolicy query to apply different staleness
checks based on gossip version. For v1 policies, continue checking
last_update timestamps. For v2 policies, check block_height instead,
using >= comparison to handle policies from the same block.
The version-specific WHERE clause ensures that policy updates are only
applied when they contain newer information according to the versioning
scheme appropriate for that gossip version.
Extend channel policy queries and structs to support v2-specific fields:
- Add BlockHeight field to track the block height for v2 policy updates.
- Add DisableFlags field for v2 channel disable messages.
Both fields are nullable (sql.NullInt64/Int16) to maintain backwards
compatibility with v1 channels. The fields are initialized as null in
updateChanEdgePolicy and threaded through all policy-related queries
(GetChannelBySCIDWithPolicies, ListChannelsByNodeID, UpsertEdgePolicy,
etc.) and the extractChannelPolicies helper.
This commit includes both the hand-written SQL query updates and the
corresponding sqlc-generated Go code.
This commit improves handling of missing channel signatures in the
database:
- Return nil from auth proof accessors instead of empty slices so that
missing signatures are stored as NULL in SQL.
- Update public channel checks to require signature length > 0, which
properly handles existing empty bytea values in the database.
- Add regression test covering empty v1 and v2 channel signatures to
prevent future issues.
Add SQL query to determine if a node has public v2 channels. Unlike
v1 which requires all four individual signatures, v2 channels are
considered public when the single aggregated signature is present.
Add SQL query to update the signature column for v2 channel auth proofs.
Unlike v1 which requires four separate signatures, v2 channels use a
single aggregated signature.
Add three new optional fields to the CreateChannel SQL query to
support v2 channel announcements:
- signature: single schnorr signature (replaces four ECDSA sigs)
- funding_pk_script: the funding output script
- merkle_root_hash: for taproot channels
These fields are NULL for v1 channels and populated for v2 channels.
Add a new migration that updates the graph tables (nodes, channels and
policies) in preparation for the new columns required for V2
announcements. This migration has to be added to the set of "live"
migrations instead of "dev only" since it edits the columns of existing
tables and so changes the existing sql models. We are going to prep the
SQLStore code to handle the V2 types in the coming commits, so we need
this migration to be in place.
In this commit we also remove the TestSchemaMigrationIdempotency test
since this test fails with the new "ALTER TABLE" migrations which dont
have "IF NOT EXISTS" options like tables and indexes do. Migrations
should be idempotent anyways due to the migration tracker file and/or
the sqlc migration tracker.
In this commit we updated the IsPublicV1Node query to use UNION
instead of OR, since sqlite struggles to efficiently use
multiple indexes in a single query involving OR conditions across
different columns.
We use UNION ALL since the query doesn't care about duplicates.
This query is less strict in terms of the latest update timestamp field.
We want to be less strict with our own node data since we always want
our own updates recorded.
In this commit, we update the SQL store implementation to support the
new iterator-based API for ChanUpdatesInHorizon. This includes adding
SQL query pagination support and helper functions for efficient batch
processing.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetChannelsByPolicyLastUpdateRange is updated to support:
- Cursor-based pagination using (max_update_time, id) compound cursor
- Configurable batch sizes via MaxResults parameter
- Efficient batch caching with updateChanCacheBatch helper
In this commit, we update the SQL store implementation to support the
new iterator-based API for NodeUpdatesInHorizon. This includes adding a
new SQL query that supports efficient pagination through result sets.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetNodesByLastUpdateRange is updated to support:
* Cursor-based pagination using (last_update, pub_key) compound cursor
* Optional filtering for public nodes only
* Configurable batch sizes via MaxResults parameter
This commit simplifies insertChanEdgePolicyMig. Much of the logic can be
removed given that this method is only used in the context of the graph
SQL migration.
This should improve the performance of the migration quite a lot since
it removes the extra GetChannelAndNodesBySCID call.
Finally, we make the channel-policy part of the SQL migration idempotent
by adding a migration-only policy insert query which will not error out
if the policy already exists and does not have a timestamp that is newer
than the existing records timestamp. To keep the commit simple, a
insertChanEdgePolicyMig function is added which is basically identical
to the updateChanEdgePolicy function except for the fact that it uses
the newly added query. In the next commit, it will be simplified even
more.
In this commit, we make the channel part of the graph SQL migration
idempotent (retry-safe!). We do this by adding a migration-only channel
insert query that will not error out if a the query is called and a
chanenl with the given scid&version already exists. We also ensure that
errors are not thrown if existing channel features & extra types are
re-added.
There is no need to use the "collect-then-update" pattern for node
insertion during the SQL migration since if we do have any previously
persisted data for the node and happen to re-run the insertion for that
node, the data will be exactly the same. So we can make use of "On
conflict, no nothing" here too.
In this commit, the graph SQL migration is updated so that the node
migration step is retry-safe. This is done by using migration specific
logic & queries that do not use the same node-update-constraint as the
normal node upsert logic. For normal "run-time" logic, we always expect
a node update to have a newer timestamp than any previously stored one.
But for the migration, we will only ever be dealing with a single
announcement for a given node & to make things retry-safe, we dont want
the query to error if we re-insert the exact same node.
We put this new schema update into the main line and change the
versions of the schema updates which are currently only available
in dev builds. The schemas need to be chronological therefore we
also need to rename the file numbers.
Finally, we update the migrateZombieIndex function to use batch
validation just like was done in the previous commits. Here, we
additionally make sure to validate the entire zombie index entry and not
just the SCID.
ziggie