For legacy payments, the HTLC Hash field may be nil in the bbolt
backend. Previously, the migration would fail with "HTLC attempt X
missing payment hash" when encountering such payments.
This commit fixes the migration by falling back to the parent payment
hash when the HTLC-specific hash is nil. This is consistent with how
the router handles legacy payments (see patchLegacyPaymentHash in
payment_lifecycle.go).
The validation logic is also updated to apply the same fallback when
comparing bbolt data with migrated SQL data, ensuring the comparison
succeeds.
Hook the payments KV→SQL migration into the SQL migration config.
The migration is still only available when building with the build tag
"test_native_sql".
Moreover a tombstone protection similar to the invoice migration is added
to prevent re-runningi with the KV backend once migration completes.
Add a developer-facing migration_external_test that allows
running the KV→SQL payments migration against a real channel.db
backend to debug migration failures on actual data. The accompanying
testdata README documents how to supply a database file and configure
the test, so users can validate their data and confirm the migration
completes successfully.
The test is skipped by default and meant for manual diagnostics.
Add test helpers plus sql_migration_test coverage for KV→SQL migration.
Basic migration, sequence ordering, data integrity, and feature-specific cases
(MPP/AMP, custom records, blinded routes, metadata, failure messages). Also
cover duplicate payment migration to payment_duplicates, including missing
attempt info to ensure terminal failure is recorded.
This gives broad regression coverage for the migration path and its edge-cases.
Implement the KV→SQL payment migration and add an in-migration
validation pass that deep-compares KV and SQL payment data in batches.
Duplicate payments are migrated into the payment_duplicates table,
and duplicates without attempt info or explicit resolution are marked
failed to ensure terminal state. Validation checks those rows as well.
Copy the core payments/db code into payments/db/migration1 and
add the required sqlc-generated types/queries from sqldb/sqlc.
This effectively freezes the migration code so it stays robust
against future query or schema changes in the main payments package.
Replace the delegation to channeldb.ReadElement/WriteElement with
self-contained, frozen implementations that only handle the exact types
required by this migration package. This removes the dependency on the
live channeldb codec so that future changes to channeldb serialization
cannot silently corrupt or break the migration.
UnknownElementType is also defined locally for the same reason.
Older LND versions could create multiple payments for the same hash.
We need to preserve those historical records during KV→SQL migration,
but they don’t fit the normal payment schema because we enforce a
unique payment hash constraint. Introduce a lean payment_duplicates
table to store only the essential fields (identifier, amount,
timestamps, settle/fail data).
This keeps the primary payment records stable and makes the migration
deterministic even when duplicate records lack attempt info. The table
is intentionally minimal and can be dropped after migration if no
duplicate payments exist.
For now there is no logic in place which allows the noderunner to
fetch duplicate payments after the migration.
- Remove duplicate compile-time interface assertion for SQLStore.
- Fix the down migration to drop payment_intents before payments to
respect the foreign key dependency order. This was not a bug in the
first place bc we have the CASCADE when deleting payments.
The SQL implementation collects payments into a map before converting
to a slice, resulting in non-deterministic iteration order due to Go's
intentional map randomisation. Sort the result by SequenceNum to produce
a deterministic, insertion-ordered output.
Note that the current sole caller (resumePayments in router.go) processes
each payment independently, so this ordering does not affect any existing
behaviour.
The SQL backend introduced in this PR was ignoring the SettleTime and
FailTime fields provided in HTLCSettleInfo and HTLCFailInfo, instead
always recording time.Now() as the resolution timestamp. The KV backend
correctly serializes and deserializes these fields.
The timestamps are set by the caller using a mockable clock
(p.router.cfg.Clock.Now() in payment_lifecycle.go), so ignoring them
means the stored timestamp reflects when the DB write happened rather
than when the event occurred, breaking deterministic testing.
This commit also extends the test assertions in assertPaymentInfo to
verify that SettleTime and FailTime are correctly stored and retrieved
by the SQL backend, and updates the relevant call sites to pass explicit
timestamps so regressions are caught.
The 000009 schema version slot is now taken by 000009_graph_v2_columns
which was merged ahead of the payments schema. Bump the payments schema
file number to 000010 to avoid the collision.
Moreover update the migration_dev.go file to reflect this change and
update the order of migration.
We add a couple of additional tests to increase the unit test
coverage of the sql store but also the kv store. We only create
db agnostic unit tests so both backends are tested effectively.
Previously we had db and application logic mixed on the db level.
We now move the config option KeepFailedPaymentAttempts to the
ChannelRouter level and move it out of the db level.
We make the TestDeleteNonInFlight and separate all the logic out
for the duplicate payment test case. The deletion of duplicate
payments is now tested in isolation only for the kv backend.
The design of the sql and kv db are a bit different. A harness
interface is introduced which allows us to unit most of the test
and keep the backend specific tests at a minimum.
In commit adds the harness which will be used to run db agnostic
tests against the kv and sql backend. We have adopted all the
unit tests so far so that with this commit all the payment tests
not specifically put into the kv_store_test.go should all pass
for all backends.
We are now not supporting the LegacyPayload for the onion packet
anymore. All payments and their onion payload need to be tlv
encoded. The sql backend assumes tlv so we have to always set the
in memory presentation of a hop where the legacy parameter is still
available but deprecated to false, otherwise the hops will not be
equal and unit tests for the sql backend will fail when switched
on in the next commits.
Since now the sql backend is more strict in using the same
session key we refactor the helper so that we can easily change
the session key for every new attempt.
Now that every method of the interface was implemented we can
remove the embedded reference we put into place for the sql store
implementation so that the interface would succeed. This is now
removed.
We rename this variable to paymentsDetailsData because we will
also need to batch load the core payment and intent data in
future commits and this renaming should make it clear that this
does match payment related data but not the core data which is
in the payment and in the intent table.
We remove the SQLStore from most of the helper functions. This
also makes sure we do not accidentally create a new db tx but use
the provided db SQLQueries parameter.
Previously a one(intent)-to-many(payment) relationship it is now
changed to a one-to-one relationship because a payment request
only can have 1 payment related to it. Looking into the future
with BOLT12 offers, the fetched invoice from the offer could be
stored here as well and the relationship would still hold.
We prepare the code for the sql payment backend. However no
payment db interface method for the sql backend is implemented
yet. This will be done in the following commits. They currently
use the embedded KVStore to satify the build environment.
Add testFetchPendingInvoicesAccepted to explicitly verify the state
filtering behaviour of FetchPendingInvoices across all four contract
states:
ContractOpen (state 0) – must be returned
ContractAccepted (state 3) – must be returned
ContractSettled (state 1) – must be excluded
ContractCanceled (state 2) – must be excluded
This directly exercises the `state IN (0, 3)` predicate introduced in
the FetchPendingInvoices SQL query and addresses the review request for
explicit test coverage of the new targeted query variants. The test
runs against the KV, SQLite, and Postgres backends.
All call sites have been migrated to the targeted replacement queries
in the previous commit. Remove FilterInvoices and its associated
params struct from the SQL source and regenerate.
Replace all four FilterInvoices call sites with the focused queries
introduced in the previous commit:
FetchPendingInvoices → FetchPendingInvoices
InvoicesSettledSince → FilterInvoicesBySettleIndex
InvoicesAddedSince → FilterInvoicesByAddIndex
QueryInvoices → FilterInvoicesForward / FilterInvoicesReverse
The first three are straight 1:1 swaps — the new params structs carry
only the fields that are actually used, and the removed fields (Reverse,
PendingOnly, unused index bounds) were always left at their zero values.
QueryInvoices is restructured more substantially. The forward/reverse
branch now selects between FilterInvoicesForward and
FilterInvoicesReverse, each of which takes a concrete id bound that is
always set:
forward: AddIndexGet = IndexOffset + 1 (≥ 1 when IndexOffset = 0)
reverse: AddIndexLet = IndexOffset - 1 (or MaxInt64 when offset = 0)
Timestamp parameters are changed from nullable (sql.NullTime with
OR-based SQL fallbacks) to always-on Go-side defaults, consistent with
the approach used by the payments query:
createdAfter → time.Unix(0, 0).UTC() (epoch, before any invoice)
createdBefore → time.Date(9999, 12, 31, …) (far future, no upper cap)
This ensures the planner always sees plain range predicates on
created_at and can use the invoices_created_at_idx index.
The SQLInvoiceQueries interface is updated to expose the five new
methods and drop FilterInvoices.
The existing FilterInvoices query uses optional parameters via the
pattern `(col >= param OR param IS NULL)` for every filter. SQLite
cannot use indexes with this pattern because the OR prevents the query
planner from determining at plan time which rows satisfy the condition,
resulting in a full table scan regardless of the available indexes
(invoices_state_idx, invoices_settle_index_idx, and the primary-key
clustered index on id).
Additionally, the conditional ORDER BY:
ORDER BY CASE WHEN reverse = FALSE ... THEN id ELSE NULL END ASC,
CASE WHEN reverse = TRUE ... THEN id ELSE NULL END DESC
prevents the planner from using the index ordering and forces an
explicit sort.
Add five focused replacements, each with a plain sargable predicate and
a direct ORDER BY so the planner can always choose an index scan:
- FetchPendingInvoices: WHERE state IN (0, 3)
- FilterInvoicesBySettleIndex: WHERE settle_index >= $1
- FilterInvoicesByAddIndex: WHERE id >= $1
- FilterInvoicesForward: WHERE id >= $1 ... ORDER BY id ASC
- FilterInvoicesReverse: WHERE id <= $1 ... ORDER BY id DESC
FilterInvoicesForward and FilterInvoicesReverse accept non-nullable
timestamp parameters (created_after, created_before) so the planner
always sees plain range predicates on created_at. Callers supply
Go-side defaults when no date filter is needed, following the same
convention already used by the payments query.
FilterInvoices is kept in this commit so all existing call sites
continue to compile. It will be removed once all callers have been
migrated.
Replace curl with wget for downloading release manifests and
signatures in verify-install.sh. wget handles redirects, retries, and
error reporting more robustly by default, which avoids silent download
failures that caused misleading "Invalid signature!" errors.
Also add error checking to all download calls so failures are reported
immediately with the URL that failed, and log which signature file and
user failed gpg verification.
When gpg --verify fails, include the signature filename, username,
and full GPG output in the error message. Previously only a generic
"Invalid signature!" was printed, making it hard to identify which
signer's signature was invalid.
Add a workflow that triggers when a release is published. It runs
verify-install.sh inside the official Docker image to validate
signatures and binary hashes. If verification fails, the release
is automatically set back to draft.
The severity bot was misclassifying CLI client code as HIGH because
filenames like cmd_walletunlocker.go matched the walletunlocker/*
auth/security keyword. Add cmd/* explicitly to the MEDIUM tier and
add a classification rule to prevent filename-based false positives.
Re-adds the old no-experimental-endorsement config option as a hidden,
deprecated alias for no-experimental-accountability. This ensures
backward compatibility for users who have the old option in their
config files after the rename it.
ziggie