This commit introduces the functionality to set a tombstone key
in the invoice bucket after the migration to the native SQL
database. The tombstone prevents the user from switching back
to the KV invoice database, ensuring data consistency and
avoiding potential issues like lingering invoices or partial
state in KV tables. The tombstone is checked on startup to
block any manual overrides that attempt to revert the migration.
Previously, we applied replacements to our schema definitions
to make them compatible with both SQLite and Postgres backends,
as the files were not fully compatible with either.
With this change, the only replacement required for SQLite has
been moved to the generator script. This adjustment ensures
compatibility by enabling auto-incrementing primary keys that
are treated as 64-bit integers by sqlc.
Previously we'd recalculate the paid amount by summing amounts of
settled HTLCs. This approach while correct would stop the SQL migration
process as some KV invoices may have incorrectly stored paid amounts.
The current sqlc GetInvoice query experiences incremental slowdowns during
the migration of large invoice databases, primarily due to its complex
predicate set. For this specific use case, a streamlined GetInvoiceByHash
function provides a more efficient solution, maintaining near-constant
lookup times even with extensive table sizes.
Certain invoices may not have a deterministic payment hash. For such
invoices we still store the payment hashes in our KV database, but we do
not have a sufficient index to retrieve them. This PR adds such index to
the SQL database that will be used during migration to retrieve payment
hashes.
Previously we intentially did not set settled_at and settle_index when
inserting a new invoice as those fields are set when we settle an
invoice through the usual invoice update. As migration requires that we
set these nullable fields, we can safely add them.
This commit separates the execution of SQL and in-code migrations
from their construction. This change is necessary because,
currently, the SQL schema is migrated during the construction
phase in the lncfg package. However, migrations are typically
executed when individual stores are constructed within the
configuration builder.
This commit introduces support for custom, in-code migrations, allowing
a specific Go function to be executed at a designated database version
during sqlc migrations. If the current database version surpasses the
specified version, the migration will be skipped.
The package provides type GoroutineManager which is used to launch goroutines
until context expires or the manager is stopped. Stop method blocks until all
started goroutines stop.
Original code by Andras https://go.dev/play/p/HhRpE-K2lA0
Adjustments and tests by Boris.
Previously, the SQL implementation of the invoice query simply
converted the start and end timestamps to time and used them
in SQL queries to check for inclusivity. However, this logic
failed when the start and end timestamps were equal.
This commit addresses and corrects this issue.
Previously SQL invoice updater ignored the set ID hint when updating an
AMP invoice resulting in update subscriptions returning all of the AMP
state as well as all AMP HTLCs. This commit synchornizes behavior with
the KV implementation such that we now only return relevant AMP state
and HTLCs when updating an AMP invoice.
When fetching an AMP invoice we sometimes filter HTLCs to selected set
IDs, however we always kept the full AMP state which is irrelevant as it
contains state for all AMP payments. This was a side effect of
UpdateInvoice needing to serialize the whole invoice when storing after
an update but it is an unwanted "feature" as users will need to filter
to relevant set when listing an AMP payment or subsribing to an update.
This commit extends our healtcheck with an optional leader check. This
is to ensure that given network partition or other cluster wide failure
we act as soon as possible to avoid a split-brain situation where a new
leader is elected but we still hold onto our etcd client.
Previously our RPC calls to etcd would hang even in the case of properly
set dial timeouts and even if there was a network partition. To ensure
liveness we need to make sure that calls fail correctly in case of
system failure. To fix this we add a default timeout of 30 seconds to
each etcd RPC call.
Previously, when using the native schema, invoice expiries were incorrectly
stored as 64-bit values (expiry in nanoseconds instead of seconds), causing
overflow issues. Since we cannot determine the original values, we will set
the expiries for existing invoices to 1 hour with this migration.
Previously paginated queries offseted the add_index_get, add_index_let,
settle_index_get and settle_index_let parameters with the paginators
current page offset, however this was incorrect as we can just use
SQL's LIMIT/OFFSET to paginate. This commit fixes this issue and adds an
optional parameter to the constructor of the invoice SQL store to set
page size. This is useful when testing as we can now cover pagination
correctly with our existing unit tests.
Previously if the `reverse` named arg was unset (value of NULL), then
SQL would order by NULL instead of ID causing undifined ordering of the
returned rows. To fix that we check for NULL and also make sure to set
the `reverse` arg in the code explicitly as it in the generated code it
is an `interface{}` instead of `bool`.
For SQL transactions, we often accumulate results in variables declared
outside the closure's scope. To eliminate the need for manually clearing
these containers, we introduce a reset function to ExecTx, mirroring the
approach already adopted in kvdb.
As SQL serializaton errors can lead to transaction retries we need to
ensure that we reset any out of scope containers where we accumulate
results. Otherwise partial data from a previously executed transacton
retry may be returned along with the latest result.
This commit adds a check to ensure that we don't start LND with native
SQL invoice DB if we already have any invoices in our KV channeldb. This
is needed as native SQL invoices is an experimental feature and we do
not yet support migration.
This commit adds an optional separate "native" SQL backend that will
host all tables supporting native SQL. For now since we don't have many
such tables we'll keep it in one file for SQLite, but it may be split up
if desired.
This commit is part of a refactor that unifies configuration of the
sqldb and kvdb packages for SQL backends.
In order to unify the SQLite and Postgres configuration under sqldb we
first need to ensure that the final config types are compatible with
the alreay deployed versions.
This change will enable us to use a single PostgreSQL container instead of
spawning new a one for each (parallel) unit test reducing overall test
runtime.
This commit attempts to fix some issues with the invoice store's schema that we
couldn't foresee before the implementation was finished. This is safe as the
schema has not been instantiated yet outside of unit tests. Furthermore the
commit updates invoice store SQL queries according to fixes in the schema as
well as to prepare the higher level implementation in the upcoming commits.
This commit extracts the InvoiceDB construction from all invoice and
registry tests such that we can later on run subtests with multiple
backends without needing to use tags.
With the introducation of the `InvoiceUpdater` interface we are now
able to move the non-kv parts of `UpdateInvoice` completely under
the invoices package. This is a preprequisite for being able to use
the same code-base for the sql InvoiceDB implementation of
UpdateInvoice.
This commit introduces the InvoiceUpdater interface which is meant
to abstract and assist the in-memory invoice update procedure with
the accompanying database updates. These abstract updater steps will
enable further refactoring later while also ensuring that a full
SQL implementation of the InvoiceDB interface will be possible.
With this commit updateInvoiceAmpState becomes getUpdatedInvoiceAmpState
which will only return the new AMP state but that needs to be applied at
the call site. This is a part of a larger refactor to gather all
mutations of an invoice update to be later applied by the invoice
updater.
This change moves the HTLC state change out of the cancelSingleHtlc
function. This is part of the larger refactor of collecting all changes
to be later applied by the invoice updater.
With this refactor updateHtlc is renamed to getUpdatedHtlcState and
changed such that it won't change the HTLC's state and resolve time but
instead returns whether the change is needed. This change is part of a
multi-commit refactor to ensure that all changes to the invoice will be
tracked individually.
