Here we add the `ForEachNodeDirectedChannel` and `ForEachNodeCacheable`
SQLStore implementations which then lets us run
`TestGraphTraversalCacheable` and `TestGraphCacheForEachNodeChannel`
against SQL backends.
Ensure that it does a sanity check on the length of its input and also
only call it if the nullable SQL string is not null.
Expand an existing unit tests to cover the DecodeHexColor such that it
would have caught the bug.
Use a length check to determine if a bitcoin signature has been set or
not. Also add a clarifying comment to explain why we only need to check
if one signature field is set to determine if we have the auth proof for
the channel or not.
In this commit, the ForEachSourceNodeChannel implementation of the
SQLStore is added. Since this is the first method of the SQLStore that
fetches channel and policy info, it also adds all the helpers that are
required to do so. These will be re-used in upcoming commits as more
"For"-type methods are added.
With this implementation, we convert the `TestForEachSourceNodeChannel`
such that it is run against SQL backends.
In this commit, the various SQL queries are defined that we will need in
order to implement the SQLStore UpdateEdgePolicy method. Channel
policies can be "replaced" and so we use the upsert pattern for them
with the rule that any new channel policy must have a timestamp greater
than the previous one we persisted.
As is done for the KVStore implementation of the method, we use the
batch scheduler for this method.
In this commit, we introduce a SQLStoreConfig struct which for the time
being only has the ChainHash of the genesis block of the chain this node
is running on. This is used to reconstruct lnwire messages from what we
have persisted in the DB. This means we dont need need to persist the
chain-hash of gossip messages since we know it will always be the same
for a given node. If a node were to be started with a different network,
the lnwire messages it reconstructs for gossip will be invalid.
Implement ForEachChannelCacheable which is like ForEachChannel but its
call-back takes the cached versions of channel info & policies. This is
then used during graph cache population. This will be useful once the
SQL implementation is added so that we can reduce the number of DB trips
on cache population.
Define a new CachedEdgeInfo type and let the graph cache's AddChannel
use this. This will let us later on (for the SQL impl of the graph db)
only load from the DB what we actually need for the graph cache.
Update the GraphCache.UpdatePolicy method to take a
`models.CachedEdgePolicy` instead of a `models.ChannelEdgePolicy`.
Doing this will allow us later on to only fetch the necessary info for
populating the CachedEdgePolicy when we are populating the cache via
UpdatePolicy.
Remove the previously added TODOs which would extract InboundFee info
from the ExtraOpaqueData of a ChannelUpdate at the time of
ChannelEdgePolicy construction. These can now be replaced by using the
newly added InboundFee record on the ChannelUpdate message.
Also add a temporariy replace to the tlv package which can be removed as
soon as the PR that includes this commit is merged and a new tag for the
tlv package has been created.
Now that we know that the InboundFee on the ChannelEdgePolicy is always
set appropriately, we can update the GraphCache UpdatePolicy method to
take the InboundFee directly from the ChannelEdgePolicy object.
In this commit, we make sure to set the new field wherever appropriate.
This will be any place where the ChannelEdgePolicy is constructed other
than its disk deserialisation.
Like the previous commit, here we can start directly using the
InboundFee on the models.ChannelEdgePolicy object since we know we read
it from disk and so the InboundFee field will be populated accordingly.
NOTE: unlike the previous commit, behaviour is slightly different here
since previously we would error out here if TLV parsing failed whereas
now, the DB call will just skip the error and return a nil policy. This
should be ok since this is explicitly only dealing with our own updates
and so our TLV should always be valid.
For any call-site where we extract inbound fees from a
models.ChannelEdgePolicy object that was deserialised from disk, we can
now just use the new InboundFee field on the object since we know that
it would have been populated at deserialisation time.
Note that for all these call-sites, if a failure previously happened on
decoding of the TLV stream, the error would be ignored and the edge
would just be skipped. This behaviour is now still the same given how
ErrParsingExtraTLVBytes is handled on the DB layer.
Here we add an explicit InboundFee field to the ChannelEdgePolicy
struct. Then, in the graph KVStore, at deserialisation time, we extract
the InboundFee from the ExtraOpaqueData. Currently we do this at higher
levels but we are going to move it to the DB layer so that when we add
the SQL implementation of the graph store, we can have explicit columns
for inbound fees. We need to account for the fact that we may have
invalid TLV already persisted though and we dont want to fail if we
deserialise those necessarily. So we return ErrParsingExtraTLVBytes now
if we fail to parse the extra bytes as TLV and then we let the callers
handle it similarly to how ErrParsingExtraTLVBytes is handled in that we
dont necessarily fail if we receive one of these errors.
As of this commit, we can now expect the InboundFee field of a
ChannelEdgePolicy to be set (if inbound fees are set on the policy) for
any update that we read from disk.
In this commit, we start validating the extra opaque data of a channel
edge policy before persisting it. We just check that the data is valid
TLV.
NOTE: we recently [started
validating](https://github.com/lightningnetwork/lnd/commit/1410a0949d555e14de11faf81a17fe64917505e3)
this at the lnwire level. So really, no new update will reach the DB
layer without this already being checked. But we check it again here so
that the DB API behaves correctly as its own unit.
This commit simplifies the code of the ForwardingLog.Query method by
removing a confusing for-loop. The for-loop makes it seem as though
multiple events could be encoded under a single timestamp. But from the
time that this forwarding log was introduced, it was never possible to
encode multiple events under the same timestamp and so this loop will
never execute successfully more than once per timestamp and can thus be
removed. This paves the way such that future expansions of the method
can be added easily.
See the initial commit that introduced this code [here](https://github.com/lightningnetwork/lnd/commit/f2cd668bcfb423fb212e78282c786dea166e0168).
In this commit you can see that from the start it was never possible to
have more than one event in a single timestamp since any previous event
in that timestamp would be overwritten. Then see [this commit](https://github.com/lightningnetwork/lnd/pull/4475/commits/97c73706b55fb5520d2f3001dfa13dc1da986972)
where even more protection was added to ensure that each event had a
unique timestamp.
In this commit, the `AddChannelEdge` method of the SQLStore is
implemented. Like the KVStore implementation, it makes use of the
available channel `batch.Scheduler` and also updates the reject and
channel caches.
This then lets us convert the following 2 unit tests to run against the
SQL backends:
- TestPartialNode
- TestAddChannelEdgeShellNodes
Expand the existing TestAddChannelEdgeShellNodes test so that we have
coverage for error we expect when AddChannelEdge is called a second time
if we already know of a channel.
In preparation for having consistency with the structs created by the
SQLStore and the KVStore (so that they have the same behaviour when
tested by the unit tests), here we make sure not to init the
ExtraOpaqueData field of the LightningNode struct unless there are
actualy bytes to set.
In this commit, we define the SQL schemas for storing graph channel
data. This includes a new `channels` table and a new `channel_features`
table along with various indices.
Add a re-usable implementation of the sqldb.TxOptions interface and make
use of this in the various spots (invoices, batch and graph/db) where we
had previously defined individual implementations that were all doing
the same thing.
In this commit, we add the `source_nodes` table. It points to entries in
the `nodes` table. This table will store one entry per protocol version
that we are announcing a node_announcement on.
With this commit, we can run the TestSourceNode unit test against our
SQL backends.
In this commit we add the necessary SQL queries and then implement the
SQLStore's NodeUpdatesInHorizon method. This lets us run the
TestNodeUpdatesInHorizon unit tests against SQL backends.
In this commit, we add the various sqlc queries that we need in order
to implement the following V1Store methods:
- AddLightningNode
- FetchLightningNode
- HasLightningNode
- AddrsForNode
- DeleteLightningNode
- FetchNodeFeatures
These are implemented by SQLStore which then lets us use the SQLStore
backend for the following unit tests:
- TestNodeInsertionAndDeletion
- TestLightningNodePersistence
In this commit, the various SQL schemas required to store graph node
related data is defined. Specifically, the following tables are defined:
- nodes
- node_extra_types
- node_features
- node_addresses
Here, we use the new options to initialise the reject and channel caches
for the SQLStore (as is done for the KVStore) and also the channel and
node batch schedulers.
Let all the NewTestDB functions return the V1Store interface type
instead of pointers. Then add a manual skip in the TestGraphLoading test
for any non-bbolt backend. We can remove this once all the methods used
by the test have been implemented by the SQLStore. We only need the
manual skip for this one test since it is the only one that doesnt use
MakeGraphTest to init the graph db.
In this commit, we update the batch schedular so that it has the ability
to do read-only calls. It will do a best effort attempt at keeping a
transaction in read-only mode and then if any requests get added to a
batch that require a read-write tx, then the entire batch's tx will be
upgraded to use a read-write tx.
Here we add a new BenchmarkBoltBatching test that helps us benchmark the
performance when writing to a bbolt backend using various different
configurations. We test using N txs for N writes, 1 tx for N writes and
then various configurations when using the TimeScheduler.
In preparation for using the same logic for non-bbolt backends, we adapt
the batch.Schedular to be more generic.
The only user of the scheduler at the moment is the KVStore in the
`graph.db` package. This store instantiates the bbolt implementation of
the scheduler.
In this commit we add more test coverage for the persistence of the
addresses of a LightningNode. This is so that we have unit test coverage
that ensures that all the various address types can be persisted and
that the order of the addresses (within a type) are preserved.
Here we expand TestEdgeInsertionDeletion to assert that the expected
error is returned when AddChannelEdge is called for a channel that has
already been persisted. We also take the opportunity to convert some of
the error checks in the test to use strict error type matching.
Finally, we test that the expected error is returned if
DeleteLightningNode is called for a node that is no longer in the DB.
Expand an existing test for ForEachNodeDirectedChannel so that it also
tests the DB method and not just the ChannelGraph method which will use
the in-memory graph cache for the query.
Add `migrations_dev.go` and `migrations_prod.go` files which each define
a `migrationAdditions` slice to be appended to the `migrationConfig`
slice. The `migrations_dev.go` file is only built if either the
`test_db_postgres` or `test_db_sqlite` build flags are used.
This slice will be used to add any migrations that are still under
development that we want access to via unit tests and itests but do not
want to expose to our release build.
In this commit, we implement the postgres and sqlite versions of the
NewTestDB function. We add the various build flags so that only one of
the three versions of this function can be active at a time.
We also introduce the SQLStore struct which is the SQL implementation of
the V1Store interface.
NOTE: it currently temporarily embeds the KVStore struct so that we can
implement the V1Store interface incrementally. For any method not
implemented, things will fall back to the KVStore. This is ONLY the
case for the time being while this struct is purely used in unit tests
only. Once all the methods have been implemented, the KVStore field will
be removed from the SQLStore struct.
Update the test methods to take a `testing.TB` param instead of a
`*testing.T` so that the test functions can be used for all the graph's
tests including benchmark tests.
We also add a temporary replace so that we can make use of these changes
and also since we will soon be adding graph related schemas and queries
in this submodule that we'll want to have access to.
In this commit, we add a `test_kvdb.go` file with a single definition of
the `NewTestDB` function. A new version of `MakeTestGraph` (called
`MakeTestGraphNew` is added which makes use of this `NewTestDB` function
to create the backing `V1Store` passed to the `ChannelGraph` for tests.
Later on, we will add new implementations of this method backed by
sqlite and postgres. When those are added, then build flags will
control which version of `NewTestDB` is called.
With this change, the only test call-site of `NewKVStore` is the new
`test_kvdb.go` file.
In this commit, we unify how all unit tests that make use of the graph
create their test ChannelGraph instance. This will make it easier to
ensure that once we plug in different graph DB implementations, that all
unit tests are run against all variants of the graph DB.
With this commit, `NewChannelGraph` is mainly only called via
`MakeTestGraph` for all tests _except_ for `TestGraphLoading` which
needs to be able to reload a ChannelGraph with the same backend. This
will be addressed in a follow-up commit once more helpers are defined.
Note that in all previous packages where we created a test graph using
`kvdb.GetBoltBackend`, we now need to add a `TestMain` function with a
call to `kvdb.RunTest` since the `MakeTestGraph` helper uses
`GetTestBackend` instead of `kvdb.GetBoltBackend` which requires an
embedded postgres instance to be running.
Currently none of the calls to MakeTestGraph make use of the
KVStoreOptionModifier options but later on we will want to make use of
the `WithUseGraphCache` ChannelGraphOption and so we take this
opportunity to switch out the functional parameters that the helper
function takes.
To clearly separate the mission control DB store (which for now will
remain kvdb only) from the graph store, we rename the `graphBackend`
variable to `mcBackend` in the `testGraphInstance` struct.
Instead of returning an error and needing to call `require.NoError` for
each call to `MakeTestGraph`, rather just used the available testing
variable to require no error within the function itself.
In preparation for our SQL Graph store which wont explicitly store the
chain hash but will instead obtain it from the runtime config, we
replace the test chainhash value with that of the mainnet genesis hash.
In preparation for a different store impl which may wrap errors, we
use `require.ErrorIs` for error assertions rather than direct "="
comparisons in tests.
Expand this existing test so that it also tests that a node's addresses
and feature are fetched correctly after insertion. With this, we ensure
that the `FetchNodeFeatures` and `AddrsForNodes` methods of the
`V1Store` interface are properly covered by unit tests.
In this commit, we check that the extra bytes appended to gossip
messages contain valid TLV streams. We do this here for:
- channel_announcement
- channel_announcement_2
- channel_update
- channel_update_2
- node_announcement
This is in preparation for the SQL version of the graph store which will
normalise TLV streams at persistence time.
Later when we introduce our SQL version of the graph store, we will
normalise the persistence of the ExtraOpaqueData using the fact that it
is always made up of TLV entries. So we update our tests here to ensure
that they use valid TLV streams as examples.
Use the new `V1Store` interface as a field in the `ChannelGraph` rather than
a pointer to the `KVStore` implementation in preparation for allowing a
SQL implementation of `V1Store` to be used by the `ChannelGraph`.
Note that two tests are adjusted in this commit to be skipped if the
V1Store is not the `KVStore` implementation since the tests are very
bbolt specific.
In this commit, we introduce the `V1Store` interface which the existing
`graphdb.KVStore` implements today. The idea is to eventually create a
SQL DB backed implementation of this interface.
Currently, a few of the graph KVStore methods take the
`batch.SchedulerOptions` param. This is only used to set the LazyAdd
option. A SchedulerOption is a functional option that takes a
`batch.Request` which has bolt-specific fields in it. This commit
restructures things a bit such that the `batch.Request` type is no
longer part of the `batch.SchedulerOptions` - this will make it easier
to implement the graph store with a different DB backend.
Remove the kvdb.Backend return type of the `makeTestGraph` helper. This
is in preparation for the helper being used to create a test graph
backed by a DB other than bbolt.
- Let it do a proper comparison of the full structs passed in.
- Pass in a testing parameter so we can remove the returned error.
- Make sure the callers are passing in the expected and result
parameters in the correct order.
- Fix a bug: the compareNodes was not comparing the Features field of
the LightningNode structs. Now that it does, one test needed to be
updated to properly set the expected Features fields.
Since we have not removed all call-sites that make use of this
parameter, we can remove it. This helps hide DB-specific details from
the interface we will introduce for the graph store.
Elle Mouton