Commit Graph
100 Commits
Author SHA1 Message Date
Andras Banki-Horvath fc139d9bb8 kvdb: add ForEachBucket implementation 2021-05-13 10:35:35 +02:00
Andras Banki-Horvath 3c6f036bee kvdb: make lnd/kvdb a submodule 2021-05-07 14:18:57 +02:00
Andras Banki-Horvath 14c851c8fc kvdb: move channeldb/kvdb to top level 2021-05-07 14:18:56 +02:00
Andras Banki-Horvath ad586c6bf1 healthcheck: separate lnd/healthcheck into submodule 2021-05-05 15:33:19 +02:00
Andras Banki-Horvath cee832b749 healthcheck: disable default healthcheck logger
Disable logging by default in the healthcheck submodule to cut
dependency on lnd/build as we are hooking the logger separately
during lnd init.
2021-05-05 15:33:14 +02:00
Andras Banki-Horvath dd14947154 lint: silence depreciation warnings 2021-05-04 17:33:14 +02:00
Andras Banki-Horvath 80bc46e614 lint: fix proto message no-copy linter warnings 2021-05-04 17:33:14 +02:00
Andras Banki-Horvath 9aacc35989 lnrpc: add GetState to stateservice (v1/state)
This commit adds a new GetState call to the state service in order to
make leader election integrate simply with k8s using a readinessProbe.
2021-05-04 17:33:13 +02:00
Andras Banki-Horvath 7caf26ce94 itest: add itest for failover after forcefull shutdown 2021-05-04 17:33:12 +02:00
Andras Banki-Horvath 5d8488871c itest: basic failover itest when using leader election on etcd 2021-05-04 17:33:12 +02:00
Andras Banki-Horvath 5e215a7a66 lnrpc: add "waiting to start" state to state service
This commit adds a new "waiting to start" state which may be used to
query if we're still waiting to become the cluster leader. Once leader
we advance the state to "wallet not exist" or "wallet locked" given
wallet availablity.
2021-05-04 17:33:11 +02:00
Andras Banki-Horvath b6a620e6b2 misc: add ability to store the wallet in the remote DB 2021-05-04 17:33:11 +02:00
Andras Banki-Horvath 08be03367a walletunlocker: extract method to create the Loader 2021-05-04 17:33:10 +02:00
Andras Banki-Horvath c190c3d582 etcd: update etcd to 3.4.14 2021-05-04 17:33:09 +02:00
Andras Banki-Horvath e5f249ad51 mod: integrate latest btcwallet and bump protobuf and grpc 2021-05-04 17:33:09 +02:00
Andras Banki-Horvath d161b34ee5 lnd: integrate leader elector to lnd main
This commit also changes the order of DB init to be run after the RPC
server is up. This will allow us to later add an RPC endpoint to be used
to query leadership status.
2021-05-04 17:33:08 +02:00
Andras Banki-Horvath 41532ce634 cluster: add etcd based leader elector and factory 2021-05-04 17:33:07 +02:00
Andras Banki-Horvath e62dbca11a cluster: add cluster package skeleton and LeaderElector interface 2021-05-04 17:33:07 +02:00
Andras Banki-Horvath 282618441d etcd: do not try authenticate when using embedded 2021-05-04 17:33:06 +02:00
Andras Banki-Horvath 5441c1bb6d etcd: fix embbeded etcd connection cfg 2021-05-04 17:33:06 +02:00
Andras Banki-Horvath 9992e5c0b5 etcd: decouple embedded etcd instance startup
This refactor changes how we instantiate the etcd based Backend to allow
separate initialization and startup for the embedded instance.
2021-05-04 17:33:05 +02:00
Andras Banki-Horvath 6757e14998 etcd: refactors to simplify etcd configuration
This refactor removes a redundancy where we had etcd configuration under
kvdb and kvdb/etcd packages.
2021-05-04 17:33:04 +02:00
Andras Banki-Horvath 44e312ace9 etcd: remove (unused) etcd root bucket prefix
This commit removes the unused prefix from the etcd backend config as
etcd key space separation in LND is implemented by using namespaces
instead.
2021-05-04 17:32:59 +02:00
Andras Banki-Horvath ec50f2ccce invoices: more verbose logging for startup invoice GC 2021-04-22 15:37:53 +02:00
Andras Banki-Horvath adbcc3f7a3 invoices: do not fail DeleteInvoice if payment addr isn't indexed
This commit relaxes DeleteInvoice failure cases by removing the
requirement that the invoice needs to be indexed by the payment address.
Since payment address index was introduced with an empty migration it
is possible that users have old invoices which were never added to
this index causing invoice garbage collection to get stuck.
2021-04-19 17:56:09 +02:00
Andras Banki-Horvath dde5750160 invoices+rpc: add missing channel graph to the AddInvoiceConfig
The Graph which is referenced later in the AddInvoice call graph is
unset when adding a hodl invoice. This resulted in a crash.
2021-02-03 11:33:27 +01:00
Andras Banki-Horvath 22711ade3a cnct: dispatch contract breach after channel close summary is serialized
This commit moves the contract breach event dispatch after the channel
close summary has been added to the database. This is important
otherwise it may occur that we attempt to mark the channel fully closed
while the channel close summary is not yet serialized.
2021-01-11 22:48:41 +01:00
Andras Banki-Horvath d3cd412137 ci: add etcd itests to our CI (bitcoind only) 2021-01-08 15:39:12 +01:00
Andras Banki-Horvath b2ab5e8af1 itests: run etcd itests with generated ports 2021-01-08 15:39:12 +01:00
Andras Banki-Horvath f1831f0581 etcd: add support for user specified ports for testing 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath ce4ca86ca6 kvdb: fix kvdb.Batch to use state reset when using etcd backend 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath a9ba5af9fd etcd: fix config in docs and extend with new options 2021-01-05 16:50:57 +01:00
Andras Banki-Horvath cb2b5efc6e etcd: add (dev only) disabletls option for etcd 2021-01-05 16:50:53 +01:00
Andras Banki-Horvath 5ee0d77414 build: extend sample-lnd.conf with the new etcd namespace 2020-12-16 18:34:41 +01:00
Andras Banki-Horvath 4a6af67241 etcd: add support for namespaces
This commit adds support for etcd namespaces. This is useful when
using a shared etcd database where separate users have access to
separate namespaces.
2020-12-16 18:25:38 +01:00
András Bánki-Horváth 669663bfc2 Merge pull request #4822 from bhandras/add_globalfeatures
routing: add missing features to NetworkNodeUpdate (and also depreacte GlobalFeatures)
2020-12-08 10:16:50 +01:00
Andras Banki-Horvath 98c61be342 routing: add missing GlobalFeatures to NetworkNodeUpdate 2020-12-07 15:47:09 +01:00
Andras Banki-Horvath 5e19c41cb8 build: extend sample-lnd-conf with db.etcd.embedded 2020-11-18 15:34:46 +01:00
Andras Banki-Horvath 5f3b800630 lint: fix error reported by the linter 2020-11-18 15:34:46 +01:00
Andras Banki-Horvath efedb5547c itest: timeouts for itets with etcd 2020-11-18 15:34:45 +01:00
Andras Banki-Horvath 369ae5e372 itest: move all test db files when using both etcd and bbolt
In some tests we moved channeld.db to a temp location in order to
"time travel". This commit extends the existing semantics by moving all
files, including embedded etcd db too besides the channeld.db file.
2020-11-18 15:34:45 +01:00
Andras Banki-Horvath 9138efc2c3 build: extend itest with etcd=1 param which will run itests on etcd 2020-11-18 15:34:44 +01:00
Andras Banki-Horvath 98342433ab itest: add etcd flag to control if new lnd nodes should run on etcd 2020-11-18 15:34:44 +01:00
Andras Banki-Horvath 511e817624 lncfg+channeldb: add config to be able to run lnd on embedded etcd 2020-11-18 15:34:43 +01:00
Andras Banki-Horvath 38e8184926 itest: update error log whitelist 2020-11-12 09:17:53 +01:00
Andras Banki-Horvath 9eac0dd3c9 itest: fix crash in parallel macaroon tests 2020-11-12 09:17:52 +01:00
Andras Banki-Horvath 979c8497b2 itest: add icase to node log filename + restart nodes before each icase
This commit adds the icase name to the log filename, to make it simpler
to find problematic tests. Additionally after this commit we'll restart
Alice and Bob (the base harness nodes) before each icase to start with a
clean state.
2020-11-12 09:17:52 +01:00
Andras Banki-Horvath d6c2ee1bbf itest: use require.NoError in TestLightningNetworkDaemon 2020-11-12 09:17:47 +01:00
András Bánki-Horváth 0c3c6e6155 Merge pull request #4705 from bhandras/kvdb_view_reset
multi: add reset closure to `kvdb.View/Update` to be able to clean up external state before retries
2020-11-06 12:47:35 +01:00
Andras Banki-Horvath adfd99ec08 etcd: allow readwrite bucket Delete with nil key (bbolt compatibility) 2020-11-05 17:57:13 +01:00
Andras Banki-Horvath e8f47cf882 invoices: channel graph reference was wrongly retrieved from remote db
This commit fixes a bug where we tried to use the graph from the remote
db instance whereas it lives in the local db instance.
2020-11-05 17:57:13 +01:00
Andras Banki-Horvath d89f51d1d0 multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath 2a358327f4 multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath ffb27284df channeldb: change walletdb dependency to kvdb 2020-11-05 17:57:08 +01:00
András Bánki-Horváth 9c9ce15ca6 Merge pull request #4708 from bhandras/expiry_watcher_opt
invoices: reduce heap usage when starting the expiry watcher
2020-10-29 10:15:35 +01:00
Andras Banki-Horvath 478cf704be invoices: reduce heap usage when starting the expiry watcher 2020-10-28 16:26:47 +01:00
András Bánki-Horváth dc16869d72 Merge pull request #4725 from halseth/chaneventstore-race
chanfitness/chaneventstore:  stop ticker after goroutine
2020-10-28 11:26:30 +01:00
András Bánki-Horváth 111db801e7 Merge pull request #4457 from bhandras/etcd_tx_queue
etcd: STM transaction queue to effectively reduce retries for conflicting transactions
2020-09-17 17:35:59 +02:00
Andras Banki-Horvath 26effca761 build: unit test on github with kvdb_etcd tag 2020-09-17 14:50:19 +02:00
Andras Banki-Horvath 357cd7d7a7 make: allow optional extra tags when running unit tests 2020-09-17 14:50:19 +02:00
Andras Banki-Horvath 9c47392dfa etcd: integrate the commitQueue to the STM commit loop
This commit integrates an externally passed commitQueue instance with
the STM to reduce retries for conflicting transactions.
2020-09-17 14:50:19 +02:00
Andras Banki-Horvath 6f3a45b75f etcd: make embedded etcd context cancelable 2020-09-17 14:50:18 +02:00
Andras Banki-Horvath 281c0b9d92 etcd: add commit queue to effectively reduce transaction retries
This commit adds commitQueue which is a lightweight contention manager
for STM transactions. The queue attempts to queue up transactions that
conflict for sequential execution, while leaving all "unblocked"
transactons to run freely in parallel.
2020-09-17 14:50:18 +02:00
Andras Banki-Horvath b4b5a9d7de etcd: increase message and transaction limits for embedded etcd 2020-09-17 14:50:14 +02:00
András Bánki-Horváth dea6d91431 Merge pull request #4603 from bhandras/fix_help
config: prevent displaying usage twice if --help was requested
2020-09-14 18:58:21 +02:00
Andras Banki-Horvath b778b25866 config: prevent displaying usage twice if --help was requested 2020-09-14 15:18:41 +02:00
András Bánki-Horváth d62b93e012 Merge pull request #4607 from bhandras/routing_experiments
lncli: add outgoing_chanid to queryroutes
2020-09-14 13:26:56 +02:00
Andras Banki-Horvath 4ffe744364 lncli: add outgoing_chanid to queryroutes 2020-09-14 11:36:06 +02:00
András Bánki-Horváth c3821e5ad1 Merge pull request #4493 from bhandras/invoice_gc
invoices: garbage collect settled/canceled invoices
2020-08-24 08:34:58 +02:00
Andras Banki-Horvath 2aa680ede2 invoices: optionally garbage collect invoices on the fly
This commit extends invoice garbage collection to also remove invoices
which are canceled when LND is already up and running. When the option
GcCanceledInvoicesOnTheFly is false (default) then invoices are kept and
the behavior is unchanged.
2020-08-05 12:24:17 +02:00
Andras Banki-Horvath a0d7877d9a multi: make canceled invoice garbage collection configurable
This commit extends the application config with a flag to control canceled
invoice garbage collection upon startup.
2020-08-05 12:24:16 +02:00
Andras Banki-Horvath 0ea763d83c invoices: attempt to delete old invoices upon start 2020-08-05 12:24:15 +02:00
Andras Banki-Horvath a5778c4673 channeldb: add DeleteInvoices call
This commit extends channeldb with the DeleteInvoices call which is when
passed a slice of delete references will attempt to delete the invoices
pointed to by the references and also clean up all our invoice indexes.
2020-08-05 12:24:15 +02:00
Andras Banki-Horvath 92f3b0a30c channeldb+invoices: add ScanInvoices and integrate with InvoiceRegistry
This commit adds channeldb.ScanInvoices to scan through all invoices in
the database. The new call will also replace the already existing
channeldb.FetchAllInvoicesWithPaymentHash call in preparation to collect
invoices we'd like to delete and watch for expiry in one scan in later
commits.
2020-08-05 12:24:14 +02:00
Andras Banki-Horvath ba3c65bfd6 invoices: re-format overreaching code lines 2020-08-05 12:24:13 +02:00
Andras Banki-Horvath 5346ed8a5c kvdb+etcd: assert on bucket/value key when putting value/bucket
This commit extends compatibility with the bbolt kvdb implementation,
which returns ErrIncompatibleValue in case of a bucket/value key
collision. Furthermore the commit also adds an extra precondition to the
transaction when a key doesn't exist. This is needed as we fix reads to
a snapshot revision and other writers may commit the key otherwise.
2020-08-01 00:29:01 +02:00
Andras Banki-Horvath b17ed28dea build: add all subservers to dev build 2020-07-29 14:33:25 +02:00
Andras Banki-Horvath 12a341ba59 etcd: remove the lock set concept
This commit removes the lock set which was used to only add bucket keys
to the tx predicate while also bumping their mod version.
This was useful to reduce the size of the compare set but wasn't useful
to reduce contention as top level buckets were always in the lock set.
2020-07-28 17:57:30 +02:00
Andras Banki-Horvath 3cdbb341da lnd: utxo nursery test to use channeldb.MakeTestDB 2020-07-28 17:57:30 +02:00
Andras Banki-Horvath 905990eb54 sweep: use channeldb.MakeTestDB 2020-07-28 17:57:30 +02:00
Andras Banki-Horvath f2a08e420e lnd: use channeldb.MakeTestDB in nursery store tests 2020-07-28 17:57:29 +02:00
Andras Banki-Horvath cbce8e8872 channeldb: move makeTestDB out of test to make it available for other tests
This commit moves makeTestDB to db.go and exports it so that we'll be
able to use this function in other unit tests to make them testable with
etcd if needed.
2020-07-28 17:57:29 +02:00
Andras Banki-Horvath 63e9d6102f kvdb+etcd: change flattened bucket key derivation algorithm
This commit changes the key derivation algo we use to emulate buckets
similar to bbolt. The issue with prefixing keys with either a bucket or
a value prefix is that the cursor couldn't effectively iterate trough
all keys in a bucket, as it skipped the bucket keys.
While there are multiple ways to fix that issue (eg. two pointers,
iterating value keys then bucket keys, etc), the cleanest is to instead
of prefixes in keys we use a postfix indicating whether a key is a
bucket or a value. This also simplifies all operations where we
(recursively) iterate a bucket and is equivalent with the prefixing key
derivation with the addition that bucket and value keys are now
continous.
2020-07-28 17:57:29 +02:00
Andras Banki-Horvath 9173958f9c kvdb: s/hu/en/g 2020-07-28 17:57:28 +02:00
Andras Banki-Horvath 70a69ce990 kvdb: make etcd tests use testify require instead of assert 2020-07-28 17:57:24 +02:00
András Bánki-Horváth 77549f1fb8 Merge pull request #4384 from bhandras/atpl_bc_topk
add modified greedy topK centrality heuristic to autopilot
2020-07-17 18:27:42 +01:00
Andras Banki-Horvath afbbeae4f2 autopilot+test: testify betweenness centrality tests
The commit also reindents the source to conform with ts=8 guideline.
2020-07-17 16:12:55 +02:00
Andras Banki-Horvath ccabad8607 autopilot: add TopCentrality to the available heuristics 2020-07-17 16:12:54 +02:00
Andras Banki-Horvath 8373b2ad20 autopilot: add greedy "TopK" centrality heuristic
This commit creates a new autopilot heuristic which simply returns
normalized betweenness centrality values for the current graph. This
new heuristic will make it possible to prefer nodes with large
centrality when we're trying to open channels. The heuristic is also
somewhat dumb as it doesn't try to figure out the best nodes, as that'd
require adding ghost edges to the graph recalculating the centrality as
many times as many nodes there are (minus the one we already have
channels with).
2020-07-17 16:12:54 +02:00
Andras Banki-Horvath 82ddccee0b autopilot+test: make centrality test data available for other tests 2020-07-17 16:12:53 +02:00
Andras Banki-Horvath 906b0b707b atpl: remove unneeded extra filter on address availability
This commit removes an extra filter on address availability which is not
needed as the scored nodes are a already prefiltered subset of the whole
graph where address availability has already been checked.
2020-07-17 16:12:48 +02:00
Andras Banki-Horvath fe51f528d1 docs: documentation for the experimental etcd feature 2020-07-07 15:59:53 +02:00
Andras Banki-Horvath 80012c3936 invoice+test: changing testitfy asserts to specific requires 2020-06-05 15:16:31 +02:00
Andras Banki-Horvath 574bbe5eba invoices+test: replace DeepEqual + spew dump with testify
This commit replaces reflect.DeepEqual tests and spew.Dump prints with
testify's require.Equal to make diffs smaller and test outputs more
readable.
2020-06-05 15:16:27 +02:00
András Bánki-Horváth 595bb7c1ff Merge pull request #4318 from bhandras/etcd_fixes
lnd+kvdb+etcd: remove retry goroutine from the STM and integrate abort context
2020-06-03 19:21:01 +02:00
Andras Banki-Horvath 3e5e60cac4 lnd+kvdb: pass LND abort context to etcd 2020-06-03 17:44:44 +02:00
Andras Banki-Horvath d3545830c9 kvdb+etcd: integrate the abort context to the STM retry loop
This commit extends the etcd.BackendConfig to also provide an abort
context and integrates it with the STM retry loop in order to be able
stop LND when conflicting transactions keep the loop running.
2020-06-03 17:44:44 +02:00
Andras Banki-Horvath 415de2f0c7 kvdb+etcd: remove retry goroutine from the STM
This commit removes the retry goroutine from the STM as the retry loop
is only running when the STM transaction is encapsulated in Update/View
whereas for self-standing transactions we use a different approach.
By removing the goroutine we won't catch panics thrown that are supposed
to be catched outside of the STM.
2020-06-03 17:44:44 +02:00
Andras Banki-Horvath 1dd43f3669 test: add error to itest whitelist 2020-06-02 15:40:20 +02:00
Andras Banki-Horvath 2a591ee2f1 build: bump etcd to v3.3.22 2020-06-02 15:40:20 +02:00