We add a WhenSome that'll pass in the actual underlying value, and not
just the record.
We also add `SomeRecord` to make it easier to create TLV records w/
`Some` set.
This will be useful when decoding optional TLV records, as we can use
this to look up in the `typeMap` for a given field to see if we decoded
it or not.
With this change, callers can now examine a struct instance of the
`TlvType` instance to figure out programmatically which integer type is
maps to. This'll be useful when decoding optional TLV types into a wire
struct (knowing when to set it to Some vs None).
In this commit, we modify the RecordT type to allow callers to re-use
the Record definition of a different type, but use the new type param to
override the integer type used on the wire.
This will let use do things like encode a signature using the same
RecordProducer instance, but with a diff type in another context.
The upcoming use for this is allowing our `lnwire.Sig` type to be
encoded in the same message using distinct TLV integer types (new co-op
close protocol).
In this commit, we add a new type, `RecordT[T, V]` to reduce some of the
common boiler plate for TLV types. This type lets you take either a
primitive type, or an existing Record, and gain common methods used to
create tlv streams.
It also serves as extra type annotation as well, since wire structs can
use this to wrap any existing type and gain the relevant record methods.
This implementation ensures that the very definition of the field also
binds the TLV type value. It does this by using the generated code to
map a struct like TlvType1 to an actually Type like Type(1).
This shouldn't need to be run again, as this implementation restricts
things to just values 0-99, due to a hard upper limit with the way Go
unions work under the hood.
In this commit, we add some new code generation to the codebase. As
we'll see in a future commit, this'll allow us to create a new Record[T,
V] type, where T is actually a concrete _struct_ that implements a
special interface that deems it as a valid TLV type.
In this commit, we make a new sub-module from the `fn` package. This
lets other projects use it w/o pulling in all of lnd, and also other
sub-packages within lnd.
We also add a temp replace directive for new module. We'll remove this
once we tag the final version.
We've only ever made macaroons with the v2 versions, so we should
explicitly reject those that aren't actually v2. We add a basic test
along the way, and also add a similar check for the version encoded in
the macaroon ID.
When we first go to boot up the syncer, when we're in the active phase,
after we do the historical sync, we'll send a timestamp message that we
want everything, then transition to the passive mode. The test didn't
account for this extra message before, as the last test was being
re-used here (ran in parallel).
We fix this by asserting that the first expected message is sent, then
also the follow up messages as well.
In this commit, [we prep for some upcoming changes in
Go](https://go.dev/blog/loopvar-preview) by running our tests with
`GOEXPERIMENT=loopvar`. This changes the loop semantics to fix a common
bug where a scoping issue causes a variable to be re-used, which can
cause unintended bugs down the line.
If everything passes with this flag on, then we can keep it on, and also
rest a bit easier knowing that the compiler will fix a class of bugs
that would previously pop up for us.
This PR is a follow up, to a [follow
up](https://github.com/lightningnetwork/lnd/pull/7938) of an [initial
concurrency issue](https://github.com/lightningnetwork/lnd/pull/7856)
fixed in the peer goroutine.
In #7938, we noticed that the introduction of `p.startReady` can cause
`Disconnect` to block. This happens as `Disconnect` cannot be called
until `p.startReady` has been closed. `Disconnect` is also called from
`InboundPeerConnected` (the case of concurrent peers, so we need to
remove one of the connections) while the main server mutex is held. If
`p.Start` blocks for any reason, then this leads to the deadlock as: we
can't disconnect until we've finished starting, and we can't finish
starting as we need the disconnect caller to exit as it has the mutex.
In this commit, we now make the call to `prunePersistentPeerConnection`
async. The call to `prunePersistentPeerConnection` eventually wants to
grab the server mutex, which triggers the circular waiting scenario
above.
The main learning here is that no calls to the main server mutex path
can block from `p.Start`. This is more or less a stop gap to resolve the
issue initially introduced in v0.16.4. Assuming we want to move forward
with this fix, we should reexamine `p.startReady` all together, and also
revisit attempt to refactor this section of the code to eliminate the
mega mutex in the server in favor of a dedicated event loop.
In this commit, we modify the incoming contest resolver to use a
concurrent queue. This is meant to ensure that the invoice registry
subscription loop never blocks. This change is meant to be minimal and
implements option `5` as outlined here:
https://github.com/lightningnetwork/lnd/issues/8023.
With this change, the inner loop of the subscription dispatch method in
the invoice registry will no longer block, as the concurrent queue uses
a fixed buffer of a queue, then overflows into another queue when that
gets full.
Fixes https://github.com/lightningnetwork/lnd/issues/7917
In this commit, we add the ability to obtain blocking and mutex
profiles. The blocking profile will show which goroutines are
consistently blocked on synchronization primitives like channels, or
I/O. The mutex profile will show which mutexes are very contested.
The blocking profile can be enabled with a new arg: `--blockingprofile`.
The mutex profile can be enabled with a new arg: `--mutexprofile`. These
are both ignored if the profile port isn't set.
Activating these profiles requires the caller to pass in a sampling
rate. For now I've set it just to `1` to test things out. Unfortunately
documentation is rather scarce, so there aren't any good guides re what
these values should be set to. AFAICT, these add more overhead than the
other prowling options, so they shouldn't necessarily be enabled
persistently in production.
In this commit, we update the set of protos to accept the local secret
nonces over RPC. This is actually a 97 byte value, as it includes the
two 32 byte nonces, as well as the 33 byte value of the public key of
the signer.
This is needed in order to be able to open taproot channels over the RPC
interface.
In this commit, we modify the musig2 interfaces to instead use an
explicit value for the local nonces. Before this commit, we used the
functional option, but we want to also support specifying this value
over RPC for the remote signer. The functional option pattern is opaque,
so we can't get the nonce value we need. To get around this, we'll just
make this an explicit pointer, then map this to the functional option at
the very last moment.
In this commit, we increase the max message size for the ws proxy. We
have a similar setting for the normal gRPC server which was tuned to be
able to support decoding `GetNetworkInfo` as the channel graph got
larger. We keep the default buffer size of 64 KB, but allow that to be
expanded to up to 4 MB (current value) to decode larger messages.
One alternative would be to modify the `Split` function to break up
larger lines into smaller ones. We'd need to double check that the
libraries at a higher level of abstraction can handle the chunks. The
scan function would look something like:
```go
splitFunc := func(data []byte, eof bool) (int, []byte, error) {
if len(data) >= chunkSize {
return chunkSize, data[:chunkSize], nil
}
return bufio.ScanLines(data, eof))
}
scanner.Split(splitFunc)
```
In this commit, update the start up logic to gracefully handle a
seemingly rare case. In this case, a peer detects local data loss with a
set of active HTLCs. These HTLCs then eventually expire (they may or may
not actually "exist"), causing a force close decision. Before this PR,
this attempt would fail with a fatal error that can impede start up.
To better handle such a scenario, we'll now catch the error when we fail
to force close due to entering the DLP and instead terminate the state
machine at the broadcast state. When a commitment transaction eventually
confirms, we'll play it as normal.
Fixes https://github.com/lightningnetwork/lnd/issues/7984
In this commit, we use exhaustive build tags to ensure that we can
always build the `sqlbase` package, independent of the set build tags.
To do this, we move the type declarations _into_ the parsing functions.
This then allows us to create two versions for each db: with the db, and
without it.
To avoid a module tag round trip to get this working, we use a local
replace for now. Once this is merged in, we can do the tag (along side
rc3), then remove the replace.
* lnwallet: fix log output msg
The log message is off by one.
* htlcswitch: fail channel when revoking it fails.
When the revocation of a channel state fails after receiving a new
CommitmentSigned msg we have to fail the channel otherwise we
continue with an unclean state.
* docs: update release-docs
* htlcswitch: tear down connection if revocation processing fails
If we couldn't revoke due to a DB error, then we want to also tear down
the connection, as we don't want the other party to continue to send
updates. That may lead to de-sync'd state an eventual force close.
Otherwise, the database might be able to recover come the next
reconnection attempt.
* kvdb: use sql.LevelSerializable for all backends
In this commit, we modify the default isolation level to be
`sql.LevelSerializable. This is the strictness isolation type for
postgres. For sqlite, there's only ever a single writer, so this doesn't
apply directly.
* kvdb/sqlbase: add randomized exponential backoff for serialization failures
In this commit, we add randomized exponential backoff for serialization
failures. For postgres, we''ll his this any time a transaction set fails
to be linearized. For sqlite, we'll his this if we have many writers
trying to grab the write lock at time same time, manifesting as a
`SQLITE_BUSY` error code.
As is, we'll retry up to 10 times, waiting a minimum of 50 miliseconds
between each attempt, up to 5 seconds without any delay at all. For
sqlite, this is also bounded by the busy timeout set, which applies on
top of this retry logic (block for busy timeout seconds, then apply this
back off logic).
* docs/release-notes: add entry for sqlite/postgres tx retry
---------
Co-authored-by: ziggie <ziggie1984@protonmail.com>
In this commit, we attempt to fix circular waiting scenario introduced
inadvertently when [fixing a race condition
scenario](https://github.com/lightningnetwork/lnd/pull/7856). With that
PR, we added a new channel that would block `Disconnect`, and
`WaitForDisconnect` to ensure that only until the `Start` method has
finished would those calls be allowed to succeed.
The issue is that if the server is trying to disconnect a peer due to a
concurrent connection, but `Start` is blocked on `maybeSendNodeAnn`,
which then wants to grab the main server mutex, then `Start` can never
exit, which causes `startReady` to never be closed, which then causes
the server to be blocked.
This PR attempts to fix the issue by calling `maybeSendNodeAnn` in a
goroutine, so it can grab the server mutex and not block the `Start`
method.
Fixes https://github.com/lightningnetwork/lnd/issues/7924
Fixes https://github.com/lightningnetwork/lnd/issues/7928
Fixes https://github.com/lightningnetwork/lnd/issues/7866
In this commit, we carry out a new notion introduced during a recent
spec meeting to use a feature bit plus 100 before the feature has been
finalized in the spec.
We split into the Final and Staging bits.
We use a helper function to ensure that anytime we're about to make a
normal sighash, we consult the channel type to check if we should use
the default value or sighash all explicitly.
In this commit, we remove the internal call to `InitRemoteMusigNonces`.
We don't need this since when we go to process the remote party's chan
reest message, we'll already call this method. Otherwise, we'll get an
error here since the pending verification nonce has been wiped out after
each call.
Olaoluwa Osuntokun