These two messages will be used to implement the new and improved co-op
closing protocol. This PR also show cases how to use the new
`tlv.OptionalRecord` type to define and handle TLV level parsing.
I think we can make one additional helper function to clean up some of
the boiler plate for the encode/decode.
In this commit, we add a new `Zero` method for the `RecordT` type. This
method allows a caller to create the zero record for a type without
needing to reference the actual TLV type.
With this we go from this:
```go
sig1 := tlv.ZeroRecordT[tlv.TlvType1, Sig]()
```
To this:
```
sig1 := c.CloserNoClosee.Zero()
```
This builds on the concurrent queue to create a generic way to allow
goroutines to pub/sub information. An example includes being notified
each time a state machine is able to carry out a new state transition.
The error was never used as the init couldn't return an error, so we do
away with that. We also modify the main event loop dispatch to more
closely match other areas of the codebase.
In this commit, we make all calls to disconnect after a ping/pong
violation is detected in the `PingManager` async. We do this to avoid
circular waiting that may occur if the disconnect call back ends up
waiting on the peer goroutine to be torn down. If this happens, then the
peer goroutine will be blocked on the ping manager fully tearing down,
which is blocked on the peer disconnect succeeding.
This is a similar class of issue we've delt with recently as pertains to
the peer and the server: sync all back execution must not lead to
a circular waiting loop.
Fixes#8379
We add a WhenSome that'll pass in the actual underlying value, and not
just the record.
We also add `SomeRecord` to make it easier to create TLV records w/
`Some` set.
This will be useful when decoding optional TLV records, as we can use
this to look up in the `typeMap` for a given field to see if we decoded
it or not.
With this change, callers can now examine a struct instance of the
`TlvType` instance to figure out programmatically which integer type is
maps to. This'll be useful when decoding optional TLV types into a wire
struct (knowing when to set it to Some vs None).
In this commit, we modify the RecordT type to allow callers to re-use
the Record definition of a different type, but use the new type param to
override the integer type used on the wire.
This will let use do things like encode a signature using the same
RecordProducer instance, but with a diff type in another context.
The upcoming use for this is allowing our `lnwire.Sig` type to be
encoded in the same message using distinct TLV integer types (new co-op
close protocol).
In this commit, we add a new type, `RecordT[T, V]` to reduce some of the
common boiler plate for TLV types. This type lets you take either a
primitive type, or an existing Record, and gain common methods used to
create tlv streams.
It also serves as extra type annotation as well, since wire structs can
use this to wrap any existing type and gain the relevant record methods.
This implementation ensures that the very definition of the field also
binds the TLV type value. It does this by using the generated code to
map a struct like TlvType1 to an actually Type like Type(1).
This shouldn't need to be run again, as this implementation restricts
things to just values 0-99, due to a hard upper limit with the way Go
unions work under the hood.
In this commit, we add some new code generation to the codebase. As
we'll see in a future commit, this'll allow us to create a new Record[T,
V] type, where T is actually a concrete _struct_ that implements a
special interface that deems it as a valid TLV type.
In this commit, we make a new sub-module from the `fn` package. This
lets other projects use it w/o pulling in all of lnd, and also other
sub-packages within lnd.
We also add a temp replace directive for new module. We'll remove this
once we tag the final version.
We've only ever made macaroons with the v2 versions, so we should
explicitly reject those that aren't actually v2. We add a basic test
along the way, and also add a similar check for the version encoded in
the macaroon ID.
When we first go to boot up the syncer, when we're in the active phase,
after we do the historical sync, we'll send a timestamp message that we
want everything, then transition to the passive mode. The test didn't
account for this extra message before, as the last test was being
re-used here (ran in parallel).
We fix this by asserting that the first expected message is sent, then
also the follow up messages as well.
In this commit, [we prep for some upcoming changes in
Go](https://go.dev/blog/loopvar-preview) by running our tests with
`GOEXPERIMENT=loopvar`. This changes the loop semantics to fix a common
bug where a scoping issue causes a variable to be re-used, which can
cause unintended bugs down the line.
If everything passes with this flag on, then we can keep it on, and also
rest a bit easier knowing that the compiler will fix a class of bugs
that would previously pop up for us.
This PR is a follow up, to a [follow
up](https://github.com/lightningnetwork/lnd/pull/7938) of an [initial
concurrency issue](https://github.com/lightningnetwork/lnd/pull/7856)
fixed in the peer goroutine.
In #7938, we noticed that the introduction of `p.startReady` can cause
`Disconnect` to block. This happens as `Disconnect` cannot be called
until `p.startReady` has been closed. `Disconnect` is also called from
`InboundPeerConnected` (the case of concurrent peers, so we need to
remove one of the connections) while the main server mutex is held. If
`p.Start` blocks for any reason, then this leads to the deadlock as: we
can't disconnect until we've finished starting, and we can't finish
starting as we need the disconnect caller to exit as it has the mutex.
In this commit, we now make the call to `prunePersistentPeerConnection`
async. The call to `prunePersistentPeerConnection` eventually wants to
grab the server mutex, which triggers the circular waiting scenario
above.
The main learning here is that no calls to the main server mutex path
can block from `p.Start`. This is more or less a stop gap to resolve the
issue initially introduced in v0.16.4. Assuming we want to move forward
with this fix, we should reexamine `p.startReady` all together, and also
revisit attempt to refactor this section of the code to eliminate the
mega mutex in the server in favor of a dedicated event loop.
In this commit, we modify the incoming contest resolver to use a
concurrent queue. This is meant to ensure that the invoice registry
subscription loop never blocks. This change is meant to be minimal and
implements option `5` as outlined here:
https://github.com/lightningnetwork/lnd/issues/8023.
With this change, the inner loop of the subscription dispatch method in
the invoice registry will no longer block, as the concurrent queue uses
a fixed buffer of a queue, then overflows into another queue when that
gets full.
Fixes https://github.com/lightningnetwork/lnd/issues/7917
In this commit, we add the ability to obtain blocking and mutex
profiles. The blocking profile will show which goroutines are
consistently blocked on synchronization primitives like channels, or
I/O. The mutex profile will show which mutexes are very contested.
The blocking profile can be enabled with a new arg: `--blockingprofile`.
The mutex profile can be enabled with a new arg: `--mutexprofile`. These
are both ignored if the profile port isn't set.
Activating these profiles requires the caller to pass in a sampling
rate. For now I've set it just to `1` to test things out. Unfortunately
documentation is rather scarce, so there aren't any good guides re what
these values should be set to. AFAICT, these add more overhead than the
other prowling options, so they shouldn't necessarily be enabled
persistently in production.
In this commit, we update the set of protos to accept the local secret
nonces over RPC. This is actually a 97 byte value, as it includes the
two 32 byte nonces, as well as the 33 byte value of the public key of
the signer.
This is needed in order to be able to open taproot channels over the RPC
interface.
In this commit, we modify the musig2 interfaces to instead use an
explicit value for the local nonces. Before this commit, we used the
functional option, but we want to also support specifying this value
over RPC for the remote signer. The functional option pattern is opaque,
so we can't get the nonce value we need. To get around this, we'll just
make this an explicit pointer, then map this to the functional option at
the very last moment.
In this commit, we increase the max message size for the ws proxy. We
have a similar setting for the normal gRPC server which was tuned to be
able to support decoding `GetNetworkInfo` as the channel graph got
larger. We keep the default buffer size of 64 KB, but allow that to be
expanded to up to 4 MB (current value) to decode larger messages.
One alternative would be to modify the `Split` function to break up
larger lines into smaller ones. We'd need to double check that the
libraries at a higher level of abstraction can handle the chunks. The
scan function would look something like:
```go
splitFunc := func(data []byte, eof bool) (int, []byte, error) {
if len(data) >= chunkSize {
return chunkSize, data[:chunkSize], nil
}
return bufio.ScanLines(data, eof))
}
scanner.Split(splitFunc)
```
In this commit, update the start up logic to gracefully handle a
seemingly rare case. In this case, a peer detects local data loss with a
set of active HTLCs. These HTLCs then eventually expire (they may or may
not actually "exist"), causing a force close decision. Before this PR,
this attempt would fail with a fatal error that can impede start up.
To better handle such a scenario, we'll now catch the error when we fail
to force close due to entering the DLP and instead terminate the state
machine at the broadcast state. When a commitment transaction eventually
confirms, we'll play it as normal.
Fixes https://github.com/lightningnetwork/lnd/issues/7984
Olaoluwa Osuntokun