This commit modifies WriteMessage to only perform encryption on the
passed plaintext, and buffer the ciphertext within the connection
object. We then modify internal uses of WriteMessage to follow with a
call to Flush, which actually writes the message to the wire.
Additionally, since WriteMessage does not actually perform the write
itself, the io.Writer argument is removed from the function signature
and all call sites.
This commit adds a Flush method to the brontide.Machine, which can write
out a buffered message to an io.Writer. This is a preliminary change
which will allow the encryption of the plaintext to be done in a
distinct method from actually writing the bytes to the wire.
This commit adds a brief delay when sending our channel reestablish
message if the link contains a restored channel to ensure we first have
a stable connection. Sending the message will cause the remote peer to
force close the channel, which currently may not be resumed reliably if
the connection is being torn town simultaneously. This delay can be
removed after the force close is reliable, but in the meantime it
improves the reliability of successfully closing out the channel and
allows the `channel_backup_restore/restore_during_creation` to pass
reliably.
This commit removes the MarkEdgeZombie method from channeldb. This
method is currently not used in any live code paths in production, and
is only used in unit tests. However, incorrect usage of this method
could result in an edge being present in both the zombie and channel
indexes, which deviates from any state we would expect to see in
production. Removing the method will help mitigate the potential for
writing incorrect unit tests in the future, by forcing zombie edges to
be created via the relevant, production APIs, e.g. DeleteChannelEdge.
The existing unit tests that use this method have been modified to use
the DeleteChannelEdge instead. No regressions were discovered in the
process.
This commit modifies FetchChanInfos to skip any channels that are not in
the graph at the time of the call. Currently the entire call will fail
if the edge is not found, which stalls a gossip sync in the following
scenario:
1. Remote peer queries for a channel range
2. We return the set of channel ids in that range
3. A channel from that set is removed from the graph, e.g. via close.
4. Remote peer queries for removed edge, causing the query to fail.
To remedy this, we will now skip any edges that are not known in the
database at the time of the query. This prevents the syncer state
machines from halting, which otherwise could only be resolved by
disconnecting and reconnecting.
This commit reduces the number of channels a syncer will request from
the remote node in a single QueryShortChanIDs message. The current size
is derived from the chunkSize, which is meant to signal the maximum
number of short chan ids that can fit in a single ReplyChannelRange
message. For EncodingSortedPlain, this number is 8000, and we use the
same number to dictate the size of the batch from the remote peer.
We modify this by introducing a separately configurable batchSize, so
that both can be tuned independently. The value is chosen to reduce the
amount of buffering the remote party will perform, only requiring them
queue 500 responses, as opposed to 8000. In turn, this reduces larges
spikes in allocation on the remote node at the expense of a few extra
round trips for the control messages. However, will be negligible since
the control messages are much smaller than the messages being returned.
This commit modifies FilterKnownChanIDs to skip edges that
we ourselves have deemed zombies. This prevents us from requesting
the updates from them, as this wastes bandwidth and cpu cycles.
This commit adds optional jitter to our initial reconnection to our
persistent peers. Currently we will attempt reconnections to all peers
simultaneously, which results in large amount of contention as the
number of channels a node has grows.
We resolve this by adding a randomized delay between 0 and 30 seconds
for all persistent peers. This spreads out the load and contention to
resources such as the database, read/write pools, and memory
allocations. On my node, this allows to start up with about 80% of the
memory burst compared to the all-at-once approach.
This also has a second-order effect in better distributing messages sent
at constant intervals, such as pings. This reduces the concurrent jobs
submitted to the read and write pools at any given time, resulting in
better reuse of read/write buffers and fewer bursty allocation and
garbage collection cycles.
This prevents a panic during test failure due to a child test calling
FailNow on a parent test context. The sub tests now capture the
testing.T object provided in each closure as opposed to ignoring it and
using the parent context.
This commit fixes a bug that would cause the
notifier not to commit spend hints for items
that are not found. This is done by calling
UpdateSpendDetails with a nil detail, permitting
the notifier to begin updating the spend hints
with new blocks that arrive at tip. The change
is designed to mimic the behavior for historical
confirmation dispatch.
The symptom of this bug is needing to do many
long rescans on startup, even if new blocks
arrive after the rescan had completed. With
this change, nodes will have to do the scans
once more before their hints will be properly
updated. Restarts from then on should not
have this behavior.
This commit introduces the Validator interface, which
is intended to be implemented by any sub configs. It
specifies a Validate() error method that should fail
if a sub configuration contains any invalid or insane
parameters.
In addition, a package-level Validate method can be
used to check a variadic number of sub configs
implementing the Validator interface. This allows the
primary config struct to be extended via targeted
and/or specialized sub configs, and validate all of
them in sequence without bloating the main package
with the actual validation logic.
Bumps the default read and write handlers to be well
above the average number of peers a node has. Since
the worker counts specify only a maximum number of
concurrent read/write workers, it is expected that
the actual usage would converge to the requirements
of the node anyway. However, in preparation for a
major release, this is a conservative measure to
ensure that the default values aren't too low and
improve network instability.
In this commit, we add a 5 minute idle timer to
the write handler. After catching the write
timeouts, it's been observed that some connections
have trouble reading a message for several hours.
This typically points to a deeper issue w/ the peer
or, e.g. the remote peer switched networks. This now
mirrors the idle timeout used in the read handler,
such that we will disconnect a peer if we are unable
to send or receive a message from the peer after 5
minutes.
We also modify the readHandler to drain its
idleTimer's channel in the even that the timer had
already fired, but we successfully sent the message.
This commit reduces the peer's write timeout to 5s.
Now that the peer catches write timeouts and doesn't
disconnect, this will ensure we spend less time blocking
in the write pool in case others also need to access the
workers concurrently. Slower peers will now only block
for 5s, after every reattempt w/ exponential backoff.
This commit modifies the writeHandler to catch timeout
errors, and retry writes to the socket after a small
backoff, which increases exponentially from 5s to 1m.
With the growing channel graph size, some lower-powered
devices can be slow to pull messages off the wire during
validation. The current behavior will cause us to
disconnect the peer, and resend all of the messages that
the remote peer is slow to validate. Catching the timeout
helps in preventing such expensive reconnection cycles,
especially as the network continues to grow.
This is also a preliminary step to reducing the
write timeout constant. This will allow concurrent usage
of the write pools w/out devoting excessive amounts of
time blocking the pool for slow peers.
This commit modifies the breach arbiter to monitor
all breached inputs for spends and remove them from
the set of inputs to be swept if they are spent to
a terminal state. Prior, we would only watch for
spends on htlcs that may need to transition and
sweep the corresponding second-level htlc.
With these changes, we will no monitor commitment
outputs for spends, as well as spends from the
second level htlcs themselves. If either of these
is detected, we remove them from the set of inputs
to sweep via the justice transaction because there
is nothing the breach arbiter can do.
This functionality will be needed when adding
watchtower support, as the breach arbiter must
detect the case when the tower sweeps on behalf of
the user and stop pursuing the sweep itself. In
addition, this now properly handles the potential
case where somehow the remote party is able to sweep
the their commitment or second-level htlc to their
wallet, and prevent the breach arbiter from trying
to sweep the outputs as it would now.
Note that in the latter event, the internal
accounting may still be incorrect, as it is assumed
that all breached funds return to the victim.
However, these issues will deferred and fixed at a
later date, as the more crucial aspect is that the
breach arbiter doesn't blow up as a result of towers
sweeping channels.
Increases the default window from 250 to 2.5k. Many
users have reported attempting recovery with the
default value only to find an empty wallet. This
change should help ensure that the first recovery
attempt succeeds for the majority of nodes that
have modest load. It might prudent to consider
increasing this value further in the future if
the issue persists or average node age increases.
Modifies the netann.ChanStatusManager to remove outpoints
from the set of tracked channels when it encounters a
channeldb.ErrEdgeNotFound when passively disabling. It is
possible for this to occur if a channel is closed w/o first
being disabled, which may happen if the remote party force
closes.
Conner Fromknecht