It can take some time to unmarshal large mission control data sets such
that the macaroon can expire during that phase. We postpone the
connection establishment to give the user more time to answer the
prompt.
Mission control may have outdated success/failure amounts for node pairs
that have channels with differing capacities. In that case we assume to
still find the liquidity as before and rescale the amounts to the
according range.
We skip the evaluation of probabilities when the amount is lower than
the last success amount, as the probability would be evaluated to 1 in
that case.
If the success and fail amounts indicate that a channel doesn't obey a
bimodal distribution, we fall back to a uniform/linear success
probability model. This also helps to avoid numerical normalization
issues with the bimodal model.
This is achieved by adding a very small summand to the balance
distribution P(x) ~ exp(-x/s) + exp((x-c)/s), 1/c that helps to
regularize the probability distribution. The distribution becomes finite
for intermediate balances where the exponentials would be evaluated to
an exact zero (float) otherwise. This regularization is effective in
edge cases and leads to falling back to a uniform model should the
bimodal model fail.
This affects the normalization to be s * (-2 * exp(-c/s) + 2 + 1/s) and
the primitive function to receive an extra term x/(cs).
The previously added fuzz seed is expected to be resolved with this.
This test demonstrates an error found in a fuzz test by adding a
previously found seed, which will be fixed in an upcoming commit.
The following fuzz test is expected to fail:
go test -v -fuzz=Prob ./routing/
The bimodal model doesn't depend on the unit, which is why updating to
more realistic values doesn't require changes in tests.
We pin the scale in the fuzz test to not invalidate the corpus.
We can remove paymentFailureInfo since we can gate the result
interpretation on the source index, meaning that if we don't have a
source index, we deal with an unknown payment outcome because we
couldn't pinpoint the failing hop.
We shift the duty of determining the policies to the backward pass as
the forward pass will only be responsible for finding the corrected
receiver amount.
Note that this is not a pure refactor as demonstrated in the test, as
the forward pass doesn't select new policies anymore, which is less
flexible and doesn't lead to the highest possible receiver amount. This
is however neccessary as we otherwise won't be able to compute
forwarding amounts involving inbound fees and this edge case is unlikely
to occur, because we search for a min amount for a route that was most
likely constructed for a larger amount.
We split up the functionality in getRouteUnifiers into checking that all
edges exist via getEdgeUnifiers and then add a backward pass that will
be responsible for determining the sender amount.
We remove the node pub key from the error string, as in route building
this is duplicate info, which can be determined from the input keys,
further it's not available in the backward pass anymore.
We refactor the BuildRoute test to use the require library and add a
test case for a max HTLC violation on the last hop.
The time lock weight for a hop is supposed to be proportional to the
amount that is sent/locked, but in a previous change we switched to the
net amount, where inbound fees aren't yet applied. This is corrected in
this commit.
When iterating edges, pathfinding checks early whether using an edge
would violate the requested total fee limit for a route. This check is
done on the net amount (an amount the inbound fee is calculated with).
However, a possible next hop's fee discount leads to a reduction in fees
and as such using the net amount leads to assuming a higher cumulative
fee than the route really has, excluding the path erroneously. We
perform the fee limit check on the amount to send, which includes both
inbound and outbound fees. This should be possible as the first hop's
outbound fee is zero and therefore doesn't have to be checked in the
end.
The process how we calculate a total probability from the direct and
node probability is modified to give more importance to the direct
probability.
This is important if we know about a recent failure. We should not try
to send over this channel again if we know we can't. Otherwise this can
lead to infinite retrials over the channel, when the probability is
pinned at high probabilities by the other node results.
Having a capacity available is important for liquidity estimation.
* We add a dummy capacity to hop hints. Hop hints specify neither the
capacity nor a maxHTLC size, which is why we assume the channel to
have a high capacity relative to the amount we send.
* We add a capacity to local edges. These channels should always have a
capacity associated with them, even in the neutrino case (a fallback
to maxHTLC is not necessary). This is just for completeness, as the
probability calculation for local channels is done separately.
* The maximal reduction in the probability is limited to 0.5 (previously
~0.05), such that we don't get too low apriori probabilities.
Otherwise, this may lead to a too strong selection of large (and maybe
expensive) channels. A two-hop path would get total probability
penalties of:
- 1000PPM/(0.6*0.6) = 2778 PPM in the unsaturated case
- 1000PPM/(0.6*(0.6*0.5)) = 5556 PPM in the saturated case, where the
second hop is saturated
The difference in PPM of 2778 PPM should be enough to bias towards the
first path.
* The smearing factor is reduced. Previously we had to keep a higher
smearing factor in order to make the capacity factor not go to zero
for high amounts, to still give a fully saturated channel a chance.
This is not needed anymore due to the capping to 0.5. A lower value of
the smearing factor lets us more precisely choose a capacity fraction
and the capacity factor is more neutral when it comes to intermediate
amounts.
We set a conservative default value for the capacity fraction, which
still has the effect of discarding exhausted channels, giving a
noticeable effect when about 90% of the capacity is being used.
We make the capacity factor configurable via an lnd.conf routerrpc
apriori parameter. The capacity factor trades off increased success
probability with a reduced set of channel candidates, which may lead to
increased fees. To let users choose whether the factor is active or not,
we add a config setting where a capacity fraction of 1.0 disables the
factor. We limit the capacity fraction to values between 0.75 and 1.0.
Lower values may discard too many channels.
We require channel updates to have the max HTLC message flag set.
Several flows need to pass that check before channel updates are
forwarded to peers:
* after channel funding: `addToRouterGraph`
* after receiving channel updates from a peer:
`ProcessRemoteAnnouncement`
* after we update channel policies: `PropagateChanPolicyUpdate`
We rename `ChanUpdateOptionMaxHtlc` to `ChanUpdateRequiredMaxHtlc`
as with the latest changes it is now required.
Similarly, rename `validateOptionalFields` to
`ValidateChannelUpdateFields`, export it to use it in a later commit.
The active probability estimator can be switched dynamically using the
`Set/GetMissionControl` API, maintaining backward compatibility. The
lncli commands `setmccfg` and `getmccfg` are updated around this
functionality. Note that deprecated configuration parameters were
removed from the commands.
We use a more general `Estimator` interface for probability estimation
in missioncontrol.
The estimator is created outside of `NewMissionControl`, passed in as a
`MissionControlConfig` field, to facilitate usage of externally supplied
estimators.
Implements a new probability estimator based on a probability theory
framework.
The computed probability consists of:
* the direct channel probability, which is estimated based on a
depleted liquidity distribution model, formulas and broader concept derived
after Pickhardt et al. https://arxiv.org/abs/2103.08576
* an extension of the probability model to incorporate knowledge decay
after time for previous successes and failures
* a mixed node probability taking into account successes/failures on other
channels of the node (similar to the apriori approach)
We introduce a probability `Estimator` interface which is implemented by
the current apriori probability estimator. A second implementation, the
bimodal probability estimator follows.
* we rename the current probability estimator to be the "apriori"
probability estimator to distinguish from a different implementation
later
* the AprioriEstimator is exported to later be able to type switch
* getLocalPairProbability -> LocalPairProbabiltiy (later part of an
exported interface)
* getPairProbability -> getPairProbabiltiy (later part of an exported
interface)
We deprecate `QueryProbability`, as it displays the same information as
`QueryMissionControl` less the probability. `QueryRoutes` still contains
the total probability of a route.
We multiply the apriori probability with a factor to take capacity into
account:
P *= 1 - 1 / [1 + exp(-(amount - cutoff)/smearing)]
The factor is a function value between 1 (small amount) and 0 (high
amount). The zero limit may not be reached exactly depending on the
smearing and cutoff combination. The function is a logistic function
mirrored about the y-axis. The cutoff determines the amount at which a
significant reduction in probability takes place and the smearing
parameter defines how smooth the transition from 1 to 0 is. Both, the
cutoff and smearing parameters are defined in terms of fixed fractions
of the capacity.
Extends the pathfinder with a capacity argument for later usage.
In tests, the inserted testCapacity has no effect, but will be used
later to estimate reduced probabilities from it.
This commit adds the maximal capacity between two nodes to the unified
edge data. We use MaxHTLC as a replacement if the channel capacity is
not available.
In tests we use larger maxHTLC values to be able to convert to a
non-zero sat capacity.
This commit refactors the semantics of unified policies to unified
edges. The main changes are the following renamings:
* unifiedPolicies -> nodeEdgeUnifier
* unifiedPolicy -> edgeUnifier
* unifiedPolicyEdge -> unifiedEdge
Comments and shortened variable names are changed to reflect the new
semantics.
We encapsulate the capacity inside a unifiedPolicyEdge for later usage.
The meaning of "policy" has changed now, which will be refactored in the
next commmit.
The bitcoind .cookie contains an autogenerated user (__cookie__) and
password (random string), which can be used instead of the rpc user name
and password. This commit allows for running against bitcoind without
having to access bitcoin.conf like in the case for pure
user/password/zmq configuration.
Changes the grpc proto file, generates the protobuf, and
enables a queried way to retrieve payments in the rpc, where
backward compatibility is enforced by returning all payments
in the database by default. Adds a payment index field to
the returned payments of the rpc call.
Adds a PaymentsQuery struct, which contains parameters to restrict the
response of QueryPayments, returning a PaymentsQuerySlice with the
payments query result. The behavior of this api is the same as
the QueryInvoices one.
This commit renames the `reversed` pagination flag to
`paginate-forwards`, which is off by default. In order to
access older invoices one can set the paginate-forwards flag,
which is more intuitive than setting the reversed flag to false.
bitromortac