Merge bitcoin/bitcoin#33629: Cluster mempool

17cf9ff7ef Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general (Suhas Daftuar)
315e43e5d8 Sanity check `GetFeerateDiagram()` in CTxMemPool::check() (Suhas Daftuar)
de2e9a24c4 test: extend package rbf functional test to larger clusters (Suhas Daftuar)
4ef4ddb504 doc: update policy/packages.md for new package acceptance logic (Suhas Daftuar)
79f73ad713 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() (Suhas Daftuar)
a86ac11768 Update comments for CTxMemPool class (Suhas Daftuar)
9567eaa66d Invoke TxGraph::DoWork() at appropriate times (Suhas Daftuar)
6c5c44f774 test: add functional test for new cluster mempool RPCs (Suhas Daftuar)
72f60c877e doc: Update mempool_replacements.md to reflect feerate diagram checks (Suhas Daftuar)
21693f031a Expose cluster information via rpc (Suhas Daftuar)
72e74e0d42 fuzz: try to add more code coverage for mempool fuzzing (Suhas Daftuar)
f107417490 bench: add more mempool benchmarks (Suhas Daftuar)
7976eb1ae7 Avoid violating mempool policy limits in tests (Suhas Daftuar)
84de685cf7 Stop tracking parents/children outside of txgraph (Suhas Daftuar)
88672e205b Rewrite GatherClusters to use the txgraph implementation (Suhas Daftuar)
1ca4f01090 Fix miniminer_tests to work with cluster limits (Suhas Daftuar)
1902111e0f Eliminate CheckPackageLimits, which no longer does anything (Suhas Daftuar)
3a646ec462 Rework RBF and TRUC validation (Suhas Daftuar)
19b8479868 Make getting parents/children a function of the mempool, not a mempool entry (Suhas Daftuar)
5560913e51 Rework truc_policy to use descendants, not children (Suhas Daftuar)
a4458d6c40 Use txgraph to calculate descendants (Suhas Daftuar)
c8b6f70d64 Use txgraph to calculate ancestors (Suhas Daftuar)
241a3e666b Simplify ancestor calculation functions (Suhas Daftuar)
b9cec7f0a1 Make removeConflicts private (Suhas Daftuar)
0402e6c780 Remove unused limits from CalculateMemPoolAncestors (Suhas Daftuar)
08be765ac2 Remove mempool logic designed to maintain ancestor/descendant state (Suhas Daftuar)
fc4e3e6bc1 Remove unused members from CTxMemPoolEntry (Suhas Daftuar)
ff3b398d12 mempool: eliminate accessors to mempool entry ancestor/descendant cached state (Suhas Daftuar)
b9a2039f51 Eliminate use of cached ancestor data in miniminer_tests and truc_policy (Suhas Daftuar)
ba09fc9774 mempool: Remove unused function CalculateDescendantMaximum (Suhas Daftuar)
8e49477e86 wallet: Replace max descendant count with cluster_count (Suhas Daftuar)
e031085fd4 Eliminate Single-Conflict RBF Carve Out (Suhas Daftuar)
cf3ab8e1d0 Stop enforcing descendant size/count limits (Suhas Daftuar)
89ae38f489 test: remove rbf carveout test from mempool_limit.py (Suhas Daftuar)
c0bd04d18f Calculate descendant information for mempool RPC output on-the-fly (Suhas Daftuar)
bdcefb8a8b Use mempool/txgraph to determine if a tx has descendants (Suhas Daftuar)
69e1eaa6ed Add test case for cluster size limits to TRUC logic (Suhas Daftuar)
9cda64b86c Stop enforcing ancestor size/count limits (Suhas Daftuar)
1f93227a84 Remove dependency on cached ancestor data in mini-miner (Suhas Daftuar)
9fbe0a4ac2 rpc: Calculate ancestor data from scratch for mempool rpc calls (Suhas Daftuar)
7961496dda Reimplement GetTransactionAncestry() to not rely on cached data (Suhas Daftuar)
feceaa42e8 Remove CTxMemPool::GetSortedDepthAndScore (Suhas Daftuar)
21b5cea588 Use cluster linearization for transaction relay sort order (Suhas Daftuar)
6445aa7d97 Remove the ancestor and descendant indices from the mempool (Suhas Daftuar)
216e693729 Implement new RBF logic for cluster mempool (Suhas Daftuar)
ff8f115dec policy: Remove CPFP carveout rule (Suhas Daftuar)
c3f1afc934 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits (Suhas Daftuar)
47ab32fdb1 Select transactions for blocks based on chunk feerate (Suhas Daftuar)
dec138d1dd fuzz: remove comparison between mini_miner block construction and miner (Suhas Daftuar)
6c2bceb200 bench: rewrite ComplexMemPool to not create oversized clusters (Suhas Daftuar)
1ad4590f63 Limit mempool size based on chunk feerate (Suhas Daftuar)
b11c89cab2 Rework miner_tests to not require large cluster limit (Suhas Daftuar)
95a8297d48 Check cluster limits when using -walletrejectlongchains (Suhas Daftuar)
95762e6759 Do not allow mempool clusters to exceed configured limits (Suhas Daftuar)
edb3e7cdf6 [test] rework/delete feature_rbf tests requiring large clusters (glozow)
435fd56711 test: update feature_rbf.py replacement test (Suhas Daftuar)
34e32985e8 Add new (unused) limits for cluster size/count (Suhas Daftuar)
838d7e3553 Add transactions to txgraph, but without cluster dependencies (Suhas Daftuar)
d5ed9cb3eb Add accessor for sigops-adjusted weight (Suhas Daftuar)
1bf3b51396 Add sigops adjusted weight calculator (Suhas Daftuar)
c18c68a950 Create a txgraph inside CTxMemPool (Suhas Daftuar)
29a94d5b2f Make CTxMemPoolEntry derive from TxGraph::Ref (Suhas Daftuar)
92b0079fe3 Allow moving CTxMemPoolEntry objects, disallow copying (Suhas Daftuar)
6c73e47448 mempool: Store iterators into mapTx in mapNextTx (Suhas Daftuar)
51430680ec Allow moving an Epoch::Marker (Suhas Daftuar)

Pull request description:

  [Reopening #28676 here as a new PR, because GitHub is slow to load the page making it hard to scroll through and see comments.  Also, that PR was originally opened with a prototype implementation which has changed significantly with the introduction of `TxGraph`.]

  This is an implementation of the [cluster mempool proposal](https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393).

  This branch implements the following observable behavior changes:

   - Maintains a partitioning of the mempool into connected clusters (via the `txgraph` class), which are limited in vsize to 101 kvB by default, and limited in count to 64 by default.
   - Each cluster is sorted ("linearized") to try to optimize for selecting highest-feerate-subsets of a cluster first
   - Transaction selection for mining is updated to use the cluster linearizations, selecting highest feerate "chunks" first for inclusion in a block template.
   - Mempool eviction is updated to use the cluster linearizations, selecting lowest feerate "chunks" first for removal.
   - The RBF rules are updated to: (a) drop the requirement that no new inputs are introduced; (b) change the feerate requirement to instead check that the feerate diagram of the mempool will strictly improve; (c) replace the direct conflicts limit with a directly-conflicting-clusters limit.
   - The CPFP carveout rule is eliminated (it doesn't make sense in a cluster-limited mempool)
   - The ancestor and descendant limits are no longer enforced.
   - New cluster count/cluster vsize limits are now enforced instead.
   - Transaction relay now uses chunk feerate comparisons to determine the order that newly received transactions are announced to peers.

  Additionally, the cached ancestor and descendant data are dropped from the mempool, along with the multi_index indices that were maintained to sort the mempool by ancestor and descendant feerates. For compatibility (eg with wallet behavior or RPCs exposing this), this information is now calculated dynamically instead.

ACKs for top commit:
  instagibbs:
    reACK 17cf9ff7ef
  glozow:
    reACK 17cf9ff7ef
  sipa:
    ACK 17cf9ff7ef

Tree-SHA512: bbde46d913d56f8d9c0426cb0a6c4fa80b01b0a4c2299500769921f886082fb4f51f1694e0ee1bc318c52e1976d7ebed8134a64eda0b8044f3a708c04938eee7

doc/policy/mempool-replacements.md

@@ -12,12 +12,7 @@ other consensus and policy rules, each of the following conditions are met:
 
 1. (Removed)
 
-2. The replacement transaction only include an unconfirmed input if that input was included in
-   one of the directly conflicting transactions. An unconfirmed input spends an output from a
-   currently-unconfirmed transaction.
-
-   *Rationale*: When RBF was originally implemented, the mempool did not keep track of
-   ancestor feerates yet. This rule was suggested as a temporary restriction.
+2. (Removed)
 
 3. The replacement transaction pays an absolute fee of at least the sum paid by the original
    transactions.
@@ -38,23 +33,16 @@ other consensus and policy rules, each of the following conditions are met:
    *Rationale*: Try to prevent DoS attacks where an attacker causes the network to repeatedly relay
    transactions each paying a tiny additional amount in fees, e.g. just 1 satoshi.
 
-5. The number of original transactions does not exceed 100. More precisely, the sum of all
-   directly conflicting transactions' descendant counts (number of transactions inclusive of itself
-   and its descendants) must not exceed 100; it is possible that this overestimates the true number
-   of original transactions.
+5. The number of distinct clusters corresponding to conflicting transactions does not exceed 100.
 
-   *Rationale*: Try to prevent DoS attacks where an attacker is able to easily occupy and flush out
-   significant portions of the node's mempool using replacements with multiple directly conflicting
-   transactions, each with large descendant sets.
+   *Rationale*: Limit CPU usage required to update the mempool for so many transactions being
+   removed at once.
 
-6. The replacement transaction's feerate is greater than the feerates of all directly conflicting
-   transactions.
+6. The feerate diagram of the mempool must be strictly improved by the replacement transaction.
 
-   *Rationale*: This rule was originally intended to ensure that the replacement transaction is
-   preferable for block-inclusion, compared to what would be removed from the mempool. This rule
-   predates ancestor feerate-based transaction selection.
+   *Rationale*: This ensures that block fees in all future blocks will go up
+   after the replacement (ignoring tail effects at the end of a block).
 
-This set of rules is similar but distinct from BIP125.
 
 ## History
 
@@ -79,3 +67,5 @@ This set of rules is similar but distinct from BIP125.
 * Signaling for replace-by-fee is no longer required as of [PR 30592](https://github.com/bitcoin/bitcoin/pull/30592).
 
 * The incremental relay feerate default is 0.1sat/vB ([PR #33106](https://github.com/bitcoin/bitcoin/pull/33106)).
+
+* Feerate diagram policy enabled in conjunction with switch to cluster mempool as of **v31.0**.

doc/policy/packages.md

@@ -38,9 +38,7 @@ The following rules are enforced for all packages:
 
    - Packages are 1-parent-1-child, with no in-mempool ancestors of the package.
 
-   - All conflicting clusters (connected components of mempool transactions) must be clusters of up to size 2.
-
-   - No more than MAX_REPLACEMENT_CANDIDATES transactions can be replaced, analogous to
+   - The number of distinct clusters containing conflicting transactions can be no more than 100, analogous to
      regular [replacement rule](./mempool-replacements.md) 5).
 
    - Replacements must pay more total fees at the incremental relay fee (analogous to
@@ -56,18 +54,6 @@ The following rules are enforced for all packages:
      result in more robust fee bumping. More general package RBF may be
      enabled in the future.
 
-* When packages are evaluated against ancestor/descendant limits, the union of all transactions'
-  descendants and ancestors is considered. (#21800)
-
-   - *Rationale*: This is essentially a "worst case" heuristic intended for packages that are
-     heavily connected, i.e. some transaction in the package is the ancestor or descendant of all
-     the other transactions.
-
-* [CPFP Carve Out](./mempool-limits.md#CPFP-Carve-Out) is disabled in packaged contexts. (#21800)
-
-   - *Rationale*: This carve out cannot be accurately applied when there are multiple transactions'
-     ancestors and descendants being considered at the same time.
-
 The following rules are only enforced for packages to be submitted to the mempool (not
 enforced for test accepts):