This allows incoming connections from peers which are only banned
due to an automatic misbehavior ban if doing so won't fill inbound.
These peers are preferred for eviction when inbound fills, but may
still be kept if they fall into the protected classes. This
eviction preference lasts the entire life of the connection even
if the ban expires.
If they misbehave again they'll still get disconnected.
The main purpose of banning on misbehavior is to prevent our
connections from being wasted on unhelpful peers such as ones
running incompatible consensus rules. For inbound peers this
can be better accomplished with eviction preferences.
A secondary purpose was to reduce resource waste from repeated
abuse but virtually any attacker can get a nearly unlimited
supply of addresses, so disconnection is about the best we can
do.
The largest sensible size for a locator is log in the number of blocks.
But, as noted by Coinr8d on BCT a maximum size message could encode a
hundred thousand locators. If height were used to limit the messages
that could open new attacks where peers on long low diff forks would
get disconnected and end up stuck.
Ideally, nodes first first learn to limit the size of locators they
send before limiting what would be processed, but common implementations
back off with an exponent of 2 and have an implicit limit of 2^32
blocks, so they already cannot produce locators over some size.
This sets the limit to an absurdly high amount of 101 in order to
maximize compatibility with existing software.
Safemode is almost useless as is-- it only triggers in limited
cases most of which aren't even concerning. There have been
several proposals to remove it. But as a simpler, safer, and
more flexible first case, simply deactivate it by default.
Anyone who wants it can re-enable and know what they've signed up for.
Some people keep thinking that MAX_BLOCK_BASE_SIZE is a separate
size limit from the weight limit when it fact it is superfluous,
and used in early tests before the witness data has been
validated or just to compute worst case sizes. The size checks
that use it would not behave any differently consensus wise
if they were eliminated completely.
Its correct value is not independently settable but is a function
of the weight limit and weight formula.
This patch just eliminates it and uses the scale factor as
required to compute the worse case constants.
It also moves the weight factor out of primitives into consensus,
which is a more logical place for it.
In Olaoluwa Osuntokun's recent protocol proposal they were using a
mod in an inner loop. I wanted to suggest a normative protocol
change to use the trick we use here, but to find an explanation
of it I had to dig up the PR on github. After I posted about it
several other developers commented that it was very interesting
and they were unaware of it.
I think ideally the code should be self documenting and help
educate other contributors about non-obvious techniques that
we use. So I've written a description of the technique with
citations for future reference.
There is no point in even hashing a submitted block which doesn't have
a coinbase transaction.
This also results in more useful error reporting on corrupted input.
Thanks to rawodb for the bug report.
This changes the logging categories to boolean flags instead of strings.
This simplifies the acceptance testing by avoiding accessing a scoped
static thread local pointer to a thread local set of strings. It
eliminates the only use of boost::thread_specific_ptr outside of
lockorder debugging.
This change allows log entries to be directed to multiple categories
and makes it easy to change the logging flags at runtime (e.g. via
an RPC, though that isn't done by this commit.)
It also eliminates the fDebug global.
Configuration of unknown logging categories now produces a warning.
The old Bitcoin alert system has long since been retired.
( See also: https://bitcoin.org/en/alert/2016-11-01-alert-retirement )
This change causes each node to send any old peers that
it connects with a copy of the final alert.
The alert it hardcode cancels all other alerts including
other final alerts.
The big notice at the top of the release note is not interesting
to most users now and apparently comes across poorly to some.
Better to provide more information about what we do support.
This disentangles the script validation skipping from checkpoints.
A new option is introduced "assumevalid" which specifies a block whos
ancestors we assume all have valid scriptsigs and so we do not check
them when they are also burried under the best header by two weeks
worth of work.
Unlike checkpoints this has no influence on consensus unless you set
it to a block with an invalid history. Because of this it can be
easily be updated without risk of influencing the network consensus.
This results in a massive IBD speedup.
This approach was independently recommended by Peter Todd and Luke-Jr
since POW based signature skipping (see PR#9180) does not have the
verifiable properties of a specific hash and may create bad incentives.
The downside is that, like checkpoints, the defaults bitrot and older
releases will sync slower. On the plus side users can provide their
own value here, and if they set it to something crazy all that will
happen is more time will be spend validating signatures.
Checkblocks and checklevel are also moved to the hidden debug options:
Especially now that checkblocks has a low default there is little need
to change these settings, and users frequently misunderstand them as
influencing security or IBD speed. By hiding them we offset the
space added by this new option.
In spite of the name FindLatestBefore used std::lower_bound to try
to find the earliest block with a nTime greater or equal to the
the requested value. But lower_bound uses bisection and requires
the input to be ordered with respect to the comparison operation.
Block times are not well ordered.
I don't know what lower_bound is permitted to do when the data
is not sufficiently ordered, but it's probably not good.
(I could construct an implementation which would infinite loop...)
To resolve the issue this commit introduces a maximum-so-far to the
block indexes and searches that.
For clarity the function is renamed to reflect what it actually does.
An issue that remains is that there is no grace period in importmulti:
If a address is created at time T and a send is immediately broadcast
and included by a miner with a slow clock there may not yet have been
any block with at least time T.
The normal rescan has a grace period of 7200 seconds, but importmulti
does not.
Previously addnodes were in competition with outbound connections
for access to the eight outbound slots.
One result of this is that frequently a node with several addnode
configured peers would end up connected to none of them, because
while the addnode loop was in its two minute sleep the automatic
connection logic would fill any free slots with random peers.
This is particularly unwelcome to users trying to maintain links
to specific nodes for fast block relay or purposes.
Another result is that a group of nine or more nodes which are
have addnode configured towards each other can become partitioned
from the public network.
This commit introduces a new limit of eight connections just for
addnode peers which is not subject to any of the other connection
limitations (including maxconnections).
The choice of eight is sufficient so that under no condition would
a user find themselves connected to fewer addnoded peers than
previously. It is also low enough that users who are confused
about the significance of more connections and have gotten too
copy-and-paste happy will not consume more than twice the slot
usage of a typical user.
Any additional load on the network resulting from this will likely
be offset by a reduction in users applying even more wasteful
workaround for the prior behavior.
The retry delays are reduced to avoid nodes sitting around without
their added peers up, but are still sufficient to prevent overly
aggressive repeated connections. The reduced delays also make
the system much more responsive to the addnode RPC.
Ban-disconnects are also exempted for peers added via addnode since
the outbound addnode logic ignores bans. Previously it would ban
an addnode then immediately reconnect to it.
A minor change was also made to CSemaphoreGrant so that it is
possible to re-acquire via an object whos grant was moved.
Performing signing in the inner loop has terrible performance
when many passes through are needed to complete the selection.
Signing before the algorithm is complete also gets in the way
of correctly setting the fee (e.g. preventing over-payment when
the fee required goes down on the final selection.)
Use of the dummy might overpay on the signatures by a couple bytes
in uncommon cases where the signatures' DER encoding is smaller
than the dummy: Who cares?
This resolves an issue where a wallet transaction which failed to
relay previously because it couldn't make it into the mempool
will not try again until restart, even though mempool conditions
may have changed.
Abandoned and known-conflicted transactions are skipped.
Some concern was expressed that there may be users with many
unknown conflicts would waste a lot of CPU time trying to
add them to their memory pools over and over again. But I am
doubtful these users exist in any number, if they do exist
they have worse problems, and they can mitigate any performance
issue this might have by abandoning the transactions in question.
When a BIP152 HB-mode peer is in the least preferred position and
disconnects, they will not be by ForNode on the next loop. They
will continue to sit in that position and prevent deactivating
HB mode for peers that are still connected.
There is no reason for them to stay in the list if already gone,
so drop the first element unconditionally if there are too many.
Fixes issue #9163.
In 0.13 orphan transactions began being treated as implicit
INVs for their parents. But the resulting getdata were
not getting the witness flag.
This fixes issue #9182 reported by chjj and roasbeef on IRC.
GetTotalBlocksEstimate is no longer used and it was the only thing
the checkpoint tests were testing.
Since checkpoints are on their way out it makes more sense to remove
the test file than to cook up a new pointless test.
This introduces a 'minimum chain work' chainparam which is intended
to be the known amount of work in the chain for the network at the
time of software release. If you don't have this much work, you're
not yet caught up.
This is used instead of the count of blocks test from checkpoints.
This criteria is trivial to keep updated as there is no element of
subjectivity, trust, or position dependence to it. It is also a more
reliable metric of sync status than a block count.
We normally prefer to connect to peers offering the relevant services.
If we're not connected to enough peers with relevant services, we
probably don't know about them and could use dnsseed's help.
Only allow skipping relevant services until there are four outbound
connections up.
This avoids quickly filling up with peers lacking the relevant
services when addrman has few or none of them.
We should learn about new peers via address messages.
An inbound peer connecting to us tells us nothing about
its ability to accept incoming connections from us, so
we shouldn't assume that we can connect to it based on
this.
The vast majority of nodes on the network do not accept
incoming connections, adding them will only slow down
the process of making a successful connection in the
future.
Nodes which have configured themselves to not announce would prefer we
not violate their privacy by announcing them in GETADDR responses.
An orphan whos parents were rejected is never going to connect, so there
is little utility in keeping it.
Orphans also helpfully tell us what we're missing, so go ahead and treat
it as INVed.
This prevents higher order orphans and other junk from
holding positions in the orphan map. Parents delayed
twenty minutes are more are unlikely to ever arrive.
The freed space will improve the orphan matching success rate for
other transactions.
always grow to its maximum size.
This does not go so far as to attempt to connect orphans made
connectable by a new block.
Keeping the orphan map less full helps improve the reliability
of relaying chains of transactions.
This reduces the rate of not founds by better matching the far
end expectations, it also improves privacy by removing the
ability to use getdata to probe for a node having a txn before
it has been relayed.
Previously the benchmark code used an integer division (%) with
a non-constant in the inner-loop. This is quite slow on many
processors, especially ones like ARM that lack a hardware divide.
Even on fairly recent x86_64 like haswell an integer division can
take something like 100 cycles-- making it comparable to the
runtime of siphash.
This change avoids the division by using bitmasking instead. This
was especially easy since the count was only increased by doubling.
This change also restarts the timing when the execution time was
very low this avoids mintimes of zero in cases where one execution
ends up below the timer resolution. It also reduces the impact of
the overhead on the final result.
The formatting of the prints is changed to not use scientific
notation make it more machine readable (in particular, gnuplot
croaks on the non-fixedpoint, and it doesn't sort correctly).
This also hoists out all the floating point divisions out of the
semi-hot path because it was easy to do so.
It might be prudent to break out the critical test into a macro
just to guarantee that it gets inlined. It might also make sense
to just save out the intermediate counts and times and get the
floating point completely out of the timing loop (because e.g.
on hardware without a fast hardware FPU like some ARM it will
still be slow enough to distort the results). I haven't done
either of these in this commit.
If a node is offline failed outbound connection attempts will crank up
the addrman counter and effectively blow away our state.
This change reduces the problem by only counting attempts made while
the node believes it has outbound connections to at least two
netgroups.
Connect and addnode connections are also not counted, as there is no
reason to unequally penalize them for their more frequent
connections -- though there should be no real effect from this
unless their addnode configureation is later removed.
Wasteful repeated connection attempts while only a few connections are
up are avoided via nLastTry.
This is still somewhat incomplete protection because our outbound
peers could be down but not timed out or might all be on 'local'
networks (although the requirement for multiple netgroups helps).
The ability to GETDATA a transaction which has not (yet) been relayed
is a privacy loss vector.
The use of the mempool for this was added as part of the mempool p2p
message and is only needed to fetch transactions returned by it.
Some developers clearly don't get this and have been posting
"improvements" that create clear vulnerabilities. It should
have been better explained in the code, since the design
is somewhat subtle and getting it right is important.
This will avoid sending more pointless INVs around updates, and
prevents using filter updates to timetag transactions.
Also adds locking for fRelayTxes.
Previously Bitcoin would send 1/4 of transactions out to all peers
instantly. This causes high overhead because it makes >80% of
INVs size 1. Doing so harms privacy, because it limits the
amount of source obscurity a transaction can receive.
These randomized broadcasts also disobeyed transaction dependencies
and required use of the orphan pool. Because the orphan pool is
so small this leads to poor propagation for dependent transactions.
When the bypass wasn't in effect, transactions were sent in the
order they were received. This avoided creating orphans but
undermines privacy fairly significantly.
This commit:
Eliminates the bypass. The bypass is replaced by halving the
average delay for outbound peers.
Sorts candidate transactions for INV by their topological
depth then by their feerate (then hash); removing the
information leakage and providing priority service to
higher fee transactions.
Limits the amount of transactions sent in a single INV to
7tx/sec (and twice that for outbound); this limits the
harm of low fee transaction floods, gives faster relay
service to higher fee transactions. The 7 sounds lower
than it really is because received advertisements need
not be sent, and because the aggregate rate is multipled
by the number of peers.
This corrects a bug the case of tying group size where the code may
fail to select the group with the newest member. Since newest time
is the final selection criteria, failing to break ties on it
on the step before can undermine the final selection.
Tied netgroups are very common.
(cherry picked from commit 8e09f914f8ec66301257358b250e9a61befadd95)
With automatic tor HS support in place we should probably not be providing
absolute protection for local peers, since HS inbound could be used to
attack pretty easily. Instead, this counts on the latency metric inside
AttemptToEvictConnection to privilege actually local peers.
(cherry picked from commit 46dbcd4833115401fecbb052365b4c7725874414)
Also renames whitelistalwaysrelay.
Nodes relay all transactions from whitelisted peers, this
gets in the way of some useful reasons for whitelisting
peers-- for example, bypassing bandwidth limitations.
The purpose of this forced relaying is for specialized gateway
applications where a node is being used as a P2P connection
filter and multiplexer, but where you don't want it getting
in the way of (re-)broadcast.
This change makes it configurable with whitelistforcerelay.
Mempool requests use a fair amount of bandwidth when the mempool is large,
disconnecting peers using them follows the same logic as disconnecting
peers fetching historical blocks.
Mruset setInventoryKnown was reduced to a remarkably small 1000
entries as a side effect of sendbuffer size reductions in 2012.
This removes setInventoryKnown filtering from merkleBlock responses
because false positives there are especially unattractive and
also because I'm not sure if there aren't race conditions around
the relay pool that would cause some transactions there to
be suppressed. (Also, ProcessGetData was accessing
setInventoryKnown without taking the required lock.)
The setAskFor duplicate elimination was too eager and removed entries
when we still had no getdata response, allowing the peer to keep
INVing and not responding.
80ae230 Improve log messages for blocks only violations. (Patick Strateman)
08843ed Add relaytxes status to getpeerinfo (Peter Todd)
d8aaa51 Bail early in processing transactions in blocks only mode. (Patick Strateman)
3587f6a Fix relay mechanism for whitelisted peers under blocks only mode. (Patick Strateman)
6531f17 Add mediantime field to getblock and getblockheader (Peter Todd)
7259769 Document new mediantime field in getblockchaininfo (Peter Todd)
c277a63 Clarify nLockTime-by-time comment in CheckFinalTx() (Peter Todd)
748321e Add mediantime field to getblockchaininfo RPC call (Peter Todd)
bbf49da Fix comment for blocksonly parameter interactions (Patick Strateman)
6a4982f Fix fRelayTxs comment (Patick Strateman)
59441a0 Display DEFAULT_WHITELISTALWAYSRELAY in help text (Patick Strateman)
71a2683 Use DEFAULT_BLOCKSONLY and DEFAULT_WHITELISTALWAYSRELAY constants (Patick Strateman)
762b13b Add help text for blocksonly and whitelistalwaysrelay (Patick Strateman)
3a96497 Add whitelistalwaysrelay option (Patick Strateman)
420fa81 Do not process tx inv's in blocksonly mode (Patick Strateman)
4044f07 Add blocksonly mode (Patick Strateman)
10953a7 Better error message for fundrawtransaction w/ empty vout (Peter Todd)
61e1eb2 Actually use includeWatching value in fundrawtransaction (Peter Todd)
Gregory Maxwell