graph/db: convert NodeUpdatesInHorizon to use iterators

In this commit, we refactor the NodeUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.

The new implementation uses Go 1.23's iter.Seq type to provide a
standard iterator interface. The method now supports configurable batch
sizes through functional options, allowing fine-tuned control over
memory usage and performance characteristics.

Rather than reading all the entries from disk into memory (before this
commit, we did consult the cache for most entries, skipping the disk
hits), we now expose a chunked iterator instead.

We also make the process of filtering out public nodes first class. This
saves many newly created db transactions later.
This commit is contained in:
Olaoluwa Osuntokun committed 2025-09-26 16:56:41 -07:00
1 parent f8ce00b84a
commit 1d6d54e5db
5 files changed
+206 -32

No files matched your search

+10 -2
View File
@@ -7,6 +7,7 @@ import (
"encoding/hex"
"errors"
"fmt"
"iter"
"maps"
"math"
"net"
@@ -553,7 +554,8 @@ func (s *SQLStore) SetSourceNode(ctx context.Context,
//
// NOTE: This is part of the V1Store interface.
func (s *SQLStore) NodeUpdatesInHorizon(startTime,
endTime time.Time) ([]models.Node, error) {
endTime time.Time,
opts ...IteratorOption) (iter.Seq[models.Node], error) {
ctx := context.TODO()
@@ -587,7 +589,13 @@ func (s *SQLStore) NodeUpdatesInHorizon(startTime,
return nil, fmt.Errorf("unable to fetch nodes: %w", err)
}
return nodes, nil
return func(yield func(models.Node) bool) {
for _, node := range nodes {
if !yield(node) {
return
}
}
}, nil
}
// AddChannelEdge adds a new (undirected, blank) edge to the graph database. An