Week 37 · 2026
Sep 7 – Sep 13, 2026 · gathered Sep 14, 2026
- 4
- PRs merged
- 3
- PRs opened
- 15
- Closed
- 10
- Opened
- 0
- Releases
- 2
- Repos
- 36
- Commits
- 9
- Comments
Highlights
4 PRs merged, 3 opened and 36 commits across two repos. A consolidation week after the 2.4.0/2.4.1 pair: the merged work is a dependency bump, a test fix and last week's two leftovers, while the substance sits in the three PRs opened, two of which have since merged. Performance also closed out as a workstream (see M6). Ten issues were opened, among them five findings on unbounded memory growth and crash loops (#2862, #2863, #2865, #2866, #2867) and, on Saturday, a user's stuck deposit (#2871) taken up in W38; fifteen were closed, three by merges (#2848, #2126, #2167) and twelve in a sweep of the backlog on Monday and Tuesday, five of those as not planned.
Reactive. The etcd network component moved up a version on both sides
(#2857). The bundled etcd goes from 3.5.25 to 3.6.6, and the gRPC client
stack from grapesy 1.0.1 to 1.2.0 and http2 5.3.9 to 5.4.4. The point of
the client bump is #2167, where putMessage blocked indefinitely after five
to ten thousand large messages: the underlying http2 connection stall is
still reproducible upstream when puts share a connection with the watch
stream, but on the new stack it surfaces as a reconnectable
ServerDisconnected error rather than blocking silently and forever. The
dedicated, recycled, timeout-guarded broadcast connections added earlier for
that issue remain the operative fix, re-verified here with repeated 8000 by
100KiB broadcast soaks. Operators upgrading should note that etcd supports
rolling upgrades from 3.5.20 onward, but that the etcd data directory upgrade
is one way. The same PR corrected the docs: hydra-node embeds etcd and uses
it by default, so the tutorial's download step is gone and a separate install
is only needed with --use-system-etcd (#2126). The nightly Blockfrost test
also got a one-line fix (#2868): it waited on a bounded waitMatch for a
tick past the contestation deadline, and that bound is shorter than the
contestation period, so it now sleeps through the period first.
Both PRs left over from last week merged on the Monday this week opened, and
were described in W36 and W35 respectively: the libs/ extraction (#2854)
and the partial fanout chunk-size search (#2849).
Of the three opened, the fanout wedge is the one to note (#2858, for #2855).
A PartialFanout naming the whole remaining set behaved differently
depending on where it arrived: from a freshly closed head it was rerouted to
a full fanout, but from FanoutProgress with nothing yet distributed the
equivalent handler had no such guard and sent it out as a non-final step.
That empties the head while the on-chain datum still says Closed, which
leaves it both unfinalizable (the next step would have to be a zero-output
final fanout, which the validator rejects) and unrevertable (the revert
requires the distributed set to be empty, and it no longer is), so the
tokens can never be burned. The fix moves the decision out of the client
handlers into nextFanoutStep, which now returns the step to take, so every
producer of a fanout step goes through the same choice and the guard cannot
be missed at one entry point again. That also covers a path no client input
reaches: a node that persisted a whole-remainder selection rebuilds it
verbatim on replay, where a rollback re-post would have fired the same
wedging step. Merged the Tuesday after the week closed.
Alongside it, adversarial model testing (#2864) adds random deposits and
decommits under layer 2 transaction spam, divergent-fork rollbacks, per-node
network delivery latency faithful to etcd's total order, and node restart
and fail-recovery, all under liveness assertions. It found and fixed a real
wedge of its own: a ReqSn arriving before a party had observed the deposit
hard-errored and stalled the head, and is now parked with a TTL. It also
fixed several fidelity bugs in the mock chain that the new tests exposed,
namely non-atomic chain steps, queue ordering and silently dropped
transactions. Merged the Wednesday after. Still open: two tests for minting
inside a head
with known and unknown minting policies, which also check that an error
thrown during transaction construction does not bring the node down (#2860, merged in W40).
KPI & Outcome Achievement (M6). More than the usual bookkeeping this
week. The W36 note published, and Performance (M2) is now marked done:
decreased memory usage (M2.2) is recorded as delivered on 2026-08-17 by the
CBOR database and binary API pair (#2767, #2762), joining the 10x TPS work
(M2.1, delivered 2026-07-23) to close the workstream, both ahead of their
deadlines. Three milestone dates also moved earlier: metrics exposed and
queryable (M3.1) from 2026-11-25 to 2026-11-10, and the hydra-recipes repo
(M3.2) and continued compatibility with cardano-node releases (M5.3) both
from 2026-12-15 to 2026-11-30.
Activity
Merged PRs, issues, and releases grouped by deliverable — the evidence behind the narrative above.
Other / Reactive
- MergedFix Nightly BF testcardano-scaling/hydra · @v0d1ch
- MergedBump etcd version to improve errors; no more etcd in tutorialcardano-scaling/hydra · @noonio
- MergedExperiment in moving some code out into libscardano-scaling/hydra · @noonio
- MergedSpeed up finding the partial fanout chunk sizecardano-scaling/hydra · @vrom911
- PR openedModel rollbacks, restarts and L2 spam with de/commitscardano-scaling/hydra · @v0d1ch
- PR openedMint inside a Head testscardano-scaling/hydra · @v0d1ch
- PR openedRoute a full selection to a full fanoutcardano-scaling/hydra · @vrom911
- Issue openedIncremental deposit not recorded and not recoverable — funds locked for 6+ hourscardano-scaling/hydra · @Fiftyw3bs
- Issue openedExtend the solvency proof to the Used redeemers and fan-out completenesscardano-scaling/hydra · @v0d1ch
- Issue openedBump to cardano-api 11.7.0.0cardano-scaling/hydra · @v0d1ch
- Issue openedWatch resume after etcd compaction silently skips protocol messages; any peer can force the compactioncardano-scaling/hydra · @noonio
- Issue openedMonitoring transactionsMap retains an entry per never-confirmed ReqTx id for the process lifetime; peers grow node heap without boundcardano-scaling/hydra · @noonio
- Issue openedOversized NewTx queued for broadcast makes the etcd put fatal and crash-loops the node on every restartcardano-scaling/hydra · @noonio
- Issue openedEvery inapplicable ReqTx persists the full local UTxO plus seven tx copies to the append-only event logcardano-scaling/hydra · @noonio
- Issue openedhydra-chain-observer retains every output of every observed head tx forever; any L1 actor grows itcardano-scaling/hydra · @noonio
- Issue openedhydraw: Don't allow web clients to send msg's to the server.cardano-scaling/hydra · @noonio
- Issue openedBatched deposits - claim multiple deposits in a single incrementcardano-scaling/hydra · @v0d1ch
- Issue closedNo e2e coverage for `--deposit-period` mismatch between peerscardano-scaling/hydra · @noonio
- Issue closed`embeddedRevision` placeholder does not match the C sourcecardano-scaling/hydra · @noonio
- Issue closedFind an alternative to the hydra-cicardano-scaling/hydra · @noonio
- Issue closedDelete hydra-cardano-api and use cardano-api directly.cardano-scaling/hydra · @locallycompact
- Issue closedFix static build on aarm64cardano-scaling/hydra · @noonio
- Issue closedImpove smoke-testscardano-scaling/hydra · @v0d1ch
- Issue closedUpstream Hydra.Cardano.Api.Pretty to `cardano-api`cardano-scaling/hydra · @locallycompact
- Issue closedHead in Headcardano-scaling/hydra · @ch1bo
- Issue closedMigrate validators script to Aikencardano-scaling/hydra · @ghost
30 commits · cardano-scaling/hydra
M6 KPI & Outcome Achievement
Not started6 commits · cardano-scaling/hydra-updates
M3 Operational Excellence
In progress- Issue closedPartial fanout does redundant work when searching for the chunk sizecardano-scaling/hydra · @vrom911
- Issue closed`putMessage` blocks when sending lots of large messagescardano-scaling/hydra · @noonio
- Issue closedApproved decommit can be unpostable and permanently wedges the head; one variant makes the head unclosable and locks all fundscardano-scaling/hydra · @hadelive
- Issue closedAdd aarch64-linux nix support and docker image 🙏cardano-scaling/hydra · @wolf31o2
- Issue closedExclude phantom tokens from snapshotscardano-scaling/hydra · @ch1bo
M5 Maintenance and DevX
In progress- Issue closedDrop etcd download step from the tutorialcardano-scaling/hydra · @noonio