Addendum — the measurements

Every hypothesis tested, including the ones that were wrong.

An investigation into why an XCache cold run of the AGC ttbar analysis was dramatically slower than the same analysis reading direct from the same origin. It ended without a proven mechanism, but it eliminated a lot and produced two solid, unrelated results. The eliminations are the most valuable part: most of them were confidently asserted first and disproved after.

Companion docs: claude-blockrun-state.md (the XrdPfc BlockRun work), xcache-step1-small-reads.md, claude-pgReadV-bootstrap.md.

The setup

Workload: root-project/analysis-grand-challenge, analyses/cms-open-data-ttbar, analysis.py -n 1 --scheduler mt --ncores 8 --no-fitting. RDataFrame, 8 samples of CMS UL16 NanoAODv9. Client always on black (home, ATT fibre).

Origins used, all with the same cache code and pfc.cschk off:

The two solid results

1. Block size dominates over-fetch, by a lot. Same analysis, prefetch off, verified by exact block accounting (9185 blocks x 1 MB = 9.0 GB, matching disk usage to the digit):

11x less data for identical physics. A synthetic file with few branches and fat baskets showed only 4.7x; it took real NanoAOD (~1500 branches, small baskets) to show the true figure.

2. On a clean path the cache costs almost nothing. UCSD, 9 ms, direct, plain xrootd, reproducible to +-1 ms:

The cache's overhead is ~2 ms, of which ~1 ms is the unavoidable +22.2% from 4 kB block rounding. Warm cache beats direct by 2.2x on the full analysis.

The unexplained part, at this point

Resolved further down. Kept as written to show what the evidence looked like before the cause was found.

Against FNAL, the cache's vector reads averaged 415 ms where ROOT's averaged 178.8 ms, reproducibly (cold analysis: 247/245/241/240/223 s vs 28 s). At UCSD the same comparison is 22.0 vs 22.2 ms — no difference at all.

So the penalty appears only at high RTT, and no client-side property explains it. Everything below was measured and found not to be the cause.

Hypotheses eliminated

xrootd.async 64 kB segmentation. do_ReadV contains no aio at all — verified in XrdXrootdXeq.cc; only do_Read has the async path. Vector reads are never segmented. xrootd.async off and nocache both changed nothing (241 and 240 s vs 247 s baseline). Only 8% of bytes took the plain-read path.

Cache-internal locking / m_block_map contention. The warm run pushes the same 666 MB through the same code, same locks, same 8 threads at 57 MB/s.

Origin request count. 595 origin readv for 587 client readv — one for one, no amplification.

Request shape. With pfc.iosize 1M the cache sends fewer elements than ROOT (33795 vs 39731 for the same 587 requests), same 140 kB max element, and +22% bytes. Exhaustive check of every request in every run: zero non-monotonic, zero duplicate offsets, zero overlaps, zero zero-length elements.

The XrdPosix / XrdCl path. XrdPosixFile::ReadV and FileStateHandler::VectorRead are both straight pass-throughs; XrdCl::Utils::SplitChunks is not on this path. What ReadOpusCoalescere builds is what goes on the wire.

XrdCl version. A synthetic 64-chunk 1152 kB readv: 71.4 ms on system 5.9.2, 71.0 ms on our 6.1.x build.

Protocol, TLS, auth, topology. Both negotiate protocol 500, both TLS to the door and neither to the pools, both gsi with the same DN, both use the same 9 pool nodes over 9 connections with matching per-pool request counts. Confirmed by socket count: 12 established sockets, one per pool host, for both.

pgReadV. Never called: cschk off takes the ReadV branch. Wire shows 595 kXR_readv, 0 kXR_pgread.

Server implementation. Initially concluded "it is the dCache door". Wrong: EOS shows the same ~2x, and so does a stock xrootd at CERN.

net.core.rmem_max. Looked compelling — 212992 on the UCSD host, and latency rose in steps of about that size. But XrdNetSocket::setWindow is only called when a window is explicitly configured, which ours is not, so SO_RCVBUF is never set and rmem_max never applies. Autotuning governs, with the net.ipv4.tcp_rmem ceiling (6 MB at UCSD, 32 MB on black). Coincidence fitting.

tcp_slow_start_after_idle. Set to 1 on both hosts, so plausible. But the cache's origin connection is idle only 17 ms between readv (p90 22 ms), and only 1% of gaps exceed a 200 ms RTO. It essentially never fires.

Multiple substreams. DefaultSubStreamsPerChannel is 1 and the cache uses one TCP connection per origin — but XRD_SUBSTREAMSPERCHANNEL=4 made no difference (324/288/332/334 ms). A single response travels one stream.

A byte cap on the origin vector request. Implemented, then reverted unmerged: with one substream, splitting a readv sends the same bytes through the same connection and window. It cannot help.

"ROOT stays under a size threshold, we cross it." Wrong. ROOT's readv sizes are data-driven and overlap ours: min 75, median 1226, p90 1626, max 2038 kB. ROOT routinely sends larger requests than our 1408 kB.

What the kernel actually shows

ss -tin on the cache's FNAL sockets, mid-transfer, 100+ MB received:

rcv_space: 1465120        receive window ~1.46 MB
rcv_ssthresh: 10599918    headroom to 10.6 MB, unused
app_limited               set on every socket
rtt: 54-72 ms

The window is nowhere near any sysctl ceiling. It is small because the flow is app-limited: bursty request/response that never sustains enough data in flight for autotuning to open further.

Important caveat against over-reading this. rcv_space tracks what the application consumes per RTT, so finding it near our ~1.4 MB request size is a consequence, not a cause. It is circular to call it the limit.

And ROOT's own sockets show rcv_space of 1.24 / 1.33 / 2.06 MB and are also app_limited — the same regime. So "ROOT fills the pipe better, so its window grows" is not supported either.

Measured request concurrency, AGC run against FNAL: cache 0.98 in flight, ROOT direct 2.45. At UCSD, where there is no penalty, both are low (0.41 and 0.58). That correlation is suggestive but unproven.

Connection and concurrency, also eliminated

The cache collapses 9 origin endpoints onto one host, so the client side drops from 12 connections to 2 -- but the cache's own outbound side keeps 10. Raising the client side with XRD_SUBSTREAMSPERCHANNEL=8 gives 16 connections and changes nothing (237.4 s vs 239.7 s). ROOT itself only ever has ~2.45 requests in flight, so extra connections add no extra requests. Concurrency is not the constraint on either leg.

pss.setopt notes, since the docs are stale: ParStreamsPerPhyConn is a v4 alias parsed in XrdOucPsx::ParseSet and forwarded to XrdCl's SubStreamsPerChannel (stock default 1, pss does not override it). pss raises WorkerThreads from XrdCl's stock 3 to 64 and sets ParallelEvtLoop to 10, matching stock -- so no pss default is lower than XrdCl's own.

Worth raising with Andy independently of any of this: do_Read has a full aio path and do_ReadV has none. Vector reads therefore cannot overlap on a link and are immune to every xrootd.async setting. That is invisible on a LAN and matters at WAN latency, and vector reads are the dominant access pattern for ROOT analysis.

Where to go next, at this point

Also historical. The packet capture below was done, and "raise origin-request concurrency" turned out to be the right instinct for the wrong reason: the cache was not failing to pipeline, it was never given more than one client vector read to work on. The note on xrootd.async segsize still holds.

The client side is exhausted. Distinguishing the remaining possibilities needs either a packet capture (in-flight bytes over time, ACK pattern, retransmits) — which needs root — or origin-side logs. Two specific things worth testing:

1. Raise origin-request concurrency in XrdPfc. The cache runs at ~1 request in flight against a path whose BDP is ~6 MB. Deep pipelining, rather than one origin readv per client readv, would keep the flow from being app-limited. This is the most promising lead and is a cache-side change.
2. Whether a sustained, non-bursty transfer to the same origin reaches full BDP. If it does, app-limiting is confirmed as the regime; if it does not, something else caps the path.

Note on Andy's 64 kB xrootd.async segsize: that governs regular read segmentation, a different mechanism which vector reads bypass entirely. It has no bearing on any of the above.

Reproducing

test/rvbench.cxx issues N vector reads of C chunks and reports per-request latency; build it against both the local tree and the system XrdCl with test/pgrv_build.sh-style flags. Measure the cache's origin-side latency from its own XrdCl log (XRD_LOGLEVEL=Debug XRD_LOGFILE=...), matching MsgHandler created to Calling MsgHandler by handler address — comparing a client-side number against an origin-side one is the easiest mistake to make here, and I made it.

Beware also: measurements against EOS and dCache are extremely noisy (EOS direct swept 159 to 639 ms non-monotonically), and anything measured through an ssh tunnel picks up OpenSSH's own channel window — a clean-looking 1.3 MB "step" measured that way turned out to be an artifact, absent at 9 ms direct.

Both legs, now actually tested

The two legs have independent concurrency limits, and the connection counts that looked asymmetric are topology, not client policy.

ROOT applies the same policy to the cache and to FNAL -- it has no idea which it is talking to. The 12-vs-2 difference is simply that FNAL redirects onto a dCache pool: 10 of the cache's 11 outbound sockets go to distinct pool nodes, one channel each.

[2620:6a:0:8420::101]:1094      dCache door
[2620:6a:0:8421::a7]            9 distinct pool nodes
[2620:6a:0:8420:f0:0:205:40]    ...

All four inside the noise band for identical configs (222.9 -- 264 s).

Two traps worth writing down:

• XRD_PARSTREAMSPERPHYCONN is not a variable. XrdCl derives env names mechanically as XRD_ + registered name (XrdClDefaultEnv.cc:415), and the registered name is SubStreamsPerChannel (:270). ParStreamsPerPhyConn exists only as a v4-compat alias in the pss config table (XrdOucPsx.cc:702-733), translated before it reaches XrdCl. Setting the env form is a silent no-op -- it produced 264.0 / 260.7 s with the connection count pinned at 2 in both runs.

• Substreams are created lazily. 8 substreams x 10 channels should allow 80 sockets; it produced one extra. XrdCl opens a substream only when a channel is genuinely backed up, and at ~2.5 requests in flight over 10 pool nodes none ever is. The knob has nothing to relieve.

Concurrency is eliminated on both legs.

Partly resolved: cwnd reset after idle (real, but not the 8x)

Packet capture on black (enp25s0, net 2620:6a::/32), cold AGC run through the cache vs ROOT direct, same box, same link, same clock. 229.0 s vs 29.0 s, so the capture caught the real thing.

RTT to FNAL is 57.8 ms (TCP handshake), not the ~9 ms of the UCSD origin.

Think time is identical and equals one RTT: dCache is not slow, and the door/pool is not the problem. Throughput while flowing is higher for the cache. The entire difference is idle wire.

The chain:

1. The cache keeps ~2.5 requests in flight, spread over 10 pool connections.
2. Each connection therefore idles ~2.1 s between requests, past the 200 ms minimum RTO on 68% of occasions.
3. net.ipv4.tcp_slow_start_after_idle = 1 resets cwnd to the initial window every time that happens.
4. The measured first round is 14 kB. IPv6, MTU 1500, TCP timestamps -> MSS 1428; IW10 = 14280 B. It is the initial window exactly.
5. A ~1 MB response then needs ~10 RTT rounds to ramp. 336 ms of ramp + 56.9 ms think = 393 ms, which is the 415 ms readv latency measured from the logs.

So it is a window effect -- but the congestion window, not the receive window, and the trigger is idleness, not buffer size. The earlier rcv_space reasoning was circular and wrong.

This also explains why every concurrency knob failed. More connections and more substreams spread the same few requests thinner, lengthening per-connection idleness. We were pushing the wrong way the whole time.

Consequences for XCache

• net.ipv4.tcp_slow_start_after_idle=0 on cache hosts is a free, one-line win on any high-RTT origin. Worth putting in the deployment notes.
• Fewer, busier connections beat more, idler ones -- the opposite of the usual instinct. Anything that keeps a pool connection warm is worth more than extra parallelism.
• This is an argument for larger pfc.iosize and for pgReadV batching beyond the obvious request-count saving: bigger requests amortise the ramp, and a steady request stream keeps cwnd from collapsing at all.
• It is a general WAN-cache result, not specific to dCache or to FNAL.

Capture and analysis scripts: test/wan-analysis/ (see its README). Instrumentation: NETTIME trace lines at pfc.trace info, File::TraceNetTime() in XrdPfcFile.cc, CLOCK_REALTIME so they join against frame timestamps.

Correction: the network was not the answer

tcp_slow_start_after_idle was set on black, which is the wrong side. It governs the local sender's cwnd; the bulk data flows FNAL -> us, so the window that resets is dCache's, on their kernel. Our request packets are the only thing our setting touches. The first round stayed at exactly 14 kB after the change, which is how we know.

Raising client concurrency (--ncores 8 -> 24) did keep the pool connections warm, and it fixed the ramp completely:

Per response the cache now matches ROOT direct exactly. The run got slower.

What actually limits it

From the NETTIME logs, across all three runs:

MEAN requests in flight: 1.00      peak 8

Exactly 1.00, with 787 / 787 / 1347 requests at medians of 393 / 286 / 168 ms. The run is n_requests x latency / 1, so halving latency bought nothing.

It is not a lock in our code (m_state_cond is released at XrdPfcFile.cc:1279, before the request is issued at :1284), not RAM (pfc.ram 4g against 862 MB fetched, no starvation in the log), and not one-file-at-a-time: 6-7 files are being fetched concurrently, peak 9.

The arithmetic closes instead on the per-file cycle. Each file issues a remote request every 2.2 s, of which 0.393 s is the fetch. 7 files x (0.393 / 2.2) = 1.0 in flight. The files are independent and each is slow. 82% of the cycle is not spent waiting on the network.

Consistent with the warm cache running this same analysis in 11.7 s -- faster than direct's 28.4 s. The steady-state read path is fine. The cost lives in the cold path, alongside 210,358 four-kilobyte block writes.

Next step

Extend the NETTIME approach to the cold path: stamp response arrival -> WriteRunToDisk -> block release -> FinalizeReadRequest, and find where the 1.8 s per file per cycle goes. That is cache-internal and entirely ours.

Resolved: concurrency equals the number of client sessions (mechanism below)

Instrumented every client read (CLIREAD) as well as every remote request (NETTIME). In the cold run there were 787 client vector reads and 787 origin requests, 1:1, and the cache logged one login.

MEAN client reads in flight : 0.99   peak 8
MEAN remote reqs in flight  : 0.99   peak 8

Splitting the identical cold workload across three independent client processes, measured over the window where all three overlap:

 t(s)  files   mean in flight
    0      6            3.22
   10      7            3.19
   15      6            2.97
   20      6            1.83   <- two processes finish
   25      5            0.99   <- one session left
   ...     5            0.99

Mean in-flight tracks the number of client processes, exactly. Open files (5-7) and RDataFrame threads (8) make no difference at all.

The arithmetic then closes on every measurement taken today:

Direct to dCache, ROOT holds a session per pool node and overlaps across them. Through the cache -- a single endpoint -- it holds one. That is the whole 8x.

Not attributed to do_ReadV. It is tempting, since do_ReadV (XrdXrootdXeq.cc:2746) has no aio path where do_Read has the full xrootd.async machinery, and xrootd.async off/nocache measurably change nothing (241.1 / 240.7 s). But with a single session the peak was 8, so the server did handle eight concurrent vector reads on one link. Whatever pins the mean at 1.0 is upstream -- ROOT's I/O path or XrdCl per-session handling -- and is not yet isolated.

Not confined to caches

The measured condition is one client, one server endpoint, several concurrent vector reads. A multi-threaded RDF/RNTuple job over one large file meets it without any cache: per-thread TFiles still pool onto a single channel, since the key is (user, host, port). Directly measured — eight XrdCl::File handles on the same path, one process, two sockets, 5.13 s; separate channels, sixteen sockets, 3.82 s. Straight to one xrootd server or EOS, likewise. Behind a redirector it depends on whether opens spread across data servers.

What this means for XCache

Interposing a single-endpoint cache in front of a distributed origin collapses the parallelism the origin's architecture was providing. Warm this is invisible (14 ms x 787 = 11 s, still beating direct). Cold over a WAN it is decisive. Directions worth exploring, roughly in order of expected value:

1. Find and remove the per-session serialization (needs isolating ROOT vs XrdCl first -- it may not be ours at all).
2. Prefetch/readahead, so a client read finds data already in RAM and the RTT is never on the critical path. pfc.prefetch is off for this use case today.
3. Multiple cache endpoints, so a client can hold several sessions.

Fewer round trips also matters more than we thought, since each one is fully exposed -- which is the strongest available argument for pgReadV batching.

SOLVED: one vector read in flight per XrdCl channel

do_ReadV() runs synchronously on the link's thread, so a link carries exactly one vector read at a time. XrdCl pools channels on (user, host, port), so a client reading N files from one cache gets one channel and therefore no read concurrency at all. Going direct to a distributed origin, the N files land on N different pool hosts, so the same client gets N channels and N-way concurrency. That asymmetry is the entire 8x.

Proof, in three steps

1. It is not ROOT. A bare XrdCl probe (test/rvconc.cxx) asking for K concurrent vector reads on one session:

conc=1  30 reqs  wall= 2.34 s   78.0 ms/req
conc=4  30 reqs  wall= 3.46 s  115.2 ms/req
conc=8  30 reqs  wall= 2.79 s   92.9 ms/req

K=8 should have been ~0.3 s. No overlap. No ROOT involved.

2. It is not pss / posix / pfc. CLIREAD counts client-facing reads inside the cache, above all of them:

MEAN client reads in flight : 0.98   peak 1
MEAN remote reqs in flight  : 0.97   peak 1

Peak 1. The cache never saw two client vector reads at once while the client had eight outstanding, so the serialization is above the cache.

3. It is not XrdCl either. Loopback capture, K=8:

0.0077 REQ->    1048     eight readv requests pipelined onto one
0.0077 REQ->    1048     socket within 100 us, before any response
   ...  (x8)
0.8678 <-RSP   32768     860 ms later, responses arrive together

XrdCl sends all eight immediately. The server takes 860 ms to do eight, where one alone takes 78 ms. The server received the concurrency and did not use it.

The workaround: distinct usernames, no code change

XrdCl's channel key includes the username, so pooling can be defeated from outside by giving each file URL a different user. Full AGC cold run:

Three cold repeats: 36.7 / 36.5 / 26.4 s. ROOT direct is 28.5 s, so the cold cache goes from 8.8x behind to parity -- one repeat beat direct outright. Nothing patched, no lock touched, no XrdCl change, and the physics is identical (124 keys, 52039 entries, integral 93931.1219 in both).

Microbenchmark agrees: 8 file handles in one process, 6.47 s shared channel vs 3.27 s with distinct usernames; eight separate processes give 2.43 s.

Thread knobs are irrelevant

ParallelEvtLoop (XrdClPollerBuiltIn.cc:708) is the poller count, default 10. WorkerThreads is the worker pool, default 3; XrdPss sets it to 64 for the cache's own client and leaves the pollers at 10. Neither moves this workload -- with one outstanding request per channel there is nothing to parallelise. (Note: a test here set ParallelEvtLoop=8, which lowers it from the default 10. It made things marginally worse, as expected.)

What to do about it

1. Now, for anyone wanting to see real XCache performance: distinct usernames per file, or several cache endpoints. Purely client-side.
2. Properly: give do_ReadV an async path like do_Read has. This is the invasive one and needs Andy.
3. Prefetch would also hide it, by taking the RTT off the critical path.

Probes: test/rvconc.cxx (K concurrent readv, one session), test/rvmulti.cxx (N file handles, optional distinct usernames).

The mechanism, exactly: readv.pathid is ignored

ClientReadVRequest in XProtocol.hh already carries a pathid byte:

struct ClientReadVRequest {
   kXR_char  streamid[2];
   kXR_unt16 requestid;
   kXR_char  reserved[15];
   kXR_char  pathid;      // <-- on the wire since forever
   kXR_int32 dlen;
};

Nothing in the tree references readv.pathid -- not the server, not the client. It is a declared, unused field. Meanwhile every other bulk request honours its pathID and offloads:

do_PgRead takes its pathID from ClientPgReadReqArgs, so a do_PgReadV modelled on it inherits offload for free.

This retroactively explains the morning's dead end: XRD_SUBSTREAMSPERCHANNEL=8 gave 16 connections and changed nothing. Substreams are parallel paths -- pathIDs -- and read/pgread/write all use them. readv discards the field, so every vector read stays on the main link. The knob worked; the request type could not use it.

The fix needs no protocol change. The byte exists; do_ReadV just has to honour it the way do_Read does.

Workarounds, ranked

XrdCl keys channels in URL::GetChannelId() on protocol://hostId/ plus a few CGI keys -- one of which is xrdcl.intent. Full AGC cold run against FNAL:

Physics is bit-identical in all cases (124 keys, 52039 entries, integral 93931.1219), and the CGI does not leak into cache file names, so cache keys and warm hits are unaffected.

Prefer xrdcl.intent over usernames: no fake identities, so no per-user authentication, mapping, quota or monitoring side effects. Under GSI or tokens N usernames would mean N authentications.

An env var was tried and is the wrong answer

Added ChannelsPerHost to XrdCl (constant, env registration, a per-URL-instance slot appended in GetChannelId). It works in a native client: sockets scale 2/4/8/16 with N, and N=8 gives 3.82 s against 5.13 s pooled, matching the URL tricks. It is useless in practice, because ROOT's libNetxNG.so binds libXrdCl.so.3 while this tree builds soname 5. Only a client rebuilt against new XrdCl would see it.

The URL tricks are pure URL content, so they work with any XrdCl version -- which is why the 249.6 -> 26 s result was obtainable at all.

Trap worth remembering: build/lib contained a stale soname generation from 2026-06-09 -- libXrdCl.so.3 and friends -- alongside today's soname 5. ROOT links the soname libXrdCl.so.3, so LD_LIBRARY_PATH=build/lib made the loader prefer the June copy over /usr/lib64, and ROOT silently ran a three-month-old client for every client-side measurement here. Cleared on 2026-09-02 into build/lib-stale-pre-soname5/.

ROOT still loads two generations at once -- soname 3 from /usr/lib64 for libNetxNG, soname 5 from our build via libXrdClHttp-5.so. So any client-side XrdCl change must be checked with strings or /proc/<pid>/maps, never assumed.

The residual limit

A single file opened once cannot be helped by anything client-side: one link, one vector read at a time, by construction. Multiple opens of one file can be split (measured: 8 handles on one path, 4.97 s pooled vs 3.91 s split), but a one-file analysis has nothing to spread. That case needs the server fix, which is the real argument for honouring readv.pathid.

Why readv was left synchronous, and where the decision belongs

Not an oversight. Throwing N concurrent vector reads at a classic xrootd data server is bad, especially several on the same file when that file lives on one spinning disk -- the resulting seek pattern is exactly what a vectored read exists to avoid. Leaving do_ReadV synchronous is a defensible default for a plain data server.

But it is a property of the storage, not of the request, and a cache cannot answer it statically either. One XCache serves:

• a cold file from a WAN origin -- concurrency is free and worth ~8x;
• a fully-cached file from a local spinning disk -- concurrency is harmful;

sometimes inside the same request. So neither the protocol layer nor a static server config can decide. The decision has to be made where the storage is known: in the OSS / cache, at request time.

That reframes the ask to Andy from "make readv async" to "make the async path reachable and delegate the choice downward". Written up in xcache-step2-analysis-environment.md.

Why a server-side knob is worth more than a client-side one

ROOT and XrdCl reach users through central builds (CERN, LCG, CMS) that a user cannot change -- they run whatever their environment ships. An XCache is always deployed by the site, so a toggle there can actually be flipped. This is why the ChannelsPerHost XrdCl prototype, even though it works, is the wrong answer: it only helps clients rebuilt against new XrdCl, and ROOT 6.38 binds libXrdCl.so.3.

The URL tricks (xrdcl.intent, usernames) are the exception that works today, because they are pure URL content and need no new client code at all.

Mechanism notes for a future attempt

• do_Read has two async routes: aio on the same link when pathID==0 (XrdXrootdXeq.cc:2597-2616, ending in return 0), and do_Offload when a pathID is supplied (:2624). The first needs no client cooperation at all -- that is the one a server-side toggle would use.
• Throttling already exists: as_maxperlnk, as_maxpersrv, from xrootd.async limit / maxtot (XrdXrootdConfig.cc:983-987).
• XrdXrootdAioFob is already per-file -- the natural home for "one outstanding vectored read per LFN", which is the granularity that recovered 7-9x via the channel workaround while keeping a single file's readvs serialised.
• XrdXrootdAioTask::Read(long long offs, int dlen) is single-extent. A vector form needs a new XrdXrootdAioTask subclass or per-element completion aggregation. This is the real work and presumably why it never happened.
• do_PgRead already offloads, so a do_PgReadV modelled on it inherits the right behaviour. The gap is specific to plain kXR_readv.

Code

Diffs against upstream master, on github.com/osschar/xrootd:

• pfc-optimize-for-small-blocks — block runs and the NETTIME/CLIREAD instrumentation (kTraceNetTime, off by default) that every in-flight measurement here is derived from.
• xrdcl-channels-per-host — the client-side channel-splitting prototype.

Probes and analysis scripts are in the working tree, not on those branches: test/rvconc.cxx, test/rvmulti.cxx, test/wan-analysis/.

Status, 2026-09-03

Andy is generally supportive; he will review the branches. Three things came out of the first conversation:

• kXR_pgreadv may not be needed. The per-element fhandle in readahead_list is dead weight in the readv response, and crc32c is exactly 4 bytes. Under a flag -- there are 15 unreferenced reserved bytes in ClientReadVRequest -- readv can carry a checksum per iochunk instead. See "Andy's alternative" in Step 1.
• A server can split channels by itself, by redirecting an open to a synthetic username. SFS_REDIRECT -> kXR_redirect -> URL::ParseHostInfo -> ComputeHostId -> channel key. Measured 2026-09-03: 276 s -> 31 s with an unmodified client fileset, mean in flight 1.00 -> 2.83, i.e. parity with not caching at all. No client change of any kind. See Step 2.
• Async readv stays open, pending review of the existing branches.

Follow-up 2026-09-03: raising the redirect inside the cache (pss layer, early open) needs no protocol change -- -EDESTADDRREQ plus FileURL in the open env is already wired through XrdOfs.cc:788. But a self-redirect to the same host:port differing only in username loops on a stock client, because RetryAtServer compares URL::GetLocation(), which omits the user. Comparing GetChannelId() too is a one-line fix. Redirecting across loopback addresses avoids the client change and does create the channels, but did not produce the concurrency. Since the same usernames placed directly in the client's fileset URLs against that same single instance do work, this is a mechanism in XrdCl's redirect processing rather than anything topological. The 276 s -> 31 s result used a genuinely different endpoint.