Step 1 — XCache for small reads

Block runs, gap coalescing, vector reads, and a vector form of pgRead.

For: Andy Hanushevsky From: Matevž Tadel (with analysis by Claude, working in the xrootd tree) Date: 2026-09-01 Context: XrdPfc / XCache at 4 kB block sizes

The problem, in one paragraph

LHC physics analysis on thin data formats reads ROOT baskets of a few kB, gathered by TTreeCache into vector reads of 200–600 elements. To avoid gross over-fetch, XCache wants to run with pfc.blocksize at 4–16 kB instead of the historical 128 kB. At 4 kB, a single client vector read expands onto ~6000 cache blocks, and because the cache has no way to batch them it issues one origin request per block. A kXR_readv fixes this for the unchecksummed case — the ~6000 requests collapse to one readv of a few hundred elements. But when pfc.cschk net is configured the cache must use kXR_pgread to get end-to-end crc32c, and pgRead has no vector form, so there is no batching available at all.

Measured, on a real CMS-style workload against a UCSD origin (8 ms RTT), serving 14.4 MB of physics data out of a 115 MB ROOT file:

The last two rows are the same number because they are the same idea. Today only the readv row is reachable, and only with checksums off.

The ask

A new request kXR_pgreadv (3033, next after kXR_clone): carries N (offset, length) pairs like kXR_readv, returns data with crc32c per 4 kB page like kXR_pgread.

Why this is cheaper than it sounds

1. The response wire format needs no change at all

do_PgRIO() (XrdXrootd/XrdXrootdXeqPgrw.cc:214-345) does not send one response. It sends a stream of kXR_PartialResult responses, each carrying its own file offset:

struct pgReadResponse { ServerResponseStatus rsp; kXR_int64 ofs; } pgrResp;
...
pgrResp.ofs = htonll(ioOffset);
dlen = xframt + (items * sizeof(uint32_t));
if ((rc = Response.Send(pgrResp.rsp, infoLen, iov, items*2+1, dlen)) < 0) return rc;

A pgreadv response is that identical stream, with ofs jumping between the requested extents instead of advancing sequentially. ServerResponseBody_pgRead is already { offset; data[] } — it is already extent-addressed. Only the sender's outer loop and the client's data-placement logic currently assume the offsets are contiguous.

2. No XrdSfs / XrdOfs / XrdOss interface change

do_PgRIO calls sfsP->pgRead(offset, buff, rLen, csVec, opts) on a single extent. do_PgReadV would loop that per element, exactly as do_ReadV (XrdXrootdXeq.cc:2746) loops read. The plugin-facing interface is untouched.

In particular XrdOssCsi needs no changes — the server calls its existing XrdOssCsiFile::pgRead() (XrdOssCsi/XrdOssCsiFile.cc:474) once per extent, and its tag-store verification works as it does now. (A vector form inside Csi would be a separate, later, origin-side optimisation; it is not needed for pgreadv to work.)

3. The client already has the scatter machinery

XrdCl::AsyncPageReader (XrdCl/XrdClAsyncPageReader.hh) already takes a ChunkList, and SetRsp() already maps an incoming partial response's offset onto a chunk index plus an offset within that chunk:

void SetRsp( ServerResponseV2 *rsp ) {
   dlen   = rsp->status.bdy.dlen;
   rspoff = rsp->info.pgread.offset;
   uint64_t bufoff = rspoff - chunks[0].offset;
   for( chindex = 0; chindex < chunks.size(); ++chindex ) {
      if( chunks[chindex].length < bufoff ) { bufoff -= chunks[chindex].length; continue; }
      break;
   }
   choff   = bufoff;
   dgindex = rspoff/XrdSys::PageSize - chunks[0].offset/XrdSys::PageSize;
}

FileStateHandler::PgReadImpl just always hands it a one-element list (XrdClFileStateHandler.cc:1281), so the multi-chunk path exists but is barely exercised. What it lacks is that it walks chunks by cumulative length, i.e. it assumes they tile one contiguous file range. Making it locate the chunk whose [offset, offset+length) contains rspoff is a handful of lines.

Estimated work, per layer

The parts we think are genuinely awkward

1. Per-chunk digest indexing. AsyncPageReader::digests is one flat vector sized from csNum(chunks.front().offset, total_length), and dgindex walks it linearly. Per-extent digest ranges are needed. This lands in the partial-read resumption path (ShiftIOV, suRetry), where bugs are load-dependent rather than reproducible.

2. Checksum-error retry. PgReadHandler (XrdClFileStateHandler.cc:150-200) verifies crc32c page by page and on mismatch retries a single page via a flat pgnb index. With N extents this needs (extent, page) addressing. The retry request itself stays a single-page kXR_pgread with kXR_pgRetry, so only bookkeeping changes — but it is the easiest thing here to get subtly wrong.

3. Capability negotiation. Needs a protocol-version or kXR_Qconfig probe so clients can detect support and fall back to per-extent pgRead. Small, but must not be forgotten.

What we are doing regardless

We are not blocked on this. The XrdPfc side is being restructured now so that a vector read is expanded into contiguous block runs rather than individual blocks, with optional coalescing across small gaps. That alone takes the checksummed path from 5990 pgReads to 435 while fetching 74 MB instead of the 109 MB that today's 128 kB configuration fetches — i.e. better than current production on both axes, with no protocol change.

So the request for kXR_pgreadv is an optimisation on top of that (435 → 8), not a rescue. Once the XrdOucCacheIO method exists, using it is a one-branch change in the cache.

Andy's alternative: crc32c in the readv response, no new request

Raised by Andy on 2026-09-02. It may remove the need for kXR_pgreadv altogether, and it looks sound. Checked against the code:

readahead_list is 16 bytes and serves as both the request element and the response header:

struct readahead_list {
   kXR_char  fhandle[4];
   kXR_int32 rlen;
   kXR_int64 offset;
};

The per-element fhandle exists to allow one readv to span several open files, which nobody does. In the response the server echoes the handle the client already sent (XrdXrootdXeq.cc:89), so those 4 bytes are dead weight — and crc32c is exactly 4 bytes wide. Under a flag, the response can carry the crc32c of each iochunk there.

There is room for the flag: ClientReadVRequest carries reserved[15], and do_ReadV references neither it nor pathid.

What it costs us

Granularity. pgRead gives a checksum per 4 kB page; this gives one per readv element, which for the cache means one per block run — up to pfc.iosize, currently 1 MB. Two consequences:

• Per-page values must be recomputed cache-side for XrdOssCsi. That is cheaper than it sounds: XrdOucCRC::Calc32C(data, count, uint32_t *csval) already fills a per-page checksum vector with hardware assist, and Calc32C(data, count, prevcs) gives the element-level value to verify against the wire. Two hardware-assisted passes over data we have just received over a 57.8 ms RTT link -- not measurable.
• A failed check invalidates a whole element rather than a 4 kB page, so a retry re-fetches up to 1 MB and cannot say which page was bad. Acceptable when checks essentially never fail; worth noting that pgRead's per-page retry is genuinely finer-grained.

Andy notes that in practice almost nobody runs XrdOssCsi, since Ceph and ZFS checksum internally, which lowers the weight of the first point considerably.

Why we like it

• No new request id, no new response struct, no XrdCl::File::PgReadV, no XrdOucCacheIO::pgReadV. It deletes most of the ~900-1300 lines estimated above.
• It still needs capability negotiation for the same reason kXR_pgreadv did: an old server ignores the reserved bytes and returns a file handle, which a new client would read as a checksum.
• It composes with the other request. If do_ReadV gains an asynchronous path, checksummed vector reads inherit it for free. A separate kXR_pgreadv would need its own.

Code

Diffs against upstream master, on github.com/osschar/xrootd:

• pfc-optimize-for-small-blocks — block runs, pfc.iosize, pfc.iogap, the pfc.flush unit fix, and the NETTIME/CLIREAD tracing (compile-time gated, off).

The pgReadV implementation described in the addendum below is on three branches that stack:

• pgreadv — 4 commits: kXR_pgreadv defined in XProtocol.hh, XrdCl::File::PgReadV (initially a loop over PgRead), pgReadV added to XrdOucCacheIO and implemented in XrdPosixFile. No wire change, so it is the part that is safe on its own.
• pgreadv-wire — the above plus 4 more: do_PgReadV in the server, sending kXR_pgreadv, per-(extent, page) checksum verify and retry, and capability negotiation with a per-extent pgRead fallback.
• pfc-pgreadv — pgreadv-wire merged with the cache work, plus the cache-side change that batches block runs into pgReadV when checksums come from the network. This is what produced the 587-vs-13,405 request counts.

Note pfc-pgreadv merged the cache branch before the tracing commit was added to it, so it does not carry NETTIME/CLIREAD.

Open questions for you

• Is reusing read_list for the request acceptable, or would you rather have a distinct arg struct?
• kXR_FinalResult semantics across extents: one final per extent, or one final for the whole request? We assumed the latter.
• Monitoring: new vType alongside XROOTD_MON_READV/READU, or fold into the existing pgRead stats (pgrOps)?
• Would you want pgWriteV for symmetry, or is read-only fine? XrdPfc only needs the read side.

Addendum: as built, 2026-09-01

Added after the implementation. This records what the code turned out to be, so the open questions below are not asked twice. One judgement above has since been revised: the closing "optimisation, not a rescue" was measured against an origin 8 ms away. At 57.8 ms, with vector reads serialised per link, the same argument becomes a feasibility one -- see the companion document, "Vector reads, serialised".

All of it is implemented and tested, on two branches off master: pgreadv (the API and a fallback, no wire change) and pgreadv-wire (the protocol). Together, 1435 insertions over 21 files, against the 900-1300 estimated. The XrdPfc side is one further commit on its own branch.

What the three "cheaper than it sounds" claims came to

1. The response wire format needed no change at all -- confirmed. A test client parses the pgreadv response stream using the existing structs and no new ones.
2. No XrdSfs / XrdOfs / XrdOss change -- confirmed, none of them were touched. do_PgRIO became do_PgRIO(bool isLast) and the extents are looped outside it, so it still sees exactly the single-extent state it always did, stack arrays and kXR_Impossible guard included.
3. XrdOssCsi needed no changes -- confirmed, untouched.

Estimates versus actual

XrdXrootd came in well under: looping the extents around do_PgRIO rather than inside it is most of why. XrdCl came in over, and the overrun is almost entirely the two things this document called awkward, plus their comments.

The open questions, answered provisionally

Each is one commit, so any of them is cheap to change.

• Reusing read_list: done, unchanged. ClientPgReadVRequest is 24 bytes like every other request header, and keeps pathid at the same offset as ClientReadVRequest so the server's handling can be shared. reqflags sits in the header at byte 18, since dlen is taken by the read_list.
• kXR_FinalResult across extents: one final for the whole request, on the last partial of the last extent, as assumed here.
• Monitoring: no new vType. Each element is reported exactly as a pgread of it would be -- Add_rd and pgrOps per element -- so the batching is not visible in the monitoring stream. Adding a vType is still your call.
• pgWriteV: not implemented. XrdPfc does not need it.

Two more decisions that were not in the list:

• Per-element length is uncapped. Unlike readv, which buffers a whole response, each extent is streamed as its own sequence of partial results, so the quantum bounds the buffer rather than the request. Only the element count is capped, at maxRvecsz, because the vector is copied to the stack so do_PgRIO can reuse the request buffer.
• pathid is rejected, not ignored. kXR_readv ignores it; do_PgReadV returns kXR_ArgInvalid for a non-zero one, so a client that asked for an alternate path gets an error instead of waiting forever on a path the response will not arrive on. Nothing in XrdCl sets it.

Capability negotiation

kXR_PROTPGRVVERSION, with the protocol version moved to 5.2.1. Which release first carries the request is yours to assign -- the number here is a placeholder for the mechanism. For reference, stock xrootd 5.9.2 reports protocol 5.1.1, so there is room below it. A client that finds the server too old, or a vector too long for one request, substitutes a pgRead per extent and returns the identical answer in more round trips. A server that passes the version gate and then rejects the request is not recovered from; the version is taken as the contract.

One unrelated bug found on the way

XrdPosixPrepIO overrode Read and ReadV but never pgRead, so pgRead fell through to the XrdOucCacheIO default, which reads with Read() and computes checksums only under forceCS. Data correct, checksums silently dropped -- and XrdPfc reads that as "this origin cannot supply checksums" and marks the file as having none, permanently. It predates this work and is reproducible with no vector read involved. Fixed on the pgreadv branch; that commit stands alone against master and is worth taking regardless of what happens to pgreadv.