# Step 1 — XCache for small reads *Block runs, gap coalescing, vector reads, and a vector form of pgRead.* **For:** Andy Hanushevsky **From:** Matevž Tadel (with analysis by Claude, working in the xrootd tree) **Date:** 2026-09-01 **Context:** XrdPfc / XCache at 4 kB block sizes --- ## The problem, in one paragraph LHC physics analysis on thin data formats reads ROOT baskets of a few kB, gathered by TTreeCache into vector reads of 200–600 elements. To avoid gross over-fetch, XCache wants to run with `pfc.blocksize` at 4–16 kB instead of the historical 128 kB. At 4 kB, a single client vector read expands onto ~6000 cache blocks, and because the cache has no way to batch them it issues **one origin request per block**. A `kXR_readv` fixes this for the unchecksummed case — the ~6000 requests collapse to one readv of a few hundred elements. But when `pfc.cschk net` is configured the cache **must** use `kXR_pgread` to get end-to-end crc32c, and pgRead has no vector form, so there is no batching available at all. Measured, on a real CMS-style workload against a UCSD origin (8 ms RTT), serving 14.4 MB of physics data out of a 115 MB ROOT file: | configuration | origin requests | bytes fetched | |---|---|---| | `blocksize 128k` (today's production) | 877 | 109.6 MB | | `blocksize 4k` (what we want) | 5990 | 23.4 MB | | `4k` + contiguous-run merging | 2366 | 23.4 MB | | `4k` + run merging + gap coalescing | 435 | 74.4 MB | | `4k` + **one readv per client readv** | **8** | 23.4 MB | | `4k` + **one pgreadv per client readv** | **8** | 23.4 MB | The last two rows are the same number because they are the same idea. Today only the readv row is reachable, and only with checksums off. ## The ask A new request `kXR_pgreadv` (3033, next after `kXR_clone`): carries N `(offset, length)` pairs like `kXR_readv`, returns data with crc32c per 4 kB page like `kXR_pgread`. ## Why this is cheaper than it sounds ### 1. The response wire format needs no change at all `do_PgRIO()` (`XrdXrootd/XrdXrootdXeqPgrw.cc:214-345`) does not send one response. It sends a **stream** of `kXR_PartialResult` responses, each carrying its own file offset: ```c struct pgReadResponse { ServerResponseStatus rsp; kXR_int64 ofs; } pgrResp; ... pgrResp.ofs = htonll(ioOffset); dlen = xframt + (items * sizeof(uint32_t)); if ((rc = Response.Send(pgrResp.rsp, infoLen, iov, items*2+1, dlen)) < 0) return rc; ``` A pgreadv response is that identical stream, with `ofs` jumping between the requested extents instead of advancing sequentially. `ServerResponseBody_pgRead` is already `{ offset; data[] }` — it is already extent-addressed. Only the sender's outer loop and the client's data-placement logic currently assume the offsets are contiguous. ### 2. No XrdSfs / XrdOfs / XrdOss interface change `do_PgRIO` calls `sfsP->pgRead(offset, buff, rLen, csVec, opts)` on a single extent. `do_PgReadV` would loop that per element, exactly as `do_ReadV` (`XrdXrootdXeq.cc:2746`) loops `read`. The plugin-facing interface is untouched. **In particular `XrdOssCsi` needs no changes** — the server calls its existing `XrdOssCsiFile::pgRead()` (`XrdOssCsi/XrdOssCsiFile.cc:474`) once per extent, and its tag-store verification works as it does now. (A vector form *inside* Csi would be a separate, later, origin-side optimisation; it is not needed for pgreadv to work.) ### 3. The client already has the scatter machinery `XrdCl::AsyncPageReader` (`XrdCl/XrdClAsyncPageReader.hh`) **already takes a `ChunkList`**, and `SetRsp()` already maps an incoming partial response's offset onto a chunk index plus an offset within that chunk: ```cpp void SetRsp( ServerResponseV2 *rsp ) { dlen = rsp->status.bdy.dlen; rspoff = rsp->info.pgread.offset; uint64_t bufoff = rspoff - chunks[0].offset; for( chindex = 0; chindex < chunks.size(); ++chindex ) { if( chunks[chindex].length < bufoff ) { bufoff -= chunks[chindex].length; continue; } break; } choff = bufoff; dgindex = rspoff/XrdSys::PageSize - chunks[0].offset/XrdSys::PageSize; } ``` `FileStateHandler::PgReadImpl` just always hands it a one-element list (`XrdClFileStateHandler.cc:1281`), so the multi-chunk path exists but is barely exercised. What it lacks is that it walks chunks by cumulative **length**, i.e. it assumes they tile one contiguous file range. Making it locate the chunk whose `[offset, offset+length)` contains `rspoff` is a handful of lines. ## Estimated work, per layer | layer | change | est. | |---|---|---| | `XProtocol.hh` | request id; `ClientPgReadVRequest` reusing the `read_list` array + a `reqflags` byte. **No new response struct.** Limits reuse `maxRvecsz` (1024 elements) and `maxPGRD` (≈2,093,056/element, already computed in `do_PgRIO`). | ~30 | | `XrdXrootd` | `do_PgReadV`: read_list decode lifted from `do_ReadV`, wrapped round the existing `do_PgRIO` body; `kXR_FinalResult` only on the last partial of the last element. Monitoring vType is your call. | 250–400 | | `XrdOssCsi`, `XrdSfs`, `XrdOfs`, `XrdOss` | **none** | 0 | | `XrdCl` | `SetRsp()` offset lookup (~10); per-chunk digest ranges instead of one flat vector, touching `CalcRdSize`/`InitIOV`/`ShiftIOV` (50–100); `ParseResponse` case `kXR_pgread` per-chunk accounting, with the `kXR_readv` case immediately below as the template (~80); `PgReadVImpl` modelled on `VectorRead` (~100); retry bookkeeping (below) (~80); `File::PgReadV` sync+async API and response type (~100). | 500–700 | | `XrdOuc/XrdOucCache.hh` | one new virtual pair, defaulting to a loop of `pgRead`, following the existing `ReadV`/async-`ReadV` pattern at lines 384/397 — so every existing cache and IO keeps working unchanged. | ~40 | | `XrdPosix` | `XrdPosixFile::pgReadV` over `XrdCl::File::PgReadV`, modelled on its `ReadV` at `XrdPosixFile.cc:714`; `XrdPss` pass-through. | ~80 | | **total** | | **~900–1300** | ## The parts we think are genuinely awkward 1. **Per-chunk digest indexing.** `AsyncPageReader::digests` is one flat vector sized from `csNum(chunks.front().offset, total_length)`, and `dgindex` walks it linearly. Per-extent digest ranges are needed. This lands in the partial-read resumption path (`ShiftIOV`, `suRetry`), where bugs are load-dependent rather than reproducible. 2. **Checksum-error retry.** `PgReadHandler` (`XrdClFileStateHandler.cc:150-200`) verifies crc32c page by page and on mismatch retries a single page via a flat `pgnb` index. With N extents this needs `(extent, page)` addressing. The retry request itself stays a single-page `kXR_pgread` with `kXR_pgRetry`, so only bookkeeping changes — but it is the easiest thing here to get subtly wrong. 3. **Capability negotiation.** Needs a protocol-version or `kXR_Qconfig` probe so clients can detect support and fall back to per-extent pgRead. Small, but must not be forgotten. ## What we are doing regardless We are not blocked on this. The XrdPfc side is being restructured now so that a vector read is expanded into contiguous *block runs* rather than individual blocks, with optional coalescing across small gaps. That alone takes the checksummed path from 5990 pgReads to 435 while fetching 74 MB instead of the 109 MB that today's 128 kB configuration fetches — i.e. better than current production on both axes, with no protocol change. So the request for `kXR_pgreadv` is an optimisation on top of that (435 → 8), not a rescue. Once the `XrdOucCacheIO` method exists, using it is a one-branch change in the cache. ## Andy's alternative: crc32c in the readv response, no new request Raised by Andy on 2026-09-02. It may remove the need for `kXR_pgreadv` altogether, and it looks sound. Checked against the code: `readahead_list` is 16 bytes and serves as both the request element and the response header: ```c struct readahead_list { kXR_char fhandle[4]; kXR_int32 rlen; kXR_int64 offset; }; ``` The per-element `fhandle` exists to allow one readv to span several open files, which nobody does. In the response the server echoes the handle the client already sent (`XrdXrootdXeq.cc:89`), so those **4 bytes are dead weight — and crc32c is exactly 4 bytes wide**. Under a flag, the response can carry the crc32c of each iochunk there. There is room for the flag: `ClientReadVRequest` carries `reserved[15]`, and `do_ReadV` references neither it nor `pathid`. ### What it costs us Granularity. pgRead gives a checksum per 4 kB page; this gives one per readv element, which for the cache means one per block run — up to `pfc.iosize`, currently 1 MB. Two consequences: * **Per-page values must be recomputed cache-side** for `XrdOssCsi`. That is cheaper than it sounds: `XrdOucCRC::Calc32C(data, count, uint32_t *csval)` already fills a per-page checksum vector with hardware assist, and `Calc32C(data, count, prevcs)` gives the element-level value to verify against the wire. Two hardware-assisted passes over data we have just received over a 57.8 ms RTT link -- not measurable. * **A failed check invalidates a whole element** rather than a 4 kB page, so a retry re-fetches up to 1 MB and cannot say which page was bad. Acceptable when checks essentially never fail; worth noting that pgRead's per-page retry is genuinely finer-grained. Andy notes that in practice almost nobody runs `XrdOssCsi`, since Ceph and ZFS checksum internally, which lowers the weight of the first point considerably. ### Why we like it * No new request id, no new response struct, no `XrdCl::File::PgReadV`, no `XrdOucCacheIO::pgReadV`. It deletes most of the ~900-1300 lines estimated above. * It still needs capability negotiation for the same reason `kXR_pgreadv` did: an old server ignores the reserved bytes and returns a file handle, which a new client would read as a checksum. * **It composes with the other request.** If `do_ReadV` gains an asynchronous path, checksummed vector reads inherit it for free. A separate `kXR_pgreadv` would need its own. ## Code Diffs against upstream `master`, on `github.com/osschar/xrootd`: * [`pfc-optimize-for-small-blocks`](https://github.com/xrootd/xrootd/compare/master...osschar:pfc-optimize-for-small-blocks) — block runs, `pfc.iosize`, `pfc.iogap`, the `pfc.flush` unit fix, and the NETTIME/CLIREAD tracing (compile-time gated, off). The pgReadV implementation described in the addendum below is on three branches that stack: * [`pgreadv`](https://github.com/xrootd/xrootd/compare/master...osschar:pgreadv) — 4 commits: `kXR_pgreadv` defined in `XProtocol.hh`, `XrdCl::File::PgReadV` (initially a loop over `PgRead`), `pgReadV` added to `XrdOucCacheIO` and implemented in `XrdPosixFile`. **No wire change**, so it is the part that is safe on its own. * [`pgreadv-wire`](https://github.com/xrootd/xrootd/compare/master...osschar:pgreadv-wire) — the above plus 4 more: `do_PgReadV` in the server, sending `kXR_pgreadv`, per-(extent, page) checksum verify and retry, and capability negotiation with a per-extent `pgRead` fallback. * [`pfc-pgreadv`](https://github.com/xrootd/xrootd/compare/master...osschar:pfc-pgreadv) — `pgreadv-wire` merged with the cache work, plus the cache-side change that batches block runs into pgReadV when checksums come from the network. This is what produced the 587-vs-13,405 request counts. Note `pfc-pgreadv` merged the cache branch before the tracing commit was added to it, so it does not carry NETTIME/CLIREAD. ## Open questions for you - Is reusing `read_list` for the request acceptable, or would you rather have a distinct arg struct? - `kXR_FinalResult` semantics across extents: one final per extent, or one final for the whole request? We assumed the latter. - Monitoring: new vType alongside `XROOTD_MON_READV`/`READU`, or fold into the existing pgRead stats (`pgrOps`)? - Would you want `pgWriteV` for symmetry, or is read-only fine? XrdPfc only needs the read side. --- ## Addendum: as built, 2026-09-01 *Added after the implementation. This records what the code turned out to be, so the open questions below are not asked twice. One judgement above has since been revised: the closing "optimisation, not a rescue" was measured against an origin 8 ms away. At 57.8 ms, with vector reads serialised per link, the same argument becomes a feasibility one -- see the companion document, "Vector reads, serialised".* **All of it is implemented and tested**, on two branches off `master`: `pgreadv` (the API and a fallback, no wire change) and `pgreadv-wire` (the protocol). Together, 1435 insertions over 21 files, against the 900-1300 estimated. The XrdPfc side is one further commit on its own branch. ### What the three "cheaper than it sounds" claims came to 1. **The response wire format needed no change at all** -- confirmed. A test client parses the pgreadv response stream using the existing structs and no new ones. 2. **No XrdSfs / XrdOfs / XrdOss change** -- confirmed, none of them were touched. `do_PgRIO` became `do_PgRIO(bool isLast)` and the extents are looped outside it, so it still sees exactly the single-extent state it always did, stack arrays and `kXR_Impossible` guard included. 3. **XrdOssCsi needed no changes** -- confirmed, untouched. ### Estimates versus actual | layer | estimated | actual | |---|---|---| | `XProtocol.hh` | ~30 | 34 | | `XrdXrootd` | 250-400 | 121 | | `XrdOssCsi`, `XrdSfs`, `XrdOfs`, `XrdOss` | 0 | 0 | | `XrdCl` | 500-700 | ~1070 | | `XrdOuc/XrdOucCache.hh` | ~40 | 87 | | `XrdPosix` | ~80 | 167 | `XrdXrootd` came in well under: looping the extents around `do_PgRIO` rather than inside it is most of why. `XrdCl` came in over, and the overrun is almost entirely the two things this document called awkward, plus their comments. ### The open questions, answered provisionally Each is one commit, so any of them is cheap to change. - **Reusing `read_list`:** done, unchanged. `ClientPgReadVRequest` is 24 bytes like every other request header, and keeps `pathid` at the same offset as `ClientReadVRequest` so the server's handling can be shared. `reqflags` sits in the header at byte 18, since `dlen` is taken by the read_list. - **`kXR_FinalResult` across extents:** one final for the whole request, on the last partial of the last extent, as assumed here. - **Monitoring:** no new vType. Each element is reported exactly as a pgread of it would be -- `Add_rd` and `pgrOps` per element -- so the batching is not visible in the monitoring stream. Adding a vType is still your call. - **`pgWriteV`:** not implemented. XrdPfc does not need it. Two more decisions that were not in the list: - **Per-element length is uncapped.** Unlike readv, which buffers a whole response, each extent is streamed as its own sequence of partial results, so the quantum bounds the buffer rather than the request. Only the element count is capped, at `maxRvecsz`, because the vector is copied to the stack so `do_PgRIO` can reuse the request buffer. - **`pathid` is rejected, not ignored.** `kXR_readv` ignores it; `do_PgReadV` returns `kXR_ArgInvalid` for a non-zero one, so a client that asked for an alternate path gets an error instead of waiting forever on a path the response will not arrive on. Nothing in XrdCl sets it. ### Capability negotiation `kXR_PROTPGRVVERSION`, with the protocol version moved to 5.2.1. **Which release first carries the request is yours to assign** -- the number here is a placeholder for the mechanism. For reference, stock xrootd 5.9.2 reports protocol 5.1.1, so there is room below it. A client that finds the server too old, or a vector too long for one request, substitutes a pgRead per extent and returns the identical answer in more round trips. A server that passes the version gate and then rejects the request is not recovered from; the version is taken as the contract. ### One unrelated bug found on the way `XrdPosixPrepIO` overrode `Read` and `ReadV` but never `pgRead`, so pgRead fell through to the `XrdOucCacheIO` default, which reads with `Read()` and computes checksums only under `forceCS`. Data correct, checksums silently dropped -- and XrdPfc reads that as "this origin cannot supply checksums" and marks the file as having none, permanently. It predates this work and is reproducible with no vector read involved. Fixed on the `pgreadv` branch; that commit stands alone against `master` and is worth taking regardless of what happens to pgreadv.