"""Multi-Family Spectral Residual & Diversified Risk-Parity Books (MFSR-DRB).

Model Name: Multi-Family Spectral Residual & Diversified Risk-Parity Books
Authors: Kush B Vaidya, Tareef Ahmed, Marco Chen, Keren Wang, Tanush Agarwal, Harry Chen

Overview
--------
We unify multi-horizon economic-graph signal extraction, heterogeneous multi-family alpha
learning across dual orthogonalized return targets, and equal-risk multi-book portfolio
construction with decoupled non-overlapping volatility targeting. Every feature, target
orthogonalization, model weight, spectral gain, risk covariance matrix, and volatility
scale at month t is estimated strictly from information realized on or before month t,
with deterministic single-threaded linear algebra and bit-for-bit reproducibility.

1. Inputs & 47 Multi-Horizon Causal Daily / Missingness Features:
   Each month t, the 402 firm characteristics from the provided feature registry are stored
   in two representations: (i) cross-sectionally percentile-ranked into [-0.5, 0.5] with
   missing entries mapped to 0.0 for linear and neural learners, and (ii) missingness-
   preserving raw and rank-NaN representations for histogram gradient boosted trees.
   From daily excess returns strictly up to month-end t (restricted to stocks with at least
   one prior observed daily return so peer sets are invariant to future listings), we
   construct 47 strictly causal daily and cross-sectional structural features:
     a) 14 Trailing 21-Day Path, Industry & kNN20 Peer Features: cumulative 5d, 10d, 21d,
        and 1d returns, intermediate path legs (days -5..-2, -10..-6, -21..-11), 5d extremes
        (max/min), 21d volatility, masked 3-digit SIC industry means of 5d and 21d returns,
        and deduplicated 20-nearest-neighbor characteristic-peer means of 5d and 21d returns.
     b) 20 Trailing 252-Day Marchenko-Pastur Factor-Residual, Attention-Peer & Coverage
        Features: On trailing 252-day standardized returns, eigenvalues exceeding the
        Marchenko-Pastur (1967) upper edge (1 + sqrt(N / T))^2 isolate systematic daily
        factors U_k and loadings B. Standardizing daily factor residuals by specific risk
        sqrt(psi_i) extracts 1d, 5d, 21d, and 63-21d MP residual shocks, residual lag-1
        autocorrelation, mean top-3 shock, exponential-decay residuals (half-lives 3d and
        5d), 5d/63d variance ratio, volatility term-structure (sigma_63 / sigma_252), 21d
        skewness, characteristic-peer dislocations, softmax-attention 40-peer spillovers,
        market-equity-weighted peer MP spillover (Hou 2007), SIC4 residual mean, and
        cross-sectional characteristic coverage rank.
     c) 13 Trailing 756-Day MP-PCA Risk, Beta-Shift, Residual IR & Asymmetry Features:
        Ledoit-Wolf shrunk 756-day MP-PCA total volatility sigma_mp756 and specific variance
        share psi_mp756, 252-day trailing daily market beta, fast beta shift (beta_63d -
        beta_252d), 756-day MP-PCA idiosyncratic residuals (5d, 21d, 63-21d, 126-21d),
        missingness rank, 252-21d and 126-21d idiosyncratic momentum information ratios,
        252-day downside-vs-upside idiosyncratic semivolatility asymmetry, and 252-day
        downside-vs-upside market beta asymmetry.

2. Triple Orthogonalized Return Targets (Strictly Used Only for tt < t):
   - Peer-Relative Target (y_peer): Forward 1-month excess returns demeaned cross-sectionally
     and within 3-digit SIC industries, passed through arctan(y / std(y)).
   - MP-PCA GLS Residual Target (y_pca_res): Forward 1-month excess returns scaled by
     1 / (sigma_mp756 * sqrt(psi_mp756)), projected via SVD orthogonal to
     [B_mp756, 1/sigma, SIC2_dummies/sigma], standardized, and clipped to [-4, 4].
   - Joint Statistical + Barra-13 GLS Residual Target (y_joint13_res): Forward 1-month
     excess returns scaled by 1 / (sigma_mp756 * sqrt(psi_mp756)), projected orthogonal to
     [B_mp756, 1/sigma, Z_barra13/sigma, SIC2_dummies/sigma], standardized, and clipped to
     [-4, 4], providing exact Grinold-Kahn target alignment for the Joint-13 Residual Book.
   - Raw Cross-Sectional Z-Score Target (y_z_raw): Within-month standardized excess return.

3. Heterogeneous Four-Family Target-Matched Alpha Ensemble:
   a) Shared-Gram Triple-Target Multi-Scale Modulated Ridge + Quadratic Tail: Refit every
      January blending expanding (70%) and 120-month EWMA (30%) sufficient statistics with
      size, arbitrage-friction, variance-matched daily-modulator, and rank-2 Kronecker-
      whitened bilinear interaction blocks, solved simultaneously for y_peer, y_pca_res,
      and y_joint13_res.
   b) Five-Year Expanding Two-Stage Smooth-SWA ReLU Residual MLP Cascades (3 seeds) on
      y_peer, y_pca_res, and y_joint13_res residuals, plus a compact Smooth-SWA Tanh MLP
      (64-32-16) on y_z_raw for complementary smooth nonlinear interactions.
   c) Five-Year Expanding Deterministic Histogram Gradient Boosted Trees
      (HistGradientBoostingRegressor, 2 regularized tree complexities per family, 90 trees,
      deterministic 260,000-row causal subagging when historical rows exceed 260k) across
      four complementary representations/targets: Rank-NaN 449-feature trees on y_pca_res,
      Rank-NaN 449-feature trees on y_joint13_res, Raw-NaN 449-feature trees on y_pca_res,
      and pure-fundamental Raw-NaN 402-feature trees on y_pca_res.
   Three book-matched alpha forecasts (s_spin, s_mvo, s_j13) align each risk engine to its
   exact factor-neutral return subspace.

4. Equal-Risk Diversified Five-Book Portfolio & Non-Overlapping Volatility Allocation:
   Five genuinely distinct risk engines are combined on an inverse-ex-ante-volatility
   (equal-risk) basis (35% / 17% / 22% / 12% / 14%):
     - Book 1 (35%): 5-Band Graph Laplacian + 3-Day Overlapping Lead-Lag Mean-Field Spin
       Book (beta = 0.60, J = 0.40, peer-pooled volatility 0.40).
     - Book 2 (17%): 3-Day Overlapping Ledoit-Wolf Daily Covariance MVO (Tangency minus GMV
       with Tikhonov ridge kappa = 1.00).
     - Book 3 (22%): Joint Statistical + Fundamental Barra-13 Residual Book fed its target-
       matched s_j13 forecast, projecting orthogonal to [B_mp756, 1/sigma, Z_barra13/sigma,
       SIC2/sigma].
     - Book 4 (12%): Pure Statistical MP-PCA + SIC2 GLS Residual Book preserving orthogonal
       fundamental style tilts absent from the Barra-neutral book.
     - Book 5 (14%): Causal 60-Month Fundamental Barra-13 Factor MVO solved via Woodbury
       inversion with trailing monthly cross-sectional factor regressions and SIC2 demeaning,
       contributing genuine monthly-horizon covariance diversification orthogonal to daily
       covariance engines.
   The combined portfolio is projected strictly orthogonal to [1, beta_252d, beta_60m,
   log(ME)] and scaled by non-overlapping ex-ante daily/diagonal/LW volatility with causal
   online realized-to-ex-ante calibration and algebraic Lp soft-capping (vol_power = 1.10).

Attribution
-----------
Builds upon the public JKP CTF submissions "Residual Signal Diversified Books" by Abhinav
Keshri (2026), "Topology-Informed Decomposition Ensemble" by Hema Srikar Ankem (2026), and
"Residual Books" by Felix Gehling (2026), alongside academic research by Gu, Kelly, and Xiu
(2020), Jensen, Kelly, and Pedersen (2023), Marchenko and Pastur (1967), Lo and MacKinlay
(1990), Hou (2007), Ledoit and Wolf (2004), Izmailov et al. (2018), and Moreira and Muir
(2017).
"""
from __future__ import annotations

from concurrent.futures import ThreadPoolExecutor
import gc
import math
import os
import sys
import time

# Force single-threaded BLAS/OpenMP prior to scientific imports for bit-for-bit determinism
# across machines regardless of physical CPU core count.
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
os.environ["NUMEXPR_NUM_THREADS"] = "1"

import numpy as np
import pandas as pd
from scipy.linalg import solve as sp_solve
from scipy.optimize import minimize
from scipy.stats import rankdata as sp_rankdata
from sklearn.ensemble import HistGradientBoostingRegressor

# Locked Pre-1990 Hyperparameters (strictly calibrated on ctff_test == '0', <= 1989-11-30)
ridge_main = 500.0
ridge_inter = 2000.0
ridge_ewma_halflife = 120.0
ridge_ewma_alpha = 0.30
modulator_rank = 2

path_features = 14
daily34_cols = 34
extra756_cols = 13
total_daily_features = daily34_cols + extra756_cols
path_block_scale = 12.0 ** 0.5

arbitrage_modulators = ("ivol_capm_252d", "ami_126d")
path_modulators = (
    "beta_60m", "turnover_126d", "dolvol_126d", "zero_trades_21d", "ami_126d",
    "bidaskhl_21d", "prc", "ivol_capm_21d", "rvol_21d", "rmax5_21d",
)
barra_style_chars = (
    "market_equity", "be_me", "ret_12_1", "ret_1_0", "ivol_capm_252d", "ope_be", "qmj",
    "at_gr1", "oaccruals_at", "debt_gr3", "at_be", "niq_be_chg1", "seas_2_5an",
)

mlp_seeds = 3
mlp_relu_hidden = (256, 128, 64)
mlp_relu_epochs = 7
mlp_relu_lr = 1.15e-3
mlp_relu_wd = 1e-3
mlp_relu_dropout = 0.2
mlp_relu_feat_frac = 0.60

mlp_tanh_hidden = (64, 32, 16)
mlp_tanh_epochs = 8
mlp_tanh_lr = 1.5e-3
mlp_tanh_wd = 1e-3
mlp_tanh_dropout = 0.10
mlp_tanh_feat_frac = 0.80

mlp_refit_years = 5
mlp_valid_years = 3
mlp_val_max_steps = 20
mlp_full_max_steps = 36
mlp_swa_temp = 150.0
mlp_stage2_offset = 100
mlp_tanh_offset = 400

hgb_refit_years = 5
hgb_min_train_rows = 2500
hgb_max_train_rows = 260_000
hgb_configs = (
    {
        "lr": 0.055, "max_iter": 90, "max_leaf_nodes": 63, "max_depth": 8,
        "min_samples_leaf": 600, "l2": 5.0, "max_features": 0.45, "max_bins": 63,
    },
    {
        "lr": 0.060, "max_iter": 90, "max_leaf_nodes": 31, "max_depth": 6,
        "min_samples_leaf": 1500, "l2": 15.0, "max_features": 0.35, "max_bins": 63,
    },
)

knn_peers = 20
band_window = 180
band_shrink = 0.5
min_band_history = 60

sample_window_days = 756
min_daily_obs = 250
psi_floor = 0.05
min_industry_size = 10
days_per_month = 21

spin_beta = 0.60
spin_j = 0.40
vol_pool = 0.40
lw_mvo_kappa = 1.00
min_names = 50

# Book-matched alpha weights (sum = 1.00)
w_spin_alpha = {
    "peer_cas": 0.28,
    "pca_cas": 0.28,
    "tanh_z": 0.10,
    "hgb_rn_pca": 0.18,
    "hgb_raw_pca": 0.10,
    "hgb_raw402": 0.06,
}
w_mvo_alpha = {
    "pca_cas": 0.36,
    "tanh_z": 0.12,
    "hgb_rn_pca": 0.26,
    "hgb_raw_pca": 0.16,
    "hgb_raw402": 0.10,
}
w_j13_alpha = {
    "j13_cas": 0.30,
    "pca_cas": 0.18,
    "hgb_rn_j13": 0.22,
    "hgb_rn_pca": 0.14,
    "hgb_raw_pca": 0.10,
    "hgb_raw402": 0.06,
}

# Equal-risk (inverse ex-ante vol) 5-book allocation weights:
# (Spin5, DailyLW3d_MVO, Joint13_Resid, PureMP_Resid, FMVO_Barra13_60m)
book_risk_weights = (0.35, 0.17, 0.22, 0.12, 0.14)
fmvo_window_months = 60

target_vol_annual = 0.10
gross_cap = 12.0
soft_cap_ceiling = 11.95
soft_cap_power = 3.50
realised_lookback_days = 252
realised_halflife_days = 21
vol_power = 1.10
vref_window_months = 48
vol_w_cross = 0.45
vol_w_diag = 0.33
vol_w_lw = 0.22
vol_cal_exp = 0.60
vol_cal_lo = 0.78
vol_cal_hi = 1.22

start_date = "1960-01-01"
target_col = "ret_exc_lead1m"
feature_list_col = "features"
seed = 42


def group_mean(v: np.ndarray, lab: np.ndarray) -> np.ndarray:
    if len(v) == 0:
        return np.zeros(0, dtype=np.float64)
    _, inv = np.unique(lab, return_inverse=True)
    cnts = np.maximum(np.bincount(inv), 1)
    return (np.bincount(inv, np.asarray(v, dtype=np.float64)) / cnts)[inv]


def masked_group_mean(values: np.ndarray, codes: np.ndarray, valid_mask: np.ndarray) -> np.ndarray:
    if len(values) == 0:
        return np.zeros(0, dtype=np.float32)
    msk = np.asarray(valid_mask, dtype=np.float64)
    vals = np.where(valid_mask, np.asarray(values, dtype=np.float64), 0.0)
    _, inv = np.unique(codes, return_inverse=True)
    num = np.bincount(inv, weights=vals)[inv]
    den = np.bincount(inv, weights=msk)[inv]
    return np.where(den > 1e-6, num / np.maximum(den, 1e-6), 0.0).astype(np.float32)


def zs(v: np.ndarray) -> np.ndarray:
    arr = np.asarray(v, dtype=np.float64)
    if len(arr) == 0:
        return np.zeros(0, dtype=np.float64)
    c = arr - float(arr.mean())
    sd = float(c.std())
    return c / sd if sd > 1e-12 else np.zeros_like(c)


def cs_rank_into(vals: np.ndarray, valid_mask: np.ndarray) -> np.ndarray:
    n = len(vals)
    out = np.zeros(n, dtype=np.float32)
    if n == 0:
        return out
    msk = np.asarray(valid_mask, dtype=bool) & np.isfinite(vals)
    cnt = int(msk.sum())
    if cnt == 0:
        return out
    rk = sp_rankdata(np.asarray(vals[msk], dtype=np.float64), method="average")
    out[msk] = (rk / float(cnt) - 0.5).astype(np.float32)
    return out


def knn_with_sim_f32(rk: np.ndarray, k: int = 20) -> tuple[np.ndarray, np.ndarray]:
    n = len(rk)
    if n <= 1 or k <= 0:
        return np.zeros((n, 0), dtype=np.int64), np.zeros((n, 0), dtype=np.float32)
    k_eff = min(int(k), n - 1)
    xf = np.ascontiguousarray(rk, dtype=np.float32)
    nrm = np.sqrt(np.sum(xf * xf, axis=1, keepdims=True))
    z = xf / np.maximum(nrm, np.float32(1e-12))
    sim = z @ z.T
    np.fill_diagonal(sim, -np.float32(np.inf))
    idx_k = np.argpartition(-sim, kth=k_eff - 1, axis=1)[:, :k_eff]
    sim_k = np.take_along_axis(sim, idx_k, axis=1)
    ord_k = np.argsort(-sim_k, axis=1, kind="stable")
    return np.take_along_axis(idx_k, ord_k, axis=1), np.take_along_axis(sim_k, ord_k, axis=1)


def knn_f32(rk: np.ndarray, k: int = 20) -> np.ndarray:
    n = len(rk)
    if n <= 1:
        return np.zeros((n, 1), dtype=np.int64)
    k_eff = min(int(k), n - 1)
    rk32 = np.asarray(rk, dtype=np.float32)
    rn = rk32 / (np.linalg.norm(rk32, axis=1, keepdims=True) + np.float32(1e-12))
    sim = rn @ rn.T
    np.fill_diagonal(sim, -np.inf)
    return np.argpartition(-sim, k_eff, axis=1)[:, :k_eff]


def inv_sqrt_psd(mat: np.ndarray) -> np.ndarray:
    evals, evecs = np.linalg.eigh(mat)
    return (evecs * (1.0 / np.sqrt(np.maximum(evals, 1e-12)))) @ evecs.T


def ledoit_wolf_f32(x32: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    t, n = x32.shape
    s64 = ((x32.T @ x32) / np.float32(t)).astype(np.float64)
    mu = np.trace(s64) / n
    ss = np.sum(s64 ** 2)
    d2 = ss - 2.0 * mu * np.trace(s64) + n * mu * mu
    rs = np.sum(x32.astype(np.float64) ** 2, axis=1)
    b2 = np.sum(rs ** 2) / (t ** 2) - ss / t
    delta = min(max(b2, 0.0), d2) / d2 if d2 > 0 else 1.0
    s64 *= (1.0 - delta)
    s64[np.diag_indices(n)] += delta * mu
    sd = np.sqrt(np.maximum(np.diag(s64), 1e-18))
    return (s64 / np.outer(sd, sd)).astype(np.float32), sd


def _pca_risk_with_factors(
    r_mat: np.ndarray,
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    """756-day MP-PCA Ledoit-Wolf shrunk statistical risk model."""
    t_w, n_m = r_mat.shape
    xc = r_mat - r_mat.mean(0, keepdims=True)
    ss = (xc * xc).sum(0)
    sigma = np.sqrt(ss / max(t_w - 1, 1) * float(days_per_month))
    xn = xc / np.sqrt(np.maximum(ss, 1e-12))
    mp_edge = (1.0 + math.sqrt(float(n_m) / float(max(t_w, 1)))) ** 2
    if n_m < t_w:
        corr_n = xn.T @ xn
        ev, v_vecs = np.linalg.eigh(corr_n)
        sel = ev > mp_edge
        if not sel.any():
            sel[-1] = True
        ev_sel = np.maximum(ev[sel], 1e-12)
        v_k = v_vecs[:, sel]
        u_k = (xn @ v_k) / np.sqrt(ev_sel)[None, :]
        b_raw = v_k * np.sqrt(ev_sel)[None, :]
        s2 = float((corr_n * corr_n).sum())
    else:
        gram = xn @ xn.T
        ev, vecs = np.linalg.eigh(gram)
        sel = ev > mp_edge
        if not sel.any():
            sel[-1] = True
        u_k = vecs[:, sel]
        b_raw = xn.T @ u_k
        s2 = float((gram * gram).sum())
    d2 = s2 - float(n_m)
    b2 = float(((xn * xn).sum(1) ** 2).sum()) - s2 / float(max(t_w, 1))
    delta = min(max(b2, 0.0), d2) / d2 if d2 > 0 else 1.0
    b_load = b_raw * math.sqrt(1.0 - delta)
    psi = np.maximum(1.0 - (b_load * b_load).sum(1), psi_floor)
    return b_load, psi, sigma, u_k, xn


def _industry_dummies(ind: np.ndarray) -> np.ndarray:
    s = pd.Series(np.asarray(ind))
    s = s.where(s.map(s.value_counts()) >= min_industry_size, -999)
    return pd.get_dummies(s).to_numpy(dtype=np.float64)


def _basis(g_mat: np.ndarray, psi: np.ndarray) -> np.ndarray:
    u_mat, s_vals, _ = np.linalg.svd(g_mat / np.sqrt(psi)[:, None], full_matrices=False)
    if len(s_vals) == 0 or s_vals.max() <= 0:
        return u_mat[:, :0]
    return u_mat[:, s_vals > 1e-10 * s_vals.max()]


def _residual_weights(mu: np.ndarray, sigma: np.ndarray, psi: np.ndarray, g_mat: np.ndarray) -> np.ndarray:
    sc = np.maximum(sigma * np.sqrt(psi), 1e-8)
    s = np.where(np.isfinite(mu), mu, 0.0) / sc
    s = s - s.mean()
    u_mat = _basis(g_mat, psi)
    p = s - u_mat @ (u_mat.T @ s)
    nrm = float(np.linalg.norm(p))
    return p / nrm / sc if nrm > 1e-14 else np.zeros(len(s), dtype=np.float64)


def tan_minus_gmv(mu: np.ndarray, sigma_mat: np.ndarray, kappa: float = lw_mvo_kappa) -> np.ndarray:
    n = mu.shape[0]
    lam = kappa * float(np.trace(sigma_mat)) / max(n, 1)
    a_mat = sigma_mat + lam * np.eye(n, dtype=np.float64)
    one = np.ones(n, dtype=np.float64)
    sol = np.linalg.solve(a_mat, np.column_stack([mu, one]))
    a_vec, b_vec = sol[:, 0], sol[:, 1]
    ob = float(one @ b_vec)
    w = a_vec - (float(one @ a_vec) / ob) * b_vec if abs(ob) > 1e-14 else a_vec
    den = float(np.abs(w).sum())
    return w / den if den > 1e-12 else np.zeros(n, dtype=np.float64)


def _woodbury_sigma_inv_mu(
    b_mat: np.ndarray, omega: np.ndarray, d_var: np.ndarray, mu: np.ndarray
) -> np.ndarray:
    """Invert (B Omega B' + diag(d_var)) against mu via Woodbury identity."""
    dinv = 1.0 / np.maximum(d_var, 1e-10)
    bt_dinv = b_mat.T * dinv
    k_f = b_mat.shape[1]
    mx = np.linalg.inv(omega + 1e-7 * np.eye(k_f, dtype=np.float64)) + bt_dinv @ b_mat
    return dinv * mu - dinv * (b_mat @ np.linalg.solve(mx, bt_dinv @ mu))


def spin_solve(h: np.ndarray, corr32: np.ndarray) -> np.ndarray:
    h = (h - h.mean()) / (h.std() + 1e-12)
    eps = 1e-6

    def free_energy(m: np.ndarray) -> tuple[float, np.ndarray]:
        cm = (corr32 @ m.astype(np.float32)).astype(np.float64)
        mc = np.clip(m, -1.0 + eps, 1.0 - eps)
        ent = (
            (1.0 + mc) / 2.0 * np.log((1.0 + mc) / 2.0)
            + (1.0 - mc) / 2.0 * np.log((1.0 - mc) / 2.0)
        ).sum() / spin_beta
        val = float(0.5 * spin_j * (m @ cm) - (h @ m) + ent)
        grad = spin_j * cm - h + np.arctanh(mc) / spin_beta
        return val, grad

    res = minimize(
        free_energy,
        0.5 * np.tanh(spin_beta * h),
        jac=True,
        method="L-BFGS-B",
        bounds=[(-1.0 + eps, 1.0 - eps)] * len(h),
        options={"maxiter": 200},
    )
    return res.x


def exante_vol_of_unit(
    w_u: np.ndarray,
    xd_c: np.ndarray,
    corr32: np.ndarray,
    own_sd: np.ndarray,
    sd21_diag: np.ndarray,
    sd63_diag: np.ndarray,
) -> float:
    w32 = w_u.astype(np.float32)
    pnl = (xd_c[-realised_lookback_days:].astype(np.float32) @ w32).astype(np.float64)
    n_pnl = len(pnl)
    if n_pnl == 0:
        return 0.0
    decay = 0.5 ** (np.arange(n_pnl - 1, -1, -1, dtype=np.float64) / realised_halflife_days)
    vol_ew21 = float(np.sqrt((decay * (pnl ** 2)).sum() / decay.sum() * days_per_month))
    pnl_21 = pnl[-21:] if n_pnl >= 21 else pnl
    vol_spot21 = (
        float(np.sqrt(np.mean(pnl_21 ** 2) * days_per_month))
        if len(pnl_21) > 0
        else vol_ew21
    )
    v_fast = max(vol_ew21, vol_spot21)
    pnl_out = pnl[:-63] if n_pnl > 84 else pnl
    vol_out63 = (
        float(np.sqrt(np.mean(pnl_out ** 2) * days_per_month))
        if len(pnl_out) > 0
        else vol_ew21
    )
    u32 = (w_u * own_sd).astype(np.float32)
    vol_lw = float(np.sqrt(max(float(u32 @ (corr32 @ u32)), 1e-18)))
    vol_lw_eff = min(vol_lw, 1.50 * max(v_fast, vol_out63))
    vol_diag21 = float(np.sqrt(np.sum((w_u * sd21_diag) ** 2)))
    vol_diag63 = float(np.sqrt(np.sum((w_u * sd63_diag) ** 2)))
    v_diag = 3.00 * max(vol_diag21, vol_diag63)
    return float(
        np.sqrt(
            vol_w_cross * v_fast * v_diag
            + vol_w_diag * (v_diag ** 2)
            + vol_w_lw * ((3.00 * vol_lw_eff) ** 2)
        )
    )


def soft_cap_lp(lev: float, cap: float = soft_cap_ceiling, p: float = soft_cap_power) -> float:
    """Strictly increasing smooth algebraic Lp soft-cap bounded in (0, cap)."""
    if lev <= 0.0 or not math.isfinite(lev):
        return 0.0
    x = lev / cap
    if x > 50.0:
        return float(cap * (1.0 - (50.0 / x) * 1e-4))
    return float(lev / ((1.0 + (x ** p)) ** (1.0 / p)))


def project_neutral_and_safe_cap(
    raw_w: np.ndarray,
    beta_v: np.ndarray,
    lme_v: np.ndarray,
    bet60_v: np.ndarray | None = None,
    max_share: float = 0.055,
) -> np.ndarray:
    n = len(raw_w)
    if n < 5 or float(np.abs(raw_w).sum()) <= 1e-14:
        return np.zeros(n, dtype=np.float64)
    cols = [
        np.ones(n, dtype=np.float64),
        beta_v - float(beta_v.mean()),
        lme_v - float(lme_v.mean()),
    ]
    if bet60_v is not None and float(np.std(bet60_v)) > 1e-10:
        cols.append(bet60_v - float(bet60_v.mean()))
    basis = np.column_stack(cols)
    w = raw_w.astype(np.float64).copy()
    cf, *_ = np.linalg.lstsq(basis, w, rcond=None)
    w = w - basis @ cf
    w = w - float(w.mean())
    g = float(np.abs(w).sum())
    if g <= 1e-12:
        return np.zeros(n, dtype=np.float64)
    w = w / g
    # Concentration guard (inactive on broad universes where max|w| ~ 0.015; bounds small-N stress tests)
    if n >= 25 and float(np.abs(w).max()) > max_share:
        for _ in range(8):
            w = np.clip(w, -max_share, max_share)
            cf, *_ = np.linalg.lstsq(basis, w, rcond=None)
            w = w - basis @ cf
            w = w - float(w.mean())
            g2 = float(np.abs(w).sum())
            if g2 <= 1e-12:
                return np.zeros(n, dtype=np.float64)
            w = w / g2
            if float(np.abs(w).max()) <= max_share + 1e-4:
                break
    return w


def rank_panel_inplace(x: np.ndarray, starts: np.ndarray, ends: np.ndarray, n_cols: int) -> None:
    if n_cols <= 0:
        return
    for a, b in zip(starts, ends):
        if b <= a:
            continue
        rk = pd.DataFrame(x[a:b, :n_cols]).rank(pct=True).to_numpy(dtype=np.float32) - np.float32(0.5)
        rk[np.isnan(rk)] = 0.0
        x[a:b, :n_cols] = rk


def _series_to_dates(s: pd.Series) -> np.ndarray:
    if "pyarrow" in str(s.dtype):
        return s.astype("timestamp[s][pyarrow]").to_numpy().astype("datetime64[D]")
    return pd.to_datetime(s).to_numpy(dtype="datetime64[D]")


def _series_to_float(s: pd.Series, dt: type) -> np.ndarray:
    if "pyarrow" in str(s.dtype):
        return s.to_numpy(dtype=dt, na_value=np.nan)
    return s.to_numpy(dtype=dt)


def build_daily_matrix(
    daily_ret: pd.DataFrame, panel_ids: np.ndarray
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    t0 = time.perf_counter()
    if daily_ret is None or len(daily_ret) == 0 or len(panel_ids) == 0:
        return (
            np.empty(0, dtype="datetime64[D]"),
            np.empty(0, dtype=np.int64),
            np.empty((0, 0), dtype=np.float32),
            np.full(1, 1, dtype=np.int64),
        )
    n_dr = len(daily_ret)
    pid = np.asarray(panel_ids, dtype=np.int64)
    chunk = 2_000_000
    col_id = daily_ret["id"]
    col_dt = daily_ret["date"]
    col_ret = daily_ret["ret_exc"]

    date_parts: list[np.ndarray] = []
    id_parts: list[np.ndarray] = []
    for i in range(0, n_dr, chunk):
        j = min(i + chunk, n_dr)
        dt_c = _series_to_dates(col_dt.iloc[i:j])
        dt_ok = dt_c[~np.isnat(dt_c)]
        if len(dt_ok) > 0:
            date_parts.append(np.unique(dt_ok))
        rid_c = _series_to_float(col_id.iloc[i:j], np.float64)
        rret_c = _series_to_float(col_ret.iloc[i:j], np.float32)
        fin_c = np.isfinite(rid_c) & np.isfinite(rret_c) & (~np.isnat(dt_c))
        if not fin_c.any():
            continue
        rid_i = rid_c[fin_c].astype(np.int64)
        keep_i = np.isin(rid_i, pid, kind="table")
        if keep_i.any():
            id_parts.append(np.unique(rid_i[keep_i]))

    all_dates = (
        np.unique(np.concatenate(date_parts))
        if date_parts
        else np.empty(0, dtype="datetime64[D]")
    )
    ids = np.unique(np.concatenate(id_parts)) if id_parts else np.empty(0, dtype=np.int64)
    del date_parts, id_parts
    gc.collect()

    m = np.full((len(all_dates), len(ids)), np.nan, dtype=np.float32)
    if len(ids) > 0 and len(all_dates) > 0:
        for i in range(0, n_dr, chunk):
            j = min(i + chunk, n_dr)
            rid_c = _series_to_float(col_id.iloc[i:j], np.float64)
            rret_c = _series_to_float(col_ret.iloc[i:j], np.float32)
            fin_c = np.isfinite(rid_c) & np.isfinite(rret_c)
            if not fin_c.any():
                continue
            dt_all = _series_to_dates(col_dt.iloc[i:j])
            fin_c = fin_c & (~np.isnat(dt_all))
            if not fin_c.any():
                continue
            rid_i = rid_c[fin_c].astype(np.int64)
            keep_f = np.isin(rid_i, pid, kind="table")
            if not keep_f.any():
                continue
            dt_k = dt_all[fin_c][keep_f]
            d_ix = np.searchsorted(all_dates, dt_k)
            i_ix = np.searchsorted(ids, rid_i[keep_f])
            m[d_ix, i_ix] = rret_c[fin_c][keep_f]

        first_obs_day = np.full(len(ids), len(all_dates), dtype=np.int64)
        for c0 in range(0, len(ids), 1024):
            c1 = min(c0 + 1024, len(ids))
            fin_blk = np.isfinite(m[:, c0:c1])
            first_obs_day[c0:c1] = np.where(
                fin_blk.any(axis=0), np.argmax(fin_blk, axis=0), len(all_dates)
            )
    else:
        first_obs_day = np.full(1, len(all_dates) + 1, dtype=np.int64)

    print(
        f"daily matrix {m.shape[0]:,} days x {m.shape[1]:,} ids in {time.perf_counter() - t0:.1f}s",
        flush=True,
    )
    return all_dates, ids, m, first_obs_day


def build_causal_features_and_targets(
    r: np.ndarray,
    miss_all: np.ndarray,
    n_char: int,
    starts: np.ndarray,
    ends: np.ndarray,
    months: np.ndarray,
    dates: np.ndarray,
    dmat: np.ndarray,
    col_all: np.ndarray,
    has_d: np.ndarray,
    first_obs_day: np.ndarray,
    sic: np.ndarray,
    y_all: np.ndarray,
    size_ix: int | None,
    barra_ix: list[int],
) -> dict[str, np.ndarray]:
    """Build 47 causal daily/756d-MP features in-place in r[:, n_char:] + triple targets & risk arrays."""
    n_rows = r.shape[0]
    r_char = r[:, :n_char]
    out_dl = r[:, n_char : n_char + daily34_cols]
    out_ex13 = r[:, n_char + daily34_cols : n_char + total_daily_features]

    y_peer = np.full(n_rows, np.nan, dtype=np.float32)
    y_pca_res = np.full(n_rows, np.nan, dtype=np.float32)
    y_joint13_res = np.full(n_rows, np.nan, dtype=np.float32)
    y_z_raw = np.full(n_rows, np.nan, dtype=np.float32)
    sig_mp756 = np.full(n_rows, np.nan, dtype=np.float32)
    psi_mp756 = np.full(n_rows, np.nan, dtype=np.float32)
    beta252 = np.zeros(n_rows, dtype=np.float32)
    beta252_rk = np.zeros(n_rows, dtype=np.float32)

    has_dmat = len(dates) > 0 and dmat.ndim == 2 and dmat.shape[0] > 0 and dmat.shape[1] > 0

    for t in range(len(months)):
        a, b = int(starts[t]), int(ends[t])
        n_m = b - a
        if n_m <= 0:
            continue
        rows = np.arange(a, b, dtype=np.int64)
        all_ones = np.ones(n_m, dtype=bool)
        out_dl[a:b, 33] = cs_rank_into(1.0 - miss_all[a:b], all_ones)

        # Targets y_peer & y_z_raw (ONLY read at future refits tt < t)
        rm = y_all[a:b]
        fin_m = np.isfinite(rm)
        if int(fin_m.sum()) > 2:
            yd = rm[fin_m] - float(rm[fin_m].mean())
            yd = yd - group_mean(yd, sic[a:b][fin_m] // 10)
            sd_p = float(yd.std())
            if sd_p > 1e-12:
                y_peer[rows[fin_m]] = np.arctan(yd / sd_p).astype(np.float32)
        if int(fin_m.sum()) >= 10:
            sd_m = float(rm[fin_m].std())
            if sd_m > 1e-12:
                y_z_raw[rows[fin_m]] = ((rm[fin_m] - float(rm[fin_m].mean())) / sd_m).astype(np.float32)

        if not has_dmat:
            out_ex13[a:b, 8] = cs_rank_into(miss_all[a:b], all_ones)
            continue

        end_d = int(np.searchsorted(dates, np.datetime64(months[t], "D"), side="right"))
        if end_d <= 0:
            out_ex13[a:b, 8] = cs_rank_into(miss_all[a:b], all_ones)
            continue

        hd = has_d[rows] & (first_obs_day[col_all[rows]] < end_d)
        rel_rw = np.flatnonzero(hd)

        # Block 1 & 2 (34 daily/peer/252d MP-PCA features matching m2_alpha_module)
        if end_d >= 25 and len(rel_rw) >= 25:
            rw = rows[rel_rw]
            lo252 = max(0, end_d - 252)
            w252 = np.asarray(dmat[lo252:end_d][:, col_all[rw]], dtype=np.float32)
            obs = np.isfinite(w252)
            t_len = w252.shape[0]

            k40 = min(40, len(rw) - 1)
            k20 = min(knn_peers, k40)
            rc_rw = np.asarray(r_char[rw], dtype=np.float32)
            nn40, sim40 = knn_with_sim_f32(rc_rw, k=k40)
            nn20 = nn40[:, :k20]

            l21 = min(21, t_len)
            fin21 = obs[-l21:]
            ok21 = fin21.sum(axis=0) >= 10
            w21 = np.where(fin21, w252[-l21:], np.float32(0.0))

            r5 = w21[-min(5, l21):].sum(axis=0)
            r10 = w21[-min(10, l21):].sum(axis=0)
            r21 = w21.sum(axis=0)
            r1 = w21[-1]
            max5 = w21[-min(5, l21):].max(axis=0)
            min5 = w21[-min(5, l21):].min(axis=0)
            std21 = w21.std(axis=0)
            r5_1 = w21[-min(5, l21):-1].sum(axis=0) if l21 >= 2 else np.zeros(len(rw), dtype=np.float32)
            r10_5 = w21[-min(10, l21):-5].sum(axis=0) if l21 >= 6 else np.zeros(len(rw), dtype=np.float32)
            r21_10 = w21[:-10].sum(axis=0) if l21 > 10 else np.zeros(len(rw), dtype=np.float32)

            s_sic3 = sic[rw] // 10
            sic2_r5 = masked_group_mean(r5, s_sic3, ok21)
            sic2_r21 = masked_group_mean(r21, s_sic3, ok21)

            m_ok21_20 = ok21[nn20].astype(np.float32)
            den20 = np.maximum(m_ok21_20.sum(axis=1), np.float32(1.0))
            knn20_r5 = (r5[nn20] * m_ok21_20).sum(axis=1) / den20
            knn20_r21 = (r21[nn20] * m_ok21_20).sum(axis=1) / den20

            blk1_raw = (
                r5, r10, r21, r1, max5, min5, std21, r5_1, r10_5, r21_10,
                sic2_r5, sic2_r21, knn20_r5, knn20_r21,
            )
            for c_i, arr in enumerate(blk1_raw):
                out_dl[rw, c_i] = cs_rank_into(arr, ok21)

            if end_d >= 65 and len(rw) >= 30 and t_len >= 63:
                n_obs = obs.sum(axis=0)
                ok63 = obs[-63:].sum(axis=0) >= 30
                ok_mp = n_obs >= 60

                x_z = np.where(obs, w252, np.float32(0.0))
                mean_s = x_z.sum(axis=0) / np.maximum(n_obs, 1).astype(np.float32)
                x_z = np.where(obs, x_z - mean_s, np.float32(0.0))
                ss252 = np.sum(x_z.astype(np.float64) ** 2, axis=0)
                sig252 = np.sqrt(ss252 / np.maximum(n_obs - 1, 1)).astype(np.float32)
                sig63 = np.sqrt(
                    np.sum(x_z[-63:].astype(np.float64) ** 2, axis=0)
                    / np.maximum(obs[-63:].sum(axis=0) - 1, 1)
                ).astype(np.float32)

                valid_mp = ok_mp & (ss252 > 1e-16)
                mp_r1 = np.zeros(len(rw), dtype=np.float32)
                mp_r5 = np.zeros(len(rw), dtype=np.float32)
                mp_r21 = np.zeros(len(rw), dtype=np.float32)
                mp_r63_21 = np.zeros(len(rw), dtype=np.float32)
                mp_ac1 = np.zeros(len(rw), dtype=np.float32)
                mp_max3 = np.zeros(len(rw), dtype=np.float32)
                mp_hl3 = np.zeros(len(rw), dtype=np.float32)
                mp_hl5 = np.zeros(len(rw), dtype=np.float32)
                mp_vr5_63 = np.ones(len(rw), dtype=np.float32)

                if int(valid_mp.sum()) >= 30:
                    xm = x_z[:, valid_mp] / np.sqrt(ss252[valid_mp]).astype(np.float32)
                    t_w, n_m_mp = xm.shape
                    evals, u_mat = np.linalg.eigh((xm @ xm.T).astype(np.float64))
                    desc = np.arange(len(evals) - 1, -1, -1)
                    evals, u_mat = evals[desc], u_mat[:, desc]
                    mp_edge = (1.0 + math.sqrt(float(n_m_mp) / float(t_w))) ** 2
                    k_ret = max(1, int(np.sum(evals > mp_edge)))
                    u_k = u_mat[:, :k_ret].astype(np.float32)
                    b_load = xm.T @ u_k
                    psi = np.maximum(np.float32(1.0) - np.sum(b_load * b_load, axis=1), np.float32(psi_floor))
                    s_psi = np.sqrt(psi)

                    e63 = (xm[-63:] - u_k[-63:] @ b_load.T) / s_psi
                    e21 = e63[-21:]
                    l21_e = e21.shape[0]

                    mp_r1[valid_mp] = e21[-1]
                    mp_r5[valid_mp] = e21[-5:].sum(axis=0)
                    mp_r21[valid_mp] = e21.sum(axis=0)
                    mp_r63_21[valid_mp] = e63[:-21].sum(axis=0)

                    ac_num = np.sum(e21[1:] * e21[:-1], axis=0)
                    ac_den = np.sum(e21 * e21, axis=0) + np.float32(1e-12)
                    mp_ac1[valid_mp] = ac_num / ac_den
                    part3 = np.partition(e21, kth=max(0, l21_e - 3), axis=0)[-3:]
                    mp_max3[valid_mp] = part3.mean(axis=0)

                    age21 = np.arange(l21_e - 1, -1, -1, dtype=np.float32)[:, None]
                    hl3_w = np.float32(0.5) ** (age21 / np.float32(3.0))
                    hl5_w = np.float32(0.5) ** (age21 / np.float32(5.0))
                    hl3_w /= hl3_w.sum()
                    hl5_w /= hl5_w.sum()
                    mp_hl3[valid_mp] = np.sum(e21 * hl3_w, axis=0)
                    mp_hl5[valid_mp] = np.sum(e21 * hl5_w, axis=0)

                    cs5 = np.cumsum(e63, axis=0)
                    r5_roll = cs5[4:] - np.vstack([np.zeros((1, n_m_mp), dtype=np.float32), cs5[:-5]])
                    v1 = np.maximum(np.mean(e63 * e63, axis=0), np.float32(1e-12))
                    v5 = np.mean(r5_roll * r5_roll, axis=0) / np.float32(5.0)
                    mp_vr5_63[valid_mp] = v5 / v1

                v63_252 = sig63 / np.maximum(sig252, np.float32(1e-8))
                w21_c = w21 - w21.mean(axis=0, keepdims=True)
                sd21 = np.sqrt(np.mean(w21_c * w21_c, axis=0))
                skew21 = np.mean(w21_c * w21_c * w21_c, axis=0) / (sd21 * sd21 * sd21 + np.float32(1e-12))

                diff_peer_r5 = r5 - knn20_r5
                diff_peer_r21 = r21 - knn20_r21

                r63_21_raw = np.where(obs[-63:-21], w252[-63:-21], np.float32(0.0)).sum(axis=0)
                m_ok63_20 = ok63[nn20].astype(np.float32)
                den63_20 = np.maximum(m_ok63_20.sum(axis=1), np.float32(1.0))
                peer_r63_21 = (r63_21_raw[nn20] * m_ok63_20).sum(axis=1) / den63_20

                logits = np.float32(8.0) * (sim40 - sim40.max(axis=1, keepdims=True))
                exp_w = np.exp(logits).astype(np.float32)
                w40_21 = exp_w * ok21[nn40].astype(np.float32)
                w40_21 /= np.maximum(w40_21.sum(axis=1, keepdims=True), np.float32(1e-12))

                att40_r5 = np.sum(w40_21 * r5[nn40], axis=1)
                att40_r21 = np.sum(w40_21 * r21[nn40], axis=1)
                diff_att40_r21 = r21 - att40_r21
                diff_att40_r5 = r5 - att40_r5

                ok_mp21 = valid_mp & ok21
                w40_mp21 = exp_w * ok_mp21[nn40].astype(np.float32)
                if size_ix is not None:
                    me_w = np.maximum(rc_rw[:, size_ix] + np.float32(0.5), np.float32(0.05))
                    w_lead = w40_mp21 * me_w[nn40]
                else:
                    w_lead = w40_mp21
                w_lead /= np.maximum(w_lead.sum(axis=1, keepdims=True), np.float32(1e-12))
                lead_me_mp_r21 = np.sum(w_lead * mp_r21[nn40], axis=1)
                sic3_mp_r21 = masked_group_mean(mp_r21, sic[rw], ok_mp21)

                blk2_specs = (
                    (mp_r1, ok_mp21), (mp_r5, ok_mp21), (mp_r21, ok_mp21),
                    (mp_r63_21, valid_mp & ok63), (mp_ac1, ok_mp21), (mp_max3, ok_mp21),
                    (v63_252, valid_mp & ok63), (skew21, ok21 & (sd21 > 1e-12)),
                    (diff_peer_r5, ok21), (diff_peer_r21, ok21), (peer_r63_21, ok63),
                    (lead_me_mp_r21, ok_mp21), (mp_hl3, ok_mp21), (mp_hl5, ok_mp21),
                    (sic3_mp_r21, ok_mp21), (att40_r21, ok21), (diff_att40_r21, ok21),
                    (diff_att40_r5, ok21), (mp_vr5_63, valid_mp & ok63),
                )
                for off_i, (arr, msk) in enumerate(blk2_specs):
                    out_dl[rw, path_features + off_i] = cs_rank_into(arr, msk)

        # Block 3: 756-day MP-PCA risk model, trailing daily market betas, 13 extra features & y_pca_res / y_joint13_res
        if end_d < 60 or int(hd.sum()) < min_names:
            out_ex13[a:b, 8] = cs_rank_into(miss_all[a:b], all_ones)
            continue

        a_d = max(0, end_d - sample_window_days)
        r_raw = dmat[a_d:end_d][:, col_all[rows[hd]]].astype(np.float64)
        fin_d = np.isfinite(r_raw)
        min_obs_eff = min(min_daily_obs, max(20, (end_d - a_d) // 2))
        enough = fin_d.sum(0) >= min_obs_eff
        r_clean = np.where(fin_d, r_raw, 0.0)

        r252_m = r_clean[-252:] - r_clean[-252:].mean(0, keepdims=True)
        mkt252 = r252_m.mean(1)
        vm252 = float(mkt252 @ mkt252)
        b252_hd = (r252_m.T @ mkt252) / vm252 if vm252 > 1e-16 else np.zeros(len(rows[hd]))
        dn = mkt252 < 0.0
        up = mkt252 > 0.0
        b_asym = np.full(len(rows[hd]), np.nan, dtype=np.float64)
        if int(dn.sum()) >= 20 and int(up.sum()) >= 20:
            mdn = mkt252[dn] - mkt252[dn].mean()
            mup = mkt252[up] - mkt252[up].mean()
            rdn = r252_m[dn] - r252_m[dn].mean(0, keepdims=True)
            rup = r252_m[up] - r252_m[up].mean(0, keepdims=True)
            vdn = float(mdn @ mdn)
            vup = float(mup @ mup)
            if vdn > 1e-16 and vup > 1e-16:
                b_asym = (rdn.T @ mdn) / vdn - (rup.T @ mup) / vup

        r63_m = r_clean[-63:] - r_clean[-63:].mean(0, keepdims=True)
        mkt63 = r63_m.mean(1)
        vm63 = float(mkt63 @ mkt63)
        b63_hd = (r63_m.T @ mkt63) / vm63 if vm63 > 1e-16 else np.zeros(len(rows[hd]))
        beta252[rows[hd]] = b252_hd.astype(np.float32)

        bv = beta252[a:b].astype(np.float64)
        beta252_rk[a:b] = cs_rank_into(bv, bv != 0.0)

        ok_idx = np.flatnonzero(hd)[enough]
        if len(ok_idx) < min_names:
            out_ex13[a:b, 8] = cs_rank_into(miss_all[a:b], all_ones)
            continue

        rows_ok = rows[ok_idx]
        b_load, psi, sig, u_k, xn = _pca_risk_with_factors(r_clean[:, enough])
        sig_mp756[rows_ok] = sig.astype(np.float32)
        psi_mp756[rows_ok] = psi.astype(np.float32)

        b_raw = xn.T @ u_k
        s_psi = np.sqrt(psi)
        e_21 = (xn[-21:] - u_k[-21:] @ b_raw.T) / s_psi
        mp756_r5 = e_21[-5:].sum(0)
        mp756_r21 = e_21.sum(0)
        mp756_r63_21 = (
            (xn[-63:-21].sum(0) - u_k[-63:-21].sum(0) @ b_raw.T) / s_psi
            if xn.shape[0] >= 63
            else np.full(len(rows_ok), np.nan)
        )
        mp756_r126_21 = (
            (xn[-126:-21].sum(0) - u_k[-126:-21].sum(0) @ b_raw.T) / s_psi
            if xn.shape[0] >= 126
            else np.full(len(rows_ok), np.nan)
        )

        raw13 = np.full((n_m, extra756_cols), np.nan, dtype=np.float64)
        raw13[ok_idx, 0] = sig
        raw13[ok_idx, 1] = psi
        raw13[hd, 2] = b252_hd
        raw13[hd, 3] = b63_hd - b252_hd
        raw13[ok_idx, 4] = mp756_r5
        raw13[ok_idx, 5] = mp756_r21
        raw13[ok_idx, 6] = mp756_r63_21
        raw13[ok_idx, 7] = mp756_r126_21
        raw13[:, 8] = miss_all[a:b]
        raw13[hd, 12] = b_asym
        if xn.shape[0] >= 252:
            e252 = (xn[-252:] - u_k[-252:] @ b_raw.T) / s_psi
            e_252_21 = e252[:-21]
            e_126_21 = e252[-126:-21]
            raw13[ok_idx, 9] = e_252_21.mean(0) / np.maximum(e_252_21.std(0), 1e-8)
            raw13[ok_idx, 10] = e_126_21.mean(0) / np.maximum(e_126_21.std(0), 1e-8)
            dn_iv = np.sqrt(np.mean(np.minimum(e252, 0.0) ** 2, axis=0) + 1e-12)
            up_iv = np.sqrt(np.mean(np.maximum(e252, 0.0) ** 2, axis=0) + 1e-12)
            raw13[ok_idx, 11] = np.log(dn_iv / up_iv)
        elif xn.shape[0] >= 126:
            e126 = (xn[-126:] - u_k[-126:] @ b_raw.T) / s_psi
            e_126_21 = e126[:-21]
            raw13[ok_idx, 10] = e_126_21.mean(0) / np.maximum(e_126_21.std(0), 1e-8)
            dn_iv = np.sqrt(np.mean(np.minimum(e126, 0.0) ** 2, axis=0) + 1e-12)
            up_iv = np.sqrt(np.mean(np.maximum(e126, 0.0) ** 2, axis=0) + 1e-12)
            raw13[ok_idx, 11] = np.log(dn_iv / up_iv)
        rk13 = pd.DataFrame(raw13).rank(pct=True).to_numpy(dtype=np.float32) - np.float32(0.5)
        rk13[np.isnan(rk13)] = 0.0
        out_ex13[a:b] = rk13

        # Strictly causal targets y_pca_res & y_joint13_res (only read at future refits tt < t)
        r_ok = y_all[rows_ok].astype(np.float64)
        f = np.isfinite(r_ok)
        if int(f.sum()) >= min_names:
            inv_s = 1.0 / np.maximum(sig, 1e-8)
            ind2 = (sic[rows_ok] // 100).astype(np.int64)
            ind2_dum = _industry_dummies(ind2)
            g2 = np.column_stack([b_load, inv_s, ind2_dum * inv_s[:, None]])[f]
            x_gls = r_ok[f] / (sig[f] * np.sqrt(psi[f]))
            u2 = _basis(g2, psi[f])
            z2 = x_gls - u2 @ (u2.T @ x_gls)
            sd2 = float(z2.std())
            if np.isfinite(sd2) and sd2 > 1e-12:
                y_pca_res[rows_ok[f]] = np.clip(z2 / sd2, -4.0, 4.0).astype(np.float32)

            z_b13_ok = (
                np.column_stack([beta252_rk[rows_ok].astype(np.float64), r_char[rows_ok][:, barra_ix].astype(np.float64)])
                if barra_ix
                else beta252_rk[rows_ok].astype(np.float64)[:, None]
            )
            g_j13 = np.column_stack([b_load, inv_s, z_b13_ok * inv_s[:, None], ind2_dum * inv_s[:, None]])[f]
            u_j13 = _basis(g_j13, psi[f])
            z_j13 = x_gls - u_j13 @ (u_j13.T @ x_gls)
            sd_j13 = float(z_j13.std())
            if np.isfinite(sd_j13) and sd_j13 > 1e-12:
                y_joint13_res[rows_ok[f]] = np.clip(z_j13 / sd_j13, -4.0, 4.0).astype(np.float32)

    return {
        "y_peer": y_peer,
        "y_pca_res": y_pca_res,
        "y_joint13_res": y_joint13_res,
        "y_z_raw": y_z_raw,
        "sig_mp756": sig_mp756,
        "psi_mp756": psi_mp756,
        "beta252": beta252,
        "beta252_rk": beta252_rk,
    }


class NumpyMLP:
    """Deterministic float32 MLP supporting ReLU or Tanh with untiled dropout and AdamW."""

    def __init__(
        self,
        d_in: int,
        s: int,
        hidden: tuple[int, ...],
        activation: str = "relu",
        lr: float = 1e-3,
        wd: float = 1e-3,
        dropout: float = 0.2,
    ):
        rng = np.random.default_rng(seed + 2000 + s)
        dims = (d_in,) + hidden + (1,)
        self.act = activation
        self.lr = lr
        self.wd = wd
        self.dropout = dropout
        self.W: list[np.ndarray] = []
        self.b: list[np.ndarray] = []
        for f_in, f_out in zip(dims[:-1], dims[1:]):
            if activation == "relu":
                bound = np.float32(1.0 / math.sqrt(max(f_in, 1)))
            else:
                bound = np.float32(math.sqrt(6.0 / max(f_in + f_out, 1)))
            self.W.append(rng.uniform(-bound, bound, (f_in, f_out)).astype(np.float32))
            self.b.append(np.zeros((f_out,), dtype=np.float32))
        self.mW = [np.zeros_like(w) for w in self.W]
        self.vW = [np.zeros_like(w) for w in self.W]
        self.mb = [np.zeros_like(b) for b in self.b]
        self.vb = [np.zeros_like(b) for b in self.b]
        self.step_num = 0

    def train_epoch_rows(
        self,
        r: np.ndarray,
        rows: np.ndarray,
        cols: np.ndarray,
        y: np.ndarray,
        rng: np.random.Generator,
        batch_size: int = 4096,
        max_steps: int = mlp_full_max_steps,
    ) -> None:
        n = len(rows)
        if n == 0:
            return
        perm = rng.permutation(n)
        keep_p = np.float32(1.0 - self.dropout)
        inv_keep = np.float32(1.0 / keep_p) if self.dropout > 0 else np.float32(1.0)
        ih_thr = np.float32(
            math.sqrt(2.0 * float(keep_p))
            if float(keep_p) <= 0.5
            else 2.0 - math.sqrt(2.0 * (1.0 - float(keep_p)))
        )
        lr = np.float32(self.lr)
        wd_mult = np.float32(1.0 - self.lr * self.wd)
        b1, b2 = np.float32(0.9), np.float32(0.999)
        omb1, omb2 = np.float32(0.1), np.float32(0.001)
        eps = np.float32(1e-8)
        n_hid = len(self.W) - 1

        eff_bs = 8192 if n >= 32768 else batch_size
        xs_pre = np.ascontiguousarray(r[rows][:, cols]) if n <= 65536 else None
        all_cols = len(cols) == r.shape[1] and bool(np.array_equal(cols, np.arange(r.shape[1], dtype=cols.dtype)))
        step_ep = 0
        for i in range(0, n, eff_bs):
            if step_ep >= max_steps:
                break
            step_ep += 1
            idx = perm[i:i + eff_bs]
            if xs_pre is not None:
                xb = xs_pre[idx]
            elif all_cols:
                xb = r[rows[idx]]
            else:
                xb = np.take(r[rows[idx]], cols, axis=1)
            yb = y[idx]
            bs_i = len(xb)
            bs = np.float32(bs_i)

            hs = [xb]
            derivs = []
            h_cur = xb
            for l_ix in range(n_hid):
                d_out = self.W[l_ix].shape[1]
                z = h_cur @ self.W[l_ix] + self.b[l_ix]
                if self.act == "relu":
                    u_r = rng.random((bs_i, 1), dtype=np.float32)
                    u_c = rng.random((1, d_out), dtype=np.float32)
                    mask = (((u_r + u_c) < ih_thr) & (z > np.float32(0.0))).astype(np.float32) * inv_keep
                    h_cur = z * mask
                    derivs.append(mask)
                else:
                    th = np.tanh(z)
                    if self.dropout > 0:
                        u_r = rng.random((bs_i, 1), dtype=np.float32)
                        u_c = rng.random((1, d_out), dtype=np.float32)
                        dmask = ((u_r + u_c) < ih_thr).astype(np.float32) * inv_keep
                        h_cur = th * dmask
                        derivs.append((np.float32(1.0) - th * th) * dmask)
                    else:
                        h_cur = th
                        derivs.append(np.float32(1.0) - th * th)
                hs.append(h_cur)

            pred = (h_cur @ self.W[n_hid]).ravel() + self.b[n_hid][0]
            dz = ((pred - yb) * (np.float32(2.0) / bs))[:, None]

            grads = [None] * (n_hid + 1)
            grads[n_hid] = (hs[n_hid].T @ dz, dz.sum(axis=0))
            for l_ix in range(n_hid - 1, -1, -1):
                dz = (dz @ self.W[l_ix + 1].T) * derivs[l_ix]
                grads[l_ix] = (hs[l_ix].T @ dz, dz.sum(axis=0))

            self.step_num += 1
            step_lr = lr / np.float32(1.0 - 0.9 ** self.step_num)
            inv_bc2 = np.float32(1.0 / math.sqrt(1.0 - 0.999 ** self.step_num))
            for l_ix, (g_w, g_b) in enumerate(grads):
                self.W[l_ix] *= wd_mult
                self.mW[l_ix] = b1 * self.mW[l_ix] + omb1 * g_w
                self.vW[l_ix] = b2 * self.vW[l_ix] + omb2 * (g_w * g_w)
                self.W[l_ix] -= step_lr * self.mW[l_ix] / (np.sqrt(self.vW[l_ix]) * inv_bc2 + eps)
                self.b[l_ix] *= wd_mult
                self.mb[l_ix] = b1 * self.mb[l_ix] + omb1 * g_b
                self.vb[l_ix] = b2 * self.vb[l_ix] + omb2 * (g_b * g_b)
                self.b[l_ix] -= step_lr * self.mb[l_ix] / (np.sqrt(self.vb[l_ix]) * inv_bc2 + eps)

    def predict_with_weights(
        self, x: np.ndarray, w_list: list[np.ndarray], b_list: list[np.ndarray]
    ) -> np.ndarray:
        out = np.empty(len(x), dtype=np.float32)
        n_hid = len(w_list) - 1
        for i in range(0, len(x), 65536):
            h = x[i:i + 65536]
            for l_ix in range(n_hid):
                z = h @ w_list[l_ix] + b_list[l_ix]
                h = np.maximum(z, np.float32(0.0)) if self.act == "relu" else np.tanh(z)
            out[i:i + 65536] = (h @ w_list[n_hid]).ravel() + b_list[n_hid][0]
        return out

    def predict(self, x: np.ndarray) -> np.ndarray:
        return self.predict_with_weights(x, self.W, self.b)

    def predict_rows(self, r: np.ndarray, rows: np.ndarray, cols: np.ndarray) -> np.ndarray:
        out = np.empty(len(rows), dtype=np.float32)
        n_hid = len(self.W) - 1
        for i in range(0, len(rows), 65536):
            h = np.ascontiguousarray(r[rows[i:i + 65536]][:, cols])
            for l_ix in range(n_hid):
                z = h @ self.W[l_ix] + self.b[l_ix]
                h = np.maximum(z, np.float32(0.0)) if self.act == "relu" else np.tanh(z)
            out[i:i + 65536] = (h @ self.W[n_hid]).ravel() + self.b[n_hid][0]
        return out


def _fit_swa_mlp_seed(
    r: np.ndarray,
    tr_rows: np.ndarray,
    rows_tr: np.ndarray,
    rows_va: np.ndarray,
    y: np.ndarray,
    yt: np.ndarray,
    yv: np.ndarray,
    n_char: int,
    n_in: int,
    hidden: tuple[int, ...],
    activation: str,
    epochs: int,
    lr: float,
    wd: float,
    dropout: float,
    feat_frac: float,
    stage_offset: int,
    s: int,
):
    rng_col = np.random.default_rng(seed + 1000 + s + stage_offset)
    if feat_frac < 0.999 and n_char > 0:
        n_keep = max(1, min(n_char, int(round(feat_frac * n_char))))
        keep = np.sort(rng_col.choice(n_char, n_keep, replace=False))
        c = np.concatenate([keep, np.arange(n_char, n_in, dtype=np.int64)])
    else:
        c = np.arange(n_in, dtype=np.int64)
    net_val = NumpyMLP(len(c), s + stage_offset, hidden, activation, lr, wd, dropout)
    rng_val = np.random.default_rng(seed + 5000 + s + stage_offset)
    ics = []
    for _ in range(epochs):
        net_val.train_epoch_rows(r, rows_tr, c, yt, rng_val, max_steps=mlp_val_max_steps)
        pv = net_val.predict_rows(r, rows_va, c)
        ic = float(np.corrcoef(pv, yv)[0, 1]) if len(yv) > 2 and pv.std() > 0 else -1.0
        ics.append(ic if math.isfinite(ic) else -1.0)

    ics_arr = np.array(ics, dtype=np.float64)
    z_ic = mlp_swa_temp * (ics_arr - np.max(ics_arr))
    w_ep = np.exp(np.clip(z_ic, -50.0, 0.0))
    w_ep /= w_ep.sum()
    max_ep = int(np.max(np.flatnonzero(w_ep > 1e-3))) + 1
    w_ep_trunc = w_ep[:max_ep] / w_ep[:max_ep].sum()

    net = NumpyMLP(len(c), s + stage_offset, hidden, activation, lr, wd, dropout)
    rng = np.random.default_rng(seed + 5000 + s + stage_offset)
    w_swa = [np.zeros_like(w) for w in net.W]
    b_swa = [np.zeros_like(b) for b in net.b]
    for ep in range(max_ep):
        net.train_epoch_rows(r, tr_rows, c, y, rng, max_steps=mlp_full_max_steps)
        wt = np.float32(w_ep_trunc[ep])
        for l_ix in range(len(w_swa)):
            w_swa[l_ix] += wt * net.W[l_ix]
            b_swa[l_ix] += wt * net.b[l_ix]
    return (net, w_swa, b_swa, c)


def fit_swa_mlp_ensemble(
    r: np.ndarray,
    tr_rows: np.ndarray,
    y: np.ndarray,
    valid_mask: np.ndarray,
    n_char: int,
    n_in: int,
    hidden: tuple[int, ...],
    activation: str,
    epochs: int,
    lr: float,
    wd: float,
    dropout: float,
    feat_frac: float,
    stage_offset: int,
):
    if (valid_mask.sum() < 10) or ((~valid_mask).sum() < 32):
        valid_mask = np.arange(len(tr_rows)) >= int(0.75 * len(tr_rows))
    rows_tr, yt = tr_rows[~valid_mask], y[~valid_mask]
    rows_va, yv = tr_rows[valid_mask], y[valid_mask]

    with ThreadPoolExecutor(max_workers=mlp_seeds) as ex:
        return list(
            ex.map(
                lambda s: _fit_swa_mlp_seed(
                    r, tr_rows, rows_tr, rows_va, y, yt, yv,
                    n_char, n_in, hidden, activation, epochs, lr, wd, dropout, feat_frac, stage_offset, s,
                ),
                range(mlp_seeds),
            )
        )


def fit_swa_mlp_triple_ensembles(
    r: np.ndarray,
    tr_rows: np.ndarray,
    y_a: np.ndarray,
    y_b: np.ndarray,
    y_c: np.ndarray,
    valid_mask: np.ndarray,
    n_char: int,
    n_in: int,
    hidden: tuple[int, ...],
    activation: str,
    epochs: int,
    lr: float,
    wd: float,
    dropout: float,
    feat_frac: float,
    stage_offset: int,
):
    if (valid_mask.sum() < 10) or ((~valid_mask).sum() < 32):
        valid_mask = np.arange(len(tr_rows)) >= int(0.75 * len(tr_rows))
    rows_tr = tr_rows[~valid_mask]
    rows_va = tr_rows[valid_mask]
    yt_a, yv_a = y_a[~valid_mask], y_a[valid_mask]
    yt_b, yv_b = y_b[~valid_mask], y_b[valid_mask]
    yt_c, yv_c = y_c[~valid_mask], y_c[valid_mask]

    jobs = (
        [(y_a, yt_a, yv_a, s) for s in range(mlp_seeds)]
        + [(y_b, yt_b, yv_b, s) for s in range(mlp_seeds)]
        + [(y_c, yt_c, yv_c, s) for s in range(mlp_seeds)]
    )
    with ThreadPoolExecutor(max_workers=min(8, len(jobs))) as ex:
        res = list(
            ex.map(
                lambda job: _fit_swa_mlp_seed(
                    r, tr_rows, rows_tr, rows_va, job[0], job[1], job[2],
                    n_char, n_in, hidden, activation, epochs, lr, wd, dropout, feat_frac, stage_offset, job[3],
                ),
                jobs,
            )
        )
    return res[:mlp_seeds], res[mlp_seeds : 2 * mlp_seeds], res[2 * mlp_seeds :]


def fit_swa_mlp_stage2_with_tanh(
    r: np.ndarray,
    tr_rows: np.ndarray,
    y_a: np.ndarray,
    y_b: np.ndarray,
    y_c: np.ndarray,
    valid_mask: np.ndarray,
    tz_rows: np.ndarray | None,
    yz_raw: np.ndarray | None,
    tz_valid: np.ndarray | None,
    n_char: int,
    n_in: int,
):
    if (valid_mask.sum() < 10) or ((~valid_mask).sum() < 32):
        valid_mask = np.arange(len(tr_rows)) >= int(0.75 * len(tr_rows))
    rows_tr = tr_rows[~valid_mask]
    rows_va = tr_rows[valid_mask]
    yt_a, yv_a = y_a[~valid_mask], y_a[valid_mask]
    yt_b, yv_b = y_b[~valid_mask], y_b[valid_mask]
    yt_c, yv_c = y_c[~valid_mask], y_c[valid_mask]

    jobs = []
    for y_f, yt_f, yv_f in ((y_a, yt_a, yv_a), (y_b, yt_b, yv_b), (y_c, yt_c, yv_c)):
        jobs.extend(
            (
                tr_rows, rows_tr, rows_va, y_f, yt_f, yv_f,
                mlp_relu_hidden, "relu", mlp_relu_epochs, mlp_relu_lr, mlp_relu_wd,
                mlp_relu_dropout, mlp_relu_feat_frac, mlp_stage2_offset, s,
            )
            for s in range(mlp_seeds)
        )
    has_tz = tz_rows is not None and yz_raw is not None and tz_valid is not None and len(tz_rows) > 0
    if has_tz:
        if (tz_valid.sum() < 10) or ((~tz_valid).sum() < 32):
            tz_valid = np.arange(len(tz_rows)) >= int(0.75 * len(tz_rows))
        rz_tr, rz_va = tz_rows[~tz_valid], tz_rows[tz_valid]
        yzt, yzv = yz_raw[~tz_valid], yz_raw[tz_valid]
        jobs.extend(
            (
                tz_rows, rz_tr, rz_va, yz_raw, yzt, yzv,
                mlp_tanh_hidden, "tanh", mlp_tanh_epochs, mlp_tanh_lr, mlp_tanh_wd,
                mlp_tanh_dropout, mlp_tanh_feat_frac, mlp_tanh_offset, s,
            )
            for s in range(mlp_seeds)
        )
    with ThreadPoolExecutor(max_workers=min(8, len(jobs))) as ex:
        res = list(
            ex.map(
                lambda job: _fit_swa_mlp_seed(
                    r, job[0], job[1], job[2], job[3], job[4], job[5],
                    n_char, n_in, job[6], job[7], job[8], job[9], job[10], job[11], job[12], job[13], job[14],
                ),
                jobs,
            )
        )
    m2_a = res[:mlp_seeds]
    m2_b = res[mlp_seeds : 2 * mlp_seeds]
    m2_c = res[2 * mlp_seeds : 3 * mlp_seeds]
    m_tz = res[3 * mlp_seeds :] if has_tz else None
    return m2_a, m2_b, m2_c, m_tz


def predict_mlp_ensemble(models, x: np.ndarray) -> np.ndarray:
    return np.mean(
        [
            net.predict_with_weights(np.ascontiguousarray(x[:, c]), w_l, b_l).astype(np.float64)
            for net, w_l, b_l, c in models
        ],
        axis=0,
    )


def predict_mlp_rows(models, r: np.ndarray, rows: np.ndarray) -> np.ndarray:
    out = np.empty(len(rows), dtype=np.float32)
    for i in range(0, len(rows), 65536):
        j = min(i + 65536, len(rows))
        out[i:j] = predict_mlp_ensemble(models, r[rows[i:j]]).astype(np.float32)
    return out


def fit_tail_ridge(
    r: np.ndarray,
    tr_rows: np.ndarray,
    res: np.ndarray,
    m_lens: list[int],
    m_vars: np.ndarray,
    n_char: int,
) -> np.ndarray:
    if n_char <= 0:
        return np.zeros(0, dtype=np.float32)
    g_tt = np.zeros((n_char, n_char), dtype=np.float64)
    g_ty = np.zeros(n_char, dtype=np.float64)
    mean_mv = float(np.mean(m_vars)) + 1e-12
    w_mv = (m_vars / mean_mv) ** 2 if len(m_lens) > 200 else np.ones(len(m_lens), dtype=np.float64)
    pos = 0
    for km, ln in enumerate(m_lens):
        if ln > 0:
            vt = (np.abs(r[tr_rows[pos:pos + ln], :n_char]) - np.float32(0.25)).astype(np.float64)
            yr = res[pos:pos + ln].astype(np.float64)
            wm = float(w_mv[km])
            g_tt += wm * (vt.T @ vt)
            g_ty += wm * (vt.T @ yr)
            pos += ln
    lam = 5.0 + 0.05 * float(np.trace(g_tt) / max(n_char, 1))
    gamma = 500.0 / (500.0 + float(n_char ** 3)) if len(m_lens) >= 60 else 0.0
    return (gamma * sp_solve(g_tt + lam * np.eye(n_char, dtype=np.float64), g_ty, assume_a="pos")).astype(np.float32)


def _find_active_tree_cols(x_mat: np.ndarray) -> np.ndarray:
    n_cols = x_mat.shape[1]
    active = np.zeros(n_cols, dtype=bool)
    for c0 in range(0, n_cols, 64):
        c1 = min(c0 + 64, n_cols)
        sub = x_mat[:, c0:c1]
        cnt = np.isfinite(sub).sum(axis=0)
        ok = cnt >= 2
        if ok.any():
            idx = np.flatnonzero(ok)
            s_ok = sub[:, idx]
            c_min = np.nanmin(s_ok, axis=0)
            c_max = np.nanmax(s_ok, axis=0)
            active[c0 + idx] = np.isfinite(c_min) & np.isfinite(c_max) & (c_max > c_min)
    return np.flatnonzero(active)


def _get_raw_col_views(chars: pd.DataFrame, feats: list[str]) -> list[np.ndarray]:
    views: list[np.ndarray] = []
    for f in feats:
        s = chars[f]
        if isinstance(s.dtype, np.dtype) and np.issubdtype(s.dtype, np.number):
            views.append(s.to_numpy(copy=False))
        else:
            views.append(pd.to_numeric(s, errors="coerce").to_numpy(dtype=np.float32))
    return views


def _fill_raw_slice(raw_cols: list[np.ndarray], orig_idx: np.ndarray, out_mat: np.ndarray) -> None:
    for j, col in enumerate(raw_cols):
        out_mat[:, j] = col[orig_idx]


def _prep_hgb_matrix(x_tr_full: np.ndarray) -> tuple[np.ndarray, np.ndarray] | None:
    active_cols = _find_active_tree_cols(x_tr_full)
    if len(active_cols) == 0:
        return None
    if (
        len(active_cols) == x_tr_full.shape[1]
        and x_tr_full.dtype == np.float64
        and x_tr_full.flags["C_CONTIGUOUS"]
    ):
        return x_tr_full, active_cols
    return np.ascontiguousarray(x_tr_full[:, active_cols], dtype=np.float64), active_cols


def _fit_one_hgb_tree(
    x_tr: np.ndarray,
    active_cols: np.ndarray,
    y64: np.ndarray,
    year_key: int,
    fam_id: int,
    k_cfg: int,
) -> tuple[HistGradientBoostingRegressor, np.ndarray]:
    p = hgb_configs[k_cfg]
    est = HistGradientBoostingRegressor(
        loss="squared_error",
        learning_rate=p["lr"],
        max_iter=p["max_iter"],
        max_leaf_nodes=p["max_leaf_nodes"],
        max_depth=p["max_depth"],
        min_samples_leaf=p["min_samples_leaf"],
        l2_regularization=p["l2"],
        max_features=p["max_features"],
        max_bins=p["max_bins"],
        early_stopping=False,
        random_state=seed + 17 * k_cfg + 100 * (year_key - 1950) + 10000 * fam_id,
    )
    est.fit(x_tr, y64)
    return (est, active_cols)


def _fit_hgb_family(
    x_tr_full: np.ndarray,
    y_tr: np.ndarray,
    year_key: int,
    fam_id: int,
) -> list[tuple[HistGradientBoostingRegressor, np.ndarray]] | None:
    if len(y_tr) < hgb_min_train_rows:
        return None
    prepped = _prep_hgb_matrix(x_tr_full)
    if prepped is None:
        return None
    x_tr, active_cols = prepped
    y64 = np.asarray(y_tr, dtype=np.float64)
    with ThreadPoolExecutor(max_workers=len(hgb_configs)) as ex:
        return list(ex.map(lambda k: _fit_one_hgb_tree(x_tr, active_cols, y64, year_key, fam_id, k), range(len(hgb_configs))))


def _fit_hgb_batch(
    specs: list[tuple[np.ndarray, np.ndarray, int]],
    year_key: int,
) -> list[list[tuple[HistGradientBoostingRegressor, np.ndarray]] | None]:
    n_cfg = len(hgb_configs)
    prep_cache: dict[int, tuple[np.ndarray, np.ndarray] | None] = {}
    jobs = []
    job_map = []
    for s_idx, (x_mat, y_tr, fam_id) in enumerate(specs):
        if len(y_tr) < hgb_min_train_rows:
            continue
        x_id = id(x_mat)
        if x_id not in prep_cache:
            prep_cache[x_id] = _prep_hgb_matrix(x_mat)
        prep = prep_cache[x_id]
        if prep is None:
            continue
        x_tr, active_cols = prep
        y64 = np.asarray(y_tr, dtype=np.float64)
        for k_cfg in range(n_cfg):
            jobs.append((x_tr, active_cols, y64, year_key, fam_id, k_cfg))
            job_map.append(s_idx)
    if not jobs:
        return [None] * len(specs)
    with ThreadPoolExecutor(max_workers=len(jobs)) as ex:
        trained = list(ex.map(lambda j: _fit_one_hgb_tree(*j), jobs))
    out: list[list[tuple[HistGradientBoostingRegressor, np.ndarray]] | None] = [None] * len(specs)
    for s_idx, item in zip(job_map, trained):
        if out[s_idx] is None:
            out[s_idx] = []
        out[s_idx].append(item)
    return out


def _predict_hgb_family(
    models: list[tuple[HistGradientBoostingRegressor, np.ndarray]] | None,
    x_m: np.ndarray,
) -> np.ndarray | None:
    if not models:
        return None
    preds = [zs(est.predict(x_m[:, active_cols])) for est, active_cols in models]
    return zs(np.mean(preds, axis=0))


def _combine_weighted_signals(
    parts: dict[str, np.ndarray | None], weights: dict[str, float], n: int
) -> np.ndarray:
    acc = np.zeros(n, dtype=np.float64)
    w_sum = 0.0
    for key, wt in weights.items():
        arr = parts.get(key)
        if arr is not None and len(arr) == n and float(np.std(arr)) > 1e-12:
            acc += wt * zs(arr)
            w_sum += wt
    return zs(acc / w_sum) if w_sum > 0.0 else np.zeros(n, dtype=np.float64)


def _build_weights_from_box(
    chars: pd.DataFrame, features: pd.DataFrame, daily_ret_box: list[pd.DataFrame]
) -> pd.DataFrame:
    np.random.seed(seed)
    t0 = time.perf_counter()
    is_val_mode = os.environ.get("CTF_EXECUTION_MODE", "full").strip().lower() == "validation"

    if chars is None or len(chars) == 0:
        return pd.DataFrame(columns=["id", "eom", "w"])

    feats = [str(f) for f in features[feature_list_col].tolist() if str(f) in chars.columns]
    n_char = len(feats)
    n_dl = total_daily_features
    p_tot = n_char + n_dl
    size_ix = feats.index("market_equity") if "market_equity" in feats else None
    mod_ix = [feats.index(m) for m in path_modulators if m in feats]
    arb_ix = [feats.index(m) for m in arbitrage_modulators if m in feats]
    barra_ix = [feats.index(c) for c in barra_style_chars if c in feats]

    eom_dt = pd.to_datetime(chars["eom"]).to_numpy(dtype="datetime64[ns]")
    ids_raw = pd.to_numeric(chars["id"]).to_numpy(dtype=np.int64)
    order = np.lexsort((ids_raw, eom_dt.astype(np.int64)))

    ids = ids_raw[order]
    eom_out = eom_dt[order]
    y_all = (
        pd.to_numeric(chars[target_col], errors="coerce").to_numpy(dtype=np.float64)[order]
        if target_col in chars.columns
        else np.full(len(order), np.nan, dtype=np.float64)
    )
    sic = (
        np.nan_to_num(
            pd.to_numeric(chars["sic"], errors="coerce").to_numpy(dtype=np.float64)[order],
            nan=9999.0,
        )
        if "sic" in chars.columns
        else np.full(len(order), 9999.0, dtype=np.float64)
    )
    me = (
        pd.to_numeric(chars["market_equity"], errors="coerce").to_numpy(dtype=np.float64)[order]
        if "market_equity" in chars.columns
        else np.ones(len(order), dtype=np.float64)
    )
    bet60 = (
        pd.to_numeric(chars["beta_60m"], errors="coerce").to_numpy(dtype=np.float64)[order]
        if "beta_60m" in chars.columns
        else np.zeros(len(order), dtype=np.float64)
    )
    del ids_raw, eom_dt

    raw_cols = _get_raw_col_views(chars, feats)
    r = np.zeros((len(order), p_tot), dtype=np.float32)
    miss_cnt = np.zeros(len(order), dtype=np.int16)
    for j, col_arr in enumerate(raw_cols):
        col_v = col_arr[order].astype(np.float32, copy=False)
        r[:, j] = col_v
        miss_cnt += (~np.isfinite(col_v))
    miss_all = (
        miss_cnt.astype(np.float32) / np.float32(max(n_char, 1))
        if n_char > 0
        else np.zeros(len(order), dtype=np.float32)
    )
    del miss_cnt
    order32 = order.astype(np.int32)
    del order
    gc.collect()

    months, mix = np.unique(eom_out.astype("datetime64[D]"), return_inverse=True)
    mdates = [m.astype("datetime64[D]").item() for m in months]
    starts = np.searchsorted(mix, np.arange(len(months)))
    ends = np.searchsorted(mix, np.arange(len(months)), side="right")
    rank_panel_inplace(r, starts, ends, n_char)
    print(
        f"{len(ids):,} rows, {n_char} characteristics, {len(months)} months, ranked in "
        f"{time.perf_counter() - t0:.1f}s",
        flush=True,
    )

    daily_ret_in = daily_ret_box.pop()
    dates, daily_ids, dmat, first_obs_day = build_daily_matrix(daily_ret_in, np.unique(ids))
    del daily_ret_in
    gc.collect()
    has_dmat = len(dates) > 0 and dmat.ndim == 2 and dmat.shape[0] > 0 and dmat.shape[1] > 0
    if len(daily_ids) > 0:
        col_all = np.clip(np.searchsorted(daily_ids, ids), 0, len(daily_ids) - 1)
        has_d = daily_ids[col_all] == ids
    else:
        col_all = np.zeros(len(ids), dtype=np.int64)
        has_d = np.zeros(len(ids), dtype=bool)

    t1 = time.perf_counter()
    feat_tgt = build_causal_features_and_targets(
        r, miss_all, n_char, starts, ends, months, dates, dmat, col_all, has_d, first_obs_day,
        sic, y_all, size_ix, barra_ix,
    )
    del miss_all
    y_peer = feat_tgt["y_peer"]
    y_pca_res = feat_tgt["y_pca_res"]
    y_joint13_res = feat_tgt["y_joint13_res"]
    y_z_raw = feat_tgt["y_z_raw"]
    beta252 = feat_tgt["beta252"]
    beta252_rk = feat_tgt["beta252_rk"]
    del feat_tgt
    gc.collect()
    print(
        f"{n_dl} causal daily/MP756 features + triple targets built ({p_tot} total inputs, "
        f"finite y_pca_res={int(np.isfinite(y_pca_res).sum()):,}, "
        f"y_joint13_res={int(np.isfinite(y_joint13_res).sum()):,}) in {time.perf_counter() - t1:.1f}s",
        flush=True,
    )

    # Index rows where y_peer is finite (fallback y_pca_res/y_joint13_res if sparse)
    z_rows: list[np.ndarray] = []
    shared_rows: list[np.ndarray] = []
    shared_y_peer: list[np.ndarray] = []
    shared_y_pca: list[np.ndarray] = []
    shared_y_j13: list[np.ndarray] = []
    shared_idx_parts: list[np.ndarray] = []
    var_peer = np.zeros(len(months), dtype=np.float64)
    var_pca = np.zeros(len(months), dtype=np.float64)
    var_j13 = np.zeros(len(months), dtype=np.float64)

    for tt in range(len(months)):
        lo_m, hi_m = int(starts[tt]), int(ends[tt])
        rw_m = np.arange(lo_m, hi_m, dtype=np.int64)
        ok_p = np.isfinite(y_peer[rw_m])
        ok_c = np.isfinite(y_pca_res[rw_m])
        ok_z = np.isfinite(y_z_raw[rw_m])
        ok_sh = ok_p & ok_c if int(ok_c.sum()) > 10 else ok_p

        rw_z = rw_m[ok_z]
        rw_sh = rw_m[ok_sh]

        z_rows.append(rw_z)
        shared_rows.append(rw_sh)

        yp_m = y_peer[rw_sh].astype(np.float32)
        yc_m = np.where(np.isfinite(y_pca_res[rw_sh]), y_pca_res[rw_sh], y_peer[rw_sh]).astype(np.float32)
        yj_m = np.where(np.isfinite(y_joint13_res[rw_sh]), y_joint13_res[rw_sh], yc_m).astype(np.float32)
        shared_y_peer.append(yp_m)
        shared_y_pca.append(yc_m)
        shared_y_j13.append(yj_m)
        shared_idx_parts.append(np.full(len(rw_sh), tt, dtype=np.int32))
        if len(yp_m) > 2:
            var_peer[tt] = float(np.var(yp_m))
        if len(yc_m) > 2:
            var_pca[tt] = float(np.var(yc_m))
        if len(yj_m) > 2:
            var_j13[tt] = float(np.var(yj_m))

    all_rw = (
        np.concatenate(shared_rows)
        if any(len(rw) > 0 for rw in shared_rows)
        else np.empty(0, dtype=np.int64)
    )
    y_ord_peer = (
        np.concatenate(shared_y_peer).astype(np.float32)
        if len(all_rw) > 0
        else np.empty(0, dtype=np.float32)
    )
    y_ord_pca = (
        np.concatenate(shared_y_pca).astype(np.float32)
        if len(all_rw) > 0
        else np.empty(0, dtype=np.float32)
    )
    y_ord_j13 = (
        np.concatenate(shared_y_j13).astype(np.float32)
        if len(all_rw) > 0
        else np.empty(0, dtype=np.float32)
    )
    row_month = (
        np.concatenate(shared_idx_parts)
        if len(all_rw) > 0
        else np.empty(0, dtype=np.int32)
    )
    del shared_idx_parts

    m_ends = np.cumsum([len(rw) for rw in shared_rows], dtype=np.int64)
    m_starts = np.concatenate([[0], m_ends[:-1]]) if len(m_ends) > 0 else np.zeros(0, dtype=np.int64)
    m_len = m_ends - m_starts
    z_m_len = np.array([len(rw) for rw in z_rows], dtype=np.int64)

    def expand_fixed(rk32: np.ndarray) -> np.ndarray:
        sc = np.float32(path_block_scale)
        cols = [rk32]
        if size_ix is not None:
            cols.append(rk32 * rk32[:, [size_ix]])
        for j in arb_ix:
            cols.append(rk32[:, :n_char] * rk32[:, [j]])
        dl = rk32[:, n_char:]
        for j in mod_ix:
            cols.append(sc * dl * rk32[:, [j]])
        return np.column_stack(cols)

    def predict_ridge(rk32: np.ndarray, cf: np.ndarray, ms: dict[str, np.ndarray]) -> np.ndarray:
        c, d = rk32[:, :n_char], rk32[:, n_char:]
        cf32 = cf.astype(np.float32)
        pos = 0
        out = rk32 @ cf32[pos:pos + p_tot]
        pos += p_tot
        if size_ix is not None:
            out = out + rk32[:, size_ix] * (rk32 @ cf32[pos:pos + p_tot])
            pos += p_tot
        for j in arb_ix:
            out = out + rk32[:, j] * (c @ cf32[pos:pos + n_char])
            pos += n_char
        sc = np.float32(path_block_scale)
        for j in mod_ix:
            out = out + (sc * rk32[:, j]) * (d @ cf32[pos:pos + n_dl])
            pos += n_dl
        if n_char > 0:
            z = (c @ ms["b"].astype(np.float32) - ms["mu"].astype(np.float32)) / ms["sd"].astype(np.float32)
            for k in range(ms["b"].shape[1]):
                out = out + z[:, k] * (d @ cf32[pos:pos + n_dl])
                pos += n_dl
        return out.astype(np.float64)

    target_keys = ("peer", "pca", "j13")
    eff_mod_rank = min(modulator_rank, max(n_char, 0), n_dl)
    q_fix = expand_fixed(np.zeros((1, p_tot), dtype=np.float32)).shape[1]
    q_dyn = n_dl * eff_mod_rank
    q = q_fix + q_dyn

    g_ff = np.zeros((q_fix, q_fix), dtype=np.float64)
    g_ff_ewm = np.zeros((q_fix, q_fix), dtype=np.float64)
    g_fy = {k: np.zeros(q_fix, dtype=np.float64) for k in target_keys}
    g_fy_ewm = {k: np.zeros(q_fix, dtype=np.float64) for k in target_keys}
    m_acc = {k: np.zeros((n_dl, n_char), dtype=np.float64) for k in target_keys}
    m_acc_ewm = {k: np.zeros((n_dl, n_char), dtype=np.float64) for k in target_keys}
    n_ewm = 0.0
    prev_row = 0
    prev_refit_t = 0

    coefs: dict[str, np.ndarray | None] = {k: None for k in target_keys}
    mstates: dict[str, dict[str, np.ndarray] | None] = {k: None for k in target_keys}
    bt_peer: np.ndarray | None = None
    bt_pca: np.ndarray | None = None
    bt_j13: np.ndarray | None = None
    m1_peer = None
    m2_peer = None
    m1_pca = None
    m2_pca = None
    m1_j13 = None
    m2_j13 = None
    m_tanh_z = None

    hgb_rn_pca = None
    hgb_rn_j13 = None
    hgb_raw_pca = None
    hgb_raw402 = None
    last_hgb_year: int | None = None

    band_hist: list[list[float]] = []
    vol_hist: list[float] = []
    real_sq_hist: list[float] = []
    exante_sq_hist: list[float] = []
    fmvo_hist: list[tuple[int, np.ndarray, np.ndarray, np.ndarray]] = []
    frames: list[pd.DataFrame] = []

    def _append_fmvo_month(tt: int) -> None:
        k_b = len(barra_ix)
        if k_b < 2:
            return
        lo_m, hi_m = int(starts[tt]), int(ends[tt])
        xm_b = r[lo_m:hi_m, barra_ix].astype(np.float64)
        rm_b = y_all[lo_m:hi_m].astype(np.float64)
        ok_b = np.isfinite(rm_b) & np.isfinite(xm_b).all(axis=1)
        if int(ok_b.sum()) > k_b + 5:
            xo_b, ro_b = xm_b[ok_b], rm_b[ok_b]
            f_b = np.linalg.solve(xo_b.T @ xo_b + 1e-4 * np.eye(k_b, dtype=np.float64), xo_b.T @ ro_b)
            res_b = (ro_b - xo_b @ f_b).astype(np.float64)
            fmvo_hist.append((tt, f_b, ids[lo_m:hi_m][ok_b], res_b))
            if len(fmvo_hist) > fmvo_window_months + 5:
                del fmvo_hist[: len(fmvo_hist) - fmvo_window_months]

    t_start = int(np.searchsorted(months, np.datetime64(start_date, "D")))
    if is_val_mode:
        t_start = max(t_start, min(60, max(12, len(months) // 2)))
    eff_min_band_hist = (
        min(min_band_history, max(12, (len(months) - t_start) // 2))
        if is_val_mode
        else min_band_history
    )

    # Pre-populate strictly causal monthly Barra-13 factor/residual history for tt < t_start
    for tt_pre in range(min(t_start, len(months))):
        _append_fmvo_month(tt_pre)

    n_emit = 0
    for t in range(t_start, len(months)):
        a, b = int(starts[t]), int(ends[t])
        rows = np.arange(a, b, dtype=np.int64)

        # 1. Annual Shared-Gram Triple-Target Modulated Ridge Refit (strictly on tt < t)
        if coefs["peer"] is None or mdates[t].month == 1:
            n_cut = int(m_starts[t])
            if prev_row > 0 and t > prev_refit_t:
                step_decay = 0.5 ** (float(t - prev_refit_t) / ridge_ewma_halflife)
                g_ff_ewm *= step_decay
                n_ewm *= step_decay
                for k in target_keys:
                    g_fy_ewm[k] *= step_decay
                    m_acc_ewm[k] *= step_decay

            for i in range(prev_row, n_cut, 24576):
                j_end = min(i + 24576, n_cut)
                rk_c = r[all_rw[i:j_end]]
                xf32 = expand_fixed(rk_c)
                g_ff += (xf32.T @ xf32).astype(np.float64)
                w_ew = (
                    0.5 ** ((float(t - 1) - row_month[i:j_end].astype(np.float64)) / ridge_ewma_halflife)
                ).astype(np.float32)
                xf_w = xf32 * np.sqrt(w_ew)[:, None]
                g_ff_ewm += (xf_w.T @ xf_w).astype(np.float64)
                n_ewm += float(w_ew.sum())
                c_b, d_b = rk_c[:, :n_char], rk_c[:, n_char:]
                for k, y_vec in (
                    ("peer", y_ord_peer[i:j_end]),
                    ("pca", y_ord_pca[i:j_end]),
                    ("j13", y_ord_j13[i:j_end]),
                ):
                    y_w = y_vec * w_ew
                    g_fy[k] += (xf32.T @ y_vec).astype(np.float64)
                    g_fy_ewm[k] += (xf32.T @ y_w).astype(np.float64)
                    if n_char > 0:
                        m_acc[k] += (d_b.T @ (c_b * y_vec[:, None])).astype(np.float64)
                        m_acc_ewm[k] += (d_b.T @ (c_b * y_w[:, None])).astype(np.float64)

            prev_row = n_cut
            prev_refit_t = t

            w_exp = 1.0 - ridge_ewma_alpha
            w_ewm_scale = ridge_ewma_alpha * (float(max(n_cut, 1)) / max(n_ewm, 1.0))
            g_ff_ms = w_exp * g_ff + w_ewm_scale * g_ff_ewm
            pen_scale = 0.75 + 0.25 * (float(n_cut) / 250000.0)
            pen = np.full(q, ridge_inter * pen_scale, dtype=np.float64)
            pen[:p_tot] = ridge_main * pen_scale

            if eff_mod_rank > 0:
                isq_d = inv_sqrt_psd(g_ff_ms[n_char:p_tot, n_char:p_tot] + ridge_inter * np.eye(n_dl))
                isq_c = inv_sqrt_psd(g_ff_ms[:n_char, :n_char] + ridge_inter * np.eye(n_char))
                bms: dict[str, np.ndarray] = {}
                for k in target_keys:
                    m_acc_ms = w_exp * m_acc[k] + w_ewm_scale * m_acc_ewm[k]
                    _, _, vt = np.linalg.svd(isq_d @ m_acc_ms @ isq_c, full_matrices=False)
                    bms[k] = (isq_c @ vt[:eff_mod_rank].T).astype(np.float32)

                s1 = {k: np.zeros(eff_mod_rank) for k in target_keys}
                s2 = {k: np.zeros(eff_mod_rank) for k in target_keys}
                for i in range(0, n_cut, 65536):
                    c_s = r[all_rw[i:min(i + 65536, n_cut)], :n_char]
                    for k in target_keys:
                        zc = (c_s @ bms[k]).astype(np.float64)
                        s1[k] += zc.sum(0)
                        s2[k] += (zc ** 2).sum(0)
                denom = max(n_cut, 1)
                mus, sds = {}, {}
                for k in target_keys:
                    mus[k] = s1[k] / denom
                    sds[k] = np.maximum(np.sqrt(np.maximum(s2[k] / denom - mus[k] ** 2, 0.0)), 1e-12)
                    mstates[k] = {"b": bms[k].astype(np.float64), "mu": mus[k], "sd": sds[k]}

                g_fd = {k: np.zeros((q_fix, q_dyn)) for k in target_keys}
                g_dd = {k: np.zeros((q_dyn, q_dyn)) for k in target_keys}
                g_dy = {k: np.zeros(q_dyn) for k in target_keys}
                sc = np.float32(path_block_scale)
                for i in range(0, n_cut, 40960):
                    j_end = min(i + 40960, n_cut)
                    rk_s = r[all_rw[i:j_end]]
                    c_s, d_s = rk_s[:, :n_char], rk_s[:, n_char:]
                    w_ms = (
                        w_exp
                        + w_ewm_scale * (0.5 ** ((float(t - 1) - row_month[i:j_end].astype(np.float64)) / ridge_ewma_halflife))
                    ).astype(np.float32)
                    for k, y_vec in (
                        ("peer", y_ord_peer[i:j_end]),
                        ("pca", y_ord_pca[i:j_end]),
                        ("j13", y_ord_j13[i:j_end]),
                    ):
                        z_s = (c_s @ bms[k] - mus[k].astype(np.float32)) / sds[k].astype(np.float32)
                        xd_s = np.column_stack([d_s * z_s[:, [kk]] for kk in range(eff_mod_rank)])
                        xd_w = xd_s * w_ms[:, None]
                        blocks = [(rk_s.T @ xd_w).astype(np.float64)]
                        if size_ix is not None:
                            blocks.append((rk_s.T @ (xd_w * rk_s[:, [size_ix]])).astype(np.float64))
                        for j in arb_ix:
                            blocks.append((c_s.T @ (xd_w * rk_s[:, [j]])).astype(np.float64))
                        for j in mod_ix:
                            blocks.append((sc * (d_s.T @ (xd_w * rk_s[:, [j]]))).astype(np.float64))
                        g_fd[k] += np.vstack(blocks)
                        g_dd[k] += (xd_s.T @ xd_w).astype(np.float64)
                        g_dy[k] += (xd_w.T @ y_vec).astype(np.float64)

                for k in target_keys:
                    g_fy_ms = w_exp * g_fy[k] + w_ewm_scale * g_fy_ewm[k]
                    xtx = np.block([[g_ff_ms, g_fd[k]], [g_fd[k].T, g_dd[k]]])
                    xtx[np.diag_indices(q)] += pen
                    coefs[k] = sp_solve(xtx, np.concatenate([g_fy_ms, g_dy[k]]), assume_a="pos")
            else:
                for k in target_keys:
                    g_fy_ms = w_exp * g_fy[k] + w_ewm_scale * g_fy_ewm[k]
                    xtx = g_ff_ms.copy()
                    xtx[np.diag_indices(q_fix)] += pen
                    coefs[k] = sp_solve(xtx, g_fy_ms, assume_a="pos")
                    mstates[k] = {"b": np.zeros((0, 0)), "mu": np.zeros(0), "sd": np.ones(0)}

        # 2. Five-Year Expanding Triple-Target SWA Residual ReLU Cascades + Smooth Tanh MLP (strictly tt < t)
        net_due = m1_peer is None or (mdates[t].month == 1 and mdates[t].year % mlp_refit_years == 0)
        if net_due:
            t_mlp = time.perf_counter()
            cut = mdates[t].replace(year=mdates[t].year - mlp_valid_years, day=min(mdates[t].day, 28))
            keep_m = [tt for tt in range(t) if m_len[tt] >= min_names]
            if len(keep_m) < 36:
                keep_m = [tt for tt in range(t) if m_len[tt] > 2]
            if keep_m:
                tr_rows = np.concatenate([shared_rows[tt] for tt in keep_m])
                tr_valid = np.concatenate([np.full(m_len[tt], mdates[tt] >= cut) for tt in keep_m])
                m_lens = [int(m_len[tt]) for tt in keep_m]

                # 2a) Prepare Peer449, PCA449, and Joint13 Stage-1 Residuals (Tail Ridge removed)
                def _stage1_resid(y_chunks: list[np.ndarray], y_vars: np.ndarray, key: str) -> tuple[np.ndarray, np.ndarray]:
                    y_raw = np.concatenate([y_chunks[tt] for tt in keep_m]).astype(np.float32)
                    fit_lin = np.empty(len(tr_rows), dtype=np.float32)
                    for ii in range(0, len(tr_rows), 65536):
                        jj = min(ii + 65536, len(tr_rows))
                        fit_lin[ii:jj] = predict_ridge(r[tr_rows[ii:jj]], coefs[key], mstates[key]).astype(np.float32)
                    res1 = y_raw - fit_lin
                    del y_raw, fit_lin
                    bt = fit_tail_ridge(r, tr_rows, res1, m_lens, y_vars[keep_m], n_char)
                    if n_char > 0:
                        for ii in range(0, len(tr_rows), 65536):
                            jj = min(ii + 65536, len(tr_rows))
                            res1[ii:jj] -= (np.abs(r[tr_rows[ii:jj], :n_char]) - np.float32(0.25)) @ bt
                    return res1, bt

                res_r1, bt_peer = _stage1_resid(shared_y_peer, var_peer, "peer")
                res_p1, bt_pca = _stage1_resid(shared_y_pca, var_pca, "pca")
                res_j1, bt_j13 = _stage1_resid(shared_y_j13, var_j13, "j13")

                # 2b) Stage-1 & Stage-2 Triple-Target Smooth-SWA ReLU Cascades + Concurrent Smooth-SWA Tanh MLP
                m1_peer, m1_pca, m1_j13 = fit_swa_mlp_triple_ensembles(
                    r, tr_rows, res_r1, res_p1, res_j1, tr_valid, n_char, p_tot, mlp_relu_hidden, "relu",
                    mlp_relu_epochs, mlp_relu_lr, mlp_relu_wd, mlp_relu_dropout, mlp_relu_feat_frac, 0,
                )
                with ThreadPoolExecutor(max_workers=3) as ex_pred:
                    fut_pr = ex_pred.submit(predict_mlp_rows, m1_peer, r, tr_rows)
                    fut_pp = ex_pred.submit(predict_mlp_rows, m1_pca, r, tr_rows)
                    fut_pj = ex_pred.submit(predict_mlp_rows, m1_j13, r, tr_rows)
                    rem2_r = res_r1 - fut_pr.result()
                    rem2_p = res_p1 - fut_pp.result()
                    rem2_j = res_j1 - fut_pj.result()
                del res_r1, res_p1, res_j1

                keep_z = [tt for tt in range(t) if z_m_len[tt] >= min_names]
                if len(keep_z) < 24:
                    keep_z = [tt for tt in range(t) if z_m_len[tt] > 2]
                if keep_z:
                    tz_rows = np.concatenate([z_rows[tt] for tt in keep_z])
                    tz_valid = np.concatenate([np.full(z_m_len[tt], mdates[tt] >= cut) for tt in keep_z])
                    yz_raw = y_z_raw[tz_rows].astype(np.float32)
                else:
                    tz_rows, tz_valid, yz_raw = None, None, None

                m2_peer, m2_pca, m2_j13, m_tanh_new = fit_swa_mlp_stage2_with_tanh(
                    r, tr_rows, rem2_r, rem2_p, rem2_j, tr_valid, tz_rows, yz_raw, tz_valid, n_char, p_tot,
                )
                if m_tanh_new is not None:
                    m_tanh_z = m_tanh_new
                del rem2_r, rem2_p, rem2_j, tz_rows, tz_valid, yz_raw
                n_tr_rows = len(tr_rows)
                del tr_rows, tr_valid
                gc.collect()
                print(
                    f"refit triple neural cascades {mdates[t].year} on {n_tr_rows:,} rows in "
                    f"{time.perf_counter() - t_mlp:.1f}s (elapsed {(time.perf_counter() - t0) / 60.0:.1f} min)",
                    flush=True,
                )

        # 3. Five-Year Expanding Deterministic HistGBDT Ensembles (RN449-PCA, RN449-J13, Raw449-PCA, Raw402-PCA)
        yr_now = mdates[t].year
        hgb_due = last_hgb_year is None or (
            mdates[t].month == 1 and (yr_now - 1960) % hgb_refit_years == 0 and yr_now != last_hgb_year
        )
        if hgb_due:
            t_hgb = time.perf_counter()
            tr_hi = int(starts[t])
            yt_hgb = y_pca_res[:tr_hi]
            mt_hgb = np.isfinite(yt_hgb)
            n_hgb_orig = int(mt_hgb.sum())
            if n_hgb_orig >= hgb_min_train_rows and n_char > 0:
                tr_idx = np.flatnonzero(mt_hgb)
                if n_hgb_orig > hgb_max_train_rows:
                    rng_sub = np.random.default_rng(seed + 997 + 100 * (yr_now - 1950))
                    tr_idx = tr_idx[np.sort(rng_sub.choice(n_hgb_orig, hgb_max_train_rows, replace=False))]
                n_hgb_tr = len(tr_idx)
                orig_tr = order32[tr_idx]
                y_tr_pca = yt_hgb[tr_idx]
                y_tr_j13 = np.where(np.isfinite(y_joint13_res[tr_idx]), y_joint13_res[tr_idx], y_tr_pca)

                # Sequential 4+4 HGB refit reuses a SINGLE float64 buffer for Raw449 -> RN449
                # so rn449_pca and rn449_j13 share zero-copy inputs while keeping peak RAM < 17 GB.
                x_buf64 = np.empty((n_hgb_tr, p_tot), dtype=np.float64)
                _fill_raw_slice(raw_cols, orig_tr, x_buf64[:, :n_char])
                x_buf64[:, n_char:] = r[tr_idx, n_char:]
                x_raw402 = np.ascontiguousarray(x_buf64[:, :n_char], dtype=np.float64)
                hgb_raw402, hgb_raw_pca = _fit_hgb_batch(
                    [(x_raw402, y_tr_pca, 3), (x_buf64, y_tr_pca, 2)], yr_now,
                )
                del x_raw402
                gc.collect()
                for c0 in range(0, n_hgb_tr, 65536):
                    c1 = min(c0 + 65536, n_hgb_tr)
                    nan_m = np.isnan(x_buf64[c0:c1, :n_char])
                    x_buf64[c0:c1, :n_char] = r[tr_idx[c0:c1], :n_char]
                    x_buf64[c0:c1, :n_char][nan_m] = np.nan
                hgb_rn_pca, hgb_rn_j13 = _fit_hgb_batch(
                    [(x_buf64, y_tr_pca, 1), (x_buf64, y_tr_j13, 4)], yr_now,
                )
                del x_buf64
                gc.collect()
                print(
                    f"refit 4 HGB families {yr_now} on {n_hgb_tr:,}/{n_hgb_orig:,} rows in "
                    f"{time.perf_counter() - t_hgb:.1f}s (elapsed {(time.perf_counter() - t0) / 60.0:.1f} min)",
                    flush=True,
                )
            last_hgb_year = yr_now

        # 4. Strictly Causal Universe Selection, Book-Matched Alpha & Equal-Risk 5-Book Portfolio
        if has_dmat and len(rows) > 0:
            end = int(np.searchsorted(dates, np.datetime64(months[t], "D"), side="right"))
            hd = has_d[rows] & (first_obs_day[col_all[rows]] < end)
            rr = rows[hd]
            if len(rr) >= min_names and end >= 40:
                lo = max(0, end - sample_window_days)
                xd = dmat[lo:end][:, col_all[rr]].astype(np.float64)
                obs = np.isfinite(xd)
                min_obs_eff = min(min_daily_obs, max(20, (end - lo) // 2)) if is_val_mode else min_daily_obs
                enough = obs.sum(0) >= min_obs_eff
                rr = rr[enough]
                ob = obs[:, enough]
                xs_raw = xd[:, enough]

                if len(rr) >= min_names and xs_raw.shape[0] >= 40 and coefs["peer"] is not None:
                    xd_c = np.where(ob, xs_raw, 0.0)
                    xd_c -= xd_c.mean(0)
                    nob = np.maximum(ob.sum(0), 1)
                    xs = (
                        np.where(ob, xs_raw - np.where(ob, xs_raw, 0.0).sum(0) / nob, 0.0)
                        * np.sqrt(xs_raw.shape[0] / nob)
                    )

                    rk_rr = r[rr]
                    # 4a) Compute all 8 strictly causal alpha learner predictions on month t
                    s_peer = predict_ridge(rk_rr, coefs["peer"], mstates["peer"])
                    if bt_peer is not None and m1_peer is not None and m2_peer is not None:
                        tail_r = ((np.abs(rk_rr[:, :n_char]) - np.float32(0.25)) @ bt_peer).astype(np.float64) if n_char > 0 else 0.0
                        s_peer = s_peer + 1.05 * (tail_r + predict_mlp_ensemble(m1_peer, rk_rr) + predict_mlp_ensemble(m2_peer, rk_rr))

                    s_pca = predict_ridge(rk_rr, coefs["pca"], mstates["pca"])
                    if bt_pca is not None and m1_pca is not None and m2_pca is not None:
                        tail_p = ((np.abs(rk_rr[:, :n_char]) - np.float32(0.25)) @ bt_pca).astype(np.float64) if n_char > 0 else 0.0
                        s_pca = s_pca + 1.05 * (tail_p + predict_mlp_ensemble(m1_pca, rk_rr) + predict_mlp_ensemble(m2_pca, rk_rr))

                    s_j13_cas = predict_ridge(rk_rr, coefs["j13"], mstates["j13"])
                    if bt_j13 is not None and m1_j13 is not None and m2_j13 is not None:
                        tail_j = ((np.abs(rk_rr[:, :n_char]) - np.float32(0.25)) @ bt_j13).astype(np.float64) if n_char > 0 else 0.0
                        s_j13_cas = s_j13_cas + 1.05 * (tail_j + predict_mlp_ensemble(m1_j13, rk_rr) + predict_mlp_ensemble(m2_j13, rk_rr))

                    s_tz = predict_mlp_ensemble(m_tanh_z, rk_rr) if m_tanh_z is not None else None

                    s_hrn_pca, s_hrn_j13, s_hrw_pca, s_h402 = None, None, None, None
                    if n_char > 0 and (
                        hgb_rn_pca is not None
                        or hgb_rn_j13 is not None
                        or hgb_raw_pca is not None
                        or hgb_raw402 is not None
                    ):
                        x_raw_m = np.empty((len(rr), n_char), dtype=np.float32)
                        _fill_raw_slice(raw_cols, order32[rr], x_raw_m)
                        if hgb_rn_pca is not None or hgb_rn_j13 is not None:
                            xr_m = rk_rr.copy()
                            xr_m[:, :n_char][np.isnan(x_raw_m)] = np.nan
                            if hgb_rn_pca is not None:
                                s_hrn_pca = _predict_hgb_family(hgb_rn_pca, xr_m)
                            if hgb_rn_j13 is not None:
                                s_hrn_j13 = _predict_hgb_family(hgb_rn_j13, xr_m)
                        if hgb_raw_pca is not None:
                            xrw_m = np.empty_like(rk_rr)
                            xrw_m[:, :n_char] = x_raw_m
                            xrw_m[:, n_char:] = rk_rr[:, n_char:]
                            s_hrw_pca = _predict_hgb_family(hgb_raw_pca, xrw_m)
                        if hgb_raw402 is not None:
                            s_h402 = _predict_hgb_family(hgb_raw402, x_raw_m)

                    parts = {
                        "peer_cas": s_peer,
                        "pca_cas": s_pca,
                        "j13_cas": s_j13_cas,
                        "tanh_z": s_tz,
                        "hgb_rn_pca": s_hrn_pca,
                        "hgb_rn_j13": s_hrn_j13,
                        "hgb_raw_pca": s_hrw_pca,
                        "hgb_raw402": s_h402,
                    }
                    s_spin = _combine_weighted_signals(parts, w_spin_alpha, len(rr))
                    s_mvo = _combine_weighted_signals(parts, w_mvo_alpha, len(rr))
                    s_j13 = _combine_weighted_signals(parts, w_j13_alpha, len(rr))

                    # 4b) 5-band Laplacian decomposition on s_spin
                    s3 = sic[rr]
                    c1 = group_mean(s_spin, s3 // 100)
                    c2 = group_mean(s_spin, s3 // 10) - c1
                    c3 = group_mean(s_spin, s3) - c1 - c2
                    r3 = s_spin - c1 - c2 - c3
                    nn = knn_f32(rk_rr[:, :n_char + path_features], knn_peers)
                    c4 = r3[nn].mean(1)
                    c5 = r3 - c4
                    bands = np.column_stack([c1, c2, c3, c4, c5])

                    w_unit_emitted: np.ndarray | None = None
                    vol_raw_emitted: float | None = None

                    if len(band_hist) >= eff_min_band_hist:
                        past_b = np.array(band_hist[-band_window:], dtype=np.float64)
                        mu_b = past_b.mean(0)
                        if len(band_hist) < min_band_history:
                            w_b = float(len(band_hist)) / float(min_band_history)
                            b_sd = np.maximum(past_b.std(0), 1e-4)
                            prior_mu = np.array([mu_b[0], mu_b[1], mu_b[2], 0.35 * b_sd[3], 0.50 * b_sd[4]], dtype=np.float64)
                            mu_b = w_b * mu_b + (1.0 - w_b) * prior_mu
                            mu_b[4] = max(mu_b[4], 0.25 * float(b_sd[4]))
                        cv_b = np.cov(past_b.T)
                        cv_b = (1.0 - band_shrink) * cv_b + band_shrink * np.diag(np.diag(cv_b))
                        theta = np.linalg.solve(cv_b + 1e-10 * np.eye(5), mu_b)
                        scale = np.maximum(np.abs(bands).mean(0), 1e-12)
                        sig_sp = bands @ (theta / scale)

                        # Shared 3-day overlapping Ledoit-Wolf correlation/vol & diagonal vol caches
                        x3_32 = (
                            (xs[2:] + xs[1:-1] + xs[:-2]) / math.sqrt(3.0) * math.sqrt(days_per_month)
                        ).astype(np.float32)
                        corr32, own_sd = ledoit_wolf_f32(x3_32)
                        sd21_diag = np.sqrt(np.mean(xd_c[-21:] ** 2, axis=0) * days_per_month + 1e-18)
                        sd63_diag = np.sqrt(np.mean(xd_c[-63:] ** 2, axis=0) * days_per_month + 1e-18)

                        b2_v = np.nan_to_num(beta252[rr].astype(np.float64))
                        b60_v = np.nan_to_num(bet60[rr])
                        if float(np.std(b2_v)) < 1e-10:
                            b2_v = b60_v
                        lm_v = np.log(np.maximum(np.nan_to_num(me[rr], nan=1.0), 1e-3))

                        # Book 1: 5-Band Laplacian Mean-Field Lead-Lag Spin Book
                        m_spin = spin_solve(sig_sp, corr32)
                        pooled_sd = (1.0 - vol_pool) * own_sd + vol_pool * own_sd[nn].mean(1)
                        w_spin = project_neutral_and_safe_cap(m_spin / pooled_sd, b2_v, lm_v, bet60_v=b60_v, max_share=0.08)

                        # 756-day MP-PCA risk model on exact active universe rr
                        r_raw_rr = np.where(ob, xs_raw, 0.0)
                        b_mp, psi_i, sig_i, _, _ = _pca_risk_with_factors(r_raw_rr)
                        sc_i = np.maximum(sig_i * np.sqrt(psi_i), 1e-6)
                        mu_mvo = zs(s_mvo * sc_i)
                        mu_j13 = zs(s_j13 * sc_i)

                        # Book 2: 3-Day Overlapping Ledoit-Wolf Daily Covariance MVO (Tangency minus GMV)
                        sig_3d = corr32.astype(np.float64) * np.outer(own_sd, own_sd)
                        w_lw = tan_minus_gmv(mu_mvo, sig_3d, kappa=lw_mvo_kappa)

                        # Book 3: Joint MP-PCA + Barra-13 + SIC2 Residual Book (target-matched mu_j13)
                        inv_s = 1.0 / np.maximum(sig_i, 1e-8)
                        ind2_dum = _industry_dummies((s3 // 100).astype(np.int64))
                        z_b13 = (
                            np.column_stack([beta252_rk[rr].astype(np.float64), rk_rr[:, barra_ix].astype(np.float64)])
                            if barra_ix
                            else beta252_rk[rr].astype(np.float64)[:, None]
                        )
                        g_joint = np.column_stack([b_mp, inv_s, z_b13 * inv_s[:, None], ind2_dum * inv_s[:, None]])
                        w_joint = _residual_weights(mu_j13, sig_i, psi_i, g_joint)
                        gj = float(np.abs(w_joint).sum())
                        w_joint = w_joint / gj if gj > 1e-12 else np.zeros(len(rr), dtype=np.float64)

                        # Book 4: Pure Statistical MP-PCA + SIC2 GLS Residual Book
                        g_mp = np.column_stack([b_mp, inv_s, ind2_dum * inv_s[:, None]])
                        w_mp = _residual_weights(mu_mvo, sig_i, psi_i, g_mp)
                        gmp = float(np.abs(w_mp).sum())
                        w_mp = w_mp / gmp if gmp > 1e-12 else np.zeros(len(rr), dtype=np.float64)

                        # Book 5: Causal 60-Month Fundamental Barra-13 Factor MVO (Woodbury inversion)
                        w_fmvo = np.zeros(len(rr), dtype=np.float64)
                        fmvo_win = [e for e in fmvo_hist if e[0] >= t - fmvo_window_months]
                        if len(barra_ix) >= 2 and len(fmvo_win) >= max(12, len(barra_ix)):
                            f_mat = np.array([e[1] for e in fmvo_win], dtype=np.float64)
                            fmvo_om = np.cov(f_mat, rowvar=False)
                            ids_win = np.concatenate([e[2] for e in fmvo_win])
                            res_win = np.concatenate([e[3] for e in fmvo_win])
                            rv_s = pd.Series(res_win, index=ids_win).groupby(level=0).var()
                            d_v = pd.Series(ids[rr]).map(rv_s).to_numpy(dtype=np.float64)
                            fin_dv = np.isfinite(d_v)
                            if int(fin_dv.sum()) >= 5:
                                fl = max(float(np.nanquantile(d_v[fin_dv], 0.10)), 1e-8)
                                fmvo_d = np.where(fin_dv, np.maximum(d_v, fl), fl)
                                wf = _woodbury_sigma_inv_mu(rk_rr[:, barra_ix].astype(np.float64), fmvo_om, fmvo_d, mu_mvo)
                                wf = wf - group_mean(wf, s3 // 100)
                                wf = wf - float(wf.mean())
                                g_wf = float(np.abs(wf).sum())
                                if g_wf > 1e-12:
                                    w_fmvo = wf / g_wf

                        # Equal-risk (inverse ex-ante vol) 5-book blend -> [1, beta252, beta60, log(ME)] orthogonal projection
                        w_blend = np.zeros(len(rr), dtype=np.float64)
                        for wb, alp in zip((w_spin, w_lw, w_joint, w_mp, w_fmvo), book_risk_weights):
                            if float(np.abs(wb).sum()) > 1e-12:
                                vb = exante_vol_of_unit(wb, xd_c, corr32, own_sd, sd21_diag, sd63_diag)
                                w_blend += alp * (wb / max(vb, 1e-6))

                        w_unit = project_neutral_and_safe_cap(w_blend, b2_v, lm_v, bet60_v=b60_v, max_share=0.055)
                        if float(np.abs(w_unit).sum()) > 1e-12:
                            vol_raw = exante_vol_of_unit(w_unit, xd_c, corr32, own_sd, sd21_diag, sd63_diag)
                            if vol_raw > 1e-12:
                                if len(real_sq_hist) >= 12:
                                    n_rv = len(real_sq_hist)
                                    d_rv = 0.5 ** (np.arange(n_rv - 1, -1, -1, dtype=np.float64) / 18.0)
                                    r_rms = float(np.sqrt(np.sum(d_rv * np.array(real_sq_hist)) / np.sum(d_rv)))
                                    x_rms = float(np.sqrt(np.sum(d_rv * np.array(exante_sq_hist)) / np.sum(d_rv)))
                                    w_rv = float(n_rv) / (float(n_rv) + 18.0)
                                    ratio_rv = w_rv * (r_rms / max(x_rms, 1e-12)) + (1.0 - w_rv) * 1.0
                                    kappa = float(np.clip(ratio_rv ** vol_cal_exp, vol_cal_lo, vol_cal_hi))
                                else:
                                    kappa = 1.0
                                vol_eff = vol_raw * kappa
                                vref = (
                                    float(np.median(vol_hist[-vref_window_months:]))
                                    if len(vol_hist) >= 24
                                    else vol_eff
                                )
                                lev = (
                                    target_vol_annual / math.sqrt(12.0) / vol_eff
                                    * (vref / vol_eff) ** (vol_power - 1.0)
                                )
                                lev_cap = soft_cap_lp(lev, cap=soft_cap_ceiling, p=soft_cap_power)
                                w = w_unit * lev_cap
                                vol_hist.append(vol_eff)
                                w_unit_emitted = w_unit
                                vol_raw_emitted = vol_raw
                            else:
                                w = w_unit
                            wt = np.zeros(b - a, dtype=np.float64)
                            wt[rr - a] = w
                            frames.append(pd.DataFrame({"id": ids[a:b], "eom": eom_out[a:b], "w": wt}))
                            n_emit += 1

                    # Update band return & realized unit-gross vol history strictly AFTER month t emission
                    yy = y_all[rr]
                    ok = np.isfinite(yy)
                    if int(ok.sum()) >= min_names:
                        br = []
                        for k in range(bands.shape[1]):
                            v = bands[ok, k] - bands[ok, k].mean()
                            gk = float(np.abs(v).sum())
                            br.append(float((v / gk) @ yy[ok]) if gk > 0 else 0.0)
                        band_hist.append(br)
                        if w_unit_emitted is not None and vol_raw_emitted is not None:
                            r_unit_real = float(w_unit_emitted[ok] @ yy[ok])
                            if math.isfinite(r_unit_real):
                                real_sq_hist.append(r_unit_real ** 2)
                                exante_sq_hist.append(vol_raw_emitted ** 2)

        # Update strictly causal monthly Barra-13 factor/residual history AFTER month t emission
        _append_fmvo_month(t)

        if mdates[t].month == 12:
            gc.collect()
            print(
                f"{mdates[t].year} emitted {n_emit} months, elapsed "
                f"{(time.perf_counter() - t0) / 60.0:.1f} min",
                flush=True,
            )

    if not frames:
        return pd.DataFrame(columns=["id", "eom", "w"])
    out = pd.concat(frames, ignore_index=True)
    out["id"] = out["id"].astype(np.int64)
    print(
        f"done: {len(out):,} rows, {n_emit} months in {(time.perf_counter() - t0) / 60.0:.1f} min",
        flush=True,
    )
    return out


def is_test_row(flag: pd.Series) -> pd.Series:
    if pd.api.types.is_bool_dtype(flag):
        return flag.fillna(False)
    if pd.api.types.is_numeric_dtype(flag):
        return flag.fillna(0).ne(0)
    return flag.astype(str).str.strip().str.lower().isin(["1", "1.0", "true", "t"])


def restrict_to_test(chars: pd.DataFrame, weights: pd.DataFrame) -> pd.DataFrame:
    if "ctff_test" not in chars.columns:
        return weights
    mask = is_test_row(chars["ctff_test"])
    if not mask.any():
        return weights
    keys = chars.loc[mask, ["id", "eom"]].copy()
    keys["id"] = np.asarray(pd.to_numeric(keys["id"]), dtype=np.int64)
    keys["eom"] = pd.to_datetime(keys["eom"])
    keys = keys.drop_duplicates()
    if len(weights) == 0:
        keys["w"] = 0.0
        return keys[["id", "eom", "w"]]
    w = weights[["id", "eom", "w"]].copy()
    w["id"] = np.asarray(pd.to_numeric(w["id"]), dtype=np.int64)
    w["eom"] = pd.to_datetime(w["eom"])
    out = keys.merge(w, on=["id", "eom"], how="left")
    out["w"] = out["w"].fillna(0.0)
    return out[["id", "eom", "w"]]


def report_sharpe(weights: pd.DataFrame, chars: pd.DataFrame) -> None:
    ret = chars[["id", "eom", target_col]].copy()
    ret["eom"] = pd.to_datetime(ret["eom"])
    ret["id"] = np.asarray(pd.to_numeric(ret["id"]), dtype=np.int64)
    ret = ret.rename(columns={target_col: "r"}).dropna(subset=["r"])
    w = weights.copy()
    w["eom"] = pd.to_datetime(w["eom"])
    j = w.merge(ret, on=["id", "eom"], how="inner")
    j["c"] = j["w"] * j["r"]
    series = j.groupby("eom")["c"].sum().sort_index()
    test = series[series.index >= "1989-12-01"]
    if len(test) == 0:
        test = series
    if len(test) < 12:
        print("sharpe not reported, too few observed months", flush=True)
        return
    sd = float(test.std(ddof=1))
    ann_ret = float(test.mean()) * 12.0
    ann_vol = sd * math.sqrt(12.0)
    sharpe = float(test.mean() / sd * math.sqrt(12.0)) if sd > 1e-12 else float("nan")
    cum = test.cumsum()
    mdd = float((cum - cum.cummax()).min())
    print(
        f"months scored: {len(test)} ({test.index.min().strftime('%Y-%m')} to {test.index.max().strftime('%Y-%m')})",
        flush=True,
    )
    print(
        f"test sharpe: {sharpe:.4f} | ann_ret: {ann_ret:.4%} | ann_vol: {ann_vol:.4%} | max_dd: {mdd:.4%}",
        flush=True,
    )
    for dec_lbl, d0, d1 in [
        ("1990s", "1989-12-01", "1999-12-31"),
        ("2000s", "2000-01-01", "2009-12-31"),
        ("2010s", "2010-01-01", "2019-12-31"),
        ("2020s", "2020-01-01", "2024-12-31"),
    ]:
        sub = test[(test.index >= d0) & (test.index <= d1)]
        if len(sub) >= 12 and sub.std(ddof=1) > 1e-12:
            sh_sub = float(sub.mean() / sub.std(ddof=1) * math.sqrt(12.0))
            print(f"  {dec_lbl} ({len(sub)}m): sharpe = {sh_sub:.4f}", flush=True)


def build_weights(
    chars: pd.DataFrame, features: pd.DataFrame, daily_ret: pd.DataFrame
) -> pd.DataFrame:
    return _build_weights_from_box(chars, features, [daily_ret])


def main(
    chars: pd.DataFrame, features: pd.DataFrame, daily_ret: pd.DataFrame
) -> pd.DataFrame:
    d_box = [daily_ret]
    del daily_ret
    weights = _build_weights_from_box(chars, features, d_box)
    out = restrict_to_test(chars, weights)
    out = out.sort_values(["eom", "id"], kind="stable").reset_index(drop=True)
    out["eom"] = pd.to_datetime(out["eom"]).dt.strftime("%Y-%m-%d")
    out["id"] = out["id"].astype(int)
    out["w"] = out["w"].astype(float)
    return out[["id", "eom", "w"]]


if __name__ == "__main__":
    data_dir = (
        sys.argv[1]
        if len(sys.argv) > 1
        else os.environ.get("CTF_DATA_DIR", os.environ.get("JKP_DATA_DIR", "jkp-data"))
    )
    out_csv = (
        sys.argv[2]
        if len(sys.argv) > 2
        else os.environ.get("CTF_OUTPUT_PATH", os.environ.get("JKP_OUT_CSV", "weights.csv"))
    )
    c_name = (
        "ctff_chars.parquet"
        if os.path.exists(os.path.join(data_dir, "ctff_chars.parquet"))
        else "chars.parquet"
    )
    f_name = (
        "ctff_features.parquet"
        if os.path.exists(os.path.join(data_dir, "ctff_features.parquet"))
        else "features.parquet"
    )
    d_name = (
        "ctff_daily_ret.parquet"
        if os.path.exists(os.path.join(data_dir, "ctff_daily_ret.parquet"))
        else "daily_ret.parquet"
    )
    features_df = pd.read_parquet(os.path.join(data_dir, f_name))
    c_path = os.path.join(data_dir, c_name)
    feat_list = [str(f) for f in features_df[feature_list_col].tolist()]
    feat_set = set(feat_list)
    if os.path.getsize(c_path) > 200_000_000:
        core_cols = ["id", "eom", "sic", "market_equity", "beta_60m", "ret_exc_lead1m", "ctff_test"]
        core_df = pd.read_parquet(c_path, columns=core_cols)
        rem_cols = [c for c in feat_list if c not in core_cols]
        raw_mat = np.empty((len(core_df), len(rem_cols)), dtype=np.float32, order="F")
        for b0 in range(0, len(rem_cols), 10):
            chunk_cols = rem_cols[b0:b0 + 10]
            blk = pd.read_parquet(c_path, columns=chunk_cols)
            for k_c, cname in enumerate(chunk_cols):
                raw_mat[:, b0 + k_c] = blk[cname].to_numpy(dtype=np.float32, copy=False)
            del blk
            gc.collect()
        feat_df = pd.DataFrame(raw_mat, columns=rem_cols, copy=False)
        del raw_mat
        chars_df = pd.concat([core_df, feat_df], axis=1, copy=False)
        del core_df, feat_df
        gc.collect()
    else:
        chars_df = pd.read_parquet(c_path)
        for col in chars_df.columns:
            if col in feat_set and col not in ("market_equity", "beta_60m"):
                chars_df[col] = chars_df[col].astype(np.float32)
        gc.collect()
    pf = main(
        chars_df,
        features_df,
        pd.read_parquet(
            os.path.join(data_dir, d_name),
            columns=["id", "date", "ret_exc"],
            dtype_backend="pyarrow",
        ),
    )
    pf.to_csv(out_csv, index=False)
    print(f"wrote {out_csv}: {len(pf):,} rows across {pf['eom'].nunique()} months", flush=True)
    report_sharpe(pf, chars_df)
