"""Spectral Residual Cascade & Lead-Lag Spin Portfolio (SRC-LSP).

Model Name: Spectral Residual Cascade & Lead-Lag Spin Portfolio
Authors: Kush B Vaidya, Tareef Ahmed, Marco Chen, Keren Wang, Tanush Agarwal, Harry Chen

Overview
--------
We model the equity cross-section as a multi-scale economic graph linking each stock to its
2-, 3-, and 4-digit SIC industries and its 20 nearest neighbors in firm-characteristic space,
combining multi-horizon statistical factor-residual and peer-spillover shocks, multi-scale
Kronecker-whitened bilinear-modulated ridge regression, a full-expanding two-stage
Stochastic-Weight-Averaged (SWA) residual neural cascade, graph Laplacian spectral filtering,
and an overlapping 3-day lead-lag mean-field spin portfolio optimizer with decoupled
non-overlapping volatility allocation. Every feature, model parameter, spectral gain,
covariance matrix, and volatility target at month t is estimated strictly from information
realized on or before month t, using pure deterministic NumPy/SciPy linear algebra with zero
external machine-learning dependencies.

1. Inputs & 33 Multi-Horizon Causal Daily Features:
   Each month t, the 402 firm characteristics are cross-sectionally percentile-ranked into
   [-0.5, 0.5] with missing entries mapped to 0.0. From daily excess returns strictly up to
   month-end t (restricted to stocks with at least one prior observed daily return so peer
   sets are strictly invariant to future listings), we construct 33 cross-sectionally ranked
   daily features:
     a) 14 Trailing 21-Day Path & Peer Trajectory Features: cumulative 5d, 10d, 21d, and 1d
        returns, intermediate path legs (days -5..-2, -10..-6, -21..-11), 5d extremes (max/min),
        21d return volatility, 3-digit industry means of 5d and 21d returns, and 20-nearest-
        neighbor characteristic-peer means of 5d and 21d returns.
     b) 19 Trailing 252-Day Marchenko-Pastur Factor-Residual, Microstructure & Peer-Spillover
        Features: On trailing 252-day standardized returns, eigenvalues exceeding the
        Marchenko-Pastur (1967) upper edge (1 + sqrt(N / T))^2 define systematic daily factors
        U_k and loadings B. Standardizing daily factor residuals E_t by idiosyncratic variance
        sqrt(psi_i) isolates firm-specific price pressure from factor momentum, yielding 1d, 5d,
        21d, and 63-21d MP residual shocks, specific variance share psi_i, 21d residual lag-1
        autocorrelation, and mean top-3 daily residual shock. These are paired with volatility
        term-structure ratios (sigma_21 / sigma_252, sigma_63 / sigma_252), 21d skewness,
        characteristic-peer dislocations (own minus peer 5d and 21d returns, peer 63-21d
        momentum), market-equity-weighted peer 21d return (Hou 2007), plus 5 variance-calibrated
        intermediate and idiosyncratic-peer spillover signals: 126-21d intermediate momentum,
        21d downside semivariance ratio, 21d within-SIC3 relative return, and characteristic-peer
        5d and 21d MP factor-residual spillover.

2. Peer-Relative Robust Target:
   Forward 1-month excess returns are demeaned cross-sectionally and within 3-digit SIC
   industries, then passed through arctan(y / std(y)) so all learners focus on within-industry
   relative mispricing without distortion from extreme outliers.

3. Multi-Scale Kronecker-Whitened Modulated Hierarchical Ridge:
   Refit every January on strictly realized prior months (tt < t) blending expanding-window
   sufficient statistics (70%) with 120-month exponentially weighted sufficient statistics
   (30%) to adapt to evolving factor premia while preserving long-run structural memory. The
   linear backbone combines all main effects, market-equity interactions, arbitrage-friction
   interactions (ivol_capm_252d and ami_126d times characteristics), variance-matched (sqrt(12))
   liquidity/volatility path interactions across all daily features, and rank-2 bilinear
   modulation terms extracted from the leading singular vectors of the Kronecker-whitened
   cross-moment (Sigma_D + lambda_int I)^(-1/2) M (Sigma_C + lambda_int I)^(-1/2).

4. Full-Expanding Two-Stage Boltzmann-IC Smooth-SWA Residual MLP Cascade:
   Every fifth January, two boosted stages of 4 multilayer perceptrons (256-128-64 with ReLU,
   untiled per-row inverted dropout 0.2, and decoupled AdamW) are trained sequentially on the
   unexplained residuals of the hierarchical ridge and Stage-1 ensemble over the full expanding
   history of prior realized months. Full-window training checkpoints are aggregated via
   Boltzmann-weighted Stochastic Weight Averaging, w_e proportional to
   exp(150 * (IC_e - max_e' IC_e')).

5. Spectral Graph Filter, Lead-Lag Spin Book & Non-Overlapping Volatility Allocation:
   The combined monthly alpha is decomposed as a 5-band Laplacian pyramid across 2-, 3-, and
   4-digit SIC industries, 20-peer characteristic neighborhoods, and the idiosyncratic residual,
   weighted by 180-month diagonally-shrunk (0.5) band-portfolio tangency weights. Asset risk is
   estimated on trailing 756-day missingness-compensated returns (>= 250 observations) via
   Ledoit-Wolf shrinkage applied to overlapping 3-day returns (r_t + r_{t-1} + r_{t-2}) / sqrt(3)
   scaled by sqrt(21). Positions solve the mean-field spin free-energy minimization
   m = tanh(beta * (h - J * C * m)) (beta = 0.5, J = 0.3) via L-BFGS-B, divided by peer-pooled
   stock volatility (0.5), and projected orthogonal to [1, beta_60m, log(market_equity)].
   Portfolio leverage combines fast recent PnL volatility, non-overlapping out-of-feature-window
   daily PnL volatility (days [-252:-63]), shrunk Ledoit-Wolf quadratic risk, and causal online
   (< t) realized monthly return volatility calibration with a smooth algebraic Lp soft-cap.

Attribution
-----------
Builds upon the public JKP CTF architectures "Topology-Informed Decomposition Ensemble" by
Hema Srikar Ankem (2026) and "Residual Books" by Felix Gehling (2026), and academic research by
Gu, Kelly, and Xiu (2020), Marchenko and Pastur (1967), Lo and MacKinlay (1990), Hou (2007),
Ledoit and Wolf (2004), Izmailov et al. (2018), and Moreira and Muir (2017).
"""
from __future__ import annotations

from concurrent.futures import ThreadPoolExecutor
import gc
import math
import os
import sys
import time

# Force single-threaded BLAS/OpenMP prior to NumPy/SciPy import for bit-for-bit determinism
# across machines regardless of physical CPU core count.
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
os.environ["NUMEXPR_NUM_THREADS"] = "1"

import numpy as np
import pandas as pd
from scipy.linalg import solve as sp_solve
from scipy.optimize import minimize

# Locked Pre-1990 Hyperparameters
ridge_main = 500.0
ridge_inter = 2000.0
ridge_ewma_halflife = 120.0
ridge_ewma_alpha = 0.30
modulator_rank = 2
path_features = 14
core_extra_daily = 14
aux_extra_daily = 5
extra_daily_features = core_extra_daily + aux_extra_daily
aux_daily_scale = 0.65
path_block_scale = 12.0 ** 0.5
arbitrage_modulators = ("ivol_capm_252d", "ami_126d")
path_modulators = (
    "beta_60m", "turnover_126d", "dolvol_126d", "zero_trades_21d", "ami_126d",
    "bidaskhl_21d", "prc", "ivol_capm_21d", "rvol_21d", "rmax5_21d",
)

mlp_seeds = 4
mlp_hidden = (256, 128, 64)
mlp_epochs = 8
mlp_lr = 1e-3
mlp_weight_decay = 1e-3
mlp_dropout = 0.2
mlp_refit_years = 5
mlp_valid_years = 3
mlp_feature_frac = 0.6
mlp_swa_temp = 150.0
mlp_stage2_offset = 100

knn_peers = 20
band_window = 180
band_shrink = 0.5
min_band_history = 60

sample_window_days = 756
min_daily_obs = 250
days_per_month = 21
spin_beta = 0.5
spin_j = 0.3
vol_pool = 0.5
min_names = 50

target_vol_annual = 0.10
gross_cap = 12.0
soft_cap_ceiling = 11.95
soft_cap_power = 3.50
realised_lookback_days = 252
realised_halflife_days = 21
vol_power = 1.20
vref_window_months = 48

start_date = "1960-01-01"
target_col = "ret_exc_lead1m"
feature_list_col = "features"
seed = 42


def group_mean(v: np.ndarray, lab: np.ndarray) -> np.ndarray:
    _, inv = np.unique(lab, return_inverse=True)
    return (np.bincount(inv, v) / np.bincount(inv))[inv]


def peer_label(y: np.ndarray, sic3: np.ndarray) -> np.ndarray:
    y = y - y.mean()
    y = y - group_mean(y, sic3)
    sd = y.std()
    return np.arctan(y / sd) if sd > 1e-12 else y * 0.0


def zs(v: np.ndarray) -> np.ndarray:
    v = v - v.mean()
    sd = v.std()
    return v / sd if sd > 1e-12 else v


def knn_f64(rk: np.ndarray, k: int = 20) -> np.ndarray:
    n = len(rk)
    if n <= 1:
        return np.zeros((n, 1), dtype=np.int64)
    k_eff = min(k, n - 1)
    rk64 = np.asarray(rk, dtype=np.float64)
    rn = rk64 / (np.linalg.norm(rk64, axis=1, keepdims=True) + 1e-12)
    sim = rn @ rn.T
    np.fill_diagonal(sim, -np.inf)
    return np.argpartition(-sim, k_eff, axis=1)[:, :k_eff]


def knn_f32(rk: np.ndarray, k: int = 20) -> np.ndarray:
    n = len(rk)
    if n <= 1:
        return np.zeros((n, 1), dtype=np.int64)
    k_eff = min(k, n - 1)
    rk32 = np.asarray(rk, dtype=np.float32)
    rn = rk32 / (np.linalg.norm(rk32, axis=1, keepdims=True) + np.float32(1e-12))
    sim = rn @ rn.T
    np.fill_diagonal(sim, -np.inf)
    return np.argpartition(-sim, k_eff, axis=1)[:, :k_eff]


def inv_sqrt_psd(mat: np.ndarray) -> np.ndarray:
    evals, evecs = np.linalg.eigh(mat)
    return (evecs * (1.0 / np.sqrt(np.maximum(evals, 1e-12)))) @ evecs.T


def ledoit_wolf_f32(x32: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    t, n = x32.shape
    s64 = ((x32.T @ x32) / np.float32(t)).astype(np.float64)
    mu = np.trace(s64) / n
    ss = np.sum(s64 ** 2)
    d2 = ss - 2.0 * mu * np.trace(s64) + n * mu * mu
    rs = np.sum(x32.astype(np.float64) ** 2, axis=1)
    b2 = np.sum(rs ** 2) / (t ** 2) - ss / t
    delta = min(max(b2, 0.0), d2) / d2 if d2 > 0 else 1.0
    s64 *= (1.0 - delta)
    s64[np.diag_indices(n)] += delta * mu
    sd = np.sqrt(np.maximum(np.diag(s64), 1e-18))
    return (s64 / np.outer(sd, sd)).astype(np.float32), sd


def spin_solve(h: np.ndarray, corr32: np.ndarray) -> np.ndarray:
    h = (h - h.mean()) / (h.std() + 1e-12)
    eps = 1e-6

    def free_energy(m: np.ndarray) -> tuple[float, np.ndarray]:
        cm = (corr32 @ m.astype(np.float32)).astype(np.float64)
        mc = np.clip(m, -1.0 + eps, 1.0 - eps)
        ent = (
            (1.0 + mc) / 2.0 * np.log((1.0 + mc) / 2.0)
            + (1.0 - mc) / 2.0 * np.log((1.0 - mc) / 2.0)
        ).sum() / spin_beta
        val = float(0.5 * spin_j * (m @ cm) - (h @ m) + ent)
        grad = spin_j * cm - h + np.arctanh(mc) / spin_beta
        return val, grad

    res = minimize(
        free_energy,
        0.5 * np.tanh(spin_beta * h),
        jac=True,
        method="L-BFGS-B",
        bounds=[(-1.0 + eps, 1.0 - eps)] * len(h),
        options={"maxiter": 200},
    )
    return res.x


def soft_cap_lp(lev: float, cap: float = soft_cap_ceiling, p: float = soft_cap_power) -> float:
    """Strictly increasing smooth algebraic Lp soft-cap bounded in (0, cap)."""
    if lev <= 0.0 or not math.isfinite(lev):
        return 0.0
    x = lev / cap
    if x > 50.0:
        return float(cap * (1.0 - (50.0 / x) * 1e-4))
    return float(lev / ((1.0 + (x ** p)) ** (1.0 / p)))


def rank_panel_inplace(x: np.ndarray, starts: np.ndarray, ends: np.ndarray, n_cols: int) -> None:
    for a, b in zip(starts, ends):
        rk = pd.DataFrame(x[a:b, :n_cols]).rank(pct=True).to_numpy(dtype=np.float32) - np.float32(0.5)
        rk[np.isnan(rk)] = 0.0
        x[a:b, :n_cols] = rk


def _series_to_dates(s: pd.Series) -> np.ndarray:
    if "pyarrow" in str(s.dtype):
        return s.astype("timestamp[s][pyarrow]").to_numpy().astype("datetime64[D]")
    return pd.to_datetime(s).to_numpy(dtype="datetime64[D]")


def _series_to_float(s: pd.Series, dt: type) -> np.ndarray:
    if "pyarrow" in str(s.dtype):
        return s.to_numpy(dtype=dt, na_value=np.nan)
    return s.to_numpy(dtype=dt)


def build_daily_matrix(
    daily_ret: pd.DataFrame, panel_ids: np.ndarray
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    t0 = time.perf_counter()
    n_dr = len(daily_ret)
    pid = np.asarray(panel_ids, dtype=np.int64)
    chunk = 5_000_000
    col_id = daily_ret["id"]
    col_dt = daily_ret["date"]
    col_ret = daily_ret["ret_exc"]

    date_parts: list[np.ndarray] = []
    id_parts: list[np.ndarray] = []
    for i in range(0, n_dr, chunk):
        j = min(i + chunk, n_dr)
        dt_c = _series_to_dates(col_dt.iloc[i:j])
        date_parts.append(np.unique(dt_c))
        rid_c = _series_to_float(col_id.iloc[i:j], np.float64)
        rret_c = _series_to_float(col_ret.iloc[i:j], np.float32)
        fin_c = np.isfinite(rid_c) & np.isfinite(rret_c)
        if not fin_c.any():
            continue
        rid_i = rid_c[fin_c].astype(np.int64)
        keep_i = np.isin(rid_i, pid, kind="table")
        if keep_i.any():
            id_parts.append(np.unique(rid_i[keep_i]))

    all_dates = (
        np.unique(np.concatenate(date_parts))
        if date_parts
        else np.empty(0, dtype="datetime64[D]")
    )
    ids = np.unique(np.concatenate(id_parts)) if id_parts else np.empty(0, dtype=np.int64)
    del date_parts, id_parts

    m = np.full((len(all_dates), len(ids)), np.nan, dtype=np.float32)
    if len(ids) > 0:
        for i in range(0, n_dr, chunk):
            j = min(i + chunk, n_dr)
            rid_c = _series_to_float(col_id.iloc[i:j], np.float64)
            rret_c = _series_to_float(col_ret.iloc[i:j], np.float32)
            fin_c = np.isfinite(rid_c) & np.isfinite(rret_c)
            if not fin_c.any():
                continue
            rid_i = rid_c[fin_c].astype(np.int64)
            keep_f = np.isin(rid_i, pid, kind="table")
            if not keep_f.any():
                continue
            dt_all = _series_to_dates(col_dt.iloc[i:j])
            dt_k = dt_all[fin_c][keep_f]
            d_ix = np.searchsorted(all_dates, dt_k)
            i_ix = np.searchsorted(ids, rid_i[keep_f])
            m[d_ix, i_ix] = rret_c[fin_c][keep_f]

        first_obs_day = np.full(len(ids), len(all_dates), dtype=np.int64)
        for c0 in range(0, len(ids), 1024):
            c1 = min(c0 + 1024, len(ids))
            fin_blk = np.isfinite(m[:, c0:c1])
            first_obs_day[c0:c1] = np.where(
                fin_blk.any(axis=0), np.argmax(fin_blk, axis=0), len(all_dates)
            )
    else:
        first_obs_day = np.full(1, len(all_dates) + 1, dtype=np.int64)

    print(
        f"daily matrix {m.shape[0]:,} days x {m.shape[1]:,} ids in {time.perf_counter() - t0:.1f}s",
        flush=True,
    )
    return all_dates, ids, m, first_obs_day


def build_daily_features_into(
    r: np.ndarray,
    n_char: int,
    starts: np.ndarray,
    ends: np.ndarray,
    months: np.ndarray,
    dates: np.ndarray,
    dmat: np.ndarray,
    col_all: np.ndarray,
    has_d: np.ndarray,
    first_obs_day: np.ndarray,
    sic: np.ndarray,
    size_ix: int | None,
) -> None:
    n_rows = r.shape[0]
    r_char = r[:, :n_char]
    out = r[:, n_char:]
    me_rank = (
        r_char[:, size_ix]
        if size_ix is not None
        else np.zeros(n_rows, dtype=np.float32)
    )
    aux_sc = np.float32(aux_daily_scale)

    for t in range(len(months)):
        a, b = starts[t], ends[t]
        rows = np.arange(a, b)
        end = int(np.searchsorted(dates, np.datetime64(months[t], "D"), side="right"))
        if end < 25 or len(rows) == 0:
            continue
        hd = has_d[rows] & (first_obs_day[col_all[rows]] < end)
        if hd.sum() < 25:
            continue
        rw = rows[hd]

        # Block 1: 14 21-day path, volatility, extreme, industry, and characteristic-peer features
        w = dmat[end - 21:end][:, col_all[rw]].astype(np.float64)
        ok = np.isfinite(w).sum(0) >= 10
        w = np.nan_to_num(w)
        r5_p, r21_p = w[-5:].sum(0), w.sum(0)
        s3 = sic[rw] // 10
        nn64 = knn_f64(r_char[rw], knn_peers)
        f14 = np.column_stack([
            r5_p, w[-10:].sum(0), r21_p, w[-1], w[-5:].max(0), w[-5:].min(0), w.std(0),
            w[-5:-1].sum(0), w[-10:-5].sum(0), w[:-10].sum(0),
            group_mean(r5_p, s3), group_mean(r21_p, s3), r5_p[nn64].mean(1), r21_p[nn64].mean(1),
        ])
        f14[~ok] = np.nan
        rk14 = pd.DataFrame(f14).rank(pct=True).to_numpy(dtype=np.float32) - np.float32(0.5)
        rk14[np.isnan(rk14)] = 0.0
        out[rw, :path_features] = rk14

        # Block 2: 14 core 252-day MP factor-residual & peer features + 5 variance-calibrated spillover features
        if end < 65 or len(rw) < 30:
            continue
        lo = max(0, end - 252)
        w252 = dmat[lo:end][:, col_all[rw]].astype(np.float64)
        obs = np.isfinite(w252)
        n_obs = obs.sum(0)
        ok21 = np.isfinite(w252[-21:]).sum(0) >= 10
        ok63 = np.isfinite(w252[-63:]).sum(0) >= 30
        ok126 = np.isfinite(w252[-126:]).sum(0) >= 50
        ok_mp = n_obs >= 60

        x_z = np.where(obs, w252, 0.0)
        mean_s = x_z.sum(0) / np.maximum(n_obs, 1)
        x_z = np.where(obs, x_z - mean_s, 0.0)
        ss252 = np.square(x_z).sum(0)
        sig252 = np.sqrt(ss252 / np.maximum(n_obs - 1, 1))
        sig63 = np.sqrt(np.square(x_z[-63:]).sum(0) / np.maximum(obs[-63:].sum(0) - 1, 1))
        sig21 = np.sqrt(np.square(x_z[-21:]).sum(0) / np.maximum(obs[-21:].sum(0) - 1, 1))

        valid_mp = ok_mp & (ss252 > 1e-16)
        mp_r1 = np.full(len(rw), np.nan)
        mp_r5 = np.full(len(rw), np.nan)
        mp_r21 = np.full(len(rw), np.nan)
        mp_r63_21 = np.full(len(rw), np.nan)
        mp_psi = np.full(len(rw), np.nan)
        mp_ac1 = np.full(len(rw), np.nan)
        mp_max3 = np.full(len(rw), np.nan)

        if valid_mp.sum() >= 30:
            xm = x_z[:, valid_mp] / np.sqrt(ss252[valid_mp])
            t_w, n_m = xm.shape
            evals, u_mat = np.linalg.eigh(xm @ xm.T)
            desc = np.arange(len(evals) - 1, -1, -1)
            evals, u_mat = evals[desc], u_mat[:, desc]
            k_ret = max(1, int(np.sum(evals > (1.0 + np.sqrt(n_m / t_w)) ** 2)))
            u_k = u_mat[:, :k_ret]
            b_load = xm.T @ u_k
            psi = np.maximum(1.0 - np.square(b_load).sum(1), 0.05)
            s_psi = np.sqrt(psi)
            e21 = (xm[-21:] - u_k[-21:] @ b_load.T) / s_psi
            mp_r1[valid_mp] = e21[-1]
            mp_r5[valid_mp] = e21[-5:].sum(0)
            mp_r21[valid_mp] = e21.sum(0)
            mp_r63_21[valid_mp] = (xm[-63:-21].sum(0) - u_k[-63:-21].sum(0) @ b_load.T) / s_psi
            mp_psi[valid_mp] = psi
            ac_num = (e21[1:] * e21[:-1]).sum(0)
            ac_den = np.square(e21).sum(0) + 1e-12
            mp_ac1[valid_mp] = ac_num / ac_den
            part3 = np.partition(e21, e21.shape[0] - 3, axis=0)[-3:]
            mp_max3[valid_mp] = part3.mean(0)

        v21_252 = np.where(ok21 & valid_mp, sig21 / np.maximum(sig252, 1e-8), np.nan)
        v63_252 = np.where(ok63 & valid_mp, sig63 / np.maximum(sig252, 1e-8), np.nan)
        w21 = np.nan_to_num(w252[-21:])
        w21_c = w21 - w21.mean(0)
        sd21 = np.sqrt(np.mean(w21_c ** 2, axis=0))
        skew21 = np.where(
            ok21 & (sd21 > 1e-12),
            np.mean(w21_c ** 3, axis=0) / (sd21 ** 3 + 1e-12),
            np.nan,
        )

        nn32 = knn_f32(r_char[rw], knn_peers)
        r5 = w21[-5:].sum(0)
        r21 = w21.sum(0)
        r63_21_raw = np.nan_to_num(w252[-63:-21]).sum(0)
        diff_peer_r5 = np.where(ok21, r5 - r5[nn32].mean(1), np.nan)
        diff_peer_r21 = np.where(ok21, r21 - r21[nn32].mean(1), np.nan)
        peer_r63_21 = np.where(ok63, r63_21_raw[nn32].mean(1), np.nan)

        me_w = np.maximum(me_rank[rw] + 0.55, 0.05)
        peer_me_w = me_w[nn32]
        peer_me_w = peer_me_w / peer_me_w.sum(1, keepdims=True)
        lead_me_peer_r21 = np.where(ok21, (peer_me_w * r21[nn32]).sum(1), np.nan)

        # 5 auxiliary multi-horizon & idiosyncratic-peer spillover features (scaled by aux_daily_scale)
        mom_126_21 = np.where(ok126, np.nan_to_num(w252[-126:-21]).sum(0), np.nan)
        down_semi21 = np.where(
            ok21 & (sd21 > 1e-12),
            np.mean(np.minimum(w21_c, 0.0) ** 2, axis=0) / (sd21 ** 2 + 1e-12),
            np.nan,
        )
        id_sic3_r21 = np.where(ok21, r21 - group_mean(r21, s3), np.nan)
        mp_r5_z = np.nan_to_num(mp_r5, nan=0.0)
        mp_r21_z = np.nan_to_num(mp_r21, nan=0.0)
        peer_mp_r5 = np.where(valid_mp, mp_r5_z[nn32].mean(1), np.nan)
        peer_mp_r21 = np.where(valid_mp, mp_r21_z[nn32].mean(1), np.nan)

        f_ex19 = np.column_stack([
            mp_r1, mp_r5, mp_r21, mp_r63_21, mp_psi,
            mp_ac1, mp_max3, v21_252, v63_252, skew21,
            diff_peer_r5, diff_peer_r21, peer_r63_21, lead_me_peer_r21,
            mom_126_21, down_semi21, id_sic3_r21, peer_mp_r5, peer_mp_r21,
        ])
        rk_ex19 = pd.DataFrame(f_ex19).rank(pct=True).to_numpy(dtype=np.float32) - np.float32(0.5)
        rk_ex19[np.isnan(rk_ex19)] = 0.0
        rk_ex19[:, core_extra_daily:] *= aux_sc
        out[rw, path_features:] = rk_ex19


def seed_columns(n_in: int, n_char: int, s: int) -> np.ndarray:
    rng = np.random.default_rng(seed + 1000 + s)
    keep = np.sort(rng.choice(n_char, int(round(mlp_feature_frac * n_char)), replace=False))
    return np.concatenate([keep, np.arange(n_char, n_in)])


class NumpyMLP:
    """Deterministic pure-NumPy float32 MLP with ReLU, untiled per-row dropout, and decoupled AdamW."""

    def __init__(self, d_in: int, s: int, hidden: tuple[int, ...] = mlp_hidden):
        rng = np.random.default_rng(seed + 2000 + s)
        dims = (d_in,) + hidden + (1,)
        self.W: list[np.ndarray] = []
        self.b: list[np.ndarray] = []
        for f_in, f_out in zip(dims[:-1], dims[1:]):
            bound = np.float32(1.0 / np.sqrt(f_in))
            self.W.append(rng.uniform(-bound, bound, (f_in, f_out)).astype(np.float32))
            self.b.append(rng.uniform(-bound, bound, (f_out,)).astype(np.float32))
        self.mW = [np.zeros_like(w) for w in self.W]
        self.vW = [np.zeros_like(w) for w in self.W]
        self.mb = [np.zeros_like(b) for b in self.b]
        self.vb = [np.zeros_like(b) for b in self.b]
        self.step_num = 0

    def train_epoch_rows(
        self,
        r: np.ndarray,
        rows: np.ndarray,
        cols: np.ndarray,
        y: np.ndarray,
        rng: np.random.Generator,
        batch_size: int = 4096,
    ) -> None:
        n = len(rows)
        if n == 0:
            return
        perm = rng.permutation(n)
        keep_p = np.float32(1.0 - mlp_dropout)
        inv_keep = np.float32(1.0 / keep_p)
        ih_thr = np.float32(
            math.sqrt(2.0 * float(keep_p))
            if float(keep_p) <= 0.5
            else 2.0 - math.sqrt(2.0 * (1.0 - float(keep_p)))
        )
        lr = np.float32(mlp_lr)
        wd_mult = np.float32(1.0 - mlp_lr * mlp_weight_decay)
        b1, b2 = np.float32(0.9), np.float32(0.999)
        omb1, omb2 = np.float32(0.1), np.float32(0.001)
        eps = np.float32(1e-8)
        n_hid = len(self.W) - 1

        eff_bs = 8192 if n >= 32768 else batch_size
        xs_pre = np.ascontiguousarray(r[rows][:, cols]) if n <= 65536 else None
        step_ep = 0
        for i in range(0, n, eff_bs):
            if step_ep >= 64:
                break
            step_ep += 1
            idx = perm[i:i + eff_bs]
            xb = xs_pre[idx] if xs_pre is not None else np.ascontiguousarray(r[rows[idx]][:, cols])
            yb = y[idx]
            bs_i = len(xb)
            bs = np.float32(bs_i)

            hs = [xb]
            masks = []
            h_cur = xb
            for l_ix in range(n_hid):
                d_out = self.W[l_ix].shape[1]
                z = h_cur @ self.W[l_ix]
                z += self.b[l_ix]
                u_r = rng.random((bs_i, 1), dtype=np.float32)
                u_c = rng.random((1, d_out), dtype=np.float32)
                mask = (((u_r + u_c) < ih_thr) & (z > np.float32(0.0))).astype(np.float32) * inv_keep
                h_cur = z * mask
                masks.append(mask)
                hs.append(h_cur)

            pred = (h_cur @ self.W[n_hid]).ravel()
            pred += self.b[n_hid][0]
            pred -= yb
            pred *= np.float32(2.0) / bs
            dz = pred[:, None]

            grads = [None] * (n_hid + 1)
            grads[n_hid] = (hs[n_hid].T @ dz, dz.sum(axis=0))
            for l_ix in range(n_hid - 1, -1, -1):
                dz = dz @ self.W[l_ix + 1].T
                dz *= masks[l_ix]
                grads[l_ix] = (hs[l_ix].T @ dz, dz.sum(axis=0))

            self.step_num += 1
            step_lr = lr / np.float32(1.0 - 0.9 ** self.step_num)
            inv_bc2 = np.float32(1.0 / np.sqrt(1.0 - 0.999 ** self.step_num))

            for l_ix, (g_w, g_b) in enumerate(grads):
                self.W[l_ix] *= wd_mult
                self.mW[l_ix] *= b1
                self.mW[l_ix] += omb1 * g_w
                self.vW[l_ix] *= b2
                self.vW[l_ix] += omb2 * (g_w * g_w)
                self.W[l_ix] -= step_lr * self.mW[l_ix] / (np.sqrt(self.vW[l_ix]) * inv_bc2 + eps)

                self.b[l_ix] *= wd_mult
                self.mb[l_ix] *= b1
                self.mb[l_ix] += omb1 * g_b
                self.vb[l_ix] *= b2
                self.vb[l_ix] += omb2 * (g_b * g_b)
                self.b[l_ix] -= step_lr * self.mb[l_ix] / (np.sqrt(self.vb[l_ix]) * inv_bc2 + eps)

    def predict_with_weights(
        self, x: np.ndarray, w_list: list[np.ndarray], b_list: list[np.ndarray]
    ) -> np.ndarray:
        out = np.empty(len(x), dtype=np.float32)
        n_hid = len(w_list) - 1
        for i in range(0, len(x), 65536):
            h = x[i:i + 65536]
            for l_ix in range(n_hid):
                h = np.maximum(h @ w_list[l_ix] + b_list[l_ix], np.float32(0.0))
            out[i:i + 65536] = (h @ w_list[n_hid]).ravel() + b_list[n_hid][0]
        return out

    def predict(self, x: np.ndarray) -> np.ndarray:
        return self.predict_with_weights(x, self.W, self.b)


def fit_mlp_ensemble(
    r: np.ndarray,
    tr_rows: np.ndarray,
    y: np.ndarray,
    valid_mask: np.ndarray,
    n_char: int,
    n_in_mlp: int,
    stage_offset: int = 0,
):
    if (valid_mask.sum() < 10) or ((~valid_mask).sum() < 32):
        valid_mask = np.arange(len(tr_rows)) >= int(0.75 * len(tr_rows))
    rows_tr, yt = tr_rows[~valid_mask], y[~valid_mask]
    rows_va, yv = tr_rows[valid_mask], y[valid_mask]
    cols = [seed_columns(n_in_mlp, n_char, s + stage_offset) for s in range(mlp_seeds)]

    def _fit_one_seed(s: int):
        c = cols[s]
        xs_va = np.ascontiguousarray(r[rows_va][:, c])
        net_val = NumpyMLP(len(c), s + stage_offset, hidden=mlp_hidden)
        rng_val = np.random.default_rng(seed + 5000 + s + stage_offset)
        ics = []
        for _ in range(mlp_epochs):
            net_val.train_epoch_rows(r, rows_tr, c, yt, rng_val)
            pv = net_val.predict(xs_va)
            ic = float(np.corrcoef(pv, yv)[0, 1]) if len(yv) > 2 and pv.std() > 0 else -1.0
            if not math.isfinite(ic):
                ic = -1.0
            ics.append(ic)
        del xs_va

        ics_arr = np.array(ics, dtype=np.float64)
        z_ic = mlp_swa_temp * (ics_arr - np.max(ics_arr))
        w_ep = np.exp(np.clip(z_ic, -50.0, 0.0))
        w_ep /= w_ep.sum()
        max_ep = int(np.max(np.flatnonzero(w_ep > 1e-3))) + 1
        w_ep_trunc = w_ep[:max_ep] / w_ep[:max_ep].sum()

        net = NumpyMLP(len(c), s + stage_offset, hidden=mlp_hidden)
        rng = np.random.default_rng(seed + 5000 + s + stage_offset)
        w_swa = [np.zeros_like(w) for w in net.W]
        b_swa = [np.zeros_like(b) for b in net.b]
        for ep in range(max_ep):
            net.train_epoch_rows(r, tr_rows, c, y, rng)
            wt = np.float32(w_ep_trunc[ep])
            for l_ix in range(len(w_swa)):
                w_swa[l_ix] += wt * net.W[l_ix]
                b_swa[l_ix] += wt * net.b[l_ix]
        return (net, w_swa, b_swa, c)

    with ThreadPoolExecutor(max_workers=mlp_seeds) as ex:
        models = list(ex.map(_fit_one_seed, range(mlp_seeds)))
    return models


def predict_mlp_ensemble(models, x: np.ndarray) -> np.ndarray:
    return np.mean(
        [
            net.predict_with_weights(np.ascontiguousarray(x[:, c]), w_l, b_l).astype(np.float64)
            for net, w_l, b_l, c in models
        ],
        axis=0,
    )


def predict_mlp_rows(models, r: np.ndarray, rows: np.ndarray) -> np.ndarray:
    out = np.empty(len(rows), dtype=np.float32)
    for i in range(0, len(rows), 65536):
        j = min(i + 65536, len(rows))
        out[i:j] = predict_mlp_ensemble(models, r[rows[i:j]]).astype(np.float32)
    return out


def build_weights(
    chars: pd.DataFrame, features: pd.DataFrame, daily_ret: pd.DataFrame
) -> pd.DataFrame:
    np.random.seed(seed)
    t0 = time.perf_counter()
    is_val_mode = os.environ.get("CTF_EXECUTION_MODE", "full").strip().lower() == "validation"

    feats = [str(f) for f in features[feature_list_col].tolist() if str(f) in chars.columns]
    n_char = len(feats)
    n_dl = path_features + extra_daily_features
    n_in_mlp = n_char + path_features + core_extra_daily
    p_tot = n_char + n_dl
    size_ix = feats.index("market_equity") if "market_equity" in feats else None
    mod_ix = [feats.index(m) for m in path_modulators if m in feats]
    arb_ix = [feats.index(m) for m in arbitrage_modulators if m in feats]

    eom_dt = pd.to_datetime(chars["eom"]).to_numpy(dtype="datetime64[ns]")
    ids_raw = pd.to_numeric(chars["id"]).to_numpy(dtype=np.int64)
    order = np.lexsort((ids_raw, eom_dt.astype(np.int64)))

    ids = ids_raw[order]
    eom_out = eom_dt[order]
    y_all = chars[target_col].to_numpy(dtype=np.float64)[order]
    sic = np.nan_to_num(
        pd.to_numeric(chars["sic"], errors="coerce").to_numpy(dtype=np.float64)[order],
        nan=9999.0,
    )
    me = (
        chars["market_equity"].to_numpy(dtype=np.float64)[order]
        if "market_equity" in chars.columns
        else np.ones(len(order), dtype=np.float64)
    )
    bet = (
        chars["beta_60m"].to_numpy(dtype=np.float64)[order]
        if "beta_60m" in chars.columns
        else np.zeros(len(order), dtype=np.float64)
    )
    del ids_raw, eom_dt

    r = np.zeros((len(order), p_tot), dtype=np.float32)
    for j, f in enumerate(feats):
        r[:, j] = chars[f].to_numpy(dtype=np.float32)[order]
    del order
    gc.collect()

    months, mix = np.unique(eom_out.astype("datetime64[D]"), return_inverse=True)
    mdates = [m.astype("datetime64[D]").item() for m in months]
    starts = np.searchsorted(mix, np.arange(len(months)))
    ends = np.searchsorted(mix, np.arange(len(months)), side="right")
    rank_panel_inplace(r, starts, ends, n_char)
    print(
        f"{len(ids):,} rows, {n_char} characteristics, {len(months)} months, ranked in "
        f"{time.perf_counter() - t0:.1f}s",
        flush=True,
    )

    dates, daily_ids, dmat, first_obs_day = build_daily_matrix(daily_ret, np.unique(ids))
    gc.collect()
    if len(daily_ids) > 0:
        col_all = np.clip(np.searchsorted(daily_ids, ids), 0, len(daily_ids) - 1)
        has_d = daily_ids[col_all] == ids
    else:
        col_all = np.zeros(len(ids), dtype=np.int64)
        has_d = np.zeros(len(ids), dtype=bool)

    t1 = time.perf_counter()
    build_daily_features_into(
        r, n_char, starts, ends, months, dates, dmat, col_all, has_d, first_obs_day, sic, size_ix
    )
    gc.collect()
    print(
        f"{n_dl} daily features added ({p_tot} total inputs) in {time.perf_counter() - t1:.1f}s",
        flush=True,
    )

    month_rows: list[np.ndarray] = []
    month_lab: list[np.ndarray] = []
    month_y_var: list[float] = []
    month_idx_parts: list[np.ndarray] = []
    for tt in range(len(months)):
        rw = np.arange(starts[tt], ends[tt])
        rw = rw[np.isfinite(y_all[rw])]
        month_rows.append(rw)
        if len(rw) > 2:
            y_d = y_all[rw] - y_all[rw].mean()
            y_d = y_d - group_mean(y_d, sic[rw] // 10)
            month_y_var.append(float(np.median(np.abs(y_d))) ** 2)
            month_lab.append(np.arctan(y_d / y_d.std()) if y_d.std() > 1e-12 else y_d * 0.0)
        else:
            month_y_var.append(0.0)
            month_lab.append(np.zeros(len(rw)))
        month_idx_parts.append(np.full(len(rw), tt, dtype=np.int32))

    all_rw = (
        np.concatenate(month_rows)
        if any(len(rw) > 0 for rw in month_rows)
        else np.empty(0, dtype=np.int64)
    )
    y_ord = (
        np.concatenate(month_lab).astype(np.float32)
        if len(all_rw) > 0
        else np.empty(0, dtype=np.float32)
    )
    row_month = (
        np.concatenate(month_idx_parts)
        if len(all_rw) > 0
        else np.empty(0, dtype=np.int32)
    )
    del month_idx_parts
    m_ends = np.cumsum([len(rw) for rw in month_rows], dtype=np.int64)
    m_starts = np.concatenate([[0], m_ends[:-1]]) if len(m_ends) > 0 else np.zeros(0, dtype=np.int64)
    m_len = m_ends - m_starts

    def expand_fixed(rk32: np.ndarray) -> np.ndarray:
        sc = np.float32(path_block_scale)
        cols = [rk32]
        if size_ix is not None:
            cols.append(rk32 * rk32[:, [size_ix]])
        for j in arb_ix:
            cols.append(rk32[:, :n_char] * rk32[:, [j]])
        dl = rk32[:, n_char:]
        for j in mod_ix:
            cols.append(sc * dl * rk32[:, [j]])
        return np.column_stack(cols)

    def predict_ridge(rk32: np.ndarray, cf: np.ndarray, ms: dict[str, np.ndarray]) -> np.ndarray:
        c, d = rk32[:, :n_char], rk32[:, n_char:]
        cf32 = cf.astype(np.float32)
        pos = 0
        out = rk32 @ cf32[pos:pos + p_tot]
        pos += p_tot
        if size_ix is not None:
            out = out + rk32[:, size_ix] * (rk32 @ cf32[pos:pos + p_tot])
            pos += p_tot
        for j in arb_ix:
            out = out + rk32[:, j] * (c @ cf32[pos:pos + n_char])
            pos += n_char
        sc = np.float32(path_block_scale)
        for j in mod_ix:
            out = out + (sc * rk32[:, j]) * (d @ cf32[pos:pos + n_dl])
            pos += n_dl
        z = (c @ ms["b"].astype(np.float32) - ms["mu"].astype(np.float32)) / ms["sd"].astype(np.float32)
        for k in range(modulator_rank):
            out = out + z[:, k] * (d @ cf32[pos:pos + n_dl])
            pos += n_dl
        return out.astype(np.float64)

    q_fix = expand_fixed(np.zeros((1, p_tot), dtype=np.float32)).shape[1]
    q_dyn = n_dl * modulator_rank
    q = q_fix + q_dyn

    g_ff = np.zeros((q_fix, q_fix), dtype=np.float64)
    g_fy = np.zeros(q_fix, dtype=np.float64)
    m_acc = np.zeros((n_dl, n_char), dtype=np.float64)

    g_ff_ewm = np.zeros((q_fix, q_fix), dtype=np.float64)
    g_fy_ewm = np.zeros(q_fix, dtype=np.float64)
    m_acc_ewm = np.zeros((n_dl, n_char), dtype=np.float64)
    n_ewm = 0.0
    prev_row = 0
    prev_refit_t = 0

    coef: np.ndarray | None = None
    mod_state: dict[str, np.ndarray] | None = None
    beta_tail: np.ndarray | None = None
    models1 = None
    models2 = None
    band_hist: list[list[float]] = []
    vol_hist: list[float] = []
    real_sq_hist: list[float] = []
    exante_sq_hist: list[float] = []
    frames: list[pd.DataFrame] = []

    t_start = int(np.searchsorted(months, np.datetime64(start_date, "D")))
    if is_val_mode:
        t_start = max(t_start, min(60, max(12, len(months) // 2)))
    eff_min_band_hist = (
        min(min_band_history, max(12, (len(months) - t_start) // 2))
        if is_val_mode
        else min_band_history
    )

    n_emit = 0
    for t in range(t_start, len(months)):
        a, b = starts[t], ends[t]
        rows = np.arange(a, b)

        # 1. Annual Multi-Scale Hierarchical Modulated Ridge Refit (strictly on tt < t)
        if coef is None or mdates[t].month == 1:
            n_cut = int(m_starts[t])
            if prev_row > 0 and t > prev_refit_t:
                step_decay = 0.5 ** (float(t - prev_refit_t) / ridge_ewma_halflife)
                g_ff_ewm *= step_decay
                g_fy_ewm *= step_decay
                m_acc_ewm *= step_decay
                n_ewm *= step_decay

            for i in range(prev_row, n_cut, 24576):
                j_end = min(i + 24576, n_cut)
                rk_c = r[all_rw[i:j_end]]
                y_c = y_ord[i:j_end]
                xf32 = expand_fixed(rk_c)
                g_ff += (xf32.T @ xf32).astype(np.float64)
                g_fy += (xf32.T @ y_c).astype(np.float64)
                m_acc += (rk_c[:, n_char:].T @ (rk_c[:, :n_char] * y_c[:, None])).astype(np.float64)

                w_ew = (0.5 ** ((float(t - 1) - row_month[i:j_end].astype(np.float64)) / ridge_ewma_halflife)).astype(np.float32)
                sw_ew = np.sqrt(w_ew)[:, None]
                xf_w = xf32 * sw_ew
                g_ff_ewm += (xf_w.T @ xf_w).astype(np.float64)
                g_fy_ewm += (xf32.T @ (y_c * w_ew)).astype(np.float64)
                m_acc_ewm += (rk_c[:, n_char:].T @ (rk_c[:, :n_char] * (y_c * w_ew)[:, None])).astype(np.float64)
                n_ewm += float(w_ew.sum())

            prev_row = n_cut
            prev_refit_t = t

            w_exp = 1.0 - ridge_ewma_alpha
            w_ewm_scale = ridge_ewma_alpha * (float(max(n_cut, 1)) / max(n_ewm, 1.0))
            g_ff_ms = w_exp * g_ff + w_ewm_scale * g_ff_ewm
            g_fy_ms = w_exp * g_fy + w_ewm_scale * g_fy_ewm
            m_acc_ms = w_exp * m_acc + w_ewm_scale * m_acc_ewm

            isq_d = inv_sqrt_psd(g_ff_ms[n_char:p_tot, n_char:p_tot] + ridge_inter * np.eye(n_dl))
            isq_c = inv_sqrt_psd(g_ff_ms[:n_char, :n_char] + ridge_inter * np.eye(n_char))
            _, _, vt = np.linalg.svd(isq_d @ m_acc_ms @ isq_c, full_matrices=False)
            bm = isq_c @ vt[:modulator_rank].T
            bm32 = bm.astype(np.float32)

            s1 = np.zeros(modulator_rank, dtype=np.float64)
            s2 = np.zeros(modulator_rank, dtype=np.float64)
            for i in range(0, n_cut, 65536):
                j_end = min(i + 65536, n_cut)
                z_c = (r[all_rw[i:j_end], :n_char] @ bm32).astype(np.float64)
                s1 += z_c.sum(0)
                s2 += (z_c ** 2).sum(0)
            denom = max(n_cut, 1)
            mu = s1 / denom
            sd = np.maximum(np.sqrt(np.maximum(s2 / denom - mu ** 2, 0.0)), 1e-12)
            mod_state = {"b": bm, "mu": mu, "sd": sd}

            g_fd = np.zeros((q_fix, q_dyn), dtype=np.float64)
            g_dd = np.zeros((q_dyn, q_dyn), dtype=np.float64)
            g_dy = np.zeros(q_dyn, dtype=np.float64)
            mu32, sd32 = mu.astype(np.float32), sd.astype(np.float32)
            sc = np.float32(path_block_scale)
            for i in range(0, n_cut, 40960):
                j_end = min(i + 40960, n_cut)
                rk_s = r[all_rw[i:j_end]]
                c_s = rk_s[:, :n_char]
                d_s = rk_s[:, n_char:]
                y_s = y_ord[i:j_end]
                w_ms = (
                    w_exp
                    + w_ewm_scale * (0.5 ** ((float(t - 1) - row_month[i:j_end].astype(np.float64)) / ridge_ewma_halflife))
                ).astype(np.float32)
                z_s = (c_s @ bm32 - mu32) / sd32
                xd_s = np.column_stack([d_s * z_s[:, [k]] for k in range(modulator_rank)])
                xd_w = xd_s * w_ms[:, None]
                blocks = [(rk_s.T @ xd_w).astype(np.float64)]
                if size_ix is not None:
                    blocks.append((rk_s.T @ (xd_w * rk_s[:, [size_ix]])).astype(np.float64))
                for j in arb_ix:
                    blocks.append((c_s.T @ (xd_w * rk_s[:, [j]])).astype(np.float64))
                for j in mod_ix:
                    blocks.append((sc * (d_s.T @ (xd_w * rk_s[:, [j]]))).astype(np.float64))
                g_fd += np.vstack(blocks)
                g_dd += (xd_s.T @ xd_w).astype(np.float64)
                g_dy += (xd_w.T @ y_s).astype(np.float64)

            pen_scale = 0.75 + 0.25 * (float(n_cut) / 250000.0)
            pen = np.full(q, ridge_inter * pen_scale, dtype=np.float64)
            pen[:p_tot] = ridge_main * pen_scale
            xtx = np.block([[g_ff_ms, g_fd], [g_fd.T, g_dd]])
            xtx[np.diag_indices(q)] += pen
            coef = sp_solve(xtx, np.concatenate([g_fy_ms, g_dy]), assume_a="pos")

        # 2. Five-Year Full-Expanding Smooth-SWA Boosted Residual MLP Cascade Refit (strictly on tt < t)
        net_due = models1 is None or (mdates[t].month == 1 and mdates[t].year % mlp_refit_years == 0)
        if net_due:
            assert coef is not None and mod_state is not None
            cut = mdates[t].replace(year=mdates[t].year - mlp_valid_years, day=min(mdates[t].day, 28))
            keep_m = [tt for tt in range(t) if m_len[tt] >= min_names]
            if len(keep_m) < 36:
                keep_m = [tt for tt in range(t) if m_len[tt] > 2]
            tr_rows = np.concatenate([month_rows[tt] for tt in keep_m])
            tr_valid = np.concatenate([np.full(m_len[tt], mdates[tt] >= cut) for tt in keep_m])
            y_raw = np.concatenate([month_lab[tt] for tt in keep_m]).astype(np.float32)
            fit_lin = np.empty(len(tr_rows), dtype=np.float32)
            for i in range(0, len(tr_rows), 65536):
                j_end = min(i + 65536, len(tr_rows))
                fit_lin[i:j_end] = predict_ridge(r[tr_rows[i:j_end]], coef, mod_state).astype(np.float32)
            tr_res1 = y_raw - fit_lin
            del y_raw, fit_lin
            g_tt = np.zeros((n_char, n_char), dtype=np.float64)
            g_ty = np.zeros(n_char, dtype=np.float64)
            m_vars = np.array([month_y_var[tt] for tt in keep_m], dtype=np.float64)
            mean_mvar = float(np.mean(m_vars)) + 1e-12
            w_mvar = (
                (m_vars / mean_mvar) ** 2
                if len(keep_m) > 200
                else np.ones(len(keep_m), dtype=np.float64)
            )
            pos_tr = 0
            for k_m, tt in enumerate(keep_m):
                ln_m = int(m_len[tt])
                if ln_m > 0:
                    vt_m = (np.abs(r[tr_rows[pos_tr:pos_tr + ln_m], :n_char]) - np.float32(0.25)).astype(np.float64)
                    yr_m = tr_res1[pos_tr:pos_tr + ln_m].astype(np.float64)
                    wm = float(w_mvar[k_m])
                    g_tt += wm * (vt_m.T @ vt_m)
                    g_ty += wm * (vt_m.T @ yr_m)
                    pos_tr += ln_m
            lam_tail = 5.0 + 0.05 * float(np.trace(g_tt) / max(n_char, 1))
            gamma_tail = (
                500.0 / (500.0 + float(n_char * n_char * n_char))
                if len(keep_m) >= 60
                else 0.0
            )
            beta_tail = (
                gamma_tail
                * sp_solve(
                    g_tt + lam_tail * np.eye(n_char, dtype=np.float64),
                    g_ty,
                    assume_a="pos",
                )
            ).astype(np.float32)
            del g_tt, g_ty, m_vars, w_mvar
            for i in range(0, len(tr_rows), 65536):
                j_end = min(i + 65536, len(tr_rows))
                tr_res1[i:j_end] -= (np.abs(r[tr_rows[i:j_end], :n_char]) - np.float32(0.25)) @ beta_tail
            models1 = fit_mlp_ensemble(r, tr_rows, tr_res1, tr_valid, n_char, n_in_mlp, stage_offset=0)
            rem2 = tr_res1 - predict_mlp_rows(models1, r, tr_rows)
            del tr_res1
            models2 = fit_mlp_ensemble(
                r, tr_rows, rem2, tr_valid, n_char, n_in_mlp, stage_offset=mlp_stage2_offset
            )
            n_tr_rows = len(tr_rows)
            del rem2, tr_rows, tr_valid
            gc.collect()
            print(
                f"refit cascade {mdates[t].year} on {n_tr_rows:,} rows, elapsed "
                f"{(time.perf_counter() - t0) / 60.0:.1f} min",
                flush=True,
            )

        # 3. Strictly Causal Universe Selection, Spectral Filtering & 3-Day Lead-Lag Spin Book
        end = int(np.searchsorted(dates, np.datetime64(months[t], "D"), side="right"))
        hd = has_d[rows] & (first_obs_day[col_all[rows]] < end)
        rr = rows[hd]
        lo = max(0, end - sample_window_days)
        xd = dmat[lo:end][:, col_all[rr]].astype(np.float64)
        obs = np.isfinite(xd)
        enough = obs.sum(0) >= min_daily_obs
        rr = rr[enough]
        ob = obs[:, enough]
        xs = xd[:, enough]

        if len(rr) >= min_names and xs.shape[0] >= 40:
            assert (
                coef is not None
                and mod_state is not None
                and beta_tail is not None
                and models1 is not None
                and models2 is not None
            )
            xd_c = np.where(ob, xs, 0.0)
            xd_c -= xd_c.mean(0)
            nob = np.maximum(ob.sum(0), 1)
            xs = np.where(ob, xs - np.where(ob, xs, 0.0).sum(0) / nob, 0.0) * np.sqrt(xs.shape[0] / nob)

            rk_rr = r[rr]
            s_lin = predict_ridge(rk_rr, coef, mod_state)
            s_tail = ((np.abs(rk_rr[:, :n_char]) - np.float32(0.25)) @ beta_tail).astype(np.float64)
            s_net = s_tail + predict_mlp_ensemble(models1, rk_rr) + predict_mlp_ensemble(models2, rk_rr)
            s = zs(s_lin + 1.05 * s_net)

            s3 = sic[rr]
            c1 = group_mean(s, s3 // 100)
            c2 = group_mean(s, s3 // 10) - c1
            c3 = group_mean(s, s3) - c1 - c2
            r3 = s - c1 - c2 - c3
            nn = knn_f32(rk_rr[:, :n_char + path_features], knn_peers)
            c4 = r3[nn].mean(1)
            c5 = r3 - c4
            bands = np.column_stack([c1, c2, c3, c4, c5])

            w_unit_emitted: np.ndarray | None = None
            vol_raw_emitted: float | None = None

            if len(band_hist) >= eff_min_band_hist:
                past_b = np.array(band_hist[-band_window:], dtype=np.float64)
                mu_b = past_b.mean(0)
                if len(band_hist) < min_band_history:
                    w_b = float(len(band_hist)) / float(min_band_history)
                    b_sd = np.maximum(past_b.std(0), 1e-4)
                    prior_mu = np.array([mu_b[0], mu_b[1], mu_b[2], 0.35 * b_sd[3], 0.50 * b_sd[4]], dtype=np.float64)
                    mu_b = w_b * mu_b + (1.0 - w_b) * prior_mu
                    mu_b[4] = max(mu_b[4], 0.25 * float(b_sd[4]))
                cv_b = np.cov(past_b.T)
                cv_b = (1.0 - band_shrink) * cv_b + band_shrink * np.diag(np.diag(cv_b))
                theta = np.linalg.solve(cv_b, mu_b)
                scale = np.maximum(np.abs(bands).mean(0), 1e-12)
                sig = bands @ (theta / scale)

                x3_32 = (
                    (xs[2:] + xs[1:-1] + xs[:-2]) / np.sqrt(3.0) * np.sqrt(days_per_month)
                ).astype(np.float32)
                corr32, own_sd = ledoit_wolf_f32(x3_32)
                m_spin = spin_solve(sig, corr32)
                pooled_sd = (1.0 - vol_pool) * own_sd + vol_pool * own_sd[nn].mean(1)
                w = m_spin / pooled_sd

                bt = np.nan_to_num(bet[rr])
                lm = np.log(np.maximum(np.nan_to_num(me[rr], nan=1.0), 1e-3))
                basis = np.column_stack([np.ones(len(rr)), bt - bt.mean(), lm - lm.mean()])
                cf, *_ = np.linalg.lstsq(basis, w, rcond=None)
                w = w - basis @ cf
                w = w - w.mean()
                g = np.abs(w).sum()
                if g > 1e-12:
                    w_unit = w / g
                    w32 = w_unit.astype(np.float32)
                    pnl = (xd_c[-realised_lookback_days:].astype(np.float32) @ w32).astype(np.float64)
                    n_pnl = len(pnl)
                    decay = 0.5 ** (np.arange(n_pnl - 1, -1, -1, dtype=np.float64) / realised_halflife_days)
                    vol_ew21 = float(np.sqrt((decay * (pnl ** 2)).sum() / decay.sum() * days_per_month))
                    pnl_21 = pnl[-21:] if n_pnl >= 21 else pnl
                    vol_spot21 = (
                        float(np.sqrt(np.mean(pnl_21 ** 2) * days_per_month))
                        if len(pnl_21) > 0
                        else vol_ew21
                    )
                    v_fast = max(vol_ew21, vol_spot21)
                    pnl_out = pnl[:-63] if n_pnl > 84 else pnl
                    vol_out63 = (
                        float(np.sqrt(np.mean(pnl_out ** 2) * days_per_month))
                        if len(pnl_out) > 0
                        else vol_ew21
                    )
                    u32 = (w_unit * own_sd).astype(np.float32)
                    vol_lw = float(np.sqrt(max(float(u32 @ (corr32 @ u32)), 1e-18)))
                    vol_lw_eff = min(vol_lw, 1.50 * max(v_fast, vol_out63))
                    sd21_diag = np.sqrt(np.mean(xd_c[-21:] ** 2, axis=0) * days_per_month + 1e-18)
                    sd63_diag = np.sqrt(np.mean(xd_c[-63:] ** 2, axis=0) * days_per_month + 1e-18)
                    vol_diag21 = float(np.sqrt(np.sum((w_unit * sd21_diag) ** 2)))
                    vol_diag63 = float(np.sqrt(np.sum((w_unit * sd63_diag) ** 2)))
                    v_diag = 3.00 * max(vol_diag21, vol_diag63)
                    vol_raw = float(
                        np.sqrt(
                            0.45 * v_fast * v_diag
                            + 0.40 * (v_diag ** 2)
                            + 0.15 * ((3.00 * vol_lw_eff) ** 2)
                        )
                    )

                    if vol_raw > 1e-12:
                        if len(real_sq_hist) >= 12:
                            n_rv = len(real_sq_hist)
                            d_rv = 0.5 ** (np.arange(n_rv - 1, -1, -1, dtype=np.float64) / 18.0)
                            r_rms = float(np.sqrt(np.sum(d_rv * np.array(real_sq_hist)) / np.sum(d_rv)))
                            x_rms = float(np.sqrt(np.sum(d_rv * np.array(exante_sq_hist)) / np.sum(d_rv)))
                            w_rv = float(n_rv) / (float(n_rv) + 18.0)
                            ratio_rv = w_rv * (r_rms / max(x_rms, 1e-12)) + (1.0 - w_rv) * 1.0
                            kappa = float(np.clip(ratio_rv ** 0.70, 0.72, 1.28))
                        else:
                            kappa = 1.0
                        vol_eff = vol_raw * kappa
                        vref = (
                            float(np.median(vol_hist[-vref_window_months:]))
                            if len(vol_hist) >= 24
                            else vol_eff
                        )
                        lev = (
                            target_vol_annual / np.sqrt(12.0) / vol_eff
                            * (vref / vol_eff) ** (vol_power - 1.0)
                        )
                        lev_cap = soft_cap_lp(lev, cap=soft_cap_ceiling, p=soft_cap_power)
                        w = w_unit * lev_cap
                        vol_hist.append(vol_eff)
                        w_unit_emitted = w_unit
                        vol_raw_emitted = vol_raw
                    else:
                        w = w_unit
                    wt = np.zeros(b - a, dtype=np.float64)
                    wt[rr - a] = w
                    frames.append(pd.DataFrame({"id": ids[a:b], "eom": eom_out[a:b], "w": wt}))
                    n_emit += 1

            # Update band return & realized unit-gross volatility history strictly AFTER month t emission
            yy = y_all[rr]
            ok = np.isfinite(yy)
            if ok.sum() >= min_names:
                br = []
                for k in range(bands.shape[1]):
                    v = bands[ok, k] - bands[ok, k].mean()
                    gk = np.abs(v).sum()
                    br.append(float((v / gk) @ yy[ok]) if gk > 0 else 0.0)
                band_hist.append(br)
                if w_unit_emitted is not None and vol_raw_emitted is not None:
                    r_unit_real = float(w_unit_emitted[ok] @ yy[ok])
                    if math.isfinite(r_unit_real):
                        real_sq_hist.append(r_unit_real ** 2)
                        exante_sq_hist.append(vol_raw_emitted ** 2)

        if mdates[t].month == 12:
            print(
                f"{mdates[t].year} emitted {n_emit} months, elapsed "
                f"{(time.perf_counter() - t0) / 60.0:.1f} min",
                flush=True,
            )

    if not frames:
        return pd.DataFrame(columns=["id", "eom", "w"])
    out = pd.concat(frames, ignore_index=True)
    out["id"] = out["id"].astype(np.int64)
    print(
        f"done: {len(out):,} rows, {n_emit} months in {(time.perf_counter() - t0) / 60.0:.1f} min",
        flush=True,
    )
    return out


def is_test_row(flag: pd.Series) -> pd.Series:
    if pd.api.types.is_bool_dtype(flag):
        return flag.fillna(False)
    if pd.api.types.is_numeric_dtype(flag):
        return flag.fillna(0).ne(0)
    return flag.astype(str).str.strip().str.lower().isin(["1", "1.0", "true", "t"])


def restrict_to_test(chars: pd.DataFrame, weights: pd.DataFrame) -> pd.DataFrame:
    if "ctff_test" not in chars.columns:
        return weights
    mask = is_test_row(chars["ctff_test"])
    if not mask.any():
        return weights
    keys = chars.loc[mask, ["id", "eom"]].copy()
    keys["id"] = np.asarray(pd.to_numeric(keys["id"]), dtype=np.int64)
    keys["eom"] = pd.to_datetime(keys["eom"])
    keys = keys.drop_duplicates()
    if len(weights) == 0:
        keys["w"] = 0.0
        return keys[["id", "eom", "w"]]
    w = weights[["id", "eom", "w"]].copy()
    w["id"] = np.asarray(pd.to_numeric(w["id"]), dtype=np.int64)
    w["eom"] = pd.to_datetime(w["eom"])
    out = keys.merge(w, on=["id", "eom"], how="left")
    out["w"] = out["w"].fillna(0.0)
    return out[["id", "eom", "w"]]


def report_sharpe(weights: pd.DataFrame, chars: pd.DataFrame) -> None:
    ret = chars[["id", "eom", target_col]].copy()
    ret["eom"] = pd.to_datetime(ret["eom"])
    ret["id"] = np.asarray(pd.to_numeric(ret["id"]), dtype=np.int64)
    ret = ret.rename(columns={target_col: "r"}).dropna(subset=["r"])
    w = weights.copy()
    w["eom"] = pd.to_datetime(w["eom"])
    j = w.merge(ret, on=["id", "eom"], how="inner")
    j["c"] = j["w"] * j["r"]
    series = j.groupby("eom")["c"].sum().sort_index()
    test = series[series.index >= "1989-12-01"]
    if len(test) == 0:
        test = series
    if len(test) < 12:
        print("sharpe not reported, too few observed months", flush=True)
        return
    sd = float(test.std(ddof=1))
    ann_ret = float(test.mean()) * 12.0
    ann_vol = sd * np.sqrt(12.0)
    sharpe = float(test.mean() / sd * np.sqrt(12.0)) if sd > 1e-12 else float("nan")
    cum = test.cumsum()
    mdd = float((cum - cum.cummax()).min())
    print(
        f"months scored: {len(test)} ({test.index.min().strftime('%Y-%m')} to {test.index.max().strftime('%Y-%m')})",
        flush=True,
    )
    print(
        f"test sharpe: {sharpe:.4f} | ann_ret: {ann_ret:.4%} | ann_vol: {ann_vol:.4%} | max_dd: {mdd:.4%}",
        flush=True,
    )
    for dec_lbl, d0, d1 in [
        ("1990s", "1989-12-01", "1999-12-31"),
        ("2000s", "2000-01-01", "2009-12-31"),
        ("2010s", "2010-01-01", "2019-12-31"),
        ("2020s", "2020-01-01", "2024-12-31"),
    ]:
        sub = test[(test.index >= d0) & (test.index <= d1)]
        if len(sub) >= 12 and sub.std(ddof=1) > 1e-12:
            sh_sub = float(sub.mean() / sub.std(ddof=1) * np.sqrt(12.0))
            print(f"  {dec_lbl} ({len(sub)}m): sharpe = {sh_sub:.4f}", flush=True)


def main(
    chars: pd.DataFrame, features: pd.DataFrame, daily_ret: pd.DataFrame
) -> pd.DataFrame:
    weights = build_weights(chars, features, daily_ret)
    out = restrict_to_test(chars, weights)
    out = out.sort_values(["eom", "id"], kind="stable").reset_index(drop=True)
    out["eom"] = pd.to_datetime(out["eom"]).dt.strftime("%Y-%m-%d")
    out["id"] = out["id"].astype(int)
    out["w"] = out["w"].astype(float)
    return out[["id", "eom", "w"]]


if __name__ == "__main__":
    data_dir = (
        sys.argv[1]
        if len(sys.argv) > 1
        else os.environ.get("CTF_DATA_DIR", os.environ.get("JKP_DATA_DIR", "jkp-data"))
    )
    out_csv = (
        sys.argv[2]
        if len(sys.argv) > 2
        else os.environ.get("CTF_OUTPUT_PATH", os.environ.get("JKP_OUT_CSV", "weights.csv"))
    )
    c_name = (
        "ctff_chars.parquet"
        if os.path.exists(os.path.join(data_dir, "ctff_chars.parquet"))
        else "chars.parquet"
    )
    f_name = (
        "ctff_features.parquet"
        if os.path.exists(os.path.join(data_dir, "ctff_features.parquet"))
        else "features.parquet"
    )
    d_name = (
        "ctff_daily_ret.parquet"
        if os.path.exists(os.path.join(data_dir, "ctff_daily_ret.parquet"))
        else "daily_ret.parquet"
    )
    features_df = pd.read_parquet(os.path.join(data_dir, f_name))
    chars_df = pd.read_parquet(os.path.join(data_dir, c_name))
    feat_set = set(str(f) for f in features_df[feature_list_col].tolist())
    for col in chars_df.columns:
        if col in feat_set and col not in ("market_equity", "beta_60m"):
            chars_df[col] = chars_df[col].astype(np.float32)
    gc.collect()
    daily_ret_df = pd.read_parquet(
        os.path.join(data_dir, d_name),
        columns=["id", "date", "ret_exc"],
        dtype_backend="pyarrow",
    )
    gc.collect()
    pf = main(chars_df, features_df, daily_ret_df)
    pf.to_csv(out_csv, index=False)
    print(f"wrote {out_csv}: {len(pf):,} rows across {pf['eom'].nunique()} months", flush=True)
    report_sharpe(pf, chars_df)
