Skip to content

Validate Palace result evidence

validate_solution reports core solver convergence, optional error estimation, adaptive sampling, process exit and OOM evidence, mode checks, and input provenance independently. A zero exit code does not establish physical validity. The function reads supplied evidence and never changes tolerances or reruns a simulation.

import json
from pathlib import Path
from gsim.palace.validation import validate_solution

output = Path("run/output/palace")
config = json.loads(Path("run/config.json").read_text())
report = validate_solution(
    (output / "palace.log").read_text(encoding="utf-8"),
    metadata=json.loads((output / "palace.json").read_text()),
    estimator_disabled=config["Solver"]["Linear"].get("EstimatorMaxIts") == 0,
    expected_excitations=[1, 2],  # Require these adaptive training outcomes.
    input_files={"config.json": Path("run/config.json")},
)
report.core.status  # "passed", "failed", or "unknown"
report.estimator.status  # Also permits "skipped" when explicitly disabled.
[(item.excitation, item.sample_count) for item in report.adaptive]
report.provenance.solver_revision  # Recorded GitTag, or None if unavailable.
report.provenance.input_check.status  # "unknown": no saved manifest was supplied.
report.warnings  # Suspicious but unproven, for example a jump of the port mode.

Only set estimator_disabled=True when the actual input disables the estimator. The parser recognizes the paired zero-iteration PCG and unit-residual warning inside estimator sections or adaptive offline training. It keeps field correction, eigen, and driven failures separate. An enabled estimator failure remains a failure of that check. A skipped estimator supplies no evidence of mesh accuracy.

Adaptive checks retain excitation IDs and complete training-frequency lists in GHz. Missing or truncated outcomes are unknown; reaching the sample limit is failed. Successful adaptive sampling cannot clear an earlier core failure. Explicit expected_excitations also exposes an entirely missing excitation. An outcome without an excitation header retains excitation=None; it cannot satisfy an explicitly requested excitation ID.

Check a selected mode

A selected mode number alone does not establish port identity. Supply reference values and tolerances from your own application. The following self-contained example rejects opposite voltage parity even though the linear solver and process exit succeeded:

from gsim.palace.validation import ModeExpectation, ModeSample, validate_solution

report = validate_solution(
    "GMRES solver converged in 11 iterations",
    exit_code=0,
    mode_samples=[ModeSample(
        port=1, mode=2, wave_number=50 - 1j, voltage_ratio=-1,
    )],
    mode_expectations=[ModeExpectation(
        port=1, mode=2,
        wave_number=50 - 1j, wave_number_rtol=0.01,
        voltage_ratio=1, voltage_ratio_atol=0.02,
    )],
)
assert report.exit_status.status == report.core.status == "passed"
assert report.mode_checks[0].status == "failed"
assert report.estimator.status == "unknown"

Wave numbers are complex values in 1/m, including attenuation and its sign. Without explicit mode_samples, observations come from driven wave-port log lines. Every matching sample, including repeated frequency/excitation solves, must satisfy the expectation. frequency_ghz can restrict a reference to one frequency; its default relative matching tolerance of 5e-4 accommodates the four-significant-digit log output. Physical comparison tolerances default to zero and should be selected by the caller.

For BoundaryMode results, construct ModeSample records from the selected mode's tables. voltage_ratio is the complex ratio of consistently oriented voltage integrals whose parity you want to check. The API does not infer that ratio from a log or prescribe a CPW convention. A requested but absent ratio is unknown. Explicit samples replace log observations; pass all observations you want checked.

Check wave-port mode continuity

Palace selects the wave-port mode only by its index. If the port eigensolve returns a different solution at some frequency (spurious solutions in exact sweeps on old Palace builds, gsim#375, or a closed-box mode with waveport_boundary="pec", gsim#408), the sweep still finishes and the S-parameters are silently wrong. A physical port mode changes slowly with frequency, so validate_solution compares the effective index n_eff = Re(k_n) / k0 of every port solve in a driven log with the median of its five nearest neighbours in frequency. This runs automatically whenever the log has wave-port solves with a frequency, and the result is in report.port_mode_checks:

report = validate_solution(log_text)
for check in report.warnings:
    print(check.name, check.status)
    print(*check.evidence, sep="\n")

For a log with spurious solves the evidence names each flagged point (port, excitation, frequency, n_eff, neighbour median):

port_mode_continuity warning
3 of 124 port solves flagged.
port 1, mode 1, excitation 2: n_eff = 2.0087 at 1 GHz, neighbour median 1.7116 (17.36 % > 1 %)
port 2, mode 1, excitation 2: n_eff = 2.2669 at 1 GHz, neighbour median 1.7117 (32.44 % > 1 %)
port 2, mode 1, excitation 2: n_eff = 3.3796 at 1.3 GHz, neighbour median 1.7117 (97.44 % > 1 %)
  • passed: every series with at least MIN_DISTINCT_FREQUENCIES (6) distinct frequencies stays within CONTINUITY_RTOL (1 %) of its neighbour median. Series that are too short are listed in the evidence as not judged.
  • warning: at least one point is outside the tolerance, or a wave number is not finite. A warning is not a failure and is not part of report.failures: a mode that really changes quickly (close to cutoff, very coarse sweeps) is flagged too. Inspect the listed frequencies and the port fields before trusting or discarding the sweep.
  • unknown: no series is long enough to judge, for example a single-frequency run. Unknown is never a pass.
  • No check is added when the log has no frequency-tagged wave-port solve (eigenmode, electrostatic, lumped-port runs).

The check only sees the real part of the wave number. It cannot find a wrong mode that is smooth in frequency, for example a sweep that is wrong at every frequency; compare against a BoundaryMode solve with ModeExpectation for that.

Ports are compared with each other only on request, because the log does not record the port geometry and ports of different width or stack legitimately have different modes. When every wave port has the same cross-section, pass ports_share_cross_section=True; ports solved at the same frequency must then agree within PORT_AGREEMENT_RTOL (0.5 %), reported as the port_mode_ports check. To run the check on your own observations, call check_port_mode_continuity(samples, compare_ports=..., rtol=..., port_rtol=...) with ModeSample records that carry frequency_ghz.

Compare meshes between runs

Gmsh gives the same mesh twice only for some settings. With the default algorithm_3d="delaunay" and surface_threads=1, the mesh does not depend on threads. surface_threads above 1 is faster but gives a different mesh on every run, and with algorithm_3d="hxt" the mesh depends on the thread count. This is a Gmsh limitation (gmsh#2255): Mesh.Reproducible only covers HXT's initial Delaunay step, and a fixed random seed cannot fix the order in which threads finish. gsim logs a warning for both non-reproducible settings.

sim.mesh() records an ID-independent mesh hash, the Gmsh options used and the Gmsh and gsim versions:

result = sim.mesh(threads=4, surface_threads=1)
result.mesh_stats["mesh_hash"]
result.mesh_stats["versions"]["gmsh"]
result.mesh_stats["gmsh_options"]["Mesh.MaxNumThreads2D"]

Equal hashes mean the same nodes (to 1e-4 um) and the same elements in each physical group. Compare hashes only between runs with the same Gmsh version on the same kind of host; the stats do not record the host. A different hash under reproducible settings points at a change in the geometry, the mesh settings or the Gmsh build.

Evidence and limits

Statuses describe supplied evidence, not a complete physical validation. In particular, a passed core check means at least one recognized convergence message and no recognized core failure. It cannot establish complete log coverage, mesh convergence, or convergence of unreported eigenpairs. Unrecognized log formats and absent metrics are not proof of success. Archived cfa430a and cfa430a-dirty coverage is limited to driven and eigenmode logs; electrostatic and magnetostatic log coverage is not established. Their standard It N/M: Index = ... header resets the estimator section, but abridged or reformatted logs may omit that boundary. Later Palace formats may require additional parsing rules.

report.failures contains explicit failures; an empty tuple can still leave unknown or omitted checks. Mode checks are omitted unless requested; the automatic port-mode continuity checks are separate (report.port_mode_checks, listed in report.warnings when they warn). Exit and OOM checks remain unknown unless independently recorded values are supplied. No resource limits, passivity limits, attenuation limits, or application-specific mode thresholds are assumed beyond the documented continuity tolerances.

input_files records current file hashes. To compare against independently saved run provenance, pass expected_input_sha256={"config.json": saved_digest}. Only named manifest entries are compared; a mismatch fails and a missing or unreadable file (including a directory) is unknown, with the OS error recorded as evidence. Other input checks still complete. Current hashes alone cannot prove which files a remote solver used.