---
title: "Binary ODA: Migraine Attacks in a Clinical Trial"
author: "oda"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Binary ODA: Migraine Attacks in a Clinical Trial}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

## Research question

Appleton (1995) reported a clinical trial in which 67 patients experiencing
migraine were randomised to one of two treatments, and the number of migraine
attacks was recorded.^[Appleton DR (1995). Pitfalls in the interpretation of
studies: III. *Journal of the Royal Society of Medicine*, 88, 241-243.] Various
conventional methods  -  Student's *t*-test (including square-root and log
transformations), the Mann-Whitney *U*-test, and a Poisson normal test  -  either
failed to reach conventional significance or violated their underlying
assumptions. The analyst ultimately discretised the count at 0 vs. >=1 and
validated the split with a one-tailed Fisher's exact test (*p* < 0.022).

Because ODA is invariant under any monotonic transformation and requires no
distributional assumptions, it can analyse the raw count directly. ODA is used
here to determine whether number of migraine attacks discriminates treatment
arm, and to quantify the strength of the association.

## Data

Treatment arm (0 = Treatment 1, 1 = Treatment 2) is the class variable; number
of migraine attacks (0-7, ordered) is the attribute. Published cell frequencies
are reconstructed directly into observation-level vectors  -  no external data
file is required.

```{r data}
library(oda)

# Cross-classification: rows = attacks (0-7), cols = treatment arm.
#          T1 (0)  T2 (1)
#  0 att:    13       5
#  1 att:     9      13
#  2 att:     4       6
#  3 att:     2       1
#  4 att:     1       2
#  5 att:     1       3
#  6 att:     3       3
#  7 att:     0       1

treatment <- c(
  rep(0L, 13), rep(1L,  5),   # attacks = 0
  rep(0L,  9), rep(1L, 13),   # attacks = 1
  rep(0L,  4), rep(1L,  6),   # attacks = 2
  rep(0L,  2), rep(1L,  1),   # attacks = 3
  rep(0L,  1), rep(1L,  2),   # attacks = 4
  rep(0L,  1), rep(1L,  3),   # attacks = 5
  rep(0L,  3), rep(1L,  3),   # attacks = 6
  rep(0L,  0), rep(1L,  1)    # attacks = 7
)
attacks <- c(
  rep(0L, 18), rep(1L, 22), rep(2L, 10),
  rep(3L,  3), rep(4L,  3), rep(5L,  4),
  rep(6L,  6), rep(7L,  1)
)

table(attacks, treatment,
      dnn = c("Migraine Attacks (0-7)", "Treatment (0=T1, 1=T2)"))
```

## Fit the ODA model

Number of attacks is an ordered integer; ODA scans it as an ordered attribute
(no categorical flag), consistent with the MegaODA reference analysis. No
directional hypothesis was specified *a priori*, so the default nondirectional
search (`direction = "both"`) is used. Leave-one-out (LOO) jackknife validity
analysis is included.

```{r fit-canonical, eval=FALSE}
# Canonical reference run (mc_iter = 25000L; not evaluated in CRAN vignette)
fit <- oda_fit(
  x         = attacks,
  y         = treatment,
  attr_type = "ordered",
  mc_iter   = 25000L,
  loo       = "on"
)
```

```{r fit}
# CRAN-safe run: mc_iter = 500L for vignette rendering speed.
# Training rule, ESS, and confusion matrix are identical to the canonical run.
fit <- oda_fit(
  x         = attacks,
  y         = treatment,
  attr_type = "ordered",
  mc_iter   = 500L,
  mc_seed   = 42L,
  loo       = "on"
)
```

## Rule and confusion matrix

```{r print-fit}
print(fit)
```

ODA identified a single cut at 0.5, consistent with Appleton's hand-chosen
spline:

- If attacks <= 0.5 (zero attacks) -> predict Treatment 1 (0)
- If attacks > 0.5 (one or more attacks) -> predict Treatment 2 (1)

```{r confusion}
# Confusion matrix: actual treatment (rows) x predicted treatment (cols)
conf_mat <- matrix(
  c(fit$confusion$TN, fit$confusion$FP,
    fit$confusion$FN, fit$confusion$TP),
  nrow = 2L, byrow = TRUE,
  dimnames = list(Actual    = c("T1(0)", "T2(1)"),
                  Predicted = c("T1(0)", "T2(1)"))
)
print(conf_mat)
```

## ESS / PAC / PV interpretation

```{r metrics}
summary(fit)
```

```{r pv}
# Predictive value: accuracy when the model makes a prediction into each class
pv_t1 <- fit$confusion$TN / (fit$confusion$TN + fit$confusion$FN)
pv_t2 <- fit$confusion$TP / (fit$confusion$TP + fit$confusion$FP)
cat("PV Treatment 1 (0):", round(pv_t1 * 100, 1), "%\n")
cat("PV Treatment 2 (1):", round(pv_t2 * 100, 1), "%\n")
```

- **PAC (sensitivity per class):** 39.4% for Treatment 1 patients, 85.3% for
  Treatment 2 patients. Because 50% correct per class is expected by chance, the
  model classifies Treatment 2 patients well above chance while Treatment 1
  classification is below chance  -  a notable asymmetry.
- **ESS = 24.69%** is marginally below the conventional 25% threshold for
  moderate effect strength.^[Yarnold, P.R., & Soltysik, R.C. (2005). *Optimal
  Data Analysis: A Guidebook with Software for Windows.* Washington, D.C.: APA
  Books.] The asymmetry reflects a greater concentration of zero-attack patients
  in Treatment 1 relative to Treatment 2.
- **PV:** When the model predicts Treatment 1, it is correct ~72.2% of the time;
  when it predicts Treatment 2, ~59.2%.

## Monte Carlo and LOO validity

The MC p-value and LOO result are shown in the `summary` output above.

- **MC p-value (one-tailed, non-directional permutation):** Each Monte Carlo
  permutation randomly shuffles class labels and refits ODA searching *both*
  directions, exactly as in the training analysis. The reported p is the
  proportion of permuted ESS values that equal or exceed the observed ESS.
  Because the permutation distribution accounts for optimizing over both
  directions, the MC p is more conservative for a non-directional analysis.
  The MegaODA reference value is p ≈ 0.086 (not significant at α = 0.05).
- **LOO stability:** The leave-one-out ESS equals the training ESS (24.69%),
  indicating the rule is completely stable — no single observation materially
  alters the model.
- **LOO Fisher exact p (one-tailed):** Per MPE p. 34, hold-out p is always
  one-tailed: "the null hypothesis is that the training model will not replicate
  when it is used to classify observations in the hold-out sample." The Fisher
  exact test (alternative = "greater") tests whether the LOO confusion matrix
  reflects above-chance classification. This test does *not* adjust for the
  direction search performed during training. Statistical significance is
  confirmed in LOO, consistent with Appleton's original one-tailed Fisher test.

**Why MC p and LOO p diverge for non-directional analyses:** MC permutation
p is more conservative than LOO Fisher p when the analysis is non-directional.
The MC test accounts for the fact that training optimized over both directions
(making the permutation baseline harder to beat); the LOO Fisher test does not
apply that adjustment. Both values are valid for their respective purposes:
MC p assesses training model significance with direction-search adjustment;
LOO Fisher p assesses replication of the fixed training rule in held-out
data. The divergence narrows or disappears when a directional hypothesis is
declared *a priori* (see Notes).

## Notes on reproducibility and current scope

**Fixture parity.** The training rule, confusion matrix, and ESS are verified
against MegaODA.exe output in the package test suite
(`tests/testthat/test-fixture-vignettes.R`, Example 4).

**MC p-value calibration.** The MC p shown here reflects `mc_iter = 500L`
for CRAN build speed and will differ from MegaODA's reported value
(p = 0.086 at 25000 iterations). With only 500 permutations the estimate
is noisy (Monte Carlo standard error ~1-2%). Use the canonical run with
`mc_iter = 25000L` (chunk `fit-canonical`, `eval=FALSE`) for
publication-quality results. Training ESS and confusion matrix are
unaffected by `mc_iter`.

**Directional analysis.** The original analysis did not specify a directional
hypothesis *a priori*; the nondirectional default (`direction = "both"`) is
therefore appropriate. If a directional ordered hypothesis had been specified
in advance (e.g., more attacks predicts Treatment 2), `direction = "greater"`
or `direction = "less"` could be used to enforce MPE Chapter 2 binary ordered
directional ODA and obtain a one-tailed p-value.
