---
title: "Multiclass ODA: Convergent Validity of Protein Classification Methods"
author: "oda"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Multiclass ODA: Convergent Validity of Protein Classification Methods}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

## Research question

Nishikawa, Kubota, and Ooi (1983) independently classified 325 proteins into
one of four mutually exclusive types using two different methods: one based on
biological characteristics and one based on amino acid composition.^[Nishikawa,
K., Kubota, Y., & Ooi, T. (1983). Classification of proteins into groups based
on amino acid composition and other characters, II: Grouping into four types.
*Journal of Biochemistry*, 94, 997-1007.] Because the two methods should
theoretically converge on the same type for each protein, the directional
hypothesis is that protein type codes are identical across methods  - 
demonstrating convergent validity.

Optimal Data Analysis (MultiODA) tests whether amino acid composition type
discriminates biological type, with the *a priori* prediction that type codes
match across methods.

## Data

Biological type (1-4) is the class variable; amino acid composition type (1-4)
is the attribute. Published cell frequencies are reconstructed directly into
observation-level vectors  -  no external data file is required.

```{r data}
library(oda)

# Cross-classification: rows = biological type, cols = amino acid type.
# (column-major reconstruction matches published Table 1)
#                  AA=1  AA=2  AA=3  AA=4   total
#  Biological=1     98    16     5     3      122
#  Biological=2     13    50     2     8       73
#  Biological=3      6     4    23    12       45
#  Biological=4      7    19    14    45       85
#  total           124    89    44    68      325

biological_type <- c(
  rep(1L, 98), rep(2L, 13), rep(3L,  6), rep(4L,  7),  # amino_acid = 1
  rep(1L, 16), rep(2L, 50), rep(3L,  4), rep(4L, 19),  # amino_acid = 2
  rep(1L,  5), rep(2L,  2), rep(3L, 23), rep(4L, 14),  # amino_acid = 3
  rep(1L,  3), rep(2L,  8), rep(3L, 12), rep(4L, 45)   # amino_acid = 4
)
amino_acid_type <- c(rep(1L, 124), rep(2L, 89), rep(3L, 44), rep(4L, 68))

table(amino_acid_type, biological_type,
      dnn = c("Amino Acid Type (1-4)", "Biological Type (1-4)"))
```

## Fit the ODA model

Amino acid type is a four-category nominal variable. ODA searches all possible
mappings from the four amino-acid-type categories to the four biological-type
classes and selects the mapping that maximises ESS. No *a priori* direction is
supplied; the search is nondirectional (`Hypothesis: NONDIRECTIONAL` in
MegaODA output). Leave-one-out (LOO) jackknife validity is requested via
`loo = "on"`; LOO confusion and ESS are reported. No LOO p-value is given
because no canonical Fisher-exact LOO p-value is defined for C > 2
multicategorical class problems.

```{r fit-canonical, eval=FALSE}
# Canonical reference run (mc_iter = 25000L; not evaluated in CRAN vignette)
fit <- oda_fit(
  x         = amino_acid_type,
  y         = biological_type,
  attr_type = "categorical",
  mc_iter   = 25000L,
  loo       = "on"
)
```

```{r fit}
# CRAN-safe run: mc_iter = 500L for vignette rendering speed.
# Training rule, ESS, and confusion matrix are identical to the canonical run.
fit <- oda_fit(
  x         = amino_acid_type,
  y         = biological_type,
  attr_type = "categorical",
  mc_iter   = 500L,
  mc_seed   = 42L,
  loo       = "on"
)
```

## Rule and confusion matrix

```{r print-fit}
print(fit)
```

ODA's nondirectional search identified the identity mapping as the optimal
categorical partition:

- If amino acid type = 1 -> predict biological type = 1
- If amino acid type = 2 -> predict biological type = 2
- If amino acid type = 3 -> predict biological type = 3
- If amino acid type = 4 -> predict biological type = 4

```{r confusion}
# Confusion matrix (actual x predicted); strip dimnames for clean display
conf_mat <- unname(fit$confusion)
rownames(conf_mat) <- paste0("Bio=", 1:4)
colnames(conf_mat) <- paste0("Pred=", 1:4)
print(conf_mat)
```

## ESS / PAC / PV interpretation

```{r metrics}
summary(fit)
```

```{r pac-pv}
m <- oda_metrics(fit)

# PAC (sensitivity) per class - pac_by_class is already on percentage scale
cat("PAC by biological type:\n")
cat("  Type 1:", round(m$pac_by_class[1], 1), "%\n")
cat("  Type 2:", round(m$pac_by_class[2], 1), "%\n")
cat("  Type 3:", round(m$pac_by_class[3], 1), "%\n")
cat("  Type 4:", round(m$pac_by_class[4], 1), "%\n")

# Predictive value: diagonal / column sums
pv <- diag(fit$confusion) / colSums(fit$confusion) * 100
cat("\nPV by biological type:\n")
cat("  Type 1:", round(pv[1], 1), "%\n")
cat("  Type 2:", round(pv[2], 1), "%\n")
cat("  Type 3:", round(pv[3], 1), "%\n")
cat("  Type 4:", round(pv[4], 1), "%\n")
```

- **PAC (sensitivity per class):** 80.3%, 68.5%, 51.1%, and 52.9% for protein
  types 1 through 4, respectively. Because 25% correct per class is expected by
  chance for a four-class problem, classification of all four types substantially
  exceeds chance.
- **ESS = 50.96%** indicates a relatively strong effect.^[Yarnold, P.R., &
  Soltysik, R.C. (2005). *Optimal Data Analysis: A Guidebook with Software for
  Windows.* Washington, D.C.: APA Books.] All four PAC values exceed the
  four-class chance benchmark of 25%; they also exceed 50%, indicating
  majority-accurate classification within each class.
- **PV:** When the model predicts type 1 it is correct ~79.0% of the time; type
  2, ~56.2%; type 3, ~52.3%; type 4, ~66.2%. All predictive values exceed
  chance.

## Monte Carlo and LOO validity

The MC p-value and LOO results are shown in the `print` and `summary` output
above.

- **MC p-value:** The printed `p(MC)` is a nondirectional Fisher-randomization
  p-value. Each permutation searches for the best categorical mapping of the
  permuted labels, matching the nondirectional search used for the training
  model. Interpret by decision threshold (e.g., p < 0.05).
- **LOO jackknife:** Leave-one-out ESS and Mean PAC are shown. Each fold holds
  out one observation, searches for the optimal categorical mapping on the
  remaining n-1 observations (nondirectional, equal priors), and classifies
  the held-out case using that fold's rule. Because the identity mapping is the
  globally optimal partition for these data, every fold recovers the same rule,
  and LOO ESS equals training ESS exactly. This confirms the model is stable
  across folds and no single observation drives the result.
- **LOO p-value:** No LOO Fisher-exact p-value is reported for multicategorical
  (C > 2) class problems. No canonical reference distribution is defined for
  the C x C LOO confusion matrix in this context. For binary class ODA, a
  one-tailed Fisher exact p-value is available (MPE p. 34).

## Notes on reproducibility

**Fixture parity.** The training rule, confusion matrix, and ESS are verified
against MegaODA.exe output in the package test suite
(`tests/testthat/test-fixture-vignettes.R`, Example 3).

**MC p-value calibration.** The MC p shown here reflects `mc_iter = 500L`
in this CRAN vignette. Use the canonical run with `mc_iter = 25000L` (chunk
`fit-canonical`, `eval=FALSE`) for publication-quality results.

**Nondirectional search.** No `direction` argument is supplied. ODA evaluates
all possible mappings from the four amino-acid categories to the four
biological-type classes and selects the mapping that maximises ESS. This matches
the MegaODA.exe gold run (`Hypothesis: NONDIRECTIONAL`).

**Optional directional analysis.** A researcher with an *a priori* convergent-
validity hypothesis (amino acid type i predicts biological type i) can supply
`direction = "ascending"` for a constrained identity-map analysis
(MPE Chapter 4 Phase 6C). For this dataset the two analyses yield identical
ESS and confusion because the identity mapping happens to be the global optimum;
they differ in MC interpretation (directional vs. nondirectional p-value).
