Model Selection

Through Model Selection, alternative model hypothesis results are competed to determine which hypothesis is best explained by the underlying data. Before conducting model selection, you should be familiar with the benefits and limitations of alternative model selection statistics.

Akaike Weights

To calculate Akaike weights for a set of hypotheses, you must first obtain your results by either loading in saved result files, or running models directly. Once you have obtained your results, weights can be saved after calculation by using the functions within the Bristlecone.Data namespace.

The ModelSelection.save function takes a sequence of tuples of four values: the subject's ID, the model or hypothesis ID, the estimated result, and the Akaike weight for that result. In the below example, we do some wrangling to get the weights into the correct format for saving:

open Bristlecone

fun modelResults subjectIds hypothesisIds ->
    modelResults
    |> ModelSelection.Akaike.akaikeWeights
    |> Seq.zip3 subjectIds hypothesisIds
    |> Seq.map (fun (s, h, (a, b)) -> (s, h, a, b))
    |> Bristlecone.Data.ModelSelection.save "/some/dir"

If you are working with ResultSets, calculating and saving the weights is easier, and can be completed as below:

fun (hypothesisResults: ModelSelection.ResultSet.ResultSet<string, Language.Hypotheses.Hypothesis> seq) ->
    hypothesisResults
    |> ModelSelection.Akaike.akaikeWeightsForSet (fun h -> h.ReferenceCode)
    |> Bristlecone.Data.ModelSelection.save "/some/dir"

Multiple items
module Bristlecone from Bristlecone
<namespacedoc><summary>The core library of Bristlecone, containing model-fitting functions.</summary></namespacedoc>
Main functionality of Bristlecone, including functions to scaffold `ModelSystem`s and for model-fitting (tests and real fits).

--------------------
namespace Bristlecone

val modelResults: ModelSystem.EstimationResult seq

val subjectIds: string seq

val hypothesisIds: string seq

module ModelSelection from Bristlecone
<summary>Contains tools for conducting Model Selection across individual subjects and hypotheses.</summary>

module Akaike from Bristlecone.ModelSelection
<summary>Functions for conducting Akaike Information Criterion (AIC).</summary>

val akaikeWeights: models: ModelSystem.EstimationResult seq -> (ModelSystem.EstimationResult * ModelSelection.Akaike.AkaikeWeight) seq
<summary>Akaike weights for a sequence of `EstimationResult`s.</summary>
<param name="models">The input model results</param>
<returns>An (EstimationResult * float) sequence of estimation results paired to their Akaike weights.</returns>
<exception name="ArgumentException">Occurs when there are no observations within an estimation result.</exception>

Multiple items
module Seq from Bristlecone

--------------------
module Seq from Microsoft.FSharp.Collections

val zip3: source1: 'T1 seq -> source2: 'T2 seq -> source3: 'T3 seq -> ('T1 * 'T2 * 'T3) seq

val map: mapping: ('T -> 'U) -> source: 'T seq -> 'U seq

val s: string

val h: string

val a: ModelSystem.EstimationResult

val b: ModelSelection.Akaike.AkaikeWeight

namespace Bristlecone.Data

module ModelSelection from Bristlecone.Data

val save: directory: string -> result: (string * string * ModelSystem.EstimationResult * ModelSelection.Akaike.AkaikeWeight) seq -> unit

val hypothesisResults: ModelSelection.ResultSet.ResultSet<string,Language.Hypotheses.Hypothesis> seq

module ResultSet from Bristlecone.ModelSelection
<summary>Organises multiple hypotheses and multiple subjects into distinct analysis groups.</summary>

type ResultSet<'subject,'hypothesis> = { Subject: 'subject Hypothesis: 'hypothesis BestResult: EstimationResult option AllResults: EstimationResult seq }
<summary>A representation of all results for a particular subject and hypothesis</summary>

Multiple items
val string: value: 'T -> string

--------------------
type string = System.String

module Language from Bristlecone
<summary> An F# Domain Specific Language (DSL) for scripting with Bristlecone. </summary>

module Hypotheses from Bristlecone.Language
<summary>Types to represent a hypothesis, given that a hypothesis is a model system that contains some alternate formulations of certain components.</summary>

Multiple items
union case Language.Hypotheses.Hypothesis.Hypothesis: ModelSystem.ModelSystem * Language.Hypotheses.ComponentName list -> Language.Hypotheses.Hypothesis

--------------------
type Hypothesis = | Hypothesis of ModelSystem * ComponentName list member Components: ComponentName list member Model: ModelSystem member ReferenceCode: string
<summary>A hypothesis consists of a model system and the names of the swappable components within it, alongside the name of their current implementation.</summary>

Multiple items
val seq: sequence: 'T seq -> 'T seq

--------------------
type 'T seq = System.Collections.Generic.IEnumerable<'T>

val akaikeWeightsForSet: getRefCode: ('a -> 'b) -> set: ModelSelection.ResultSet.ResultSet<'c,'a> seq -> ('c * 'b * ModelSystem.EstimationResult * ModelSelection.Akaike.AkaikeWeight) list (requires equality)
<summary>Akaike weights for a result set.</summary>
<param name="getRefCode">A function that gets a short reference code from a hypothesis.</param>
<param name="set">A sequence of `ResultSet`s, within each the 1 .. many results of a particular subject * hypothesis combination.</param>
<returns>An `(EstimationResult * float) seq` of estimation results paired to their Akaike weights.</returns>
<exception cref="ArgumentException">Occurs when there are no observations within an estimation result.</exception>

val h: Language.Hypotheses.Hypothesis

property Language.Hypotheses.Hypothesis.ReferenceCode: string with get
<summary>Compiles a reference code that may be used to identify (although not necessarily uniquely) this hypothesis</summary>
<returns>A string in the format XX_XXX_YY_YYY... where XX_XXX is a singe component with XX the component and XXX the implementation.</returns>