bristlecone
Loading and Saving Data
The Bristlecone.Data namespace includes methods for saving Bristlecone
results and diagnostics, and loading saved results for further analysis
at a later date.
Individual estimations results
The estimation result itself
Saving and loading estimation results is conducted as follows:
open Bristlecone
open Bristlecone.Data
open Bristlecone.Language
let resultsDirectory = "/some/data/dir"
let thinTraceBy = Some 1000 // Only trace every n iterations to save disk space
let subject = "Mosquito population" // The name of the 'subject' of analysis
let modelName = "Logistic growth"
fun result -> EstimationResult.saveAll resultsDirectory subject modelName thinTraceBy result
This save function outputs three files: one each for the maximum likelihood estimate, a trace of the optimisation process, and the estimated vs observed time series.
Other statistics and estimates
Some other Bristlecone functions have functions to load and save their outputs to CSV files.
Save confidence intervals:
Bristlecone.Data.Confidence.save
Save n-step ahead predictions made using Bristlecone.oneStepAhead and similar functions:
Bristlecone.Data.NStepAhead.save
Ensemble-level results
Some processes, for example model-selection, work across many model results. Bristlecone includes loading and saving functions (to and from CSV files) for many of these procedures.
Model-selection
For model selection, you may calculate Akaike weights either from a sequence of EstimationResult (for simpler tasks)
or from a sequence of ResultSet (for working with many hypotheses):
fun modelResults subjectIds hypothesisIds ->
modelResults
|> ModelSelection.Akaike.akaikeWeights
|> Seq.zip3 subjectIds hypothesisIds
|> Seq.map (fun (s, h, (a, b)) -> (s, h, a, b))
|> Bristlecone.Data.ModelSelection.save "/some/dir"
fun (hypothesisResults: ModelSelection.ResultSet.ResultSet<string, Hypotheses.Hypothesis> seq) ->
hypothesisResults
|> ModelSelection.Akaike.akaikeWeightsForSet (fun h -> h.ReferenceCode)
|> Bristlecone.Data.ModelSelection.save "/some/dir"
Convergence of Monte-Carlo based optimisation approaches
You can assess the convergence of multiple 'chains' between MCMC or Monte-Carlo based
optimisation methods using the functions in the Bristlecone.Diagnostics namespace, such as
the per-parameter R-hat values based on the optimisation trace.
// Given a set of results on a per-subject and per-hypothesis basis:
fun resultSet ->
resultSet
|> Bristlecone.Diagnostics.Convergence.gelmanRubin 10000 "some subject" "some hypothesis"
|> Option.iter (Bristlecone.Data.Convergence.save "/some/dir")
Summarise Root Mean Squared Error for many hypotheses / subjects
Given individual n-step ahead predictions made using Bristlecone.oneStepAhead
and similar functions, you can save the resultant RMSE statistic (an indication
of model performance) in a summary table across all hypotheses and subjects:
fun nStepFits -> Bristlecone.Data.NStepAhead.saveAllRMSE "/some/dir" nStepFits
Multiple items
module Bristlecone from Bristlecone
<namespacedoc><summary>The core library of Bristlecone, containing model-fitting functions.</summary></namespacedoc>
Main functionality of Bristlecone, including functions to scaffold `ModelSystem`s and for model-fitting (tests and real fits).
--------------------
namespace Bristlecone
module Bristlecone from Bristlecone
<namespacedoc><summary>The core library of Bristlecone, containing model-fitting functions.</summary></namespacedoc>
Main functionality of Bristlecone, including functions to scaffold `ModelSystem`s and for model-fitting (tests and real fits).
--------------------
namespace Bristlecone
namespace Bristlecone.Data
module Language
from Bristlecone
<summary> An F# Domain Specific Language (DSL) for scripting with Bristlecone. </summary>
<summary> An F# Domain Specific Language (DSL) for scripting with Bristlecone. </summary>
val resultsDirectory: string
val thinTraceBy: int option
union case Option.Some: Value: 'T -> Option<'T>
val subject: string
val modelName: string
val result: ModelSystem.EstimationResult
module EstimationResult
from Bristlecone.Data
val saveAll: directory: string -> subject: string -> modelId: string -> thinTraceBy: int option -> result: ModelSystem.EstimationResult -> unit
<summary>Save the Maximum Likelihood Estimate, trace of the optimisation procedure, and time-series.</summary>
<param name="directory">Relative or absolute directory to save files to</param>
<param name="subject">An identifier for the subject of the test</param>
<param name="modelId">An identifier for the model used</param>
<param name="thinTraceBy">If Some, an integer representing the nth traces to keep from the optimisation trace. If None, does not thin the trace.</param>
<param name="result">The estimation result to save</param>
<summary>Save the Maximum Likelihood Estimate, trace of the optimisation procedure, and time-series.</summary>
<param name="directory">Relative or absolute directory to save files to</param>
<param name="subject">An identifier for the subject of the test</param>
<param name="modelId">An identifier for the model used</param>
<param name="thinTraceBy">If Some, an integer representing the nth traces to keep from the optimisation trace. If None, does not thin the trace.</param>
<param name="result">The estimation result to save</param>
module Confidence
from Bristlecone.Data
val save: directory: string -> subject: string -> modelId: string -> runId: System.Guid -> result: CodedMap<Confidence.ConfidenceInterval> -> unit
module NStepAhead
from Bristlecone.Data
<summary>Loads and saves one-step-ahead and n-step-ahead prediction results into and out of simple CSV files</summary>
<summary>Loads and saves one-step-ahead and n-step-ahead prediction results into and out of simple CSV files</summary>
val save: directory: string -> subjectId: string -> modelId: string -> analysisId: System.Guid -> stepsAhead: int -> result: CodedMap<ModelSystem.FitSeries * Statistics.NStepStatistics> -> unit
<summary>Save an individual n-step prediction (excluding statistics) for an individual subject and hypothesis / model.</summary>
<param name="directory">The file directory to save to</param>
<param name="subjectId">Subject ID</param>
<param name="modelId">Model ID or Hypothesis ID</param>
<param name="analysisId">The unique reference of the estimation result used to generate n-step predictions</param>
<param name="stepsAhead">The number of time-steps ahead predicted</param>
<param name="result">The n-step prediction from Bristlecone.oneStepAhead or similar</param>
<summary>Save an individual n-step prediction (excluding statistics) for an individual subject and hypothesis / model.</summary>
<param name="directory">The file directory to save to</param>
<param name="subjectId">Subject ID</param>
<param name="modelId">Model ID or Hypothesis ID</param>
<param name="analysisId">The unique reference of the estimation result used to generate n-step predictions</param>
<param name="stepsAhead">The number of time-steps ahead predicted</param>
<param name="result">The n-step prediction from Bristlecone.oneStepAhead or similar</param>
val modelResults: ModelSystem.EstimationResult seq
val subjectIds: string seq
val hypothesisIds: string seq
Multiple items
module ModelSelection from Bristlecone.Data
--------------------
module ModelSelection from Bristlecone
<summary>Contains tools for conducting Model Selection across individual subjects and hypotheses.</summary>
module ModelSelection from Bristlecone.Data
--------------------
module ModelSelection from Bristlecone
<summary>Contains tools for conducting Model Selection across individual subjects and hypotheses.</summary>
module Akaike
from Bristlecone.ModelSelection
<summary>Functions for conducting Akaike Information Criterion (AIC).</summary>
<summary>Functions for conducting Akaike Information Criterion (AIC).</summary>
val akaikeWeights: models: ModelSystem.EstimationResult seq -> (ModelSystem.EstimationResult * ModelSelection.Akaike.AkaikeWeight) seq
<summary>Akaike weights for a sequence of `EstimationResult`s.</summary>
<param name="models">The input model results</param>
<returns>An (EstimationResult * float) sequence of estimation results paired to their Akaike weights.</returns>
<exception name="ArgumentException">Occurs when there are no observations within an estimation result.</exception>
<summary>Akaike weights for a sequence of `EstimationResult`s.</summary>
<param name="models">The input model results</param>
<returns>An (EstimationResult * float) sequence of estimation results paired to their Akaike weights.</returns>
<exception name="ArgumentException">Occurs when there are no observations within an estimation result.</exception>
Multiple items
module Seq from Bristlecone
--------------------
module Seq from Microsoft.FSharp.Collections
module Seq from Bristlecone
--------------------
module Seq from Microsoft.FSharp.Collections
val zip3: source1: 'T1 seq -> source2: 'T2 seq -> source3: 'T3 seq -> ('T1 * 'T2 * 'T3) seq
val map: mapping: ('T -> 'U) -> source: 'T seq -> 'U seq
val s: string
val h: string
val a: ModelSystem.EstimationResult
val b: ModelSelection.Akaike.AkaikeWeight
module ModelSelection
from Bristlecone.Data
val save: directory: string -> result: (string * string * ModelSystem.EstimationResult * ModelSelection.Akaike.AkaikeWeight) seq -> unit
val hypothesisResults: ModelSelection.ResultSet.ResultSet<string,Hypotheses.Hypothesis> seq
module ResultSet
from Bristlecone.ModelSelection
<summary>Organises multiple hypotheses and multiple subjects into distinct analysis groups.</summary>
<summary>Organises multiple hypotheses and multiple subjects into distinct analysis groups.</summary>
type ResultSet<'subject,'hypothesis> =
{
Subject: 'subject
Hypothesis: 'hypothesis
BestResult: EstimationResult option
AllResults: EstimationResult seq
}
<summary>A representation of all results for a particular subject and hypothesis</summary>
<summary>A representation of all results for a particular subject and hypothesis</summary>
Multiple items
val string: value: 'T -> string
--------------------
type string = System.String
val string: value: 'T -> string
--------------------
type string = System.String
module Hypotheses
from Bristlecone.Language
<summary>Types to represent a hypothesis, given that a hypothesis is a model system that contains some alternate formulations of certain components.</summary>
<summary>Types to represent a hypothesis, given that a hypothesis is a model system that contains some alternate formulations of certain components.</summary>
Multiple items
union case Hypotheses.Hypothesis.Hypothesis: ModelSystem.ModelSystem * Hypotheses.ComponentName list -> Hypotheses.Hypothesis
--------------------
type Hypothesis = | Hypothesis of ModelSystem * ComponentName list member Components: ComponentName list member Model: ModelSystem member ReferenceCode: string
<summary>A hypothesis consists of a model system and the names of the swappable components within it, alongside the name of their current implementation.</summary>
union case Hypotheses.Hypothesis.Hypothesis: ModelSystem.ModelSystem * Hypotheses.ComponentName list -> Hypotheses.Hypothesis
--------------------
type Hypothesis = | Hypothesis of ModelSystem * ComponentName list member Components: ComponentName list member Model: ModelSystem member ReferenceCode: string
<summary>A hypothesis consists of a model system and the names of the swappable components within it, alongside the name of their current implementation.</summary>
Multiple items
val seq: sequence: 'T seq -> 'T seq
--------------------
type 'T seq = System.Collections.Generic.IEnumerable<'T>
val seq: sequence: 'T seq -> 'T seq
--------------------
type 'T seq = System.Collections.Generic.IEnumerable<'T>
val akaikeWeightsForSet: getRefCode: ('a -> 'b) -> set: ModelSelection.ResultSet.ResultSet<'c,'a> seq -> ('c * 'b * ModelSystem.EstimationResult * ModelSelection.Akaike.AkaikeWeight) list (requires equality)
<summary>Akaike weights for a result set.</summary>
<param name="getRefCode">A function that gets a short reference code from a hypothesis.</param>
<param name="set">A sequence of `ResultSet`s, within each the 1 .. many results of a particular subject * hypothesis combination.</param>
<returns>An `(EstimationResult * float) seq` of estimation results paired to their Akaike weights.</returns>
<exception cref="ArgumentException">Occurs when there are no observations within an estimation result.</exception>
<summary>Akaike weights for a result set.</summary>
<param name="getRefCode">A function that gets a short reference code from a hypothesis.</param>
<param name="set">A sequence of `ResultSet`s, within each the 1 .. many results of a particular subject * hypothesis combination.</param>
<returns>An `(EstimationResult * float) seq` of estimation results paired to their Akaike weights.</returns>
<exception cref="ArgumentException">Occurs when there are no observations within an estimation result.</exception>
val h: Hypotheses.Hypothesis
property Hypotheses.Hypothesis.ReferenceCode: string with get
<summary>Compiles a reference code that may be used to identify (although not necessarily uniquely) this hypothesis</summary>
<returns>A string in the format XX_XXX_YY_YYY... where XX_XXX is a singe component with XX the component and XXX the implementation.</returns>
<summary>Compiles a reference code that may be used to identify (although not necessarily uniquely) this hypothesis</summary>
<returns>A string in the format XX_XXX_YY_YYY... where XX_XXX is a singe component with XX the component and XXX the implementation.</returns>
val resultSet: ModelSelection.ResultSet.ResultSet<'a,'b>
module Diagnostics
from Bristlecone
<summary>Diagnostic techniques for determining the suitability of results obtained with Bristlecone.</summary>
<summary>Diagnostic techniques for determining the suitability of results obtained with Bristlecone.</summary>
module Convergence
from Bristlecone.Diagnostics
<summary>Convergence diagnostics for monte-carlo markov chain (MCMC) analyses.</summary>
<summary>Convergence diagnostics for monte-carlo markov chain (MCMC) analyses.</summary>
val gelmanRubin: nMostRecent: int -> subjectId: string -> hypothesisId: string -> result: ModelSelection.ResultSet.ResultSet<'subject,'hypothesis> -> Diagnostics.Convergence.ConvergenceStatistic seq option
<summary>Calculate the Gelman-Rubin statistic for each parameter in the given `ResultSet`. The statistic tends downwards to one, with one indicating perfect convergence between all chains.</summary>
<param name="nMostRecent">How many recent iterations to use from the trace.</param>
<param name="subjectId">The subject identifier</param>
<param name="hypothesisId">The hypothesis identifier</param>
<param name="result">A result set (of 1 .. many results) for a particular subject and hypothesis</param>
<returns>If more than one replicate, the R-hat convergence statistic across replicates</returns>
<summary>Calculate the Gelman-Rubin statistic for each parameter in the given `ResultSet`. The statistic tends downwards to one, with one indicating perfect convergence between all chains.</summary>
<param name="nMostRecent">How many recent iterations to use from the trace.</param>
<param name="subjectId">The subject identifier</param>
<param name="hypothesisId">The hypothesis identifier</param>
<param name="result">A result set (of 1 .. many results) for a particular subject and hypothesis</param>
<returns>If more than one replicate, the R-hat convergence statistic across replicates</returns>
module Option
from Microsoft.FSharp.Core
val iter: action: ('T -> unit) -> option: 'T option -> unit
module Convergence
from Bristlecone.Data
val save: directory: string -> result: Diagnostics.Convergence.ConvergenceStatistic seq -> unit
val nStepFits: (string * string * CodedMap<ModelSystem.FitSeries * Statistics.NStepStatistics>) seq
val saveAllRMSE: directory: string -> results: (string * string * CodedMap<ModelSystem.FitSeries * Statistics.NStepStatistics>) seq -> unit
<summary>Saves a full ensemble overview of the root mean squared error of n-step ahead predictions.</summary>
<param name="directory">The file directory to save to</param>
<param name="results">A sequence of subjectID * hypothesisID * n-step result</param>
<summary>Saves a full ensemble overview of the root mean squared error of n-step ahead predictions.</summary>
<param name="directory">The file directory to save to</param>
<param name="results">A sequence of subjectID * hypothesisID * n-step result</param>