Updates, additions and fixes

Current stable version: v1.00 (main downloads page)

v1.00 (31-May-2024)

This major release contains a number of new analysis features as well as support for a Python interface to Luna, lunapi. It also contains a number fixes and smaller additions.

New interfaces

a new lunapi Python package provides bindings to the Luna library; it also contains a number of convenience functions and an interactive viewer for signal and annotation data

New commands

new EDF-MINUS command to collapse gapped EDF+D whilst aligning records, stage annotations and epochs
new PCOUPL command for generic phase-event coupling
new SVD commad for time-series PCA via SVD

Annotations

the ANNOTS command now takes annot classes to output only a subset (w/ wildcard character * allowed to subset, in form root* only)
date formats: two new options read-mdy-annot-dates and read-mdy-edf-dates allow for non-European (month-day-year) format dates, instead of the default day-month-year (adopted as European Data Format files specify this convention). The first form relates to annotation files (`.annot); the second form relates to values in the EDF header (i.e. if they have been incorrectly specified).
OVERLAP now has option to output a shuffled set of annotations (given the full shuffling scheme - shuffle, event-based, constrained, background-filtered, etc): e.g. add-shuffled-annots=A1,A2 adds s_A1, s_A2 as new, shuffled versions of A1 and A2; the default tag s_ can also be altered with add-shuffled-annots-tag=s_

Epoch definitions

EPOCH now takes fixed and trunc ; output gives FIXED_DUR from EPOCH
generic epochs can now be shifted left/right (backwards/forwards) using shift (-ve/+ve values)
new EPOCH outputs: TOT_DUR, TOT_PCT, TOT_REC, TOT_SPANNED and TOT_UNSPANNED
added SEGMENTS largest annotation (largest1), (largest2) etc; the 1,2,3,... index is in the instance ID of the annotation (class = largest); also, adding e.g. requires-min=120 enforces that largest segments must be at least 2 hours -- 1,2,3 is in INST ID

Spectral analysis

fixed MTM options; added speckurt for fbins (MTM)
PSD and MTM now have band (=T/F) to drop band output altogether
allow skip-bands (e.g. skip-bands=SIGMA,GAMMA,TOTAL) for PSD to omit certain bands from the output
to support ISO analyses: PSD now has an add option to emit (currently 1Hz) values (requries EPOCH dur=4 inc=1 and PSD segment-size=4). This creates a new 1 Hz time-series in the EDF.
COH now allows generic epochs - as long as they are of fixed size (which requires adding fixed along with annot when applying the EPOCH command)

Spindle/SO analyses

now so-annots (instead of annots) is needed to emit slow oscillation annotations (as is called by SPINDLES)
SPINDLES/SO coupling issue fixed if a record is above epoch span

Data manipulation

COPY allows new to specify a full name (when used with a single channel); pretag adds the tag to the front (better for [xx][1:8]); also, tag no longer adds "_" in between, this must be explicitly specified, e.g. tag=_TAG

Scripting: filters and sequence expansions

all sig options allow [inc] and [-exc] matching:
```
sig=C3,C4,F3,F4[F]
```
implies
```
sig=C3,C4
```
i.e. must match a C character. This can be useful when working with larger sets and you want to select a subset of sig=${s} where ${s} may be a very large but structured list
we now allow for more generic [][] expansion syntax, i.e. [seq1][seq2] where the terms or sequences are either in the form n:m (where n and m are integers, with n <= m) or as a commad-delimited list.
no nesting of terms if allowed, but they can be applied sequentially:
```
   ${s=[x][a,b,c]} & PSD [${s}][1:5]
```
implies
```
xa1,xa2,xa3,xa4,xa5,xb1,xb2,xb3,xb4,xb5,xc1,xc2,xc3,xc4,xc5
```
Note: conditions are evaluated first, based on prior set variables; then line-by-line we evaluate a) swap in variables, wildcards,replace variables, b) expand sequences
fixed a bug that occurred when using [n:n]

Misc. fixes and changes

added output-both option to CORREL
fixed TOT_DUR_HMS when >24 hrs in HEADERS
WRITE now resets EDF start date if needed (i.e. when changing start time if collapsing to standard EDF)
fixed the ids=<list> option (to swap in alternate IDs on reading a sample list), as it was previosly not working
POPS ignore-obs-staging option is now fixed
SIMUL can zero-out frequencies above or below a certain value (i.e. after interpolation) with zero=lwr,upr

v0.99 (5-Dec-2023)

This release (leaping from v0.28 to v0.99 in a single bound) contains a large number of incremental fixes and additions, as well as a few more major features including: 1) automatically generated hypnogram-based annotations, 2) reworked MASK interface to simplify masks based on multiple annotations, 3) allowing generic (variably-sized) epochs based on annotations, 4) a new framework to implement model-based prediction given PSG-derived features, in the PREDICT command, also suppored in Moonlight, 5) epoch-wise analysis using the mutlitaper command MTM, 6) ability to generate new annotations conveniently on-the-fly based on pairwise contrasts of existing ones, 7) the Moonbeam utility, that allows NSRR data to be pulled directly into Moonlight (for NSRR users) and 8) a prototype of a new utility for viewing hypnograms (Hypnoscope).

Hypnograms

major: the HYPNO now has the annot option, which adds a series of hypnogram-based annotations, e.g. that can be used in subsequent mask operations, or output, etc.
HYPNO epoch-level output now has PRE, SPT, and POST flags (0/1) to reflect epochs that are pre-sleep, during the sleep period time, or post-slep; also, the base-level output has two new variables, PRE and POST to give the recording time pre- and post-sleep (anchored on EDF/recording start/top times, rather than lights off, i.e. the difference between SOL and PRE is that SOL is defined relative to lights off.
if a recording is entirely LightsOut (L stage annotation), then Lights Off and Lights On times will be now be correctly shown as NA
changed HYPNO FLANKING_MIN to FLANKING, which is now an epoch count
fixed a minor issue when running with a gapped (EDF+D) recording: added new variables TGT (total gap time). Gaps count as missing and so do not enter the denominator for metrics (i.e. not in TRT or TIB). For cycle-level output, NREMC_MINS includes gaps, whereas NREMC_N does not include gaps.
epoch-level elapsed time outputs are now encoded from the start of that epoch, i.e. 0 to N-1 rather than 1 to N

Prediction models

major: initial implementation of a new PREDICT command

Moonlight

major: added a new Moonbeam feature to directly pull NSRR data into Moonlight for NSRR users
major: added a new Models tab to support model-based prediction of metrics, initially an estimate of adult "brain-age" based on the NREM sleep EEG, implementing a model described here

Hypnoscope

an initial implementation of the Hypnoscope utility for viewing multiple hypnograms and generating hypnogram-based metrics

Signal processing

major: the MTM command now allows epochwise analyses (as well as segments within those epochs); epoch-spectra/epoch-output and segment-spectra/segment-output control levels of epochwise and segmentwise outputs; renamed epoch-slopes to segment-slopes
MTM now precomputes tapers, which greatly speeds up epochwise analysis
added bandpower outputs for MTM (B-stratified)
added kurt/kurt3 options to PSD, to give epoch-level kurtosis (of power averaged over channels)
added speckurt/speckurt3 options to MTM, to give segment-level kurtsosis (of power averaged over channels, within epochs if any epochs are specified). Also, added the alternate-speckurt to support an alternate definition of these (only used for Sun et al model)
added ratio/ratio1 options to both PSD and MTM to give ratios of bandpowers for prespecifed pairs of bands; ratio1 implies a/(1+b).
both PSD and MTM now allow setting user-defined power bands on the fly, e.g. PSD sigma=12-16. (Previously, this could only be done as a special variable setting, which would be constant across all analyses.)
added a new PSD_CV output (w/ both dB and sd specified) for the PSD command; this uses the calculation for coefficients of variation assuming log-normal data, i.e. CV=sqrt(exp(s^2)-1) where s^2 is the natural log-scaled variance.
new more efficient cross-correlation XCORR implementation and new outputs

Artifact and alignment utilities

new EDGER command, designed to identify leading/trailing portions of recordings likely to be artifactual
new ALIGN-EPOCHS command
new ALIGN-ANNOTS command
new INSERT command, with new FFT-based cross-correlation analysis to align signals from different EDFs and insert them, optionally adjusting timings

Filters:

FILTER now accepts separate transition band widths (tw) and ripples (ripple) for the lower and upper edges of a Kaiser-window FIR filter, e.g. by tw=0.5,5 and ripple=0.01,0.01, corresponding to lower and upper edges respectively; the new filter is generated by convolution of two separate highpass and lowpass filters with those parameters.
added butterworth and chebyshev IIR filters to the FILTER command
added ngaus option to FILTER to implement narrow-band filter via a frequency-domain Gaussian (parameters: frequency domain central mean and FWHM of the Gaussian, ngaus=11,2)
fft is now the default for FILTER

Masks, epochs, freezes and caches

major: variable-sized ("generic") epochs are now supported by EPOCH, defined based on existing annotations: e.g. EPOCH annot=X. In this context adding else=Y will generate a new annotation (Y) which flags not being in an defined epoch (based on X) but respects gaps. Multiple annotations are allowed: annot=A,B,C. Labels are given to epochs in the output now.
for either conceptual or practical reasons, not all command currently support variable-sized annotations (e.g. COH or HYPNO) but this is flagged if attempting to run such a command in the presence of variable-sized (generic, annotation-based) epochs.
major: reworked the primary MASK command to allow multiple annotations to be specified with implied OR / AND logic: e.g. MASK ifnot=N2,N3. The type of logic can be controlled by explicitly specifying ifnot-any=N2,N3 or ifnot-all=N2,arousal to support OR versus AND logic respectively (the former is the same as the default ifnot). These are applied to all masks, e.g. mask-if, etc, as described here.
generic (annotation-based) masks can now use wildcards to complete annotation names: MASK ifnot=artifact_*
new MASK syntax ifnot=+annot means to match the epoch only if it is completely spanned by that annotation. (One cannot use both + and * symbols together, however.)
added the option EPOCH table which dumps ouutput (same as verbose) but does not change the epoch structure, i.e. it only gives output. The extra argument masked will make the table output for both masked and unmasked epochs; the verbose option also gives more output than previously
the CACHE command has a new record option, which can be used to pull any arbitrary outputs into the cache (e.g. for use with a subsequent PREDICT command)
the THAW command has a new preserve-cache option so that the cache is not over-written; i.e. it can be allowed to build up between successive FREEZE / THAW operations
the RE command has a new preserve-cache option, which does not wipe the cache when restructuring a dataset
added a new C2A cache to annotation command

Annotations

major: added a new MAKE-ANNOTS command, to generate new annotations based on pairwise comparisons of existing ones (union, intersection, if overlapped, if not overlapped, using the syntax A|B, A*B, A+B and A-B respectively); this also includes special behaviors if given the options epoch, epoch-num, flatten or split
the WRITE-ANNOTS now accepts prefix to match w/ all annots starting with that; useful for prefix=h_
added raw-annot (and raw-annots) to specify subset of annots to load - but unlike annot this does not apply sanitization first (if that is the global default)
added bins and bins-label options to the S2A command: given bins=min,max,n to make n bins of equal span, to make annots B1, B2, etc... Bn (or bins-label instead of B)

Interval-based analyses:

the OVERLAP command now accepts * wildcards in seed/other/fixed specifications to match annots seed=sp*,so* i.e. matches sp_15, sp_11 etc
the OVERLAP command now has contrasts, event-perm, offsets options. Now one should add seed-seed=T to get seed-seed pairs; adding seed-seed pileup is now turned off by default, unless pileup=T is added
the OVERLAP command now accepts rp=<annot>|<tag>,... to get rp_tag=xxx from meta data for that annotation, in which case it will set a 0-duration time-point at that position

Spindle/slow oscillation detection:

added COUPL_ALL statistics to SPINDLES, which performs spindle/SO analysis under both default (only using spindles that overlap a detected SO) and all-spindles (calculating COUPL_ANGLE and COUPL_MAG metrics based on all spindles); if the all-spindles option is explicitly set, then these two sets are identical, and only a single set is reported).
Slow oscillation output (SO/SPINDLES commands) is changed: now SO_SLOPE is output by default instead of SLOPE_NEG2. The other slopes are only generated w/ the verbose option. Additionally, it now doesn't output _neg and _pos halfwaves as annotations, but instead adds rp_mid rp_pos and rp_neg as relative positions within the interval (scored 0..1); the OVERLAP command is now able to use these rp (relative position) terms. Finally, SO annotations do not output dur now, only frq.
SPINDLES annotations (from the annot argument) now add rp_mid (as a 0-1 proportion) to indicate the mid/peak of the spindle, i.e. rather than in the old time-point based form mid=tp:123456789
the SPINDLES command has a new cache-peaks-sec option
SPINDLES now gives a message if fc is out-of-range given q. In this case, added the option noq (i.e. equivalent to setting q=-999)
fixed the SPINDLES option stratify-by-phase to avoid double-counting

Minor additions and fixes

the SEGMENTS command now outputs the largest segment size (for an EDF+D w/ gaps); also, option largest=L1 adds an annotation spannong the largest segment called L1; in the case of ties, only the last will be added
fixed bug in EDF - when forcing a standard EDF structure, this now updates the internal timetrack correctly, so that any subsequent call to timeline.wholetrace() uses the correct time-points
added min and max options to MINMAX, to clip EDF physical min/max values; can specify both or either; other wise MINMAX sets all sig channels to the be same PMIN/PMAX and DMIN/DMAX. If force is specified with min or max then that value is always set; otherwise, the physical min/max values are only changed if they are smaller, i.e. clipping, not expanding the min/max range
channel locations are now populated by default if not otherwise specified (for a standard 64-channel 10/10 EEG only)
automatically sets any EDF header reserved field characters 5-44 to null (space) (also the first 5 chars if not EDF+C or EDF+D)
new srand=XXXX special argument to set the RNG seed
added REC_DUR_SEC and REC_DUR_HMS to HEADERS which used to be TOT_*. Now, TOT_DUR_SEC and TOT_DUR_HMS reflect the full duration, including any gaps (i.e. if an EDF+D)
the RESAMPLE command has a new downsample argument: only channels with rates above the sr value will be altered
Luna can now correctly take .edfz and .edf.gz files on the command line (versus from sample list)
fixed issue with the CPT command when multiple classes of DV are specified; also added a all-dvars (or dv=*) option to select all named DVs from the DV files; also, can cluster based on time (T) as well as F, CH, CH1 and CH2; because of this, STAT output variable is now the t-statistic, and T means any time-level stratifier
the STATS command has renamed MEDIAN.X to X_MD; also added X_MN
STATS has a kurt3 to specify unadjusted kurtosis values (i.e. veresus excess values), such that X~N(0,1) has an expected value of 3.0, not 0;
PEAKS now output annotatons (annot) as well as cache; can include w to add window (sec) around each point

Interal changes/library upgrades

upgraded to Eigen library to 3.4.0
upgraded to sqlite library v3.41.2
reorganized some of the codebase, in particular, moving most TF-analyses into single spectral/ folder

v0.28 (10-Apr-2023)

Moonlight

new interactive Moonlight viewer (public demostration host: http://remnrem.net)
extended tutorial that uses Moonlight to recapitulate the prior Luna/lunaR tutorials

Vignettes

new vignette on merging EDFs and working with EDF+D files
new vignette on how Luna expects time/date information in annotation files

Major new functionality

new FREEZE and THAW commands to make and revert to prior snapshots of the in-memory dataset
new TABULATE option to summarize channels with discrete sets of values, e.g. body position
new MOVING-AVERAGE command
major improvements to --merge which can now handle gaps between consecutive files and will generate EDF+Ds
new add option for MTM to generate a new signal (with the same sample rate as the original); also, MTM now outputs estimates of the spectral slope as well as PSD and IRASA.
new SET-TIMESTAMPS utility command
new annot option for the SEGMENTS command; as demo'ed in this vignette, it is useful when making standard EDF from EDF+D files
for WRITE-ANNOTS we now make as default no-specials=T; i.e. add options specials instead of no-specials; and likewise headers from no-headers. We've also added dhms and minimal options.
added impulse option to [SIGGEN]( (T,A,D) and add not clear; makes new channle, or updates/replaces an existing one.
new REMAP command to perform annotation remapping on _already-loaded_x data;
new EXE option representative to extract K examplar epochs via a clustering heuristic based on permutation distribution clustering

Minor changes/additions

added starttime and startdate command-line special variables to force EDF header changes (prior to attaching annotations)
new anon=T special variable to set EDF headers to null values upon loading; this is similar to runnong ANON, although ANON allows a few extra options); as with starttime and startdate, this setting is enacted before attaching annotations (i.e. and so can influence how annotations are aligned)
added support for dates in annotation files, e.g. dd-mm-yyy-hh:mm:ss, as illustrated in this vignette
added the pct option to STATS to avoid calculation of percentiles pct=F. Also min=T will only give the mean.
SOAP now respects lights on (L) epochs, i.e. unlike for missing (?) epochs, it completely excludes them
new offset option for WRITE-ANNOT
Luna now gives a warning message to the console if looking at an etnire signal (e.g. from FILTER) but some epochs are masked - i.e. as this means that those epochs will be included in that step (if not running RE beforehand to remove them)
added annot feature to DUMP-MASK to create annotation based on the mask (as well as output=F and annot-unmasked=T)
added file option to RENAME - takes a file of tab-delimited old/new channel name pairs instead of sig/new from the command line
added the xbg option to specify intervals to excise certain intervals from the background in OVERLAP
offset option of EPOCH can now take hh:mm and hh:mm:ss arguments, as well as elasped seconds
now gives an error if specified filename for a new EDF file (e.g. from WRITE) is the same as the input EDF. (This was allowed previously, but could cause problems and lead to corrupt outputs...)
anchor argument for SPINDLES to specify between -1 and +1 for spindle start stop (0 = midpoint) for SO-coupling analysis. If not specified, default is the point of maximal CWT (i.e. typically, but not necessarily near the middle); offset=0.1 (-N ro +N seconds, default = 0) adds offset to above anchor for SO-coupling, i.e. can be combined with couple, i.e. look at SO phase and overlap 0.2 seconds before the start of a spindle; Gives new ANCHOR strata in output for coupling analyses. Can specify multiple anchor/offsets.

EEG microstates --label-maps command now outputs

ID  KT  FLIP    K1  MAPPED  SPC
.   D   0   1   1   0.994548052142096
.   C   0   2   1   0.972204262924326
.   A   0   3   1   0.988131018834142
.   B   0   4   1   0.996294406621555

and (will) add a message if match is not highest correl SPC OPTIMAL == 0

also, it now allows to matching on the minimum sum (1-r)^p where p = 2 by default; this is now also the default stat for compare-maps (perm.) tests; added verbose option for info to the log
for EEG microstate analysis, added a grouped option to allow A/a --> 'A' two-group mappings (i.e. case-invariant analysis, if K = 4 + 4 = 8

Misc. fixes

fixed issue with ECG-SUPPRESS when high sample rate used (insufficient smoothing -> no R-peaks detected)
HYPNO now outputs proper epoch time, i.e. if used EPOCH align so that epochs are not starting from 0; (as is, it assumes epoch 1 starts at 0 sec); also note that MINS is the elapsed time since epoch #1, not necessarily EDF start under these conditions
annotation class/instance delimiter has been changed to have a default value of : (rather than / as before)
NSRR automatic-remapping is now turned off by default: enable with nsrr-remap=T; added annot-remap to separately control remapping of stage annotations
automatic sanitization of annotation labels is now on by default
changed behavior of keep-spaces and sanitize to be similar for both channel and annotation labels; also, labels are now always trimmed on both whitespace and sanitized insert character (underscore by default)
stopped the HEADERS command from writing out EDF Annotation channels
changed lunaR to attach EDF+ annotations on attaching an EDF+

v0.27 (27-Sep-2022)

In addition to multiple smaller fixes and modifications, this release contains a number of major new additions including: 1) revised macro-architecture summary statistics, 2) the SOAP and POPS tools for automated prediction/evaluation/manipulation of sleep stages, 3) new tools for assessing temporal coupling of annotation events, and 4) new signal processing tools including IRASA. Major changes are noted below:

Macro-architecture

revised and expanded hypnogram summary metrics, and improved documentation for the HYPNO command including: bout number/duration metrics, better incorporation of lights out markers, a heuristic to handle excessive leading/trailing WASO, revised calculation of total persistent sleep time (now TST_PER), renamed sleep efficiency/latency (SE, SME and SOL), added final wake time (FWT).
added a new annot-cycles option for HYPNO, to add an annotation indicating which NREM cycle an epoch belongs to
fixed an issue when HYPNO is called twice on the same EDF
added first (mins) and anchor (T0, T1 or T2) options, to report hypnogram stats for only the first N minutes (setting the rest to L); also, added last (which takes T4 (default) or T5 or T6 as anchor values); also, clock which, along with first specifies an arbitrary anchor, based on the clock time; this may come before the EDF start; the variable SHORT will be flagged if the requested window does not fully overlap the available staging
added elapsed stage duration times, i.e. seconds after anchor that we see N minutes of N2 sleep, etc. If indiv does not have N mins of N2 sleep, a flag will be set; SS x DUR.. T and SHORT. Note that E_* epoch level output gives the elapsed duration at the START at that epoch; in contrast, this is indexed at the end...
fixed a bug impacting some of the elapsed time metrics from HYPNO (E_* etc) if the recording started at or after midnight
added and documented the new SOAP, REBASE and PLACE stage evaluation/manipulation commands
added a new automated staging command: POPS - POPulation-based Staging that implements a flexible syntax for model specification. Internally, POPS uses Microsoft's popular open-source C/C++ LightGBM library for machine learning
Internal: have depreciated the older (but still experimental) SUDS stager; previously, we had added some new features, including a canonical correlation based routine.

Data manipulation/handling

major new implementation and syntax for the CANONICAL command
new SET-HEADERS command to directly edit EDF header values
new ENFORCE-SR command
new fix-edf option, to allow (i.e. ignore) truncated final records of a trivially corrupted EDF
new --merge command to aggregate multiple EDFs
the --repath utility now accepts . as first character, meaning always append the second argument if (and only if) the sample list has a relative path
new SET-VAR command to set individual-level variables from within a script
new RENAME command to rename channels (unlike signal aliases, this command can accept variables)
HEADERS now outputs STOP_TIME
added pairwise option to REFERENCE
new RECTIFY command
new REVERSE command

Micro-architecture/signal processing

new IRASA command, to implement Irregular-Resampling Auto-Spectral Analysis
new prototype/alpha implementation of detrended fluctuation analysis via the DFA command
added Petrosian fractal dimension and permutation entropy (pe, with pe-m and pe-t options) to SIGSTATS
new prototype ASYMM command to evaluate (stage-specific) (regional) asymmetries (e.g. inter-hemispheric differences) in derived signal metrics
new Z-PEAKS command to make annotations based on peak finding in signals
added segment-median and segment-sd to PSD
added precomputed function to SPINDLES
added ch-median ch-epoch ch-spatial-threshold ch-spatial-weight to CORREL, which now also prints out disjoint sets of "high corr" channels (CHS)
added OTSU and --otsu commands; changed Otsu implementation
added the --ftt command
added the same-channel option for TLOCK to constrain output for when CH == sCH

Annotations

new OVERLAP command to evaluate enrichment of overlap/locality of annotations, in both single- and multi-sample contexts; it can also produce new annotations based on overlap with other annotations
added the collapse argument for WRITE-ANNOTS which can be used with an EDF+D if an EDF will be output (i.e. changes the times as if collapsing discontinuities)
currently largely for internal development use, a new A2C command to convert sample points (ints) in annotation meta-data to a cache store
EPOCH align now works for EDF+D - will align the first epoch at the start of each segment, i.e. still assuming uniform epochs after that (within each segment)
skip-edf-annots now still reads time-track for EDF+D from EDF Annotations channels
added the numeric-inst argument to A2S, to set a signal to a numeric-valued instance ID (i.e. not just 0/1)
when creating a set of epoch-level sleep stages, Luna will automatically detect if the stage annotation are 0-duration change-points (as is often the case in EDF+ stage annotations) and will extend those new signals up until the next stage annotaiton encountered (or the end of the recording)
alternatively, unless assume-stage-duration=F, if sleep stages have 0 duration (e.g. markers for change) assume they have the default epoch duration length (currently only for EDF+ embedded annotations)
added the add-ellipsis option (for zero-duration annotations) -- this impacts WRITE-ANNOTS .annot format files only
for EDF+ annotations, added edf-annot-class special variable, to make these annots classes (instead of edf_annot_t instanes) e.g. edf-annot-class=N1,N2,N3,R,W

Microstate analysis

the EEG microstates command MS has a new w argument for --kmer analysis, which performs local shuffling of microstate sequences (i.e. only within N states)
the microstate --kmer command will now automatically splice out ? states, and reduce adjacent states to one e.g. AAA??AABBBCCC -> ABC
a new --label-maps command assign labels for a microstate map sol file, given a labelled canonical/template sol file (template, file, new)
new --cmp-maps command to test for differences between maps, either at the group or individual (one versus all others) level, using a permutation procedure
new --correl-maps to print spatial correlations for a map (from a text file)

Misc

added --options command to allow command-line functions to take args either from stdin OR from command line args (following --options)
changed parameter parsing to include added signals when no sig specified (i.e. match all) in the case when sig was specified as an initial, top-level special variable
param files have +group and -group flags. // +group include only if matches group, otherwise skip // -group exclude if matches group, otherwise parse
IF and IFNOT (or ENDIF / FI )
fixed bug in within-record interval offset calculation for EDF+D when gaps present that are fractions of a record duration (i.e. and so record start time-points are no longer multiples of the EDF record size)

v0.26 (29-Nov-2021)

New functionality

TRANS supports arbitrary transformations of signal data
SIMUL simulates time-series data given a power spectrum
PSD now has peaks and slope options
FFT performs basic discrete Fourier transform (DFT) via the FFT
HEAD shows one epoch of data (requires same SR for selected channels)
ZC mean-centers signals, and ROBUST-NORM performs robust normalization (by median & IQR)
upated the EMD (empirical mode decomposition) command
--repath convenience sample-list function
prototype ALTER command to perform reference-channel, regression-based artifact removal
peaks and slope options for PSD and MTM
REBASE, which adopts the SOAP framework to (probabilistically) re-estimate sleep stages using a different epoch duration (e.g. to translate from 20-second manually scored datasets to 30-second epochs) given a) manual staging in the original epoch duration, and b) one or more signals (i.e. EEG) that are expected to encode sleep stage information well (i.e. have a high kappa from the original SOAP command).

Annotation format modifications/extensions

.annot format now allows key=value meta-data to be specified; also, you can have fewer meta-data terms than expected (but not more), assuming order is as header; all fields still are required to be in the header; now WRITE-ANNOTS always writes meta-data as key=value pairs
new + and - options to turn on/off @includes (e.g. alias, remap); note, the variable must be specified before the relevant @include on the command line, i.e. as command-line arguments are processed in left-to-right order
added a check for pipe (|) characters in annotation primary names (in a remap)
annotation times can now include AM/PM modifiers (otherwise assumes 24-hour clock)
new annot-whitelist option, such that annotations are only accepted if they appear in the remap list (either as aliases or primaries);
new annot-unmapped option to skip if the annotation is on the whitelist (i.e. complement of behavior with annot-whitelist alone)
if remapping an annotation to ABD/DEF|XYZ form (i.e. with a / delimiter) then for the class XYZ, it is set to ABC and instance ID is set to DEF. If there was an existing instance value (non-null), a text meta-tag of _inst= is added. The delimiter character can be changed from / with class-inst-delimiter=X (although this only works for annot files currently, and is not generally recommended, i.e. as sanitization of labels respects / for annotations, etc)
added combine-annots option to merge class and instance identifiers. It accepts a character argument but is _ by default ); this sets the annotation class to class_inst (and sets inst to '.' )
added the skip-sl-annots option to skip all SL-attached annotations; i.e. if wanting to only load annotations from an alternate, explicitly referenced annot file
added the interval option to the EVAL command, to generate new interval-level annotations based on eval expressions

EEG microstates

MS has new add-spc-sig option to add spatial correlations as new EDF channels (instead of 0/1 binary variable, as per add-sig)
MS has new canonical option to specify a file definining canonical microstates
MS solutions now always have a header row; you cannot extract based on sol=file,A,B,C,D; also, 'unassigned' states are labeled 1,2,3, etc not A,B,C,...
new --cmp-maps command to compare (spatial correlation) EEG microstates
--kmer command takes options req-len (only analyse first N sequences) and indiv-enrichment

Other fixes, minor modifications and new features

added ability to specify an empty EDF (--nr, --rs and filename equals '.' )
added 1st, 2nd, 5th and 10th percentiles to STATS
added transition frequency to slow oscillation output (SPINDLES, SO)
fixed bug in flanked mask option (e.g. flanked=W,1)
made CANONICAL definition file format more flexible: 1) it now allows whitespace, not just tab-delimitation; 2) only the first three fields are required now; if not given, the latter fields/columns will be set to .; 3) CANONICAL now respects the order of canonical signals (i.e. rather than processing things alphabetical); this allows multi-stage definitions, e.g. to first map S1, then S2, and then apply a rule such as S3 = S1 - S2.
eval syntax takes { and } instead of ' to delimit strings; it allows nesting, but can also be handy on the command line (i.e. if already using -s '' form)
added drop and keep options to the PSC command
added import=file.txt command to [CACHE]( to read from destrat output; can take factors and v param (as well as required cache=)
added MASK epoch=all to set a MASK but have it all empty ; i.e. to trim records not in an epoch
(for .annot only) added align option : given list of annots (or *) for all, align-annots-on=N1,N2, etc... if not specified, find first instance of this annot, then align with 1 second boundary (or align-annots-res=X if given) align /all/ annots with this offset (bound at 0); i.e all records beforehand will be skipped if subsequent MASK/WRITE commands are applied
added pick option to SIGNALS to pick first of pick=a,b,c that is present, and drop the rest; can map with 'rename' to rename the pick
allow sample-lists to have a comma-delim list of annotations, or '.' to denote no data; in this way, we can have a fixed width sample list (i.e. three tab-delimited columns), making it easier to parse (case in point: lunaR lsl() was broken if fewer than 3 columns were found after first five rows, reflecting how R read.table() works)
CANONICAL does not now need an explicit GROUP to be specified; the file must still have a first col, it is just ignored now; also, new 'drop-originals' option to drop all original (non-CANONICAL) signals after making the new signals; matches case-insentive
changed epoch-check to accept number of .eannot epochs that are different from expected; default is 5; only stops is absolute greater than this; otherwise writes warning to log; i.e. set to 0 for an exact match
CONTAINS can now skip to the next EDF (rather than alter the return code), if the option skip is given
check for whether an ID contains the ID-wildcard character (by default, ^) and reports an error if it does; added the wildcard option to specify an alternate character

v0.25.5 (24-May-2021)

Major new functionality

Major changes to the epoch-wise artifact detection via SIGSTATS, which is now implemented via the CHEP-MASK and CHEP commands, as illustrated in this vignette
Significant speed improvements for spectral analyses (PSD and especially MTM) and other (e.g. PSC, SUDS, ICA, CPT) commands through code refactoring, tweaking our use of FFTW and the incorporation of the Eigen matrix library
A beta-version of our new Luna-powered NAP (NSRR Automated Pipeline) cloud-based portal, described here, and available for testing from this URL http://remnrem.net/nap/ populated with public Sleep-EDF dataset.
Principal spectral components analysis via the PSC command
Phase slope index connectivity via the PSI command
Epoch-wise spherical spline interpolation via the INTERPOLATE command
Independent components analysis via the ICA and ADJUST commands
EEG microstate analysis via the MS command
Time-series clustering of epochs and/or channels via the EXE command
Cluster-based non-parametric linear models for association analysis via the CPT command

Other new commands/functionality

added DUPES command to find flat signals and digital duplicates
Added cache-metrics options for PSD, MTM, COH, PSI, SPINDLES and SO (i.e. primarily for use with the PSC command)
SO caches negative peaks (with the cache option)
New MEANS command to give signal means conditional on annotations
New special variables describing topographical EEG 64-channel groups, e.g. ${anterior}, ${midline}, etc,
Find peaks in signals and subsequently produced peak-locked averages across other channels, via the PEAKS command and the TLOCK command
Cross-correlation and phase synchrony metrics, via the new TSYNC command
New A2S and S2A commands to convert anntotations to binary EDF signals channels, and vice versa
Restructure EDFs to align records, annotations and epochs via the ALIGN command
Added the EDF command to convert EDF+C and EDF+D to standard EDF
Added the LINE-DENOISE command, using spectrum interpolation to reduce line noise
Added the SEDF command to produce thumbnail-like summary EDFs (to be used in future NAP iterations)
Added a simple SIGGEN command to generate test signals (currently only sine waves)
Added the CONTAINS command that indicates (via Luna's exit code) whether one or more signals are present in the EDF (i.e. helps guide programmatic driving of Luna pipelines, to be used in future NAP iterations)
new output/options for the HYPNO command: 1) stage transition counts and probabilities, with PRE and POST factors; also with conditional probabilities P_PRE_COND_POST and P_POST_COND_PRE; 2) it now outputs epoch-level stage transition information in full and with different naming scheme: TR_NR2REM, etc and TOT_NR2REM. 3) FLANKING_MIN and FLANKING_ALL instead of FLANKING_SIM; 4) option flanking-collapse-nrem (default T), i.e. flank based on any NREM stage (1, 2 or 3) 4) option req-pre-post, to only consider stage transitions that have FLANKING_ALL >= x for the POST stage; defaults to 4 (2 mins). Added 'CONF' and OTHR outputs.

Minor modifications/fixes

the epoch argument of MASK accepts end as the final epoch number
added the force-edf and EDF+D options to WRITE to force writing as EDF (or EDF+D); this command also now properly supports EDF+C, EDF+D files properly (e.g. in how the EDF start time is changed, etc)
one can now add generic canonical signals via CANONICAL; it will not change the SR if set to missing (.); the canonical signal definition now includes units
channel types now include specific reference and independent component types; further, channel type variables are now automatically updated after adding new channels (e.g. added IC to types; when channels added, now they are typed (i.e. the variable is updated by any commands that add channels, ICA --> ${ic} will be available afterwards to match on IC_1, IC_2, etc).
we now allow .eannot to be attached with EDF+C but not EDF+D files
SPINDLES now has an annot function to generate an internal annotation track (which should be used by WRITE-ANNOTS but can also be used by, e.g. TLOCK, etc. The old ftr format is now retired.
all commands that use epoch-level sleep stages (SOAP, HYPNO, STAGE, etc) will note if an epoch has multiple spanning stage annotations (i.e. which might happen if stages and epochs are not temporally well-aligned); it now reports tje CONF variable describing the overlap
fixed an issue in selecting ranges when annotations do not align with sample points (fixed for continuous EDFs; need to check whether EDF+D requires additional tweaks)
added the epoch option to HYPNO, which now give less verbose output by default
added offset and align to EPOCH
added annot output for the SOAPcommand
added a regional MASK which masks epochs surrounded by masked epochs
added annot, hms and no-specials options to WRITE-ANNOTS
added date-time support for annotations (in clocktime_t) and dhms flag for WRITE-ANNOTS
added test for non-integer sample rates
now the HEADERS command respects the sig option; also, some changes to variable output names
fixed the mkdir system call during WRITE for the Windows platform
the ANON command now conforms with EDF+ specifications for the null ID
added the root option to the ANON command, to specify "dummy" IDs, in the form root_N where N is 1, 2, 3, etc. As the N count is always from 1 for a given run of luna, this can be inconvenient if splitting a sample-list and running in parallel: thus the ids option as above is also given.
added the ids command-line option to supply an ID mapper [ old -> new , tab-delim file ]; this can be used with the ANON to set the EDF header IDs
--build has an option to add quotes around file paths if they contain spaces
the CLOCS command now has an option verbose, to dump pairwise channel distances
multiple changes to the .annot file format: allow durations (+seconds); allow hh:mm:ss elapsed time (0+hh:mm:ss offsets); allow fractional seconds in hh:mm:ss specfication; added channel labels and six-column format (but allowing reduced 3-col and 4-col formats for backwards compatibility); headers now optional; allow space/tab or tab-only delimiters; fixed an off-by-one-time-point glitch with clocktime specification; now fully allow 0-duration annotations [a,a); allow ... in the stop field to read until the start of the next; in all imports, times are scalled to 1/1000th second resolution to avoid floating-point nastiness to cause too much trouble; added sep-dp=N option to control the decimal place in .annot outputs;
added fac instead of bin for PSD (but, in general, removing support/documentation around this).
for SIGNALS command, added the req option to require that specified signals are present/kept; in contrast, the original keep option now will not complain if a requested signal is not present (i.e. keep implies keep if present)
added the id command-line option, so that a numeric ID can be specified (i.e. and not be interpreted as a sample-list line number)
added a min option to EPOCH, which gives minimal output: just the epoch count to standard output
added a dump option to STAGE to write stages to standard out (i.e. parallels eannot option, but does not write to a file)
added guess (and eeg) options to the CANONICAL command, to guess csEEG without a file/group specified

Internal

added cache_t for PSC command SPINDLES/SO, and [in flight] adding a COUPL command to separate out spindle/SO detection from coupling analyses
trims trailing whitespace from the physical dimension and transducer type fields in the EDF header; this was already done for channel labels
swapped in functions and classes from the Eigen matrix library for many numerical intensive procedures
added a runs test to the stats helper function library

v0.24 (22-August-2020)

New commands

ALIASES command to list any channel/annotation aliases done for that individual
MINMAX command to set EDF header digital/physical min/max to same values across multiple channels
TYPES command and accompanying options ch-match, ch-exact and ch-clear, which groups channels by types (e.g. EEG, EMG, respiratory effort, etc) and defines correspondong automatic variables (e.g. ${eeg} and ${emg}); the HEADERS command now also outputs a TYPE field
CANONICAL command to generate canonical signals, cs_EEG, cs_LOC, cs_ROC, cs_EMG and cs_ECG
VARS command to output all variables defined for a given individual (designed to be paired with the new individual-level variables feature, via the vars option)
CC command for connectivity and coupling statistics
ACF command to calculate signal autocorrelation

New options/behaviors for existing commands

added SENS and POS variables to HEADERS channel-level output
added flat and max options to SIGSTATS
CWT-DESIGN now outputs FWHM in time and frequency domains, and accepts newer FWHM-basewd specification (instead of cycles); see this manuscript
added the wrapped and fwhm options to CWT
added the fft option to FILTER for FFT convolution
FILTER and FILTER-DESIGN can now read a FIR from a file; also, instead of Kaiser windows to design a FIR, one can now fix the filter order and specify the type of window (e.g. Hamming, Blackman, etc)
added a new option for the REFERENCE command, and the ability to specify . as a reference (i.e. to do nothing, to facilitate automatic processing)
added summaries to CORREL, and improved the speed of this command
allow now a comma-delimited list when defining stage labels for STAGE and HYPNO, e.g. N1=n1,NREM2
added epoch and per-spindle options to the SPINDLES, making this level of output optinal, and omitted by default (i.e. for ExFxCH and NxFxCH)
new added cstats and astats options for SIGSTATS
added the channels option to the DESC command (to write a simple list of all channel labels to standard output)
the ANNOTS command now outputs START_HMS/STOP_HMS and START_ELAPSED_HMS/STOP_ELAPSED_HMS under ANNOT x INST x T output strata, which are the hh:mm:ss version of clock-time and elapsed time since EDF start, respectively

Other changes

default log output is now less verbose (unless verbose=1 set)
small fixes in processing EDF fields: the reserved field is now set to space (rather than null) characters; also, all EDF header fields (including the reserved field) are now checked for being within the 7-bit US-ASCII range 32-126 (any character outside this range is changed to a ? character)
changed behavior to replace spaces in channel names with underscore (_) characters for ease of processing; setting the spaces variable can specify alternate replacement characters; keep-spaces option, if true, means that spaces are retained
added support for individual-level variables to be loaded from a file (via the vars option)
added automatic special variable ${id}, which can act in the same manner as the ^ special character in scripts, i.e. to represent the current EDF ID
added the add= option to turn on/off conditional blocks
added a check that any variable defined in a script does not clash with a special variable (i.e. command line option).
added include={file list} option, to mirror exclude={file list} (nb. you cannot have both exclude and include)
Luna can now read gzipped (compressed) ASCIIs directly
added a check that named TAGs do not clash with existing, internal tags, e.g. F or CH
fixed an issue with 0-duration annotations, e.g. as may occur with marks from EDF Annotations channel
fixed an issue with the ANNOTS command not reporting all output if the EPOCH command hadn't been explicitly called first
Luna now explicitly gives an error message if trying to use an .eannot file (or e:1 notation in an .annot file) with EDF+ (i.e. which may be discontinuous). That is, if the input file is an EDF+, Luna requires annotation formats with exact times (i.e. either seconds elapsed since the start of the EDF, or hh:mm:ss format).
added EDF+ Annotation parsing; Luna now reads all EDF Annotations events as an annotation track, which is automatically combined with any other annotations (e.g. from XML or .annot files). Currently, all annotations have the class edf_annot, which values are instance IDs/label.
added new options to force/skip annotations: force-edf, skip-annots, skip-edf-annots and skip-all-annots.
added support for using braces { and } instead of ' to denote string literals in eval expressions (i.e. which can be convenient if writing expressions on the command line with -s where ' is already used)
changed formatting of -t text table file names; added --tt-prepend and --tt-append options (equivalently, --tt-prefix and --tt-suffix)
SPANNING now works on EDF+D files
HEADERS now reports EDF_TYPE

v0.23 (15-Jan-2020)

new help function (-h command line argument), to list commands, parameters and their output tables/variables
new spindle/slow oscillation coupling permutation option and output variables
provisional support for text-table output mode with -t option, optionally writing gzipped text (with corresponding lunaR ltxttab() function
new option to read data from a plain text file rather than an EDF
automatically generate sample lists with a new --build option, i.e. to recursively find all EDFs and match annotation files in a set of folders
added support to read and write compressed EDF files, via the EDFZ format, as highlighed in this vignette
new scripting features: command files now allow conditional clauses, via the [[var syntax, within-script variable definitions, via the ${var=xyz}syntax, and expanding numeric sequences, e.g. [ICA][1:10] expands to ICA1,ICA2,ICA3,...
new SEGMENTS command to show contiguous time intervals within a discontinuous EDF+
can now write EDF+ files
new experimental commands for multi-channel EEG: CLOCS (to read channel locations), CHEP (channel/epoch masks), INTERPOLATE (to interpolate missing channels/epoch based on spherical splines), SL (to compute the surface Laplacian), and ICA (implementation of the fastICA algorithm).
added individual-ID wildcards in the specification of output database names (-o or -a), i.e. to write to a separate database for each EDF/individual, e.g. -o path/to/out-^.db
COH performance has increased, and it now reports imaginary and lagged coherence; also, can accept sig1 and sig2 parameters to more flexibly specify which pairs of channels are considered
the CHS output factor is now split into two separate factors, CH1 and CH2, for COH CORREL and MI
added SPANNING command to report on "coverage" of an EDF by one or more annotations
added special variables silent to turn off all console logging
added --xml2 command line option for verbose view of XML tree
removed the skip-annots special variable; replaced with two new variables: skip-edf-annots and skip-all-annots
the MASK epoch and mask-epoch parameters can now take comma-delimited lists of ranges, e.g. epoch=1,6,8-10,22
changed the behavior of assume-pm-start; it is now off by default, and accepts a time parameter to define whether ambiguous times are assigned AM or PM values
added START_TIME, START_DATE and (optionally) SIGNALS output variables to the HEADERS command
now make new, unique labels for any duplicate channel labels found in an EDF (e.g. EEG, EEG.1, EEG.2, etc) and writes a message to the log
now check whether the same channel alias points to more than one channel, and if so, will give an error message
can now use hh:mm:ss clock-time format in .annot files (assumes 24 hour; requires start & stop specified)
changed clock-time format from hh:mm:ss to hh.mm.ss for compliance with the EDF spec; Luna can still read clock-times with the colon-delimiter however
new mode (-a instead of -o) to append to, rather than overwrite, output databases
fixed an issue with the MTM command's epoch option, when used with multiple signals
fixed issue with command parameters not being recorded in the output database
added the nsrr-remap option to turn off auto-remapping of NSRR annotations
new RECS and SEGMENTS commands to dump basic EDF record and interval info to stdout
added a fail-list option [note: need to check implementation]
fixed issue with ANNOTS where an annotation was flagged as overlapping a region/epoch is it ended exactly 1 time-unit beforehand (i.e. ignored convention that intervals are internally represented as [start] to [one past the end] of the interval).

v0.22 (31-March-2019)

Initial public release.