Skip to content

Interval annotations

Here we describe the formats of annotation files, how to attach them to EDFs, and how to view and summarize their contents.

Command Description
Luna annotations Overview of annotations in Luna
NSRR XML files Format of NSRR XML annotation files
--xml Quickly view an NSRR XML annotation file
.annot files .annot file format
.eannot files .eannot file format
FTR files Format of FTR annotation files
ANNOTS Tabulate all annotations

Luna annotations

Annotations are a key part of Luna, and in combination with the MASK command and so-called eval expressions, provide a flexible way to manipulate EDFs.

Luna can represent a number of different types of annotation. Annotations may be represented as arbitrary intervals of time (defined with respect to the start of the EDF), or can be specified at the per-epoch level in a number of ways. First, we introduce some terminology:

A class of annotations is a set of arbitrary time intervals (e.g. such as "spindles")

Each instance of a class (e.g. one particular spindle) is defined by a time interval and an identifier (which may or may not be unique).

Optionally, each instance can also have associated meta-data, stored as typed key/value pairs.

This table summarizes these levels of annotation:

Type Level Example concept Specific example
class name Generic set of annotations All spindles spindles
instance Specific instance of a class
defined by a time interval
A single spindle 873.2 to 874.1 seconds
instance names ID (or non-unique label) for each instance Spindle number
(or type of spindle)
spindle-1, spindle-2, ...
(or fast-spindle, slow-spindle)
instance meta data Associated information Duration, amplitude, etc,
for an individual spindle
dur=0.88 amp=12.2 frq=11.1

Annotations can be represented in a number of different file formats, all described below:

  • NSRR-format XML files, where each ScoredEvent parent node in one XML is treated as a distinct annotation class
  • FTR files, where each FTR file is a single annotation class, within which the instance IDs are the row-level labels in that file
  • .annot files, where each name in the header specifies a distinct annotation class
  • .eannot files containing simple per-epoch labels, where each distinct label is treated as a distinction annotation class

Note

Currently masks only operate on class and instance level data, not meta-data. Annotation meta-data is currently only for display purposes, e.g. in Scope. Also, including or excluding certain annotations with the annot option works at the class level only. That is, either all instances of a given class are loaded in, or none are.

NSRR XML files

Luna accepts XML annotation files, as used by the National Sleep Research Resource. These are based on Compumedics Profusion files, as described here. Any files that end in a .xml or .XML extension are assumed to be in this format, and Luna will attempt to read it as an annotation file. Luna maps each ScoredEvent node to a single annotation class.

--xml

To quickly view the Scored Events in a single NSRR XML file (sorted by clock time):

luna --xml my-annotations.xml
.    .      EpochLength 30
0 - 30      (30 secs)   SleepStage  Wake
30 - 60     (30 secs)   SleepStage  Wake
60 - 90     (30 secs)   SleepStage  Wake
90 - 120    (30 secs)   SleepStage  Wake
120 - 150   (30 secs)   SleepStage  Wake
150 - 180   (30 secs)   SleepStage  NREM1
180 - 210   (30 secs)   SleepStage  Wake
210 - 240   (30 secs)   SleepStage  Wake
... (etc) ...

.annot files

Generic annotation files

Text-based, tab-delimited files that can contain either interval-level annotations: that is, internally, annotations that reflect an interval of time along with certain meta-data.

Note

.annot files should be plain-text but need not have the .annot file extension. Any annotation file that is neither .xml, .ftr nor .eannot is assumed to be of generic .annot file format.

An .annot file can describe one or more annotations classes. Each class can have one or more instances, where each instance corresponds to an interval of time and a single row of the .annot file. .annot files require header rows that first define the classes in that file. Each header row starts with a # character and contains between one to three |-delimited fields, which are class name, description and meta-data types respectively. For example

# a1 
# a2 | Unlike the first, this annotation has a description field
# a3 | This annotation also specifies meta-data types | val1[txt] val2[num] val3[bool]

In the contrived example above, there are three annotation classes: a1, a2 and a3. Only a2 and a3 have description fields (these are currently not used, but will feature in Scope as labels for visualizations). The a3 class also expects some meta-data for each instance: three variables named var1, var2 and var3, each with a specified type.

Type Description
num Numeric (i.e. any floating point number)
int Integer
bool Boolean yes/no, true/false (with values y, yes, Y or 1 versus n, N, no or 0)
txt Any text string

Subsequent data rows of the .annot file specify instances of one of these three classes, along with the necessary meta-data in the case of a3. For example:

a1  i1  10.00   15.00   
a1  i2  92.10   105.22  
a1  i3  108.5   123.11
a2  .   e:2
a2  .   e:7
a2  .   e:10    e:12
a3  A   0   30  W   0.88    Y
a3  A   30  60  W   0.98    Y
a3  B   60  90  N1  0.23    N

That is, the each data row must be at least three tab-delimited columns:

  • first column: class name that must match one of the header rows (i.e. a1, a2 or a3 in this example)
  • second column: instance ID that can be unique or not with respect to its annotation class (or even missing, as for a2)
  • third (and fourth) column(s): these define the interval for this annotation instance, as described below
  • remaining columns: if the header specified meta-data for that annotation class, these values must be listed here, in the same order as specified in the header row

As in the example above, there are three different ways to define intervals in an .annot file:

  • start and stop times in seconds in columns 3 and 4 (as for a1 above)
  • a single epoch code in column 3, starting with e: (as for a2 above)
  • a pair of epoch codes in columns 3 and 4, both starting with e: (as for a3 above)

You can mix and match these different formats with the same annotation class. In the example above, we map them to a1, a2, and a3 simply to make the example clearer.

Interval encoding

Defining intervals by start and stop times in seconds is the most flexible: these annotations can take on any values that can be smaller than an epoch, or can span multiple epochs.

Intervals are defined to be inclusive of the start but exclusive of the stop, i.e. the interval from a to b is [a,b). In other words, b is the first point past the end of the interval. Interval duration is therefore defined as b-a. This means that two 30-second epochs specified below are non-overlapping epochs: rather, they are contiguous despite 30.00 appearing in both definitions:

class1  interval1   0.00 30.00 
class1  interval2  30.00 60.00 
Epoch encoding

Epoch encoding is provided as a convenience feature as many annotations are in fact specified in terms of regular-sized epochs and it might be awkward to always have to list the start and stop times of each epoch. These codes (that start with the characters e: to distinguish them from times in seconds) are converted to the equivalent interval when reading the file.

If the value in the third or fourth column starts with the characters e:, then Luna assumes that epoch-notation is being used to specify the interval. By default, Luna assumes non-overlapping 30-second epochs, whereby e:1, e:2, e:3, etc, refer to the first, second, third, etc, epochs. If the fourth column also starts e: then Luna assumes the interval is from the start for the first epoch to the end of the second epoch.

For example, the following two lines specify identical intervals:

class1       instance1       0        30
class1       instance1       e:1
as do these two lines:
class1       instance1       30       120
class1       instance1       e:2      e:4

Different epochs definitions can be specified by explicitly appending the epoch duration (in seconds) and increment (in seconds) as colon-delimited values, as shown in the Table below.

Example Description Implied interval (sec)
e:2 Second epoch; non-overlapping 30-second epochs [30.0,60.0)
e:2:20 Second epoch; non-overlapping 20-second epochs [20.0,40.0)
e:2:30:15 Second epoch; 50% overlapping 30-second epochs [15.0,45.0)

If not specified, the increment is assumed to be the same as the epoch duration, i.e. no overlap of consecutive epochs.

Note

As epochs are defined within the .annot file itself, the actual EDF need not be epoched (i.e. from the EPOCH command). In fact, the EDF may even have epochs of a different duration specified. Also, unlike .eannot files, not every epoch needs to be specified in an .annot file, i.e. if there are 1200 epochs, you do not need to have exactly 1200 rows in this file. This is because instances specified with epoch-encoding are automatically converted to interval-encoding upon loading.

.eannot files

This is the simplest format for epoch-level annotations. Epoch annotations in .eannot files are simple labels attached to individual epochs. The format is as follows:

  • one row per epoch
  • each row contains a single label, that is attached to that epoch
  • for each distinct label in the file, a new annotation class is generated
  • each instance is assigned the same ID as the label name (i.e. same as the class name)

When an .eannot is specified in the sample-list, it is attached prior to loading the EDF. By default, Luna assumes epochs are 30-seconds in duration and do not overlap when using .eannot files.

Hint

To work with .eannot files but use different epoch definitions, you have three options:

  1. use the EPOCH and EPOCH-ANNOT commands to attach the file after initially attaching the EDF (i.e. instead of specifying the .eannot file in the sample-list)

  2. Set the special variable epoch-len variable if the .eannot is specified in the sample list (i.e. and so loaded when the EDF is first attached, prior to running any EPOCH command)

  3. Use the e: epoch encoding notation in a generic .annot file instead, as described above.

Unlike other types of annotation, these can be loaded via a Luna command, EPOCH-ANNOT, potentially after the EDF has been loaded and manipulated. In that case, i.e. when using the EPOCH-ANNOT command, the number of rows (epochs) in the .eannot file must match the number of epochs that currently exist in the in-memory representation of the EDF (i.e. which may be different from the on-disk version).

A common use of an .eannot file could be to store manually-scored sleep stages:

wake
wake
N1
N1
wake
N1
N1
N2
N2
... (etc) ...
where the number of rows of this file corresponds to the number of 30-second epochs in the EDF. One could then use these annotations in a MASK command, such as:
MASK if=wake
RESTRUCTURE
to exclude wake epochs from analysis.

FTR files

Feature files are a simple annotation format generated by Luna after certain commands (e.g. the ftr option of the SPINDLES). Their filenames must conform to a special format:

id_{indiv_id}_feature_{label}.ftr 

where {indiv_id} should be replaced by the individual/EDF ID, and {label} should be replaced by the name of the feature. (Note, IDs and feature labels can contain underscore characters, but they should not contain the phrase _feature_.)

For example:

id_subj00001_feature_spindles11hz.ftr 

means that the contents of this file are the annotation class spindles11hz for the individual/EDF with ID subj00001.

If the sample-list points to a folder, e.g.

id001   /path/to/id001.edf     /path/to/annots/
or a global annot-folder folder is specified via the command-line or parameter file, e.g.
luna s.lst annot-folder=/path/to/annots/ < command.txt 

then all FTR files that match the ID of the EDF being processed are loaded.

Hint

To turn off automatic loading of all FTR files, add the following to the command line

luna s.lst ftr=N < commands.txt
or set that variable in a parameter file
ftr    N

FTR files have the following format:

  • tab-delimited plain-text
  • no header line
  • creates a single annotation class with the name from the FTR filename (i.e. after _feature_)
  • columns 1 and 2 are interval start and stop in time-point units of each instance
  • column 3 is a text label that specifies the instance ID
  • further columns are key=value pairs become instance meta-data (all encoded as string values)

For example, for the file id_subj00001_feature_spindles.ftr:

20733175781250       20733707031250       sp-1   amp=3.34   dur=0.53   frq=11.21   nosc=6
21057488281250       21058015625000       sp-2   amp=4.59   dur=0.53   frq=11.29   nosc=6
21139898437500       21140558593750       sp-3   amp=3.59   dur=0.66   frq=10.54   nosc=7
... (etc) ...

These lines list information on detected spindles; here only three spindles are listed, labeled sp-1, sp-2, etc. Each spindle has some arbitrary information encoded, on spindle amplitude (amp), duration (dur), frequency (frq) and number of oscillations (nosc). Currently, these additional fields (column 4 onwards) are not used internally by Luna. They are, however, displayed when looking at annotations in Scope for example, and are accessible when using the Luna R extension library.

When loaded in, the annotation class will be spindles. There will be as many instances as rows in the FTR file, and the instance IDs will be the third column of the FTR file (sp-1, sp-2, etc). As noted, instance IDs need not be unique, and they can be included in MASK commands.

Note

Although not currently used, there are two reserved keywords for FTR meta-data that will have special meanings, i.e. in future releases of Scope. These key values are _rgb_ (which expects a RGB value, encoded 0..255) and _value, which expects a floating-point number that will be taken to be a primary value, e.g. the one that will be used if plotting annotations in certain contexts.

_rgb=255,255,255
_value=0.234

ANNOTS

Tabulate and summarize annotation information

Produces information about the number and total duration of annotations in an EDF, at the whole-file level and optionally per-epoch.

By default, ANNOTS will only show annotations that span at least one unmasked epoch. The definition of whether or not an annotation instance is masked or not can be varied. Depending on the context, this can be useful to generate different types of summaries, e.g. the number of respiratory events in REM versus NREM sleep.

Parameters

Parameter               Example               Description
epoch epoch Show epoch-level summaries
show-masked show-masked Show masked annotations (default it not to do so)
any any Keep annotations that have any overlap with one or more unmasked epochs (default)
all all Only keep annotations that are completely within unmasked epochs
start start Keep annotations that start in an unmasked epoch

The epoch option produces lists, for each epoch, all the annotations that overlap with that epoch, given the specified overlap definition (any, all or start). If the show-mask option is given as well, all epochs and annotations are shown; otherwise, only unmasked epochs and annotations are shown. The output contains two variables (EPOCH_MASK and ANN_MASK) that indicate whether a given annotation instance is masked or not.

Output

Class-level annotation summary (strata: ANNOT)

Variable Description
COUNT Number of instances of that annotation class
DUR Combined duration (seconds) of all instances of that annotation class (does not account for potential overlap)

Instance-level annotation summary (strata: ANNOT x INST)

Variable Description
COUNT Number of instances of that annotation class and instance ID
DUR Combined duration (seconds) of all instances of that annotation class and instance ID (does not account for potential overlap)

Instance-level annotation tabulation (strata: ANNOT x INST x T)

Variable Depends on Description
START Start time (seconds) of this instance
STOP Stop time (seconds) of this instance
VAL The meta-data for this instance, if any exists (otherwise missing NA)
ALL_MASKED show-masked
ALL_UNMASKED show-masked
SOME_MASKED show-masked
SOME_UNMASKED show-masked
START_MASKED show-masked

Per-epoch instance-level annotation tabulation (strata: E x INTERVAL x INST)

Variable               Depends on Description
ANNOT_MASK epoch Flag whether this annotation instance is included or excluded (1 means masked or excluded)
EPOCH_MASK epoch Flag whether this epoch is included or excluded (1 means masked or excluded)

Example