Intraobserver error

Intraobserver error, the variation in placement of a given landmark from trial to trial by the same person, was measured as the Euclidian distance between the identified coordinates of a landmark during two landmark placement trails. After determining that left and right versions of bilateral landmarks showed similar patterns of intraobserver error, calculations of intraobserver error for bilateral landmarks include values from both sides. The median intraobserver error for landmarks taken by the more experienced landmarkers on E10.5 specimens (Figure 3A) tend to be well below 0.1 mm, usually closer to 0.05 mm, with the less experienced landmarker showing higher median and variance of error overall. While the more experienced landmarkers display low landmark placement error, the less experienced landmarker displays high median intraobserver error for landmark 2 and high variance for 17/34.For E11.5 embryos (Figure 3B), the median intraobserver error for the more experienced landmarkers is still well under 0.1 mm for most landmarks, although there are some landmarks that display higher error. Looking at the more experienced landmarkers, point 2 shows the highest median value, while point 4 is also high for one experienced observer. Of the lateral points, 17/34, 18/35, and 21/38 median values are above 0.1 for at least one of the experienced landmarkers. The median intraobserver errors are higher for the E12.5 embryos then the other two ages (Figure 3C), although this might be expected given that the overall dimensions of the head have increased substantially since E10.5. For E12.5, landmark 21/38 consistently displays the highest level of error for the experienced landmarkers, while the points 17/34, 18/35, and 13/30 display median close to 0.1 mm for at least one experienced landmarker.

Interobserver error

Interobserver error was calculated for every landmark of every specimen as centroid size of the average landmark locations chosen by the three experienced landmarkers (average between trials 1 and 2). Although centroid size is not directly interpretable as a Euclidian distance, centroid size values are comparable between ages in our analysis (Figure 4).

Overall, the median and variance of interobserver error is lower for E10.5 and E11.5 data than for E12.5 data. Landmarks that were identified as having relatively high intraobserver error across ages, including 2, 13/30, 17/34, 18/35, 21/38 are among the landmarks with higher mean interobserver error at E10.5 and E11.5. However, the landmark that displays the most interobserver error at E10.5 and E11.5 is 7/24, which showed relatively low intraobserver error. At E12.5, landmark 17/34, 18/35 and 21/38 display he highest median interobserver error, although 18/35 seems to be much more problematic on the right side than the left.

Significance test

Ontogenetic analysis

We identified a list of trouble landmarks, based on their high median intra- or inter-observer error values, for which we did not believe a clarification of landmark definition would necessarily reduce error (see Discussion). In order to determine how removing these trouble landmarks or 3 other morphologically defined landmark groups from our sample would modify the results of a simple ontogenetic analysis, we completed separate regressions of landmark coordinates on centroid size for five subsets of landmarks within MorphoJ (Table 1, Figure 2).The association between a summary regression score and centroid size is roughly linear for each of the five regressions completed (Figure 5). The proportion of the total variation for which the regression including all landmarks accounts is approximately 82.5%, while it is 78.9% when the trouble landmarks are removed, 83.8% with nasal landmarks removed, 81.1% with maxillary/mandibular landmarks removed, and 73.3% with non-facial landmarks removed. Based on a subjective comparison of the shape vectors associated with each regression, the removal of any of these subsets of landmarks does not grossly change the nature of the allometric shape change of a given landmark during the developmental period under study (Figure 6). As the embryo develops between E10.5 and E12.5, the bilateral landmarks of the face become relatively more rostral and closer together. The facial landmarks closer to the midline also become relatively more ventral by E12.5. The landmarks found away from the face are relatively more caudal and dorsal in the older specimens.

Intra- and inter-observer error

Bookstein [28] defined a commonly used landmark classification system based on the features used to determine landmark location. Type 1 landmarks are defined as the location of a discrete anatomical structure, independent of other anatomical features. Several of the landmarks in our set (Figure 2) are Type 1, including 8/25 that is found at the intersection of the eye, forebrain, and lateral nasal prominence. Type 2 landmarks are extreme points along curves or grooves and represent the majority of landmarks defined on the surfaces of embryos for this and previously defined sets (eg. [6, 25]). Type 3 landmarks, those defined with respect to relatively distant anatomical structures, do not necessarily share biological meaning and homology across specimens. They are considered mathematically deficient, because their locations are partially dependent on the locations of other anatomical structures [28, 33]. However, because Type 1 and Type 2 landmarks cannot comprehensively represent the shape of the early facial prominences [3], we defined a few landmarks on the extreme of craniofacial bulges in order to improve our coverage of the anatomical features of interest. Because the face develops quickly between E10.5 and E12.5 (Figure 1), biologically homologous landmarks may switch between landmark types across this period of development. For example, landmark 17/34 is the extreme lateral edge of the maxillary prominence at E10.5 and E11.5 (type 3), but is defined based on location between whisker rows at E12.5 (type 2). Even if a landmark does not change type, the error associated with its identification may change across this developmental period.

Our landmark set was compiled with the goals of 1) creating an explicit homologous association between each landmark and a biological feature across E10.5-E12.5 and 2) serving as the basis for ontogenetic analyses that include datasets produced by multiple observers. Therefore, we have attempted to choose landmarks that represent relevant biological features well and which can be repeatably placed between trials and by multiple observers. However, we acknowledge that a given landmark may not necessarily be more biologically relevant or repeatable than other nearby locations, particularly if the landmark is defined along a curve or curved surface (i.e. type 2 and 3 landmarks). We have endeavored to transparently report our biological interpretations of landmarks and the repeatability of landmark placement in order to allow other researchers to judge the usefulness of this landmark set for their own studies.

None of the landmarks identified as having high intra- or inter-observer error in this study are Type 1 landmarks. Instead, they tend to be Type 2 landmarks along curves with wide inflection (eg 2, 7/24, 18/35) or Type 3 landmarks on the extreme extents of bulges (eg. 17/34). After reviewing the landmarks identified as having high error (the subset of trouble landmarks) we are confident that we can clarify the definitions of 10/27 and 13/30 in order to reduce landmark placement error, based on noted systematic differences in placement between landmarkers. However, we suggest that 2, 7/24, 17/34, 18/35, and 21/38 be removed from our standardized landmark set for E10.5-E12.5 mice. This does not mean that these landmarks are necessarily inappropriate for a specific embryonic age or as an additional landmark for studies that include only one observer. For instance, landmark 7/24 displays high interobserver error and relatively low intraobserver error for the experienced observers, suggesting that observers developed their own stable (although different) interpretations of the landmark. Similarly, the median intraobserver error across the three experienced landmarkers increases for landmark 21/38 from E10.5 to E12.5 as the forebrain expands, suggesting that the landmark may be appropriate for use at E10.5 but not at E12.5.

After removing trouble landmarks from our landmark set, and assuming minimal modification of craniofacial shape during preparation and imaging of specimens [24], median intraobserver error of our experienced landmarkers is below 0.1 mm. In the case of E10.5 specimens, it tends to be below 0.05 mm. Therefore, the application of this landmark set by an experienced landmarker should allow the capture of differences in facial shape between groups of specimens that are significantly larger than 0.1 mm. For comparative purposes, the width of the ridge lateral to the nasal aperture is 0.13 mm at E10.5, while the width of the nasal aperture measured from the rostro-ventral extent of the lateral and medial ridges is 0.14 mm at E12.5. The significantly higher intraobserver error noted for the first time landmarker highlights the need for landmark training and experience in order to reduce error to acceptable levels.

One of the major reasons to design this standard set of landmarks was to allow landmarks taken by multiple observers to be combined in comparative analyses. Unfortunately, the interobserver error noted for many landmarks indicates significant differences in the interpretation of landmark definitions even though we provided precise biological and practical landmark definitions, examples of landmarks identified on a 3D craniofacial surface for each age, illustrations of standard head orientation, and face to face discussion. Measures of interobserver error can illustrate the need for further clarification of certain landmark definitions or the removal of a landmark from analysis entirely. Alternatively, assuming that there is high interobserver and low intraobserver error, it may be possible to remove typical differences in interobserver landmark placement via regression of landmark coordinate values on observer identity. While completely removing interobserver landmark error may not be possible, comparative error studies like this are highly recommended in order to identify and address problematic landmarks as a way to reduce the level of landmark placement error included within subsequent analyses.

The variation in landmark placement, measured as the distance between an observer’s landmark coordinates and mean landmark coordinates, is strongly associated with the factors of landmark and observer identity at E10.5, E11.5, and E12.5, according to our two-way ANOVA analysis. Specimen identity is associated with landmark variation at E10.5 and E11.5, although with reduced significance at E11.5 (Table 2). The significant effect of observer identity on landmark coordinate variation indicates that significant differences in landmark placement exist between observers for at least some of the landmarks. The significant effect of landmark identity suggests that this interobserver variation in landmark placement is higher for some landmarks than others. Although significant at all three embryonic days, the sum of squares values for both observer and landmark increase from E10.5 through E12.5, which matches the observation that mean inter- and intra-observer error values tend to increase from E10.5 to E12.5 (Figures 3 and 4). The significant effect of specimen identity in the earlier ages does not suggest that interobserver error is higher for younger specimens. It merely suggests that the level of interobserver error varies between specimens at the younger ages. This difference in landmark placement error may be associated with larger relative differences in morphology between specimens at slightly different stages of development within the younger age categories. It may also be due to relatively more significant fixation based changes in morphology within some younger specimens.

Interpreting ontogenetic analyses

Another major reason to define this set of landmarks was as a basis for ontogenetic analyses of craniofacial shape in mice between E10.5 and E12.5. Therefore, we carried out a series of linear regressions between size and shape in order to determine how the selection of landmarks might influence the results and interpretation of this type of analysis. Omitting major groups of landmarks from these regressions did not substantially modify the nature of the craniofacial shape change associated with an increase in centroid size between E10.5 and E12.5, suggesting that the specific landmarks selected should not influence interpretation of shape change across this ontogenetic period.

However, removing the non-facial or trouble landmarks from the dataset reduced the linearity between landmark coordinate regression coefficients and centroid size, as well as the total landmark coordinate variation explained by the regression on centroid size. In our regressions, centroid size is a proxy for developmental age. Between E10.5 and E12.5, the growth of the facial prominences outward is not as pronounced as the growth and definition of the various regions of the brain [34] (Figure 1). The lower explanatory power of the regression when non-facial landmarks were removed suggests that including these landmarks improves the usefulness of centroid size as a proxy for developmental age, because it is linked to the overall change in craniofacial size over this developmental period. The intermediate reduction in explanatory power for the regression when the trouble landmarks were removed is likely based on the fact that a high proportion of the trouble landmarks are non-facial landmarks.

We recommend that landmarks across the head be included in ontogenetic analyses in order to better estimate centroid size as a measure of developmental age, but also to provide a broader anatomical context within which to interpret morphological changes of the face or another specific region of interest. If possible, using a measure of developmental age that is independent of the landmarks under analysis, such as tail somite number, may be preferable [6].

Future directions

Great progress has been made in the automation of craniofacial surface quantification and comparison ([35–41]), which might make issues of intra- and inter-observer landmark placement error irrelevant. However, despite significant effort, an adequate automatic method has not yet been developed that allows for the quantification of subtle craniofacial variation in young embryos. This is partially due to difficulty in producing standardized and complete ectodermal surfaces from μCT images, as well as the fact that craniofacial features change very quickly during embryonic development to the point where a great deal of variation exists within a sample of specimens from a single embryonic day. The human eye remains, unfortunately, superior to computational methods in locating homologous features on incomplete surfaces and across gradations of developmental age. The authors are involved in work to automate embryonic landmark placement, but must continue to rely on manual placement of landmark sets like this for the time being.

Landmark definitions and data collection

This study was performed using a sample of 10 E10.5, 9 E11.5, and 10 E12.5 CT images heads from mice of various backgrounds that were collected for other studies. None of these mice display gross dysmorphology during the early embryonic period. Care and use of mice for this study were in compliance with relevant animal welfare guidelines approved by the University of Colorado - Denver and the University of Calgary. Fixation in 4%PFA/5% Gluteraldehyde was carried out according to protocols designed to minimize the level of desiccation and craniofacial shape change in these embryonic specimens [24]. All μCT images chosen for this study were produced with a Scanco μ35 at the University of Calgary with 45 kV/177uA for images of 0.007 mm (E10.5 and E11.5) or 0.012 mm (E12.5) voxel size.

All landmarks in our landmark set (Table 1; Figure 2) were given a biological definition representing a biological interpretation of that landmark across all three ages of interest. A practical landmark was defined for each landmark at each age in order to guide the placement of the landmark so that its location at each age would match the biological definition. Because some of these landmarks are defined as the extremes of curved surfaces, the practical definitions may include instruction on the orientation of the surface image that a landmark should be taken from. 2D lateral images of an embryonic head from each age were produced to define a standard orientation for landmarking. Terms of anatomical direction used within the definitions are in reference to a mouse head in standard adult position and may not match the directions as defined by the embryonic body. Full landmark definitions and orientation images are available in Additional files 1, 2, 3, 4 and 5. Last, each landmark was placed on the 3D surface of a reference specimen of each embryonic age as an example to be referenced during the process of data collection. Because of the relatively large file sizes involved, these surface files and associated landmark coordinates are available upon request to the authors.

After a brief orientation and access to all landmark reference materials, four observers completed two landmark trials within MeshLab [42] on minimum-threshold based ectodermal surfaces produced from μCT images of each specimen. Trials 1 and 2 were completed at least one week apart in order to reduce the influence of memory on landmark placement during the second landmarking trial. The first two observers (1,2), who had experience collecting landmarks from CT images of mice, collaborated to develop the landmark set definitions, the third (3) was also an experienced mouse landmarker, and the fourth (4) was a first time landmarker (numbers as defined in Figure 3.

Landmark error analysis

Intraobserver landmark placement error was calculated for each landmarker (1-4) as Euclidian distances (mm) between the location of a landmark taken during landmark trials 1 and 2. Boxplots were used to visualize the median and variation of intralandmark error for each landmarker (Figure 3). After determining that the left and right versions of bilateral landmarks show similar levels of intraobserver error, we decide to combine these values for both sides when calculating intraobserver error for the bilateral landmarks.

Inter-observer landmark placement error was first calculated as the centroid size of the mean landmark coordinates defined by each observer for each landmark (mean of trials 1 and 2). Because of the high intraobserver error noted for the less experienced observer, our interobserver error and all further analyses focus on the data from the three more experienced observers.

The mean landmark coordinates for a given specimen were calculated as the average of the mean landmark coordinates calculated from the two trials of each observer. Variation in landmark coordinates were calculated as the Euclidian distance between the coordinates collected by a given observer on a given specimen and the mean coordinates for that specimen. A two-factor analysis of variance (ANOVA) was completed for each embryonic age to test whether specimen, landmark, or observer identity are significantly associated with this variation in landmark coordinates.

Ontogenetic analysis

Regressions of procrustes coordinates against centroid size, a better proxy for developmental age than embryonic day, were performed in MorphoJ [43] to determine the strength and nature of ontogenetic change in craniofacial shape from E10.5 to E12.5. This regression was performed for the whole dataset and a dataset from which trouble landmarks with high intra- or inter-observer error were removed (Table 1; Figure 2). To further investigate how the use of different subsets of landmarks might influence the results of an ontogenetic analysis, regressions of procrustes coordinates against centroid size were also calculated for the dataset after removing nasal, maxillary/mandibular, or non-facial landmarks (Table 1; Figure 2). A summary regression score representing the regression coefficients of each specimen [44] was plotted against centroid size in order to visualize the strength and linearity of this association between craniofacial shape and centroid size. An estimate of the proportion of total variation for which a regression accounts, an analogue of an R-squared value, serves as an informal measure of the strength of this association.

The vectors of landmark change associated with centroid size provide a summary of the nature of the ontogenetic changes in craniofacial shape between E10.5 and E12.5. These vectors are calculated as regression coefficients of each axis of each landmark added to the mean procrustes coordinate of the landmark along each axis. A figure displaying the relative ontogenetic vectors associated with four of the landmark subsets required the use three landmarks to define parallel axes for procrustes superimposition of each subset as the basis for each regression against centroid size (these three landmarks were not found in the fifth subset of landmarks). The resulting coefficient vectors were plotted relative to the mean shape of the dataset including all landmarks for the purpose of visualization in R [45].

Availability of supporting data

The full landmark definitions, the original embryo orientation images, and an image illustrating the placement of the landmarks at all three embryonic days are available as Additional files 1, 2, 3, 4 and 5. Because of the size of the 3D surface meshes, these meshes and associated landmark coordinates for reference specimens at E10.5, E11.5, and E12.5 are available upon request to the authors.