Results reference

Guide for cdid 1.3.0.0 (R). Recorded examples retain their stated versions.

On this page

Use summary() to read a fit, tidy() to extract a table and ggdid() to plot it. This page describes the reported effects, their uncertainty and the diagnostics you can use to investigate a result.

Result classes and functions

Function Result in chained mode Methods
att_gt(gname = ...) cdid_MP, which inherits did’s MP print(), summary(), tidy(), glance(), ggdid(), chain_report(), and aggte() for aggregation
att_gt(dname = ...) cdid_static_att_gt print(), summary(), tidy(), glance(), ggdid() and plot(); the aggregate is stored in the result
aggte() cdid_AGGTEobj, which inherits did’s AGGTEobj print(), summary(), tidy(), glance(), ggdid() (except for the simple type)
chain_report() cdid_chain_report print(), as.data.frame()
conditional_did_pretest() cdid_MP.TEST, which inherits did’s MP.TEST print(), summary()

In ordinary mode each function returns did’s own object, unchanged. For the cohort chained classes, print() and summary() give the same display. The att_gt() and aggte() displays show did’s summary table under a heading that names the chained estimator, without did’s citation line, followed by a short chained record. For att_gt() the record gives the available and requested cells with the reasons for unavailable ones, the units of the unit table, the correction for missing outcomes, the parallel trends assumption and base_period, the GMM weighting and any covariance regularization, the inference actually performed and the pretrend test or the reason it is unavailable. For aggte() it gives the available and intended aggregates, every restricted or unavailable target with its reason, the bins used and excluded by the overall summary, the parallel trends assumption and the inference, and, when they apply, nondefault event-time selectors, unknown cohort shares or shares taken as masses at the first stored date. The heading shows the package versions and both paper references once. Interval family codes, the numerical policy and resource profile remain in the result without being printed, as listed below. The conditional-test display reports its CvM test and status.

Static entry and exit results

With dname, att_gt() returns switching-date effects rather than cohort effects. summary(fit) displays them, tidy(fit) exports the transition table, glance(fit) gives one summary row, and ggdid(fit) plots the available entry and exit estimates.

Field Contents
target_id, t, t_prev, d, direction Transition identifier, switching date, preceding stored date, prior treatment state and entry/exit direction
att, theta_hat Treatment effect on the treated-versus-untreated scale, and the raw switcher-minus-stayer contrast; the exit contrast is sign-reversed in att
available, support_status, support_reason Whether each declared transition can be estimated and why a target is unavailable
se, analytical_se, bootstrap_se Reported, analytical and multiplier standard errors, according to the inference requested
V_analytical Estimated covariance of the static cell estimates; analytical_se is sqrt(diag(V_analytical)). This differs from the cohort matrix scaling described below.
inffunc, unit_id, n One influence-function row per original study unit, aligned with unit_id, and the full study-unit count
aggregate Switcher-weighted estimate, standard error, intervals, availability and full-unit influence function for the declared transition family
request, diagnostics, graphs, inference, references Active settings, unit/role counts, whether the fit used direct adjacent comparisons or chained GMM (gmm_executed), pair-graph support, inference record and route-specific references

Use fit$aggregate for the static average. aggte() remains a cohort aggregation function. An unavailable declared cell also makes the declared static average unavailable; the package does not silently change its target family. See the entry and exit guide for an example and the sign convention.

chain_report() describes the comparisons and identifying paths in cohort gname fits. Use control = list(keep_nuisance = TRUE) to inspect fitted model equations and coefficient tables with nuisance_models(fit) on supported cohort and static fits. The static models label their switcher and stayer populations. The cohort-by-date model identifies the reference levels and its covariate basis. The report indicates when a model was unnecessary, a fit was requested without capture, or coefficient standard errors are unavailable.

The remaining sections describe cohort gname objects and their aggregation and pretrend tests.

Group time results

Fields inherited from did

Field Contents in a chained result
group, t cohort and period of each reported cell
att estimates; NA for unavailable cells, never zero filled; exactly zero for the reference cell under base_period = "universal"
se standard errors, analytical or from multiplier draws according to the inference actually performed; NA for unavailable cells
c critical value: the pointwise normal value qnorm(1 - alp / 2) or a simultaneous multiplier quantile
V_analytical estimated asymptotic covariance of sqrt(n) * (estimate - target), equal to n times the estimated covariance of the cell estimates; analytical standard errors are sqrt(diag(V_analytical) / n)
inffunc influence function matrix with one row per unit of the unit table (row names are the unit identifiers) and one column per cell; columns of unavailable cells are NA
n number of units in the unit table
W, Wpval unconditional pretrend Wald statistic and p value; NULL when the test is not computed
alp, DIDparams significance level and settings

The chained record

fit$chained holds everything specific to chained estimation.

For a worked inspection of unit table coverage, missingness, selected-pair counts and these fitted records, see Investigate missingness and data structure.

Element Contents
estimator, references, versions estimator label, the two references, and the versions of cdid and did with the tested did version
target_id, target_meta, role, available identifiers and descriptions of the reported cells; role is "pre", "post" or "reference"; available is TRUE or FALSE
support_status, support_reason, support, support_ledger why each cell is or is not available, with the supporting records
deterministic_reference cells that are exact zeros by construction, which have no test statistic
z_stat, p_value, analytical_se, ci_lower, ci_upper, cband_lower, cband_upper statistics, analytical standard errors, pointwise intervals and band limits for each cell, named by target_id
band_family, band_families the interval family of the reported cells; values beginning with "simultaneous" denote a simultaneous band, values beginning with "pointwise" pointwise intervals
inference the inference actually performed: kind ("analytic", "multiplier_bootstrap_iqr_se" or "not_computed"), the effective bstrap and cband, the number of draws, clustering and number of clusters, n_total, the random number record and the requested flags
pretest status and details of the unconditional pretrend test
frame the unit table as used: id, cohort, weight, cluster and eligible
base_period, pt the reporting choice and the parallel trends record
covariance_handling the GMM weighting request and the covariance regularization record
selection the handling of missing outcomes and model descriptions. Fit counts are nested in selection$models and depend on the active route. Observation weighting also records its fitting universe and active cells.
nuisance_diagnostics, outcome_design the fitted treatment, outcome, observation and imputation models, and the design columns
chain_info the comparisons, the paths that identify effects and the comparison weights used by chain_report()
numerical_certificate a numerical certificate when one is produced; the current numerical policy uses conventional computation without per fit error certification
request, public the honored computation flags with the admitted multiplier workers, and the resource profile with the numerical policy
basis optional separate contrasts, kept apart from the cells

Settings in DIDparams

fit$DIDparams keeps did’s settings as requested, including est_method, control_group, anticipation, base_period, xformla, bstrap, cband, biters, clustervars and compute_inffunc. fit$DIDparams$cdid adds the mode, the estimator label, the versions, the requested and effective inference flags and the number of draws actually made. It also records data_retained = FALSE: a chained result does not store the panel data, so it must be aggregated with cdid::aggte().

Tables from tidy and glance

tidy() of a group time result returns one row per reported cell, with did’s columns (including estimate, std.error, conf.low and conf.high) and the following chained columns or updates:

Column Meaning
statistic, p.value test of a zero effect for the cell; NA for exact zero reference cells
target.id, role cell identifier and role ("pre", "post" or "reference")
available, support.status, support.reason availability and its reason
certified whether a numerical certificate covers the cell; NA when no certificate was produced, the usual case
estimator estimator label, present for OR and DR fits and for change imputation and the augmented correction
band.family interval family

The statistic and p.value columns update the inherited values. P-values remain pointwise even when conf.low and conf.high use a simultaneous critical value; the inherited point.conf.low and point.conf.high are always pointwise normal intervals.

glance() of a group time result returns one row with did’s columns plus estimator, n.requested, n.available, band.family, inference and clustered.

tidy() of an aggregated result updates the inherited statistic and p.value and adds label, available, restricted, n.intended, n.used, excluded.ids, reason, certified, band.family and, as above, estimator. Its rows depend on the type: the group type starts with the overall summary (label equal to "overall") followed by one row per cohort; the simple type has one row; the dynamic and calendar types have one row per event time or period and no overall row. glance() of an aggregated result adds estimator, status, na.rm, n.bins, n.bins.available, overall.status, band.family and inference.

Unavailable effects

A cell is unavailable when the comparisons available to the estimator cannot support it, for example because no chain of observed changes connects the needed dates for the cohort or its controls, or because the parallel trends choice does not cover a needed step. Its estimate, standard error, influence function column and covariance entries are NA, never zero, and support_status and support_reason give the reason. The summary lists unavailable cells. An unavailable cell reflects the information in the available comparisons; whether the effect is identified in the population is a separate question.

Failures of a model fit, a covariance calculation or a numerical solve are different: the fit stops with an error that names the problem, and nothing is dropped or relaxed to obtain a result.

Unavailable cells also affect other outputs. A simultaneous band covers only the available cells. The pretrend test is unavailable when any of its cells is. An aggregate that needs an unavailable cell is NA with the missing cells named, unless na.rm = TRUE requests the restricted target over the available cells, which is then marked in restricted and excluded.ids.

Reference covariate exclusions

If a fit encounters a required reference covariate that is missing, inspect fit$chained$reference_eligibility. The targets table reports the reference date and eligible treated count for each cohort. Its counts of missing values by group describe the supplied units at that date and can include units outside the comparison groups. restricted_treated_cohorts lists the cohorts whose effects describe fewer treated units. auxiliary_unavailable explains any pretrend comparison that could not use its required covariates.

Aggregated results

Four aggregation types

aggte() follows did’s definitions. Cohort shares are estimated from the unit table, and the standard errors include their uncertainty.

Type Separate estimates (egt, att.egt) Overall summary (overall.att)
"simple" none average of the intended effects after treatment, each weighted by its cohort’s share; missing cells require explicit na.rm = TRUE for a restricted target
"group" one per cohort: the average of the cohort’s effects after treatment average of the cohort estimates, weighted by cohort shares
"dynamic" one per event time e = t - g, including event times before treatment average of the estimates for event times from 0 onward
"calendar" one per calendar period: the share weighted average over the cohorts treated by then average of the calendar period estimates

With several cohorts the overall summaries answer different questions. The simple type gives more weight to cohorts observed for more periods after treatment, the group type weights each cohort by its share whatever the number of its periods, and the dynamic and calendar types weight event times or periods equally. With one treated cohort, as in Get started, they coincide in the population. For fits with changing covariates or weights, the cohort shares are masses at the first stored date, as described in Changing covariates and weights.

Share uncertainty enters only when the target depends on shares. For one-cohort bins and a one-bin group overall summary the ratio cancels shares exactly, so missing first-date masses do not by themselves make those targets unavailable.

Fields of an aggregated result

The fields inherited from did are overall.att, overall.se, type, egt, att.egt, se.egt, crit.val.egt, inf.function, call and DIDparams, and for the dynamic type min_e, max_e and balance_e. For the simple type the separate estimates are NULL and overall.att holds the estimate. inf.function holds complete influence functions with one row per unit of the unit table, named as in did: simple.att; selective.inf.func.g and selective.inf.func; dynamic.inf.func.e and dynamic.inf.func; calendar.inf.func.t and calendar.inf.func.

agg$chained adds:

Element Contents
type, status, na_rm, reason, target_definition what was aggregated and whether it is available
cell one row per aggregate bin, including the scalar simple target: label, key, available, restricted, share_dependent, n_intended, n_used, intended_ids, used_ids, excluded_ids, reason
cell_target_ids, cell_used_target_ids the cells each estimate intended to use and actually used
overall the overall summary: status, reason, intended_bins, used_bins, excluded_bins, weights, inference_kind, n_draws, ci_lower, ci_upper and related records
shares_known, cohort_shares, weights_theta the cohort shares and the weights of the cells
z_stat, p_value, ci_lower, ci_upper, cband_lower, cband_upper statistics and limits of the separate estimates
se_fixed_shares_only, if_fixed_full, if_share_full, decomposition standard errors treating the shares as fixed, kept as a diagnostic, and the parts of the influence function
band_family, inference, requested the inference performed and the effective request, with whether each setting was inherited or explicit
selectors how min_e, max_e and balance_e were applied
share_law, cohort_mass_complete present for fits whose shares are first date masses

Uncertainty

Which interval each output reports

The defaults are bstrap = TRUE, cband = TRUE, biters = 1000 and alp = 0.05. The behavior depends on the output, following did 2.5.1:

bstrap cband Cells of att_gt() aggte() simple Separate estimates of aggte() group, dynamic and calendar
FALSE FALSE analytical standard errors, pointwise normal intervals the same the same
TRUE FALSE multiplier standard errors, pointwise normal intervals the same the same
TRUE TRUE multiplier standard errors, simultaneous band over the available cells multiplier standard error, pointwise normal interval multiplier standard errors, simultaneous band over the available estimates
FALSE TRUE analytical standard errors, pointwise normal intervals, no draws analytical standard error, pointwise normal interval, no draws analytical standard errors with a simultaneous critical value from multiplier draws, and did’s warning

These flag conventions describe supported nondegenerate families. Empty or exactly zero families may require no draws; read the effective flags and draw count in the result rather than assuming biters draws ran. Separate interval families, including any retained basis contrasts, do not form one simultaneous band over their union.

The overall summaries of the group, dynamic and calendar types always have pointwise normal intervals; their standard errors are multiplier or analytical according to the effective bstrap. aggte() inherits the settings of the fit unless you override them.

Multiplier draws

The multiplier bootstrap reuses the estimated influence functions; it does not refit the estimator. Each draw multiplies the influence function rows of each unit, or the row sums of each cluster, by a random sign of plus or minus one (Rademacher multipliers). A multiplier standard error is the interquartile range of the draws divided by the interquartile range of the standard normal distribution, on the scale of the estimate. A simultaneous critical value is the 1 - alp empirical quantile of the largest absolute standardized draw across the covered estimates, and the band is the estimate plus or minus that value times the standard error. In the bstrap = FALSE, cband = TRUE case of the aggregate bins, the draws are standardized by their own interquartile range scale and the resulting critical value is applied to the analytical standard errors. Pointwise intervals use the normal critical value.

The signs come from your session’s random number stream, so set.seed() before att_gt() or aggte() reproduces the draws; there is no fixed seed. Parallel workers only multiply the signs drawn in your session, so the results equal those of cores = 1.

Analytical standard errors and clustering

Analytical standard errors come from the complete influence functions, with one row for every unit of the unit table, including units without an observed change. They account for the estimation of the treatment, outcome, observation and imputation models, the response rates and the cohort shares used by the chosen method.

With clustervars, influence function rows are summed within clusters, including clusters without any unit that contributes a change, and the multiplier signs are drawn by cluster. Clustering changes the standard errors and bands; the point estimates stay the same, because the GMM weights are based on units. A cluster scheme that aggte() cannot honor is refused.

What the uncertainty assumes

The reported uncertainty rests on large sample approximations: enough independent units or clusters, overlap, the model conditions of the chosen methods and regularity conditions. A completed fit does not establish these conditions, and no finite sample, few cluster or nominal coverage guarantee is made. Computation is conventional, with conditioning and residual checks, and no per fit error certificate is produced.

Point estimates only

With compute_inffunc = FALSE, att_gt() returns the same point estimates as with TRUE, without allocating unit-level influence functions or analytical covariance matrices. For cohort results, inffunc, V_analytical, W and Wpval are NULL, se is NA, and c is the pointwise normal value. The recorded bstrap and cband are FALSE; fit$chained$inference$kind and the pretrend test status are "not_computed". aggte() refuses such a result.

Static dname results record fit$inference$compute_inffunc = FALSE and fit$inference$se_reported = "none_compute_inffunc_false". Their influence functions, covariance and standard errors are unavailable, as described in the static result fields above.

Pretrend tests

Unconditional Wald test

att_gt() tests jointly all cells before treatment (role "pre") of the chosen base_period, excluding exact zero reference cells, with did’s Wald statistic. W and Wpval are set only when every one of these cells is available and the full covariance passes the existing Cholesky/solve and conditioning policy. Otherwise they are NULL and fit$chained$pretest records the status ("unavailable", "not_applicable" or "not_computed") with its reason. A test on the available subset is never substituted. When the test is computed, fit$chained$pretest has status = "computed", the tested cells, the degrees of freedom df, W and Wpval.

A test that does not reject does not establish parallel trends. Under PT-all, cells before the chosen start can reject without contradicting the assumption, as explained in Parallel trends.

Conditional test

conditional_did_pretest() tests a conditional null: for every cell before treatment of the chosen base_period (reference zeros excluded), the mean of the cell’s comparison given the xformla design row of the unit at the earlier endpoint of its auxiliary date pair is zero. This can be weaker than conditioning on every relevant variable. The statistic is the unstandardized Cramér von Mises (CvM) statistic over a grid with one point for every unit of the unit table. Its multiplier draws use one matrix of Rademacher signs shared by all cells and grid points, at the unit or cluster level. The p value is the share of draws that exceed the statistic, and the critical value is their empirical type-1 1 - alp quantile.

Field Contents
CvM test statistic
CvMb the statistic in each multiplier draw
CvMcval, CvMpval critical value and p value
KS, KSb, KScval, KSpval NULL; this test is not computed
clustervars, xformla settings of the test
chained the null, status, reason, base period, tested cells with their estimates, route, grid, statistic and multiplier conventions, clusters, random number record, execution and resources

status is "available", "unavailable" (with NA statistics and a reason) or "not_applicable" for a custom est_method function. The test is unavailable when a needed cell is unavailable, when a covariate needed on the grid is missing or has a level absent from the fitted design, when there is no cell before treatment to test, when clustered inference has fewer than two clusters, or when the estimated process is exactly zero everywhere. Estimated effects are never affected. The conditional test is separate from W and Wpval. Its calibration requires uniform process and multiplier regularity conditions, including a growing number of independent clusters for clustered inference.

Plots

ggdid() returns did’s ggplot2 plots with intervals that use the result’s critical values. For a group time result, the plot is built on the grid of cohorts and periods; attr(p, "cdid_presentation_grid") lists each grid point with its group, time, target_id, att, se, status ("available", "unavailable" or "unrequested"), support_status and support_reason. For an aggregated result, attr(p, "cdid_presentation_bins") lists key, label, att, se, status ("available", "restricted" or "unavailable") and reason. As in did, ggdid() refuses the simple type; it has no plot. For OR and DR fits and for change imputation and the augmented correction, the plot caption shows the estimator label.

Saving and exporting

  • Tables. tidy() and glance() return data frames that write.csv() exports, with unavailable values as NA. The overall summary of the group type is the first row of its tidy table; for the dynamic and calendar types use overall.att and overall.se, as in Get started.
  • Plots. Save with ggplot2::ggsave().
  • Fitted objects. saveRDS() keeps the complete result, including influence functions, for later inspection with the same package versions; fit$chained$versions records them. The panel data are not part of the result.

Chain and combination reports

chain_report(fit, cohort = NULL, time = NULL, include_pre = FALSE, tol = 1e-10) inspects how each reported cell is identified and reconstructed from the retained moments in fit$chained$chain_info:

  • cr$summary (or as.data.frame(cr, what = "summary")) returns one row per selected cell with target_id, g, time, event_time, ref, role, available, status, requires_chaining (whether no single-pair comparison spans from the baseline date directly to time in the unreduced catalog), n_witness_links, n_active_gmm_links, estimate, se, chain_path, and support_reason.
  • as.data.frame(cr, what = "witness") returns a short path of comparisons that identifies the selected effect (s, t, link_role, control_pool, sign, link_estimate, contribution) bridging the reference period to the target date.
  • as.data.frame(cr, what = "gmm_weights") (or print(cr, show_gmm_weights = TRUE)) returns all retained comparison moments with |gmm_weight| > tol and their linear contribution = gmm_weight * link_estimate, which sum to the reported estimate.

Inspect fitted models

Set control = list(keep_nuisance = TRUE) in the estimation call to keep the fitted equations and coefficient tables. Use nuisance_models(fit) to see which models are available. Select a component and a fitted block to see the equation, its sample and its coefficient estimates with standard errors.

Component What it models
"treatment" Which units enter the treatment group in a comparison
"outcome" The untreated change used as the counterfactual
"selection" Whether the outcomes needed for a change are observed
"imputation" The actual outcome change within the relevant group

For a doubly robust treatment fit with augmented missing-outcome adjustment, all four models can be active. The hands-on notebook shows a coefficient table for each in its section on the four fitted models. Use ?nuisance_models for options and supported fits.

Keep three kinds of weights apart. Observation weights adjust for missing outcomes. Combination coefficients (gmm_weight in chain_report()) reconstruct an effect from comparisons; they can be negative and are not population shares. Aggregation weights combine effects into summaries.


Reading path, step 19 of 22 · Previous: Options · Next: Diagnose errors and unavailable results

All steps of the reading path
Step Page
1 Get started
2 Moving from did
3 Incomplete panels
4 Identify effects when the observed units change composition
5 How to compare precision
6 Prepare your panel and unit table
7 Treatment adjustment
8 Choose IPW, OR or DR in an incomplete panel
9 Handling missing outcomes
10 Sampling information for the observation model
11 Choose an observation model
12 Impute actual changes
13 Parallel trends
14 Add PT-all restrictions
15 GMM weighting and covariance regularization
16 Robustness
17 Inference
18 Options
19 Results (this page)
20 Diagnose errors and unavailable results
21 Cross-pair chaining and control parallel trends
22 Short pre-treatment windows

Guide home · Package overview

View this page at its source version · Report a problem