Placebo tests and extended plotting functionality by n-delaney · Pull Request #88 · ebenmichael/augsynth

n-delaney · 2023-08-25T01:27:05Z

The proposed changes:

Extend the functionality of the augsynth package by including the options for permuted placebo tests (raw and RMPSE-adjusted)
Move inference out of summary.augsynth() and into the augsynth() function by default
Extend plotting options, including placebo permutation plots and gap plots showing the divergence between the treated units and synthetic control over time

All of the changes to the augsynth package itself take place in augsynth.R and a new file, permutation.R.

The tobacco_replication directory contains a vignette demonstrating the new functionality and replicating the results of Abadie et al. (2010).

Rmd file and data for replication of Abadie et al. (2010)

-updates functions in augsynth.R -adds permutation.R (also contains some plotting functions) -updates package documentation

Updates the github directory in tobacco replication vignette

Replaces convention of inf = F with inf_type = 'none'.

- Separates out inference from estimation. Sets augsynth(inf = "none") by default, which should be backwards compatible. - Adds plot_type as an argument for plot.augsynth (allows user to specify the "name" of the desired outputted plot). - Adds the option to generate a donor table (synthetic control weights and RMSPEs for any inference type) - Ensures that all existing tests execute successfully.

Updates to abadie et al 2010 replication notebook

Adds warnings to multi-outcome and multisynth models if a user attempts to pass in `inf_type` at the time of model fitting.

Minor edits to tobacco replication notebook

…synth class methods for accessing data characteristics.

- converts the tobacco replication notebook to a vignette - removes cache folders and plot folder from the Abadie et al. (2010) tobacco replication notebook

…ation

…ome of the MDES calculation stuff from the permutation package to clarify inference side.

- addresses number formatting in print.summary.augsynth

…ation

…verage.

…tribution

ebenmichael

Overall I think there needs to be a structural change to keep inference calculations in summary.augsynth. The new add_inference function seems to be necessary only to add inference output to the augsynth function, but that should be restricted to the summary.augsynth function. I've called out parts of the code where this can be changed, but mostly it means that all of the new plotting work should happen in the plot.summary.augsynth function, and then the augsynth object never needs any inference added to it.

R/augsynth.R

R/permutation.R

ebenmichael · 2024-06-13T15:01:39Z

tests/testthat/test-permutation_inference.R

+
+
+
+test_that( "MDES_table corresponds to default treatment table", {


Can this test be broken up into subtests, since it looks like there are a few distinct things.

Another good test would be that the placebo gaps agree with manually changing the treatment variable in the data and re-running augsynth.

…nor tests make sense.

… into tobacco example more.

…utation into pr_merge

- adds a test for update_augsynth() ensuring that non-binding update criteria (ex. RMPSE multiple of infinity) return the same donor weights as the original model. - removes superfluous datasets froom the tobacco_replication.Rmd file - updates a test in test-donor_control.R to resolve an error from a variable that is called but not defined (or used as part of the test, as far as I can tell)

ebenmichael

Thanks for all of these changes. I left reviews throughout. Once those are done (and the merge conflict is fixed) we can fold this in.

R/augsynth.R

ebenmichael · 2025-01-29T19:27:48Z

tests/testthat/test-permutation_inference.R

+
+    # These should be the same?  Or close to the same?
+    # (They were not under ridge, probably due to the cross-validation procedure.)
+    gaps7 <- augsynth:::get_placebo_gaps(syn7)


Can you test this with ridge with a fixed lambda parameter? If they should be same then the test should pass.

@n-delaney @lmiratrix the question marks here make me nervous

tests/testthat/test-summary.augsynth.R

tests/testthat/test_general.R

tobacco_replication/beer_troubles.R

tobacco_replication/inference_blogpost.Rmd

lmiratrix · 2026-01-23T19:29:12Z

Trying to put in my comments

…utation into pr_merge

ebenmichael

An overarching request: Please make sure that the top-level calls to the summary and plot functions don't change their default behavior and that the exisiting vignette still works as it should (it doesn't give the same results as-is). If the vignette needs to be updated (e.g. to incorporate the new plot_type argument, please do that).

ebenmichael · 2026-03-02T17:51:13Z

R/augsynth.R

+plot.augsynth <- function(augsynth,
+                          cv = FALSE,
+                          plot_type = 'estimate',
+                          inf_type = NULL, ...) {
+
+    plot_augsynth_results( augsynth=augsynth, cv=cv, plot_type=plot_type, inf_type=inf_type, ... )


My understanding is that the cv option no longer does anything, and instead that is subsumed into plot_type correct? Can you update that throughout the code so that it isn't passing in cv if that isn't doing anything?

ebenmichael · 2026-03-02T17:54:55Z

R/augsynth.R

+#' @param ... Optional arguments
+plot_augsynth_results <- function( augsynth,
+                                   plot_type = 'estimate',
+                                   inf_type = NULL, ...) {


The default behavior of the package is to do conformal inference. Can you make whatever changes are needed so that this continues to be the case? Right now if you follow the vignette it doesn't produce the prediction intervals.

Also, the plots used to have the inf argument that governed whether uncertainty intervals were included. That should stay in to not break any existing code. It also simplifies the plot_type argument because then there doesn't need to be the 'estimate only' option.

ebenmichael · 2026-03-02T18:01:26Z

R/augsynth.R

+                                aes(x = !!as.name(augsynth$time_var), y = !!as.name(measure),
+                                    color = trt_status, linetype = !!as.name(augsynth$unit_var))) +
+        ggplot2::geom_line() +
+        ggplot2::geom_vline(lty = 2, xintercept = treat_year) +


For some reason the vertical line doesn't appear when you get here by running plot rather than permutation_plot.

e.g. the following creates two slightly different plots

syn <- augsynth(lngdpcapita ~ treated, fips, year_qtr, kansas, progfunc = "None", scm = T) # no vertical line at treatment time plot(syn, plot_type = "placebo") # yes vertical line at treatment time permutation_plot(syn)

ebenmichael · 2026-03-02T18:07:21Z

R/augsynth.R

+    } else if (plot_type == 'outcomes') {
+        p <- augsynth_outcomes_plot(augsynth, measure = 'synth')
+    } else if (plot_type == 'outcomes raw average') {
+        p <- augsynth_outcomes_plot(augsynth, measure = c('synth', 'average'))


These should have the option to add the uncertainty intervals (i.e. with inf = TRUE following the original logic of the code)

ebenmichael · 2026-03-02T18:09:57Z

R/augsynth.R

+            p <- p + ggplot2::geom_ribbon(ggplot2::aes(ymin = Estimate - 2 * Std.Error,
+                                                       ymax = Estimate + 2 * Std.Error),
+                                          alpha = 0.2)


Something is happening here where if you just run plot(syn) for a augsynth object syn, the default behavior doesn't include uncertainty quantification but this line of geom_ribbon is running with a warning about all missing inputs. This might go away if the default behavior is to plot conformal inference intervals, but still it comes up using plot(syn, inf_type = NULL)

ebenmichael · 2026-03-02T18:25:43Z

R/permutation.R

+
+#' Calculate MDES
+#'
+#' Use our method for Calculating SEs, p-values, and MDEs by looking


Ok can you just strip this out here then? Alternatively, you can out in a citation etc. Something more than "our" method, since I'd like for it to be clear what different functions are doing.

ebenmichael · 2026-03-02T18:26:34Z

R/permutation.R

+
+    del = as.numeric( diff( q ) )
+    z = -qnorm( alpha )
+    SE_imp = del / (2*z)


Please remove it if it is not being used.

ebenmichael · 2026-03-02T18:30:35Z

R/permutation.R

+    units$weights[ units$ever_Tx == 1 ] = 0
+
+    # confirm that we have placebos for every entity over every observed time period
+    stopifnot("Placebos do not cover every entity over every observed time period." = (ncol(augsynth$data$X) + ncol(augsynth$data$y)) * nrow(augsynth$data$y) == nrow(lest))


@lmiratrix @n-delaney does this ever happen or were you adding in a hypothetical check?

ebenmichael · 2026-03-02T18:32:24Z

R/permutation.R

+
+
+    # Impact is difference between observed and imputed control-side outcome
+    lest$impact = lest$Yobs - lest$Yhat


Please remove it if it isn't needed

ebenmichael · 2026-03-02T18:34:03Z

tests/testthat/test-permutation_inference.R

+
+    # These should be the same?  Or close to the same?
+    # (They were not under ridge, probably due to the cross-validation procedure.)
+    gaps7 <- augsynth:::get_placebo_gaps(syn7)


@n-delaney @lmiratrix the question marks here make me nervous

n-delaney added 10 commits August 23, 2023 17:18

add tobacco_replication files

bb79c20

Rmd file and data for replication of Abadie et al. (2010)

Update package files

ff13d1c

-updates functions in augsynth.R -adds permutation.R (also contains some plotting functions) -updates package documentation

update text in Rmd file

1a3ebd0

Updates the github directory in tobacco replication vignette

default inference

a1b858d

Replaces convention of inf = F with inf_type = 'none'.

update tobacco replication vignette

d5b21d7

add testing, address join warning

b6d0959

replication edits

ec8d52d

Updates to abadie et al 2010 replication notebook

update warnings

4ade107

Adds warnings to multi-outcome and multisynth models if a user attempts to pass in `inf_type` at the time of model fitting.

Merge branch 'master' into master

9d4af54

n-delaney closed this Feb 14, 2024

n-delaney reopened this Feb 14, 2024

n-delaney and others added 14 commits February 14, 2024 12:10

notebook updates

15c368f

Minor edits to tobacco replication notebook

Edit pass on full package with focus on summary method and making aug…

451d0b7

…synth class methods for accessing data characteristics.

tobacco vignette

b010e16

- converts the tobacco replication notebook to a vignette - removes cache folders and plot folder from the Abadie et al. (2010) tobacco replication notebook

Update .gitignore

4032043

clean up docs and whatnot

a6d447a

Merge branch 'master' of https://github.com/n-delaney/augsynth_permut…

9c534ca

…ation

Cleaned up documentation and added a few utility functions. Removed s…

e495521

…ome of the MDES calculation stuff from the permutation package to clarify inference side.

Merge branch 'master' into pr_merge

2da0e41

patch

2e57f4b

- addresses number formatting in print.summary.augsynth

cleaning up printout. Adding a small DGP function for testing.

d400901

Merge branch 'master' of https://github.com/n-delaney/augsynth_permut…

fac62f4

…ation

Cleaning up documentation and adding some testing code to increase co…

6505c82

…verage.

made plotting options work with both summary and augsynth objects

190d1f5

Clean up tobacco replication, added a helper function for placebo dis…

f7be966

…tribution

ebenmichael requested changes Jun 13, 2024

View reviewed changes

n-delaney added 3 commits August 1, 2024 11:05

plotting from summary

a17d8d8

update tests

efd7305

update plotting tests

eaca494

lmiratrix and others added 14 commits August 28, 2024 18:06

tweaks

7f35bb8

Final tweaks--made synth calls go with nonnegative weights to make do…

927904c

…nor tests make sense.

updated tobacco so it runs.

57aacc0

update from requested changes

622e95a

clean up a few small bugs on arguments with SCM=FALSE. Started to dig…

37b9cb2

… into tobacco example more.

Merge branch 'pr_merge' of https://github.com/n-delaney/augsynth_perm…

44a0de5

…utation into pr_merge

Merged in results, tried to repair some test cases (one test is failing)

d2f0c38

Final small cleanup to package based on eli comments.

ecadfe2

Cleaning up tobacco replication script and blog post on inference

76f6b72

update tests

d8e4a45

reintegrate augsynth_class

aed0742

update docs

6e35b6f

cleaning up the vignettes and initial blog post

f62134d

ebenmichael requested changes Jan 29, 2025

View reviewed changes

n-delaney added 3 commits January 21, 2026 16:39

update per pull request

c2e8941

responses to pull request

8ee3f98

remove tobacco replication

5eb991c

lmiratrix added 3 commits January 23, 2026 14:32

Merge branch 'master' into pr_merge

a419d39

redid documentation, mild auto updates

47e7e5b

Merge branch 'pr_merge' of https://github.com/n-delaney/augsynth_perm…

5187a2e

…utation into pr_merge

ebenmichael reviewed Mar 2, 2026

View reviewed changes




		test_that( "MDES_table corresponds to default treatment table", {



		# Impact is difference between observed and imputed control-side outcome
		lest$impact = lest$Yobs - lest$Yhat

Conversation

n-delaney commented Aug 25, 2023

Uh oh!

ebenmichael left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebenmichael left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lmiratrix commented Jan 23, 2026

Uh oh!

ebenmichael left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants