Dear reviewer #1,

We sincerely appreciate your constructive review and the supportive score.

Weakness 1: for experiments on a broader range of llms

We agree with the reviewer's perspective that experimenting with a wider range of LLMs would make our work more convincing. It's indeed part of our plan for future versions of the manuscript. However, we must emphasize that such experiments are substantially resource and time-intensive. We sincerely request the reviewer's understanding and patience with these practical limitations.

Besides, we also wish to clarify that：

• Accessibility for model editing: model editing studied in this paper involves modifying model parameters directly. Given that ChatGPT is a proprietary LLM and does not offer the open access required for such modifications, it falls outside the scope of our current study's methodology.
• Representativeness of selected models: our chosen models are among the most representative and extensively used in current model editing research, with Llama2-7B being one of the largest model that has been practically applied. This careful selection ensures that our experiments are sufficiently persuasive within the field.
• Primary Study Objective: we want to emphasize that our study's main goal is to unveil and highlight the potential risks associated with editing methods, rather than to perform an exhaustive evaluation across all models, which would exceed the scope of a typical conference paper.

Weakness 2: for detail introduction of the datasets In response to the insightful comments regarding the clarity of the datasets used in our study, we provide the following clarifications:

Dataset Overview:

• ME-PPL: This is a text dataset we developed, consisting of sentences to calculate the models' perplexity for normal text. It serves to quickly identify whether the edited model is prone to collapse, thereby preventing the need for time-consuming evaluations in downstream tasks.
• Editing Datasets: The remaining datasets serve as editing datasets. Each sample within these datasets represents an edit request. Specifically:
  • COUNTERFACT and ZsRE: as introduced in sec 3.1 and appendix A.2.2, are widely recognized editing datasets in the model editing domain.
  • HardCF is a curated subset of COUNTERFACT that we developed, featuring challenging samples that have been observed to induce model collapse upon a single edit.
  • HardEdit: building on the foundation laid by HardCF, we use gpt-3.5 to create a refined collection of samples designed to rigorously evaluate current editing methodologies.

We will take this valuable feedback into account and commit to providing more detailed explanations of each dataset's respective purpose in the revised version of our manuscript.

Suggestion1: discussion with recent works We thank the reviewer for the insightful comments and suggested references. Due to space constraints, we had to move the discussion of related work to the appendix, which may have caused some confusion. We apologize for this and will strive to improve the presentation of related work in our revised manuscript.

1. Model Editing at Scale leads to Gradual and Catastrophic Forgetting: Actually, we have cited the contemporaneous work in Appendix A.1.3 (L957-L970) and made the comparison. While this study foucses on the impact of large-scale edits on models, we focus on exploring the possibility of model collapse caused by a small number of edits and how to efficiently detect potential collapses in practical applications.
2. UNVEILING THE PITFALLS OF KNOWLEDGE EDITING FOR LARGE LANGUAGE MODELS: Their research focuses on Knowledge Conflict and Knowledge Distortion within edits, which are less related to the impact of edits on the downstream task performance of models that we study. We will include a citation and discussion of their findings in the context of our research focus in the revised manuscript.

Suggestion2: whether this issue has been well addressed in powerful LLM？

As explained in our response to Weakness 1, we are actively pursuing this research direction. However, based on our research over the past months, we believe that the model collapse issues we have identified might be inherent to the editing methods themselves, suggesting that these challenges are likely to persist even in more advanced LLMs. Due to resource limitations, we hope to obtain results on such larger models in a subsequent version of our work.

We value the reviewer's understanding as we navigate these limitations and areas for future exploration

Suggestion3: writing issues

We sincerely appreciate you for highlighting these issues regarding the clarity of terminology and the need for proper citations. We will fix them in the next revision.

We hope we have addressed all your concerns. If you have any further suggestions, please do not hesitate to reach out.

Dear Reviewer,

I hope this message finds you well and greatly appreciate your willingness to contribute to the reviewing process. The authors have submitted their response to your comments and is unsure if it has dispelled your doubts. If possible, please respond promptly to the author's rebuttuals. Your feedback is crucial in the next steps in the review process.

Best, AC

Dear (S)ACs,

We hope this message finds you well.

We are writing out to respectfully request your help in encouraging reviewer #2 to check our response. We have provided detailed clarifications and conducted supplementary experiments to address their concerns comprehensively. Yet, as the discussion period draws to a close, we have not received reviewer #2's feedback on our rebuttal.

We would like also to bring the attention of the (S)ACs to the quality of the review by #2, as their reasons for rejection (weakness) go against the reviewing guidelines.

• 2. shortcuts (The authors could also do [extra experiment X])

  • Regarding Weaknesses 4-5, the call for further experimental validation, it may be more appropriately considered as suggestions rather than weaknesses. Our manuscript proactively discusses these limitations, especially in term of expanding the range of models. It's important to note that our paper have already conducted very extensive experiments, incorporating four SOTA editing methods and three llms widely recognized in the field of model editing research. This extensive experimental setup already represents one of the most thorough configurations in the field, effectively illuminating the critical risks associated with current model editing techniques. We note that other reviewers (#1 and #3) acknowledge the value of our work and the validity of our experiments. Furthermore, to proactively address these considerations, we have conducted additional experiments. The results of these supplementary experiments further corroborate the findings presented in our paper.

• E. The review does not evince expertise (comments seem to be not based on a deep understanding of the submission) We noticed that Weaknesses 1-3 seem to arise from misunderstandings not just about our work, but also about the model editing field at large, such as misconceptions regarding the portability metric and the MEMIT method.

  • For weakness 1, portability metric measures performance of an edited model on specific facts related the editing request and required reasoning, has no direct relation with the downstream task performances which is the focus in our study.
  • For weakness 2, lacking of benchmarking, we rigorously test the downstream task performance of these models across a comprehensive suite of benchmarks, represented in Figures 1(b) and 2(b), Figure 3, along with Table 3 and Table 4.
  • For weakness 3, MEMIT's contribution is recognized for improving the performance of multiple edits on traditional editing metrics, not the model collapse we identified in this paper. In fact, our research distinctively points out MEMIT's vulnerability to model collapse. This particular focus of our study diverges from the objectives of MEMIT, thereby not undermining the contributions of our paper. This perspective has been acknowledged by the other two reviewers, #1 and #3.

Overall, we firmly believe that the weakness do not justify a 2.5 overall score. We have tried our best to inguire reviewer #2 for discussion. Unfortunately, we did not receive any further engagement or discussion from Reviewer #2 in return.

Given these unfair criticisms, especially considering her/his low score and high confidence rating assigned, we respectfully request a careful reevaluation of their feedback in conjunction with our detailed rebuttal during the meta-review process.

Thank you for considering our request and for your guidance throughout this process.

Best regards

Dear reviewer #2,

We sincerely appreciate your recognition of our efforts in bridging the significant gap in evaluating model editing algorithms through the creation of the ME-PPL and HardEdit datasets, and for acknowledging our work in highlighting the limitations of current evaluation metrics.

Allow us to further address two concerns raised in the summary:

"However, the paper could further benefit from a deeper theoretical exploration of why perplexity is a more suitable metric compared to others, including a discussion on its limitations and potential biases."

We value your suggestion on the need for a more comprehensive theoretical exploration of perplexity as a superior metric for evaluating model collapse. Due to space constraints, we only managed to briefly touch upon this in our paper (L347-351). We described how perplexity, by exhibiting an exponential relationship with the unsupervised pre-training loss, acts as a surrogate metric for monitoring a model's status. The comparison with other metrics, such as locality, was aimed at highlighting perplexity's advantages but, as noted, might benefit from a deeper discussion.

• Sensitivity and Potential Bias: We acknowledge the sensitivity of perplexity, where minor variations might not directly correlate with model performance shifts. Yet, in the context of model collapse, its utility is undeniable, as it sharply distinguishes collapsed models from functioning ones. This is because collapsed models usually exhibit extremely higher perplexity, not just subtle variations. Regarding potential biases due to the text dataset ME-PPL's composition, we have made concerted efforts to ensure its diversity and representativeness, drawing from widely used pre-training corpora to mitigate bias.

We will include a discussion on this topic in our revised manuscript.

Additionally, while the extensive experiments provide valuable insights, the methodologies employed in constructing the HardEdit dataset using GPT-3.5 could be detailed more thoroughly to ensure reproducibility and to understand the selection criteria for challenging samples.

We apologize for the overly concise discussion on the construction of the HardEdit dataset, a limitation imposed by space constraints. Actuality, we have elaborated on this in Appendix A.6 and Figures 10 and 12, providing a thorough presentation of the dataset creation process, including the rationale behind our prompt designs and the selection criteria for challenging samples. These appendices aim to facilitate replication and offer insights into our methodologies, further contributing to the field's understanding of model editing challenges. In future versions, when more space is available, we intend to enrich this discussion in the main body of the text to make the information more accessible and clear

• Reproducibility: For the HardCF and HardEdit datasets, we are committed to their release to ensure reproducibility and support research on editing-induced model collapse and the development of robust model editing methods. Currently, we are actively expanding the HardEdit dataset, resulting in its continuous evolution. It will be larger than initially described in our manuscript. To ensure the datasets' completeness and reliability, we've decided to delay their public release until the end of the review period.

We thank you again for your recognition of the contribution and potential impact of our paper, and hope we have addressed all your concerns. If you have any additional concerns or suggestions, we welcome you to reach out.

Dear reviewer #2,

We sincerely appreciate your valuable comments.

Weakness1: lacking of Portability metric

According to the definition in EasyEdit, Portability examines the performance of an edited model on facts related to the editing request and required reasoning, aiming to assess robust generalization. However, our study focuses on examining the downstream task performance of edited LLMs to determine their usability in practical scenarios. This is crucial for the deployment of model editing in real-world applications, where performance of edited models is essential. Certainly, while Portability offers insights into an editing method's effectiveness, it does not align with the specific objectives of our research. Therefore, this metric was not included in our study.

Weakness 2: lacking of benchmarking score

First, we agree with the reviewer's perspective that perplexity alone is not sufficient for evaluating llms.

In our paper, we primarily used perplexity as a quick way to identify potential model collapses from the editing process. However, our analysis does not stop there; we rigorously test the downstream task performance of these models across a comprehensive suite of benchmarks. As detailed in our paper:

• Figures 1(b) and 2(b), along with Table 3, present the models' benchmarking scores before and after a single edit.
• Table 4 offers a direct comparison between the models after sequential edits and their original states, highlighting the issues in sequential edits.
• Figure 3 illustrates how varying levels of perplexity influence the edited models' benchmarking scores, providing insight into the relationship between perplexity and task performance.

We apologize for any misunderstanding that may have arisen regarding the evaluation metrics used in our study. We will refine our presentation to ensure our experimental outcomes are clear.

Weakness 3: contribution is already made in MEMIT & mitigate the model collapse

We appreciate the reviewer’s attention to the contributions outlined in our paper and the comparison with MEMIT.

MEMIT's contribution is recognized for improving the performance of multiple edits on traditional editing metrics, not the model collapse we identified in this paper. In contrast, our third contribution uncovers model collapse as a critical issue in sequential editing settings——a critical insight not recognized by MEMIT. In fact, as demonstrated in Table 4, our research distinctively points out MEMIT's vulnerability to model collapse. This particular focus of our study diverges from the objectives of MEMIT, thereby not undermining the contributions of our paper.

Furthermore, while developing strategies to mitigate model collapse is undoubtedly important, our study currently focuses on identifying and understanding the model collapse within existing editing methods, including MEMIT. Our exploration into efficient detection mechanisms aims to pave the way for future solutions. Addressing and resolving model collapse, although beyond the scope of our present work, remains a critical objective for our further research.

Weakness 4: experiments on more editing methods

We acknowledge and appreciate the reviewer's suggestion that incorporating a broader range of editing methods could make our work more comprehensive. However, Given the considerable resource and time demands of our experiments, we prioritized the most commonly used setups in the field. We selected four representative and widely-adopted editing methods in current literature, spanning three key categories of model editing techniques.

• PEFT Methods: Our focus on these selected methods stems from their proven impact and the limited effectiveness of fine-tuning in model editing contexts. PEFT approaches, designed for enhancing fine-tuning efficiency, do not directly address the core challenges of editing with constrained data volumes, hence their exclusion from our primary analysis.

• Other METs: The absence of methods like KN and KE from our study is due to their comparatively lower performance within their respective categories (comparing with ROME and MEND), as evidenced by recent research findings. However, to address concerns regarding the diversity of METs explored, we conducted additional tests with KN on three LLMs (GPT-2-XL, GPT-J, and Llama2-7b):

  • An selected edit sample from the COUNTERFACT dataset resulted in significantly diminished downstream task performance for Llama2-7b.

    Model	PIQA	LAMBADA	Hellaswag
    Original_Llama2-7b	0.7845	0.6814	0.5706
    Edited_Llama2-7b	0.5256	0.0000	0.2583
    Random_guessing	0.5000	0.0000	0.2500

  • The three models which are sequentially edited by KN on HardCF all exhibits severe collapse:

    Model	PIQA	LAMBADA	Hellaswag
    Original_GPT-2-XL	0.7084	0.4461	0.4004
    Edited_GPT-2-XL	0.5332	0.0000	0.2610
    Original_GPT-J	0.7541	0.6136	0.4953
    Edited_GPT-J	0.5174	0.0000	0.2561
    Original_Llama2-7b	0.7845	0.6814	0.5706
    Edited_Llama2-7b	0.5152	0.0000	0.2591
    Random_guessing	0.5000	0.0000	0.2500

In summary, our comprehensive approach ensures a robust examination of model editing's impacts, aligning with the methodologies and models frequently discussed in published works within our domain.

Weakness 5: more backbone models

We recognize and value the reviewer's interest in assessing the applicability of our findings across different model architectures.

Our focus on decoder-only models, prominently represented by the GPT and LLaMA series, is driven by their prevailing status as the mainstay in both model editing research and the broader NLP field. Their wide applicability and strong performance underscore the relevance and importance of our findings within this mainstream context.

While encoder-only models are indeed instrumental for specific tasks like classification, their deployment is comparatively narrower than that of decoder-only models.

To address reviewer's concern, we take the encoder-decoder T5 model as an example. Since ROME and MEMIT cannot be applied to architectures other than decoder-only, we opted to try the same-category method KN recommended by the reviewer. In the experiments where KN edited T5, we observed phenomena of single edit collapse and sequential edit collapse (five sequential edits on the HardCF dataset). The results are as shown in the following table:

Weakness 6: future work

We appreciate your suggestions on future work writing, and we will polish this part accordingly.

Suggestions

Thank you for pointing out the issues related to jargon and repetition within the manuscript, as well as the formatting concern regarding Figure 7. We will refine our writing in subsequent revisions to ensure clarity and conciseness throughout the paper.

The extra space beside Figure 7 arises from the layout constraints of LaTeX dual-column formatting. Since subsequent figures are all large cross column figure, there was no suitable content to fill the adjacent space in the layout for Figure 7. We will adjust the layout to better utilize the space and prevent such formatting inconsistencies in future versions.

We hope we have addressed all your concerns. If things are clearer now we kindly request you to reconsider the score you have assigned. Thanks!

Dear reviewer #3,

We sincerely appreciate your time and effort in providing insightful feedback on our manuscript.

Weakness 1: Isn’t perplexity as a proxy already suggested by the ROME paper (Fluency)?

We thank the reviewer for recognizing the distinction between our use of perplexity as an evaluation metric and its application in the ROME paper, and for highlighting the importance of this discussion within our field.

It is critical to clarify that the fluency metric employed by ROME focuses on identifying repetitive word patterns via bi- and tri-gram entropies, which is distinct from our use of perplexity. The fluency metric in ROME can be considered a simplified version of perplexity, not a direct measurement thereof.

In contrast, we use perplexity as a proxy to monitor the overall downstram task performances for quickly indentifying the collapse caused by editing methods.

We appreciate the opportunity to further delineate this distinction and will make appropriate enhancements to our manuscript to reflect this discussion.

Weakness 2: the statistical validity of the correlation experiment

We are deeply grateful for the reviewer's constructive feedback, which we acknowledge as highly valuable in strengthening our manuscript. Our initial exclusion of a detailed correlation analysis was due to the clear patterns in Figure 3. However, we recognize that this omission may led to concerns regarding the empirical rigor of our claims and the statistical foundation of our analysis.

To address this concern, we conduct correlation experiment using Spearman's rank correlation (using scipy.stats.spearmanr) between perplexity (ppl) and performance across three downstream tasks.

1. We add the correlation analysis about the results in figure 3, as shown in the follwoing table:

   GPT-J

   	ppl and PIQA	ppl and LAMBADA	ppl and Hellaswag
   Spearman Rho	-1.0	-0.991	-1.0
   p-value	0.0	1.456e-05	0.0

   GPT-2-XL

   	ppl and PIQA	ppl and LAMBADA	ppl and Hellaswag
   Spearman Rho	-1.0	-1.0	-1.0
   p-value	0.0	0.0	0.0

   Llama2-7b

   	ppl and PIQA	ppl and LAMBADA	ppl and Hellaswag
   Spearman Rho	-0.929	-0.929	-0.964
   p-value	2.519e-03	2.519e-03	4.541e-04

2. Furthermore, we have enhanced our analysis by expanding to 20 data points (edited models), chosen to cover a broad perplexity range up to 1000, as models beyond this threshold showed diminished downstream capabilities.

   Llama2-7b

   Perplexity	37.25	92.25	131.00	199.44	241.96	296.25	342.81	403.34	445.54	477.37	566.90	601.56	636.18	708.46	738.16	796.80	834.88	911.18	948.06	988.97
   PIQA	0.7845	0.7334	0.7116	0.6861	0.6757	0.6730	0.6643	0.6572	0.6540	0.6502	0.6360	0.6355	0.6306	0.6251	0.6219	0.6197	0.6181	0.6066	0.6050	0.6039
   LAMBADA	0.6814	0.1285	0.0417	0.0126	0.0078	0.0054	0.0041	0.0029	0.0021	0.0017	0.0008	0.0010	0.0006	0.0002	0.0002	0.0002	0.0002	0.0002	0.0002	0.0002
   Hellaswag	0.5706	0.4837	0.4505	0.4113	0.4036	0.3942	0.3827	0.3674	0.3609	0.3586	0.3448	0.3449	0.3450	0.3372	0.3349	0.3332	0.3323	0.3278	0.3253	0.3238

   	ppl and PIQA	ppl and LAMBADA	ppl and Hellaswag
   Spearman Rho	-1.0	-0.977	-0.994
   p-value	0.0	1.465e-13	9.578e-19

Additional results are currently underway. Running LM-eval is time-intensive; we kindly ask for your patience as we complete these experiments.

The analysis clearly reveal a very strong correlation between perplexity and downstream task performance, with Spearman's Rho values approaching -1 for all tasks, underscoring a significant inverse relationship: as perplexity increases, performance on downstream tasks decreases. These findings serve as a robust validation of perplexity on ME-PPL as a surrogate measure for downstream task performance.

We will add these additional experiments into subsequent versions of our manuscript. We appreciate the opportunity to clarify these aspects and thank the reviewer for prompting this significant enhancement to our work.

Weakness 3: the prevalence of model collapse

We apologize for any confusion caused by our narrative and appreciate the opportunity to clarify the prevalence of model collapse as observed in our experiments. Our intentions are to present our findings accurately and contribute meaningful insights to the field. In light of the reviewer's feedback, we will refine our wording to better articulate the conditions under which model collapse occurs and its implications for large-scale model editing. Below, we attempt to address and clarify these concerns:

• prevalence of model collapse:
  • First, we would like to clarify and precisely state: model collapse is rare in single editing but common in sequential editing.
  • In single editing setting, for Llama2-7b, the model collapse rate is 0.1%, not 0.01% (21 out of 21k samples); for GPT-J, the collapse rate is 0.4%; for GPT-2-XL, the collapse rate is 0.35%. These ratio, while seemingly small, represent a significant concern in real-world applications where the volume of edits can be vast.
  • Single editing setting is mainly designed for an ideal investigation into the effects of each edit, isolated from the impacts of other edits. It's important to recognize the limitations of this setup. In real-world usage, models are not edited merely once but are continuously edited based on evolving requests.
  • More critically, in the sequential editing scenario, the likelihood of collapse becomes markedly higher as shown in figure 5 and 6. This setup more accurately reflects real-world editing practices and demonstrates that, under these conditions, model collapse is not just a possibility but a common outcome.
  • We claim the prevalence of model collapse mainly within the context of sequential editing. We apologize for any confusion this may have caused and will clarify our text to prevent such misunderstandings in the future.
• worry about confounding factors:
  • For the collapse samples of ROME, we had run multiple experiments repeatedly to ensure that these findings are stable and reproducible, eliminating the influence like hardware, random seeds.
  • Meanwhile, similar phenomena have also been independently discovered in contemporaneous work, Model Editing at Scale leads to Gradual and Catastrophic Forgetting https://arxiv.org/html/2401.07453v2. It foucses on the impact of large-scale edits on models, while we focus on exploring the possibility of model collapse caused by a small number of edits and how to efficiently detect potential collapses in practical applications.

In summary, we acknowledge that model collapse in a single edit setting is rare. However, our findings demonstrate that certain edits, such as the altered fact "Twitter was acquired by Elon Musk," can indeed induce collapse in GPT-J, reflecting a real risk in real-world editing scenarios. Crucially, when extensive consecutive edits are necessary for practical applications, model collapse could become very common. The primary objective of our paper is to unveil the existence of this risk, as we believe the likelihood of such risk occurring is much higher than expected in real-world scenarios.

Weakness4: concerns about the credibility of single-edit results

We fully appreciate and understand the reviewer's concerns and doubt regarding the results of single editing experiments. When we first encountered the collapse phenomenon, our reaction mirrored yours. Here, we address the points raised:

• Regarding the “Long-form evaluation of model editing” study: we carefully reviewed their experimental setup and noted that their analysis was based on a relatively small sample of 100 instances from the COUNTERFACT dataset, which is likely insufficient to include the hard samples we identified, as detailed below:

  "We perform these evaluations on 100 randomly sampled edits from Counterfact and zSRE. For the zSRE setting, ...... . In total, we assess 300 samples (600 passages)" in page 5.

• Fluency and Consistency in the ROME:

  • As we claimed before, these metrics can not capture the full spectrum of model behavior after editing.
  • Notably, the fluency metric in ROME is calculated by the edited models themselves, assessing the coherence of texts they generate. We found that collapsed modele tend to assign lower perplexity scores to their own incoherent outputs, as these are consistent with their compromised state. This discrepancy is a significant factor why the collapse phenomenon was not detected in previous studies.
  • Moreover, the evaluation metrics of current works are averaged across all results. Such averaging metrics mask the collapse of individual samples, failing to highlight instances of complete failure.

• the credibility of single-edit results: as highlighted in response to Weakness 3, our contemporaneous work, Model Editing at Scale leads to Gradual and Catastrophic Forgetting also independently discovered model collapse:

  "Finally, we take a deeper look at the specific edits that cause the inflection point in ROME. We call these edits disabling edits, as **they disable the model and make it unusable for downstream tasks**. ...... This shows that the disabling edits in ROME are not a result of continuous sequential editing of the model, but a fundamental limitation of ROME." in sec 3.4.1

  Unlike they merely observed the phenomenon of collapse caused by a single edit, we conducted extensive experiments to collect such hard samples to build the HardEdit dataset. This dataset enables a thorough evaluation of mainstream editing methods and llms.

For suggestion about "add an appendix to show what a range of text generation outputs across perplexities on ME-PPL": we will incorporate more detailed cases showing text generation quality at varying perplexity levels in the appendix in the revised manuscript. The following is a simple case:

Regarding the selection of a threshold of 1000, it is based on Figure 3, which shows that the llms evaluated perform poorest on downstream tasks when perplexity surpasses 1000.

Weakness5: the impact of Decoding strategy on text generation

We thanks the reviewer's insightful comments on the impact of decoding strategy. Decoding strategies indeed play a role in the performance of language models. However, such impact is typically limited and unlikely to dramatically improve the performance of a collapsed model in downstream tasks.

Here, we address the concerns raised:

• Our experiments use standard settings prevalent in the field, specifically temp=0.0, do_sample=false.
• We have tested various parameter settings, including temperature, top_k, and num_beams, and observed that models experiencing collapse maintain their collapsed state regardless of the decoding strategy employed. This confirms that the observed model collapses are not mitigated by altering decoding strategies.

We will discuss the impact of decoding strategy in the revised manuscript.

Weakness6: the differences between samples that cause model collapse and samples

We thank the reviewer for the constructive feedback. To better elucidate the difference between the "hard examples" that cause model collapse and normal counterfactual examples, we plan to make the following additions in the revised manuscript:

1. Comparison between Hard and Normal Samples: Our paper features Table 2 and Figure 11 to showcase hard examples from the COUNTERFACT dataset for clarity. To elucidate the difference between hard and normal samples, we offer a direct comparison:

   • Hard Samples often involve subjects that are single, commonly used words, such as 'France', 'Scotland', 'DVD', 'iPhone', and 'Xbox'.
   • Normal Samples, in contrast, typically relate to more specific entities or less common terms like 'Kieran Millan', 'Battle of Arausio', 'Microsoft Expression Blend', and 'Madhan Bob'.

   Constructing our dataset with GPT-3.5 using patterns of commonly used single words revealed that a notable quantity of these inputs could induce model collapse, thereby illustrating the effectiveness of such patterns. We will release the HardCF and HardEdit datasets we developed, to enable analysis and reproducibility by the research community.

2. Spatial Distribution and Editing Challenges: Further analysis, using GPT-2-XL as an example, shows that the "keys" of hard samples (as defined in the ROME paper) are spatially dispersed and significantly distant from those of normal samples, which tend to cluster more closely in the space.

Our primary goal is to unveil this phenomenon through extensive exploration. Unveiling why the key differences between these two types of samples cause editing methods to trigger model collapse, are what we currently study. However, it is beyond the scope of the current paper.

Suggestions And Typos

Firstly, we sincerely appreciate for the reviewer's detailed suggestions on our paper. We apologize for the excessive abbreviation of text due to space limitations, which resulted in unclear expressions.

• 145: I find this notation confusing
  • Your understanding is correct; what we intended to convey is that we obtain the parameters of edited model using the editing algorithm. We will revise the expression here.
• 159: portability is an important property
  • We appreciate for pointing out this deficiency; we will add an introduction for portability.
• 167: A constraint such as what?
  • We apologize for the lack of clarity; the explanation of constraint has been placed in the appendix (L996-L999) due to space limitations. For example, a ${\ell }_{\mathrm{\infty }}$ norm constraint on the fine-tuning loss, limiting the difference between the original and edited model's parameters, to reduce side effects.
• “Does Localization Inform Editing?...”
  • Thank you for your supplement; indeed, an analysis of the localization methods should be incorporated into the discussion.
• Case ID? What is the perplexity measured on? the baseline perplexity?
  • The Case ID refers to the index of each edit sample.
  • The perplexity of edited model was measured on sentences from ME-PPL dataset. And the baseline perplexity is about 65.60.
  • This figure demonstrate that, in single editing setting, the edit samples from ZsRE will not lead model to exhibit high perplexity on normal text, indicating that the model remains stable. We will refine our writing to make them clear.
• What is perplexity calculated on? the red bar is perplexity of the baseline model？
  • Each point in Figure 2a shows a model edited by one sample from COUNTERFACT, with its perplexity calculated on sentences from the ME-PPL d.
  • The red line is not the baseline. It appears like a line because there are too many points connected together.
• 304: what generation settings?
  • We set the temperature as 0 to generate text, which is standard settings in the field. And we will polish this part to make it clear.
• 325: I don’t think locality is the only measure of ... model collapse. I mentioned portability ... Fluency and Consistency ...
  • For Fluency and Consistency, we have answered in response to Weakness4.
  • For portability, it examines the performance of an edited model on facts related to the editing request and required reasoning, aiming to assess robust generalization.
  • We will add discussion about these metrics in the revised manuscript.
• Table 1+327: I don’t know what locality of 1 means?
  • You're correct: 1 means the edited model provides the correct answer to the question corresponding to the locality metric. However, this model performs poorly on PIQA. This indicates that locality alone is not sufficient to identify model collapse.
• 348: What is the theoretical perspective you are referring to from Radford?
  • The theoretical perspective means the unsupervised pre-training loss has an exponential relationship with the perplexity metric. We will make this clear later.
• 433: I think it is critical that you do provide a set of tables for these experiments you can use the appendix...
  • We will put all the perplexity results on two datasets in the appendix in the next version of the paper.
• 448-449 How is this different from the rest of the samples like what were the particular formats?...
  • This can be referred to the response to Waekness 6.
• Table 3: I think it would enhance the paper if you also presented lowest perplexity edit and potentially a perplexity between lowest and highest...
  • Actually, the results depicted in Figure 3, which illustrate the performance of downstream tasks at various levels of perplexity, can address this question. Due to space constraints, we presented a limited set of results in Table 3. With more space available in future, we intend to include more experimental results according to the suggestions.
• 498: What is occuring few times? Less than 60 in the normal cases?
  • Your understanding is correct. 60 edits is indeed a small number in practical applications.
• Table 4 - 520-536: I really like this analysis. I think my worry is the phrasing “pose a substantial risk of collapsing under sequential editing” since the hard samples are a very very very small part of Counterfact. I do think ...
  • We thank the reviewer for acknowledging the value of the analysis. For this concern, we believe we have provided sufficient discussion in response to Weakness 3.
• How was ii) determined?
  • To ensure that the generated data meets condition ii), we instructed GPT-3.5 to produce statements that are clearly counterfactual. Detailed prompt used for data construction and examples of the generated data can be found in the appendix.

For other writing suggestions like:

298: Maybe “top 30 post-edited models” for clarity
322: “Proven crucial” is maybe to strong for me at this point. “Shows promise” is more appropriate
333: I don’t see locality as a QA task?...
377-384: I understand what you are saying here but I think it could be clearer that you are selecting models which achieve these perplexity values on ME-PPL and then benchmarking them according to LM-eval - it took me a bit to get to this understanding
Figure 5: I am not sure what to do here but I can see how this chart could be a bit misleading since the y axis are all different. Maybe you can mention this in the caption and caution the reader to pay careful attention to the differences.

We will fix them in the revised manuscript.

We hope we have addressed all your concerns. If things are clearer now we kindly request you to raise the score. Thanks!