Metatuning: An Empirical Study of Judge-Guided Prompt Refinement and Its Boundary Conditions

Tracking #: 920-1940

Flag : Awaiting Camera-Ready

Authors: 

Aniruddha Chattopadhyay
Raj Dandekar
Kaushik Roy

Responsible editor: 

Guest Editors NeSy 2025

Submission Type: 

Article in Special Issue (note in cover letter)

Full PDF Version: 

PDF icon nai-paper-920.pdf

Supplementary Files: 

PDF icon nai-supplementary-920.pdf

Cover Letter: 

Dear Editors, We are pleased to submit a revised version of our manuscript, “Metatuning: An Empirical Study of Judge-Guided Prompt Refinement and Its Boundary Conditions,” for consideration as an article in the Neurosymbolic Artificial Intelligence Journal special issue. We are grateful for the reviewers’ constructive feedback. In this revision, we have substantially updated the manuscript to align its framing with the empirical evidence and to address all points raised in the decision letter and reviews. The paper is now explicitly positioned as an empirical investigation of judge-guided prompt refinement (metatuning) and its limits. We emphasize the negative findings as valuable scientific contributions that identify boundary conditions, particularly the limited additivity with strong prompting baselines (Chain-of-Thought and self-reflection) and the lack of measurable benefit in spatio-temporal video reasoning (CLEVRER). Key changes in the revised manuscript include: A narrative pivot in the title, abstract, introduction, and conclusion to foreground empirical findings and boundary conditions rather than broad conceptual claims. - Strengthened analysis and discussion of Section 5.2, including a hypothesis for model-dependent behavior and performance degradation when combining metatuning with CoT/self-reflection. - Clarification that our current judge corrects final answers rather than intermediate reasoning traces, and explicit positioning of trace-level feedback as future work. - Clear differentiation between metatuning and standard in-context learning, emphasizing error-driven example selection and learned prompt construction. - Expanded discussion situating metatuning within related work on prompt optimization and meta-prompting. - Correction of the editorial and formatting issues noted by the reviewers. We have uploaded our detailed point-by-point responses as a supplementary file (“response_to_reviewers.pdf”), as indicated. We confirm that this manuscript is submitted as a revision to the Neurosymbolic Artificial Intelligence Journal special issue. We also disclose that an earlier version of this work was previously posted as an arXiv preprint; the present submission represents a substantially revised and expanded version with updated framing, analysis, and clarifications in response to the reviews. Thank you for your time and consideration. We would be happy to provide any additional information if needed. Sincerely, Authors

Approve Decision: 

Approved

Previous Version: 

Learning and Reasoning with Model-Grounded Symbolic Artificial Intelligence Systems

Tags: 

  • Reviewed

Decision:
Accept

Solicited Reviews: