I. From Tomer Wolfson:
On July 2nd, I traveled to San Diego for ACL 2026. I presented two works, our TACL paper on the MoNaCo benchmark and a joint work, led by Dhruv Madhwal (ASU) and David Zhang (UPenn), on a new method for validating LLM factual consistency through Decomposed Prompting. You can also check out their abstracts elsewhere on the CCG blog (thanks, Jen!).
This conference saw slightly fewer attendees than past conferences (~3.5K), mostly due to the clash with July 4th celebration of 250 years, and the proximity to ICML. Still, there were quite a few thought provoking talks during the conference. I particularly enjoyed Noah Smith’s talk at “The Big Picture Workshop“, on how to do impactful research in a fast-moving field like AI. You can actually watch his talk for free using this link.
Another interesting talk was Philip Resnik’s keynote, which can be found here. The talk raises questions on the role of *ACL conferences relative to AI and Computational Linguistics. It is a thought provoking talk that raises many questions on the future of *ACL conferences and their place within the broader AI research community.
Below, are three of the papers which I found relevant to my research:
1. ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery (Hebrew University)
ScheMatiQ is an interactive LLM-based system that helps researchers transform natural language research questions and document collections into structured, evidence-backed datasets by automatically discovering task-specific schemas and extracting grounded data. Its key contribution is treating schema discovery as part of the research workflow, allowing expert users to iteratively refine both the schema and extracted data instead of requiring a predefined ontology. Experiments with legal and biomedical research tasks show that ScheMatiQ produces schemas and structured outputs that closely align with expert annotations while substantially reducing manual data curation effort. Overall, this work demonstrates how combining LLM-assisted schema induction with human oversight can make structured evidence extraction more efficient, flexible, and accessible for open-ended scientific research.
2. OctoTools: A Multi-Agent Framework with Extensible Tools for Complex Reasoning (Stanford)
OctoTools is a prompt-based multi-agent system that enables LLMs to solve complex reasoning tasks by orchestrating external tools through hierarchical planning. Its key contribution is a modular architecture based on standardized “Tool Cards”, allowing new tools to be integrated without retraining while separating high-level task planning from low-level tool execution. Across 16 benchmarks (MMLU-Pro, MedQA, GAIA-Text, etc.), OctoTools improves average accuracy by 9.3% over GPT-4o and outperforms prior agent frameworks such as AutoGen, LangChain, and GPT-Functions under comparable tool settings. My main takeaway is the introduction of Tool Cards, as another layer for tool-using agentic systems, very similar to Anthropic’s “skills”. The usage of tool cards encapsulates heterogeneous tools while providing the planner agent with metadata such as input-output formats, usage constraints, and best practices for ideal use cases. This standardized approach enables an easy extension of the agentic system by adding new tools and replacing or expanding existing ones.
3. Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants (USC)
As its name indicates, this work aims to use LLMs to assist public health experts in uncovering new opportunities for suicide prevention. The methods are designed to assist human experts, not replace them, with a strong emphasis on expert oversight. They use a public dataset, The National Violent Death Reporting System (NVDRS) with 270K suicide cases labeled with structured variables and narrative text describing the death. The annotation of such data is emotionally taxing and prone to inconsistencies. To address these challenges, they incorporate LLMs as peer validators for NVDRS annotations. They instruct LLMs to predict the label for a variable given the NVDRS codebook in the prompt and find that LLMs can match existing data annotations 85% of the time across 50 variables. My main takeaway from this work is how LLMs can be effectively integrated in the annotation pipeline in domains outside of code, Wikipedia or financial workflows. There are many data-intensive jobs that could greatly benefit from AI systems, and gauging the performance of current LLMs on these new domains is clearly important.
II. From Alon Jacoby:
As ACL concluded this year, I wanted to share a few impressions based on the papers I saw and conversations at the conference.
These topics are close to my interests, especially principled agent design and reliable evaluation, so this is not a representative survey. Still, a few trends stood out.
ACL may have suffered from unfortunate timing, with ICML nearby and the July 4th holiday competing for attention. The conference was active, but attendance and poster interaction seemed lower than they might otherwise have been.
The poster sessions also reflected the growing number of papers applying LLM agents to specific domains, especially biology and medicine. This is valuable, but it raises questions about ACL’s role as the field expands, echoing Philip Resnik’s discussion of ACL’s place in the broader research community.
At the same time, traditional computational linguistics remains present. I saw work on syntax, multilinguality, and linguistic structure, including a paper using syntactic information to improve Coptic translation through in-context learning. The shift is therefore not a replacement of computational linguistics by LLM applications, but in the balance among computational linguistics, general-purpose LLM research, and domain-specific work.
Three papers stood out to me.
1. Grammar Search for Multi-Agent Systems – Singh et al.
This paper searches over multi-agent architectures using a context-free grammar rather than unconstrained code or natural-language generation. The grammar specifies component types and valid compositions, for example allowing a multi-input, single-output component such as a voter or consensus builder to connect to other compatible components.
This separates the question of which architectures are admissible, from how to search among them. It also encodes structural knowledge directly instead of asking an LLM to “rediscover” it.
The search itself is simple: architectures are sampled from the grammar, with forced sampling used to ensure that less frequently encountered components are also explored, and each new architecture is evaluated on a validation set. The final procedure does not use architecture performance to adapt the sampling distribution, although the authors report experimenting with more sophisticated strategies such as Thompson sampling and Monte Carlo Tree Search without seeing gains. Still, I think an interesting direction would be to make production probabilities explicitly task-dependent, for example by learning which components tend to be useful for particular classes of tasks.
2. When Benchmarks Leak: Inference-Time Decontamination for LLMs – Chai et al.
This paper addresses benchmark contamination, where a model’s performance may be inflated because it encountered the test data during training. The authors perturb inputs in embedding space so that a contaminated model behaves more like a less-contaminated reference model, while leaving the discrete input text unchanged.
This avoids the obvious problem with paraphrasing or rewriting questions, where changing the text may change the task. The authors provide empirical evidence that their perturbations largely preserve the original input, including high similarity between original and perturbed embeddings and relatively small utility degradation on clean examples. They also explicitly acknowledge, however, that they do not have a formal guarantee that semantics or difficulty are preserved for every input.
I think this remains an important point. Closeness in embedding space is not obviously equivalent to equivalence in effective task difficulty, so it is difficult to completely rule out the possibility that some of the apparent decontamination comes from a more generic change in the model’s ability to solve the example. Still, the empirical results are promising under the paper’s evaluation metrics: the method brings contaminated-model behaviour substantially closer to that of the reference model while generally avoiding large degradation on clean data.
3. Do Generalisation Results Generalise? – Boglioni et al.
This paper asks whether strong performance on one OOD test set predicts strong performance on another. Raw OOD correlations are confounded by general model quality: later or better checkpoints may simply perform better on everything.
To address this, the authors regress each OOD test-set performance on in-domain performance and correlate the resulting residuals across OOD datasets. These residuals capture whether a checkpoint performs better or worse OOD than its in-domain performance would predict.
The resulting relationships are unstable. Two OOD datasets can correlate positively for one model family and negatively for another. Thus, a checkpoint that appears to generalize unusually well under one distribution may not do so under another, and more generally conclusions about OOD generalization can depend substantially on the particular test distribution.
This is a useful reminder that “OOD generalization” is not a single model property, but rather that it depends on the distribution shift being tested. More generally, benchmark scores are often treated as measurements of broad abilities such as reasoning, robustness, or generalization. These results suggest we should be cautious about assuming that different operationalizations measure the same underlying property.