Reviewer has chosen to be Anonymous

Overall Impression: Good

Content:
Technical Quality of the paper: Good
Originality of the paper: Yes
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Limited
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Good

Detailed Comments:

The paper is well written and addresses a relevant topic for neurosymbolic AI: the evaluation of long-term memory in partially observable environments using explicit temporal knowledge-graph representations. The proposed RoomKG benchmark is interesting, and the comparison between symbolic KG/TKG agents and neural sequence-based baselines is potentially valuable. The paper also makes a clear effort to define the benchmark, its scope, its metrics, and its reproducibility artifacts.

However, the presentation would benefit from better organization and from more explicit explanations of several preliminary concepts. At present, the manuscript sometimes assumes background knowledge that not all readers of a neurosymbolic AI venue may have. In particular, the benchmark interface and the types of agents that can operate in it should be introduced earlier and more systematically.

A key point that should be clarified is what kind of agents the benchmark is intended to support. The manuscript later compares symbolic KG/TKG agents with LSTM and Transformer agents trained with a DQN objective, but this comes after the benchmark has already been introduced. Earlier in the paper, it would help to include a short subsection or introductory paragraph explaining the agent interface: what the agent observes, what it must answer, what action it chooses, what private memory it may maintain, and whether the benchmark is intended for symbolic agents, RL agents, neural sequence models, LLM-based agents, or any agent satisfying the observation–query–action protocol. This would make the benchmark definition easier to understand before the experimental agents are introduced.

The discussion of 'memory queries and exploitation policies' needs more explanation. The paper mentions that TKG agents use policies such as most recently added, most recently used, and most frequently used, and eviction policies such as FIFO, LRU, and LFU. These are important components of the benchmarked agents, but they are not properly introduced.
The paper should explain earlier what kinds of memory queries are possible, how annotations such as :time_added, :last_accessed, and :num_recalled are updated, and why these quantities are meaningful for answering object-location queries. It would also help to distinguish clearly between:

1. policies used to answer a query,
2. policies used for exploration or frontier selection, and
3. policies used for memory eviction.

At present these are all described, but the organization makes the distinction hard to follow.

The sentence: 'At each timestep the agent first answers the query by returning the room of the specified object (reward +1 if correct, otherwise 0) [...]'

is also somewhat confusing in context. The mention of reward appears before the paper has clearly introduced reinforcement learning or explained whether the benchmark itself is an RL environment. Since the symbolic agents appear to use deterministic memory/query rules, while the neural baselines are trained with DQN, the paper should clarify whether the reward is part of the benchmark interface for all agents, only used for RL-trained baselines, or simply an evaluation signal (as it seems). A clearer formulation would be to first define the correctness metric, then separately explain how the neural agents use the reward signal for training.

The paper would also benefit from a more explicit description of the query types. The current benchmark appears to use object-location queries of the form '(X, :at_location, ?)', but this should be emphasized as the primary query class. If the benchmark supports only this query type, that should be stated clearly. If additional query types are possible or planned, they should be described. Similarly, the exploitation policies should be introduced as concrete mechanisms for selecting among candidate stored answers.

Overall, the manuscript presents a promising and useful benchmark, but it should be revised to make the benchmark protocol and agent assumptions clearer. In particular, I recommend adding an early, compact 'Agent interface and task protocol' subsection, followed by a clearer explanation of the supported query type, memory-access policies, exploration policies, eviction policies, and the role of reward. These changes would make the contribution more accessible and would improve the organization of the paper substantially.

Minor points:

- Please define 'lcm' when it first appears in the expression `Pw = lcm(|p1|, ..., |pW|)`.
Many readers will infer that this means “least common multiple,” but it should still be stated explicitly.

- There are several small typographical and language issues that should be corrected during revision, for example around 'question-answer' vs. 'question answering', and consistency of notation such as 'time_added' / 'time added'.

Reviewer has chosen to be Anonymous

Overall Impression: Average

Content:
Technical Quality of the paper: Good
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Good
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Average

Detailed Comments:

The submission proposes a benchmark for evaluating memory and navigation of temporal systems, through a configurable grid system with dynamic walls and objects (which change at cyclic intervals). At each position, the agent receives only information of its location and the local view. Queries are made about object locations, and navigation to their location. The benchmark, which is publicly available through three locations (the general software, the benchmark, and the agent implementations), is evaluated over 4 agents: two symbolic, one LSTM, and one Transformer. An analysis suggests that symbolic systems with explicit memory allocation behave better on these tasks.

The proposed benchmark is interesting and evaluates a relevant problem. From the description, it is well constructed and designed. The text is in general clear and well-presented. I have two main concerns about it.

1. First, many of the details in the construction and the evaluation are provided externally to the paper. The paper promises several properties, but one has to go to the benchmark artifacts to even see how these details come into place. A partial example of the deterministic replays, or details about the room coverage, or the general navigation, or some partial evolution images, would help understanding the details.

2. Second, the experiments are, to my view, underspecified and incomparable between neural and symbolic systems. The systems are evaluated with a memory capacity ranging from 0 to 512; 0 means no memory, but what does the 512 mean? 512 what? In the symbolic systems, are these 512 triples? 512 observations? 512 bits? Similarly for the neural system: is 512 the size of the context allowed? The dimension of the embedding? How can the two very different representations be compared? In addition, the authors present, for each capacity, the best-performing system out of 27 options (which could differ from capacity to capacity) thus giving an unsafe advantage to that method. A way to alleviate this would be to provide also information about the worst, or at least the median system.

Smaller issues:

- the results are provided without any analysis or reading of the numbers provided.
- In page 7 the authors cllaim that the Transformer agent outperforms the LSTM agent, but this is not supported by the figure
- if the run is deterministic, how could the neural methods not visit all rooms?

- figure 4, put a) and b) on top, c) and d) below.

Reviewer has chosen to be Anonymous

Overall Impression: Average

Content:
Technical Quality of the paper: Average
Originality of the paper: Yes, but limited
Adequacy of the bibliography: Yes, but see detailed comments

Presentation:
Adequacy of the abstract: Yes
Introduction: background and motivation: Good
Organization of the paper: Needs improvement
Level of English: Satisfactory
Overall presentation: Average

Detailed Comments:

The paper introduces RoomKG, a deterministic room-grid benchmark for evaluating long-term memory in partially observable neurosymbolic agents. The environment is represented in RDF/KG form, and the authors compare symbolic KG/Temp. KG agents with LSTM and Transformer baselines. The topic is relevant and the benchmark direction is promising. However, I recommend major revisions, because several methodological and representational issues currently limit how the results can be interpreted confidently.

The temporal RDF/KG representation should be clarified and positioned more carefully. The paper repeatedly refers to RDF 1.2-annotated triples, but the concrete representation remains hard to inspect from the manuscript alone. Since no serialized RDF/Turtle artifacts seem to be provided, the claimed RDF representation is difficult to verify without reconstructing the graph generation from code. In the supplement, the implementation appears to use old-style RDF blank-node reification, with rdf:Statement, rdf:subject, rdf:predicate, and rdf:object, plus qualifier triples. This is valid RDF, but it is different from the claimed RDF 1.2 triple terms/reifiers annotation syntax. The paper should include a small side-by-side example showing the intended RDF 1.2-style representation and the concrete artifact representation. It should also provide concrete RDF/Turtle examples for a hidden-state fragment, a local observation, and a temporal memory entry.

The paper should also better position its temporal representation against prior temporal and spatio-temporal RDF work, including Temporal RDF, named graphs/quads, RDF-star/RDF 1.2 annotations, stRDF/stSPARQL, OWL-Time, and RDF stream processing. In particular, temporal validity should be distinguished from memory-management metadata such as last_accessed and num_recalled.

The temporal memory operations are under-specified. The paper should define how insertion, repeated observation, update, query-time metadata update, eviction, deletion, and outdated or contradictory facts are handled. This is central for object-location tasks, where multiple observed locations of the same object may coexist unless the update semantics are explicitly defined.

The experimental protocol should be clarified. A compact train/test table should specify which environment components are shared and which are changed, including layout, wall dynamics, object behavior, query order, and random seeds. Without this, the reported generalization result remains ambiguous.

The reported results would benefit from stronger calibration. The currently best TKG result is better than the neural agent baselines, but its absolute score remains modest. The paper should add baselines such as a hidden-state oracle, a perfect-memory symbolic agent, an observation-only baseline, or a baseline siply using last-seen facts. These are important because the current absolute QA scores are otherwise difficult to interpret.

The symbolic vs neural comparison is substantially confounded. The symbolic agents use explicit graph retrieval and hand-designed exploration (BFS) policies, while the neural agents appear to learn a joint movement-and-answering policy. The neural baselines process observations in sequence, so temporal order is present implicitly, but they do not appear to receive explicit time, recency, currentness, or access-count features comparable to time_added, last_accessed, and num_recalled. Thus, the experiments compare KG versus TKG on the symbolic side, but do not include an analogous "temporal informed neural agent". Such a control would help separate the benefit of temporal information from the benefit of symbolic graph operations and hand-designed policies.

The hidden state notation should be introduced more carefully. The term s_t should be clearly distinguished from the local observation o_t and the agent's internal memory M_t. The deterministic question-move loop could also be formalized, since the order of answering, reward computation, object movement, agent movement, wall updates, observation, and memory update is currently too vague. In general, the paper would beenfit from more formal rigor, by introducing the used concepts with proper formal definitions.

Finally, the room-coverage results should be interpreted more cautiously. Room coverage is likely strongly influenced by the exploration policy and thus should not be treated as direct evidence for memory quality alone. The figures should also be improved for readability, especially Figures 1 (increase font size), 2 (maybe better show subgraph ur turtle example), 3 (distinguishable colors), and 5. Overall, the current experiments are informative for this controlled benchmark setting, but they do not yet justify broad claims about the effectiveness of temporal KG memory across agent architectures.