New ask Hacker News story: Ask HN: Have you noticed an improvement in AI responses with memory disabled?
Ask HN: Have you noticed an improvement in AI responses with memory disabled?
2 by zbikowski | 0 comments on Hacker News.
I use Claude on the $20/mo plan, mainly for research, "rubber duckying," and as an idea soundboard (I was using Fable for this but am now relegated to Opus 4.8 or Sonnet 5), and recently noticed a severe downturn in Opus 4.8 response quality in high-context situations. To attempt to remedy this, I disabled the memory system (now labeled "Legacy" even though to my knowledge there is no replacement). With this setting disabled, aside from the slight tedium of inputting the necessary context each time for task continuations, I have noticed a drastic improvement in response quality and accuracy—I'm one of those shmucks who whined a bit after Opus 4.7 came out, and my theory is some change was made to increase memory context that resulted in a decrease in actual response quality. In general, I wonder if the high-context abilities of SOTA thinking models are overstated. As a brief anecdote, I was working through a research task with upwards of 30 prompts. Switching to a new chat and condensing the gist of the discussion into a single new prompt yielded superior results than continuing to prompt the prior chat, despite (or due to, I would argue) the prior conversation's wealth of context on the subject. Aside: I wish that "project" and "chat" memory were separate settings. Disabling memory boosts the capabilities of chat, but severely nerfs projects by disabling their internal memory, leading the model to grope around for files each time you ask it to reference one. I suppose a hacky solution could just be to create a project for everything, but this seems tedious. Is this a known phenomenon? Have recent changes exacerbated things? Any thoughts and tips are appreciated. Thanks!
2 by zbikowski | 0 comments on Hacker News.
I use Claude on the $20/mo plan, mainly for research, "rubber duckying," and as an idea soundboard (I was using Fable for this but am now relegated to Opus 4.8 or Sonnet 5), and recently noticed a severe downturn in Opus 4.8 response quality in high-context situations. To attempt to remedy this, I disabled the memory system (now labeled "Legacy" even though to my knowledge there is no replacement). With this setting disabled, aside from the slight tedium of inputting the necessary context each time for task continuations, I have noticed a drastic improvement in response quality and accuracy—I'm one of those shmucks who whined a bit after Opus 4.7 came out, and my theory is some change was made to increase memory context that resulted in a decrease in actual response quality. In general, I wonder if the high-context abilities of SOTA thinking models are overstated. As a brief anecdote, I was working through a research task with upwards of 30 prompts. Switching to a new chat and condensing the gist of the discussion into a single new prompt yielded superior results than continuing to prompt the prior chat, despite (or due to, I would argue) the prior conversation's wealth of context on the subject. Aside: I wish that "project" and "chat" memory were separate settings. Disabling memory boosts the capabilities of chat, but severely nerfs projects by disabling their internal memory, leading the model to grope around for files each time you ask it to reference one. I suppose a hacky solution could just be to create a project for everything, but this seems tedious. Is this a known phenomenon? Have recent changes exacerbated things? Any thoughts and tips are appreciated. Thanks!
Comments
Post a Comment