From learned representations to engineered context: what Richard Socher's research tells us about working with AI

Every engineer I know who works with AI coding assistants has hit the same wall. The first session is productive. The second session is decent. By the fourth or fifth session, the AI is generating code that conflicts with decisions made three sessions ago. Naming conventions drift. Architectural patterns diverge. The codebase starts to feel like it was written by a team of contractors who never talked to each other.

Better prompts help. Detailed instructions can resolve a misunderstanding in the current session. But the next session still needs the decisions behind those instructions, and the codebase needs a way to keep them consistent.

I've been thinking about why this happens, and I keep returning to the work of Richard Socher. His research, from his PhD at Stanford through GloVe and DecaNLP to his work at You.com, offers a useful way to think about the infrastructure we need when building software with AI.

Learning representations that other tasks can use

Many NLP systems relied on hand-engineered features. A sentiment classifier might use a lexicon of positive and negative words; a parser might use grammar rules. Designing those features required knowledge of the task and domain. Socher's work helped advance an alternative: learn representations that capture useful relationships in the data.

His Stanford PhD thesis won the 2014 Arthur L. Samuel Best Computer Science PhD Thesis Award. His recursive models built phrase and sentence representations by composing word-level vectors along parse trees. The sentiment work showed why structure matters for negation: a bag of words can contain both “not” and “good” without representing how negation changes the phrase. The recursive model improved on the compared bag-of-features baselines, including on short phrases where composition matters.

GloVe (Pennington, Socher, and Manning, 2014) took a different route. It learned word vectors from global co-occurrence statistics, providing representations that other systems could reuse. It did not compose sentence meaning along parse trees. The ACL 2024 Test-of-Time Paper Award recognized its lasting contribution.

These approaches give engineers two useful ideas to work with: represent relationships explicitly, and invest in representations that can serve more than one task.

Reusable context for software development

When a team relies on isolated prompts, each session becomes another attempt to specify the project from scratch. The developer describes a task, reviews the generated code, accepts or modifies it, and moves on. Unless the decisions become part of the project's shared context, the next session has to rediscover them.

Persistent instruction files and architecture records give that knowledge somewhere to live. The engineering work is deciding what belongs there, keeping it current, and making sure it reaches the sessions that need it. A growing file of stale instructions can create its own confusion.

The connection I draw from Socher's research is reuse. Learned representations let multiple NLP tasks benefit from an earlier investment. Engineered context lets multiple development sessions build on architectural decisions, conventions and working examples.

This is what synthesis coding addresses: persistent context files that encode architectural decisions, quality gates that apply regardless of who or what wrote the code, and a foundation-first pattern in which humans build the initial architecture by hand. Those exemplars teach the AI how we build here.

Structure gives generated work a place to fit

Socher's recursive neural tensor networks learned how to compose representations along structured trees. The tree determined which smaller pieces were combined into larger ones. Structure participated in the computation.

In software, architecture decisions, naming conventions, data contracts and integration patterns define how pieces fit together. When those constraints stay implicit, separate coding sessions can produce individually plausible components that disagree with one another. Writing the constraints down gives both the assistant and the reviewer a common basis for evaluating a change.

That is why human architectural authority is a first principle of synthesis coding. Humans make the strategic decisions that define the system's structure; AI implements within those constraints. I think of the architecture as the parse tree and the generated components as the pieces composed within it. The analogy makes the architect's job concrete: decide how the parts belong together, then review whether the implementation honors that design.

From DecaNLP to disciplined prompting

In 2018, McCann, Keskar, Xiong and Socher published the DecaNLP paper. They cast ten NLP tasks, including question answering, translation, summarization and sentiment analysis, as question answering over context. Their shared model learned the tasks without task-specific modules or parameters in the multitask setting. The question was part of the input, directing what the model should do with the context.

Socher has described this as inventing "part of prompt engineering." His 2025 discussion of the work emphasizes that shift: put the task into the input. DecaNLP is an early demonstration of a general interface through which natural language specifies different tasks. Modern prompting has multiple lineages; the useful connection here is the move from a separate interface for each task to a shared model directed by a question.

For engineers, that opens the next problem: how do we make repeated use of such an interface dependable in a changing codebase? A natural language request starts the work. The surrounding discipline must carry it across sessions:

  • Versioned context artifacts preserve architectural decisions and specifications.
  • Systematic review applies the same standards to AI-generated and human-written code.
  • Foundation-first development establishes the patterns the AI will follow.
  • Iterative context building carries what the team learns into subsequent work.

Prompt engineering is a skill within synthesis coding. The discipline also includes the context, review and maintenance that make those prompts useful over time.

The "managers of AI" connection

Late in 2025, Socher posted on LinkedIn a view that I wrote about at the time because it articulated something I'd been trying to systematize:

"The future of work is all of us becoming managers of AI. Similar to moving from individual contributor to people manager: learning to delegate clearly, specify requirements, build trust. That's the skill we need now."

I apply that management idea to engineering through five practices. The table extends the idea into the responsibilities I see in synthesis coding:

My engineering applicationSynthesis coding principle
Delegate clearlyHuman architectural authority — define the structure, let AI implement
Specify requirementsIterative context building — accumulate specifications in persistent artifacts
Build trust through verificationSystematic quality standards — review, test, validate regardless of origin
Maintain understanding of the workActive system understanding — you must be able to debug what you ship
Enable team collaborationTransferable knowledge — ensure the work is comprehensible by others

Socher approaches production quality through building and deploying AI systems, first at Salesforce and then at You.com. His warning captures a familiar gap: "It's easy to make a quick prototype with an LLM. It's difficult to make them accurate at scale." His last-mile discussion focuses on the work needed to make impressive demonstrations reliable. I reach the same engineering priority from using AI assistants to build production software and discovering, through practice, what works and what doesn't.

Transfer learning and the foundation-first pattern

In 2017, McCann, Bradbury, Xiong and Socher published CoVe, Contextualized Word Vectors. They used a deep encoder trained for machine translation to produce representations for downstream NLP tasks. The paper drew an analogy to ImageNet pretraining in computer vision: invest in a useful representation, then let other tasks benefit from it.

That investment-and-reuse pattern is also central to foundation-first development. I recommend reserving roughly 10–20% of project time for building the core architecture by hand: exemplars that establish patterns, conventions and quality standards. AI can then extend those patterns across the implementation.

CoVe transfers learned model representations; foundation-first development supplies architectural examples and instructions to a coding workflow. They use different mechanisms, but the practical lesson travels well: the quality of a reusable foundation shapes the work built on it.

Capability and responsibility

Socher expects AI to surpass humans in domains that allow verification. He has said: "Any domain where you can verify and/or simulate that domain, AI will dominate it and get eventually better than humans. And programming you can verify."

I regard human architectural authority as a permanent responsibility in production engineering. Someone must decide what the system is for, which tradeoffs it should make, and who bears the consequences when it fails. A model becoming better at implementation does not settle those questions.

This is a productive tension with Socher's forecast. Better verification can expand the work we delegate to AI. Architectural judgment also involves organizational goals, team responsibilities and non-functional requirements that people must choose and reconcile. I want more capable AI to help with that reasoning while keeping human accountability explicit.

If AI can maintain architectural coherence autonomously across long development efforts, our working methods should evolve. Systematic quality, explicit structure and verified output will remain necessary wherever the human-AI boundary settles.

What this means for your work

Stop treating each session as independent. Use CLAUDE.md files, architecture decision records, or whatever your toolchain supports to accumulate knowledge across sessions. Record the decisions and reasons a future session will need. Keep that context current as the system changes.

Invest in your foundation. Write the core architecture by hand. Establish patterns, conventions and quality standards before asking AI to generate at scale. Give it examples worth repeating.

Review AI-generated code with the same rigor you'd review human code. The prototype-to-production gap makes quality standards essential. Speed without review lets defects and conflicting patterns spread through more code before anyone understands them.

Maintain understanding. If you can't debug the code at 2 AM when production breaks, either you need to understand it better or it needs to be simpler. That is an operational requirement.

The lesson I take from Socher's research is to put reusable knowledge into the structure of the work. In synthesis coding, that means engineered context, explicit architecture and review against a shared standard. Each session should leave the next one a stronger foundation to build on.

This is the first in a series of three articles connecting Richard Socher's work to synthesis engineering and synthesis coding. The second article traces the research lineage in more depth for an academic audience. The third frames the connection for business and technology leaders.