Jane Street: Socratic Tutor (github.com)

442 pointsby tosh2 days ago99 comments

sigbottle 5 minutes ago

I don't think a simple prompt will exceed any kind of memory system. Nothing fancy, but you need the agent to tune to your quirks; I find that a prompt to try and distill everything a priori is too annoying.

Been just using less of chat recently and using more codex for learning - a direct shared interactable terminal is nice, it's still relatively cheap, and importantly you can log specific thoughts to say a git repo for memory.

It's not attempting to be some kind of knowledge ontology graph like many businesses want - you just want it to understand the specific quirks of your specific learning path, which may not be formalizable or worth be spending time to step back and formalize every little thing.

Tar90 an hour ago

I've been experimenting with a lot of these learning skills (see also, e.g., https://github.com/mattpocock/skills/blob/main/skills/produc..., which I think addresses the progress tracking part better than this skill) and they're great for getting simple answers, getting unstuck on exercises, getting some intuition through simulations, etc, but I still feel some more structure is needed for effective self-teaching. Has anyone tried using a hard book as a sort of course structure and asking the LLM to teach you through it? You could of course just read the book, but a lot of advanced subjects don't have great learning materials.

vaskebjorn an hour ago

I just used this skill with a dense philosophical text this morning and I think it did a great job. I was exhausted after 15 minutes.

pkoird an hour ago

Is it only me who's thinking what's so special about this and whys everyone oohing and aahing over this? Afaik, people I know (and me) we've been doing this ever since LLMs became capable enough.

baby an hour ago

SKILLs are often underwhelming I find

frangonf 39 minutes ago

Is it wise to believe that every user of LLMs know about Socrates and his impertinent questioning he spent his time on around Athens? Is it possible that some millionaire quants math wizards found that llmwashing this ancient techniques is food for humanity in this uncertain times, or if anything, for their portfolio?

alephnerd 37 minutes ago

Pressure testing ideas and general discussion. You know - like the Socratic method.

A lot of people don't know about the Socratic method either so this is their first applied introduction to it.

As a meta-comment, this kind of reflexive dismissiveness is why HN has gone to shit. Zero curiosity, only dismissiveness.

natebc 27 minutes ago

TBH I've always given Jane Street a little side-eye. They position themselves as some haven of academic research/study/growth when in reality they're just another HFT pouring accelerant on the economy.

quibono 7 minutes ago

I was doing something similar before but - as a +1 - I find the clear and condensed description valuable, even if only to use as a foundation. It expresses cleanly what I've ended up rewriting in a prompt many times.

zkmon 7 hours ago

This is really cool. I think we will see more of these, inching towards 'refined' thought and expression percolating from AI models into human minds.

I noticed that the quality of email text has gone up around my work these days. everyone is sounding like a master of drafting emails. Ofcourse Ai has a role in it. But real cool thing about this is, this is pulling up people's thinking to have a structure, minimalism and sequencing of expression. Same as how textile industry has enabled people to have an style and expression in clothing.

prolly97 7 hours ago

I love this positive take :)

iugtmkbdfil834 6 hours ago

And, hilariously, as an AI booster, I personally dislike it, because now everyone sounds like they have a PR company ( and the same PR company come to think of it ). It was bad before in corporate land, but the level of standardization for optimized response is aggravating to me, when I am simply trying to have a short conversation.

Case in point, I reached about syllabus to my kid's teacher. Long story short, email ended with their boss, who sent me long winding non-answer that I had to resist to not knock down.

I dunno, generic executive summaries can be canned, but I would like to hope I pay them enough for an person to person exchange.

Arkhaine_kupo 5 hours ago

I have resorted to asking some people "just send me the prompt", like as messy as it might be, I will get the gist much faster than reading through Claudish or Gptinglsh which my eyes gloss over half the text

close04 5 hours ago

> everyone is sounding like a master of drafting emails

> this is pulling up people's thinking to have a structure, minimalism and sequencing of expression

I see the same for the former but the opposite on the latter. Everyone sounds like expert drafters because the AI is doing all of that.

True to the "use it or lose it" principle, the people's thinking skills about the emails have weakened. They rely on giving the AI some bits of info, a document they never read, some vague instructions, and the AI molds the message with structure and sequence. Then I get to experience live the joy of discussing with these people face to face and they still just spit out disparate bits of information with no fluid structure and sequence for the ideas because the only time they thought about the topic was when the AI wrote an email about it.

gregorygoc an hour ago

> because the only time they thought about the topic was when the AI wrote an email about it.

Calling this “thought” is an injustice.

elcritch 4 hours ago

> Same as how textile industry has enabled people to have an style and expression in clothing.

I'd tend to agree with what you said, except this part. Mass produced clothing is fantastic for many reasons, but not for style and expression. Look at the traditional clothing from almost any part of the world and it was gorgeous and colorful. Now it's all just a t-shirts.

williamdclt 4 hours ago

It was gorgeous and colourful, but highly impersonal. Traditional clothing, almost by definition, isn't really an enabler of personal style and expression. And you can't really compare traditional clothing to day-to-day clothing, these are entirely different purposes and codes.

Day-to-day clothing is definitely not "all just tshirts" nowadays, even in the mainstream. Women fashion is very varied, and while men's fashion is a lot poorer there's still a huge spectrum of expressive clothing that's available. I don't think personal style and expression has ever been as available and encouraged as it is today.

troupo 3 hours ago

> It was gorgeous and colourful, but highly impersonal.

So is mass-produced clothing. Go into any store, and you will see the same shit with very few variations. For actual personality you have to search for niche/hand-made/small-productions etc.

watwut an hour ago

Huh? At least where I from, you got cloth decorated by a person who did not produced copies the way machines do. That person was frequently you yourself.

> And you can't really compare traditional clothing to day-to-day clothing, these are entirely different purposes and codes.

I would interpret traditional clothing as "clothing they traditionally wore". It was not a uniform.

ben_w 3 hours ago

> Look at the traditional clothing from almost any part of the world and it was gorgeous and colorful.

Was it?

There's a lot of survivorship bias in old things; we don't see the things which weren't recorded. Old things had their style, that much is unavoidable, but before mass production normal people inherited their clothes because it was just that hard to create.

Also, a lot of the pigments we take for granted today didn't exist at any price in the pre-industrial era, and some which did involved trekking to a mine in Afghanistan: https://en.wikipedia.org/wiki/Ultramarine

This also means (in general, not specifically clothing) a lot of "traditional" things are post-industrial recreations or re-imaginings of the past.

ineedaj0b 3 hours ago

spoke to a clothing historian in Europe years ago. Pigments, shoes, and washing have improved… But all clothes being tailored and more craftsman made the experience better.

She said she’d only choose the modern era because of the washing machine.

pbh101 an hour ago

Clothing was extremely expensive historically relative to wages due to high labor requirements, so until the Industrial Revolution most clothing we now look back to and admire was the domain of the relatively wealthy. It might have had better craftsmanship, but many were literally in rags.

remixff2400 4 minutes ago

There are some inaccuracies/missed parts here:

One of the main differences was that clothing was expected to last longer. Heck, I haven't heard of hand-me-downs in a while now, but you used to have multiple generations of hand-me-downs in the past.

Two, it was more of the social expectation (in the west at least) that you'd dress up to go out (and moreso to be photographed). These outfits are overrepresented in surviving photos.

Third, yes, clothing was expensive, but that was a function of the expected life of the clothing (plus labor). Plus, clothing held value, so you could reasonably resell it at a relatively high value still. And, it's reasonably a higher priority purchase (as opposed to walking outside with your arse out). But no, it was not _extremely_ (read: prohibitively) expensive to obtain a couple basic outfits (leisure, work) leading up to the industrial revolution, due to the aforementioned reasons. The biggest disparity would usually be in winter, where people would lack the heavier layers that wealthier people would have, and in the condition of clothing (many generations of use vs. selling old clothing and obtaining new clothing).

For what it's worth, I'm not a clothing historian, but I did want to push back on the notion that people were walking around in rags and tatters and that a full outfit was only for the (relatively (relative to what?)) wealthy. I did also make a broad assumption here that we're referencing the rise of the suit leading up to, and during the advent of, the industrial revolution, and not the preceding periods with tailcoats and frocks and such (late 1800s vs late 1700s/early 1800s).

zkmon 2 hours ago

Before the stitching machine arrived in my village, all you can see was large rectangular pieces of cloths, delivered directly from the weaver craftsmen of the village, for men and women alike (only color difference - white for men and mostly dark green for women). The washermen still delivered clothes correctly across dozen's of families everyday, after washing. The famous Indian Saree is an example of unstitched clothing. Dhoti, lungi, turban and towel are other forms rectangular clothing.

People had cultivated an art in how the rectangular cloths are wrapped around them and they clothes are secured at their waist, or around their head. Talk to a Sikh or Afghan about turban.

tarkin2 4 hours ago

If they are copying and pasting the answers and replies how will it "pull up people's thinking"?

Lithuanian 4 hours ago

It's not really good that email (and written content overall) quality has gone up. The problem is that there's tons of high quality content but no way to know which content is useful or good. Before we could know someone's personality from their style, now everyone's just wearing Balenciaga.

ineedaj0b 3 hours ago

I have Ai read those emails and condense into one sentence summaries.

I’m not reading all that. I want one word answers.

Also noticed huge uptick in “call me” emails.

maurits an hour ago

> But real cool thing about this is, this is pulling up people's thinking to have a structure, minimalism and sequencing of expression.

I don't agree. Writing introduces friction that forces you to think and reflect. If you let AI do that part, you are not only skipping the writing, but more importantly, the thinking.

Christopgr an hour ago

Also, having a system hone people into the same writing style kills creativity and diversity.

Very interestingly I was looking for the word "diversity" and so I was asking my mother for the word as she happens to be visiting and also told her what I was writing. She said that it's like taking a rainbow and making all its color gray, she is the artistic type. :)

bsenftner 43 minutes ago

I've written an entire Socratic AI based office suite with project management and media creation, current count over 1500 agents:

https://midombot.com/b1/home

It's comprehensive, it was written at an immigration law firm, so it has all kinds of language and other perspective understanding tools, as well as a complete media suite to aid in the creation of visual explainers, courseware, and marketing media.

https://midombot.com/b1/blog/MI/method-intelligence-presents...

Mikhail_Edoshin 7 hours ago

Socratic questions are not asked blindly. They are meant to help the student to realize the incompleteness or falsety of what they think they know. That us, to realise they do not actually know something and thus put them into a position of not knowing, which is the only way to gain understanding.

One man said, that enlightement (as in Zen) cannot be faked. Whoever had it, will be able to tell if someone else had it or not. There is no way to merely sound like it. Socratic method is like enlightement.

zacian 5 hours ago

Socratic questioning needs a lot of thinking to arrive at the realization (and be able to accept it) that one has incomplete knowledge.

I think about this, whenever I have questions which are related to morality and how people act.

andai 4 hours ago

The problem is that everyone else can't tell, and that there's a lot of incentive to fake it. See Saints and Psychopaths by Hamilton.

I think this same phenomenon — faking competence, or good intentions — extends beyond spirituality as well.

awongh 2 hours ago

I think there could be some ways to fake this, using RAG or some other kind of agentic process, but in terms of the true meaning of the phrase, this is one of the things that is very far from being replaced by AI- an interaction between two people where a question that seems simple on the face of it is actually exactly what the learner needed to hear at the exact right time.

Very few teachers get to do this or are good enough, but it's just as much a question of relationships than anything else- how do you pose the exact right problem so that the learner can discover something for themselves.

genxy 6 hours ago

Socratic tutoring is the first thing I did when ChatGPT first shipped. Your liberal arts education is a "huge unlock" if you just apply it.

See also https://en.wikipedia.org/wiki/Mastery_learning

https://en.wikipedia.org/wiki/Programmed_learning

Make your own custom skinner boxes. https://en.wikipedia.org/wiki/Operant_conditioning_chamber

RugnirViking 4 hours ago

its offtopic but that Programmed learning wikipedia article is really bad. It's clearly written by someone with personal stake in it and isn't remotely encyclopedic in tone.

Not sure what to do about it because I don't care enough to learn everything about this topic in order to fix it.

Phemist an hour ago

Comparing the opening sentences:

> Programmed learning (or programmed instruction) is a research-based system which helps learners work successfully. The method is guided by research done by a variety of applied psychologists and educators.

I have no idea what programmed learning is, only that it has an alternative name, that it has scientific street cred and that it is effective according to some nebulous definition of successful.

as opposed to:

> Mastery learning is an instructional strategy and educational philosophy that emphasizes the importance of students achieving a high level of competence (e.g., 90% accuracy) in prerequisite knowledge before moving on to new material.

which is gives me a more clear idea of what the core philosophy is behind it.

Can the programmed learning page be marked as low-quality, non-encyclopedic? That is already a good signal to have.

lemax 4 hours ago

I’ve struggled to get AI to teach me things well without spilling hints or giving answers away. Assuming this was built on some solid evals this could be awesome.

> When the user is wrong, do not give the answer. Ask a narrower question that exposes the contradiction in their reasoning

vkgcr 4 hours ago

It's a prompt - the LLM will always inevitably veer off path and stop following instructions / start hallucinating.

jan_m_savage 5 hours ago

Oh this is so good. I asked it to teach me programming in Logo; though the model is not the best (GLM 4.7, on pi), the tutor is asking very good questions and is not too pedantic about exact answers.

For those new (beginners), install pi from pi.dev and then your favorite AI model (ask your model how to set up pi and it). Then ask your pi-based AI how to set up the tutor (just give it this link, it will set it up for you).

Then start pi by typing pi in the terminal; and to launch the tutor preface your question with: /skill:socratic-tutor. Like so:

/skill:socratic-tutor teach me Python.

koolba 3 hours ago

This is a cool idea. Though trying this out using the claude CLI is kind of funny because by default it's suggestion for the next prompt is the answer to the question.

nathell 3 hours ago

Indeed! To turn that off, open /config and turn off Prompt suggestions.

Imanari 5 hours ago

Socratic tutoring with LLMs works great for me and I have used this skill and it also works great (can you do the same with a simple prompt? probably)

Using LLMs this way was really refreshing, I could really feel how I started to think on my own again.

pimeys 5 hours ago

I also like this skill. It's one of the few I've used the longest: more than a year now which is a century in LLM years. Although you have to remember to ask the LLM to fill its context before starting the questions.

Especially nice when you are not so sure how something should look like, so it makes you stop and spend a few hours answering questions and thinking about what you want to do. Combine that with the RFC skill and you get a good basis for your feature and then a skill to split the RFC to issues on Linear.

orbis-tertius 2 days ago

I love the idea but Jane Street might have copied (or independently rediscovered) https://github.com/bevibing/socrates-skill

beans6543 7 hours ago

If you read them, the skills are completely different (with the Jane Street skill being more complex). Socratic questioning [0] is a well-known technique. Indeed, a Github search returns 92 repositories for "socratic tutor skill" [1], suggesting that the well-known technique has been added to LLM tools independently by many people.

[0]: https://en.wikipedia.org/wiki/Socratic_questioning [1]: https://github.com/search?q=socratic+tutor+skill&type=reposi...

vintermann 7 hours ago

Same with dialog-tuning in the first place. That you could manipulate the context of unicorn-text generators like GPT-2 in order to actually have something like a conversation with them, or that a model could generate "private thoughts" before answering, were ideas which weren't as self-evident as they seem now. But still ideas a lot of people had at the same time.

prolly97 7 hours ago

Yep - just looked at my projects in Claude, and my first "socratic tutor" was registered on 2nd of December 2024. I forget where I got the inspiration though (probably X)

orbis-tertius 5 hours ago

Yes, the comment was intended as a playfully hyperbolic nod to the lack of novelty and originality in the idea, but I guess the intended tone didn't carry through :)

arshxyz 6 hours ago

Been using this with Opus 5.5 on Medium for a week or so. I've used the same model for learning before and I definitely prefer using this skill because it stops the model from the usual handholding. Although I'd say most of the effect can be replicated with this line: "When the user is wrong, do not give the answer. Ask a narrower question that exposes the contradiction in their reasoning."

poilcn 7 hours ago

what am i missing, why does it have 434 forks and 6.3k starts? is it botted? what's the point

ngruhn 6 hours ago

Pure markdown skills are trivially recreatable. I think the value of them is more the idea than the artifact.

michelb 5 hours ago

I think a lot of people use 'fork' to bookmark a repo

amelius 3 hours ago

I want an AI that I can give a paper, and then sits with me to understand it, maybe even implement it (though this is the easy part for the LLM).

lucrbvi 3 hours ago

You should probably try [AlphaXiv](https://www.alphaxiv.org/), they are building what you want.

I use it as a better PDF reader for arxiv paper but they are going all in on AI teaching/explaining things.

olmo23 12 minutes ago

Existing AI can already do this.

bilalabdullahi 4 hours ago

I'd be curious to see whether learners can still solve similar problems on their own a week later

nilsherzig 5 hours ago

someone at jane street likes https://www.mathacademy.com/how-it-works

quibono 5 hours ago

Love this. There is so much value when using LLMs this way, it needs effort and consistency from the user but the results are really good.

A vanilla version of this is, at the end of a session, ask the model to generate a couple of "ultimate" questions to really flex your understanding. Obviously, the quality depends on the model but I've found it useful.

bhouston 5 hours ago

Reminds me of Keating AI. Highly recommend: https://keating.help/

tigitouchdown 5 hours ago

I’ve been building a harness for just this all year, they’re great. One of the main use cases for LLMs I think is largely overlooked.

flipbrad 5 hours ago

Do you have something you could share, and can it be repurposed for use with kids?

slaser79 an hour ago

Good thing with skills being just markdown text you can use them in any harness. Just imported into my phone harness (I already had a Socratic tutor like skill), but this crispier.

andai 4 hours ago

Has anyone tried ChatGPT's study mode? I thought it was supposed to be just like this, but I was underwhelmed when I tried it, and they ended up removing it. (It might be back now?)

I have the leaked prompt around here somewhere...

anonymousDan 2 hours ago

Copilot has a study and learn mode. It's ok.

burrish 4 hours ago

Socratic tutor skill isn't new at all, why did this reach the front page

igleria 5 hours ago

Trying it right now. it's pretty decent and it does sort of make my brain work. Pondering right now when the "bitter lesson" will make these skills useless.

Paracompact 5 hours ago

"These skills" being making your brain work, or this Socratic tutor SKILL.md?

spwa4 4 hours ago

First one, then the other.

igleria 4 hours ago

hahaha, I meant the skill.md from Jane Street. But I'm currently fighting for my brain every day now too!

SkyBelow 3 hours ago

I've tried something like this and ran into a problem which I'm not seeing a fix for. Generally, to really apply this method, you need to understand the person asking the question.

>Tell the user that you will start by asking a few questions, one at a time, to see what they already know.

This puts it into the session each time. Do this once and it seems to work. By the 5th time, you are getting tired of reconfirming your mastery. This ends up depending upon the user to have a pretty solid foundation of what they know and then use this skill to expand the boundary a bit, and even then the AI can end up making so many assumptions that a person has trouble following it along.

I was thinking some sort of concept mapping kept long term might help, but such a mapping can become really difficult to build. Which should be expected when we consider that teaching isn't just its own discipline, but has many sub-disciplines targeted to specific areas (even if some of it, like classroom management, doesn't apply to this scenario).

I made a few attempts at a knowledge tracking tool for personal use to track how much I knew Japanese beyond a simple JLPT level or similar with the plan to tie it into something like this, but it wasn't great. I could 'cheat' and just tap into my Wanikani API to see my progress, but that still skips so many other areas of Japanese, and even when WaniKani says something is burned, I really want AI to still use it until I truly internalize it like I do words in English. I suspect other areas of knowledge are similar.

pbronez an hour ago

Yes, lots of LLM Skills quickly turn into applications. The skill is the program and the LLM is the compute, and that leaves the harness to manage the state. I’ve found that these skills need explicit instructions on how to manage state across sessions.

In my work, this typically means an opinionated local folder structure and a dedicated /init skill that implements it. Immediately this means we have more than one skill so we’re talking about a plugin now.

Then I think about meta-state. Not just “what is on disk” but “where are we in this application’s multi-session lifecycle?” For Socratic tutoring, that means checking to see if we’ve completed an initial intake, then what skill levels the user has demonstrated, and finally what’s next on the menu.

For this specific skill, I’m surprised it doesn’t say anything about curriculum. It just assumes the model knows the topic and can improvise both a curriculum and credible assessments of progress in that curriculum. That can’t be true for all things people want to learn, so we need a /build-curriculum skill. And probably a /validate-curriculum skill for actual experts to use before launching a given curriculum.

Anyway, I’ve been thinking a lot about this kind of thing as I roll out skills to my firm.

orbis-tertius 3 hours ago

This works really well.

A positive take: this works much better than the many projects on gh that did this before Jane Street. This would be evidence that creativity and execution still matter, even when the product is just an md file

A cynical take: network effects are everything. tens of projects doing the same thing and this only gains traction when an already popular org does it