How I use LLMs to learn complex topics

Personally, I find that its generated prose tends to have an undue weight to it, almost as if every topic I ask about somehow bears a heavy burden, or is otherwise load-bearing, to use its parlance.

Quite puzzling, really.

So I have the LLM offer a very short explanation of something, and from there's it's just me asking questions. Anything that feels fuzzy or not fully internalized is something I poke at until I'm satisfied.

It really has helped me develop a sensitivity to what I understand vs what I don't, and the ability to drill into any part of it is amazing.

I'm not sure I'm better off with humans though -- I'm not qualified to judge whether a source is a proper authority, not an I qualified to judge whether someone knows enough to point me to a reliable source.

It seems this is a fundamental epistemological problem to which there may never be an answer.

I’ve found the tone of Kimi K3 to be less obnoxious. Unfortunately it doesn’t wholly solve the issue, I don’t think any LLMs out there have a truly pleasant writing style, but at least not every assumption is “load bearing”.

Like I said, I'm essentially continually prompting to refine the material. LLMs certainly continue to append, and never cut back. It just keeps spitting out additional content at me. So that's a bit annoying too. But I can basically get figure out what's going on with a few extra promps.

If youre curious what i've got so far... just be warned it is quite literally AI slop plus me continually prompting for clarification/cleanup etc. : https://github.com/cmoscardi/ai-for-ai

I've stopped using CC because of it. I find it insufferable.

It generates tutorials for you, and serves a webpage that lets you complete them. It does a remarkable job.

It still has a bit of the LLM prose problem, but it does help you fine tune the ‘voice’ it uses.

Often "be concise, to the point." is enough, but you can also paste it some stuff you like as an example text and ask to do style transfer.

I'm actually going to make a prediction here as well. I think you will soon realize that using LLMs to clarify certain questions or ideas you have will turn out to have frustrations as well. And that you will soon direct those questions to either peers you know in real life or internet forums which are very likely to have a non-AI policy.

It's just long. It just doesn't shut up. It's overly verbose. And you can't tell it to be concise or you degrade its quality.

If I ask what an integral is, the correct answer is that it is the continuos analog of a sum, generally used to calculate areas and volumes.

It should really be a single sentence, and then let me ask more about the terms I don't understand, and here's the beauty, in the previous one there can be only 5 terms I cannot know.

An LLM will vomit an entire page or more of explanation which isn't bad per se, but is an answer to something different: "give me a short introductory explanation to integrals". And that's not what I asked.

Opus 5 first built me a detailed plan, but a couple important details were either obviously wrong or felt unnecessary. I went back and forth asking for sources and more information probably like 4 times and every time it did the "in looking at things in more detail it appears my previous advice was incorrect" spiel. It just became exhausting at some point because it feels like it really lays bare how LLMs are just minimizing that loss function but don't actually "understand" anything. It was really useful as a search engine (it correlated some highly relevant source docs), but I just couldn't trust it to believe it was actually done at any step.

i can't even get agents to remember core instructions like "use jq instead of writing a python script to parse some json"..

Trust me when I say that in the hands of someone who doesn't have your experience, the LLMs would not be getting the results you get.

You might think what you're doing is trivial, it may be sessions that flow roughly, "Instrument this, okay this part is slow, profile this part, OK read the profile output and suggest a better approach".

But your experience will be steering it in the right direction, and you're probably unaware of just how much your experience is doing that guiding, as the LLM shoots off at 100mph, you feel like it's taking you with it, but you will be guiding it a lot more than you realise, and that's where learning and experience comes in, even if you're no longer operating at the lowest depth, your knowledge of that layer will be helping.

If nothing else, the experience to know when something is actually slow is a skill in itself. If a function takes 200ms, sometimes that's as quick as it can realistically go, and sometimes that's literally a million times slower than it could be, and there's actual skill and experience wrapped up in knowing what "slow" looks like.

What is fascinating is how you can witness it at so many levels of organization. One example: Employer executive get enamored with moving from labor to capital. They believe that by using LLMs, they can replace a lot of workers. At my place of employment, we have people that are surprised they can't file a Jira ticket describing a product ask, and have it kick off an implementation. You can build the skill to attempt that, but invariably you'll get back questions like "what do you mean by <x>" and "what do you want to do in this case, a, b, or c?"; questions that a product person or an exec are not well suited to answer.

In the past, programmers did that kind of interpretation and judgment call. So then you're in a quandary; who should do that work? Work that previously, you never imagined was an inherent part of what the replaceable code monkeys do at your beck and call?

And then, how do you hire for that? How do you find the training for the people that are experienced enough with... something... to know what a cohesive error response is, or what kind of telemetry strategy is best for that particular product and organization, what collection of product asks are incredibly complicated for what they're asking and can deliver 95% of the benefits at 5% of the work if we just do this instead, and whether you want to aim more towards thick or thin clients?

Who are those people? Wait, those are programmers? Wait, there's this whole collection of inherently human skills that we devalued, by not appreciating they were always quietly doing that for us in the past?

That's just one example. There's a repeating pattern of discovering where the work truly is, work that was embedded in manual patterns we might not have to involve ourselves with anymore, but is yet still essential. So the nature of our jobs changes massively, but the overall level of employment does not.

At least, not in the medium to long term. There is a lot of painful churn we have to suffer through first.

I have stuff to do now, the value of the knowledge in a year or two isn't important if it solves the issues I have today.

I wasn’t even really concerned with optimizing low level code before LLMs and that wasn’t why I was hired either.

However following that low level thread: We can look at the reasonable options and immediately know if they’re reasonable or nonsense. Why? We know the code. Now zoom a level out, where I think our expertise really lies.

Building a complex system isn’t easy. There are customers with requirements, there are budgets, SLAs etc. Sometimes one customer needs X and one needs Y. Our expertise is taking all of this in, and producing something that balances all the different variables. It’s knowing that we’ll expect X events a second so we’ll need Y to ensure we can tolerate failure.

Is it possible LLMs will be able to do all of that too? Maybe. But then why would our customers need the enterprises they pay for?

I'm not 'wasting time' but I'm also not really learning.

The more things you understand, the higher the chance you'll spot a situation to use them in the future.

I think the best innovations come from times when someone is uniquely able to combine two of their previous experiences together. The more experiences you have in your back pocket the more combinations you have access to and the more likely you'll have a unique combination when the right problem comes along.

In particular it might be valuable to be in the habit of learning things that one is bad at doing.

Or not.

This is silly. This would be like arguing that encyclopedias made knowing things pointless. I learn new stuff for me.

Professionally, it's important to know enough to know if you're going in the correct direction. Practically, tokens are going to continue to cost money and knowledge can save you tokens.

1. It satisfies you curiosity (and curiosity is always valuable)

2. You can better utilize the LLM to expedite something you now have knowledge about

3. You still improve as an engineer/programmer/prompter/whatever

I still think it's very important not to outsource everything to AI because there is a lot of value in learning and doing things yourself which is an important part of life.

I can tell you that there is an enormous gap in ability between them despite them both using LLMs for daily IR work.

The reasons aren’t complicated. The senior responders have tacit knowledge of how breaches evolve and what to look for which gives them a much better framework for where to employ the LLM.

The juniors will normally start from “here are some logs, look for weird” which is fine but leads to tunnel vision and a lack of confidence in their reporting.

I don’t mandate that anyone do work with or without an LLM. I hire seniors based on experience and juniors based on interest. But my experience has so far been that our best up and comers focusing more on learning the technologies instead of leaving those details to the LLM are developing their intuition and understanding faster and in a more robust manner.

In my experience it's infinitely easier and faster to learn deep, "boring" things when you understand how they relate to your shallow and wide understanding of all of the related components.

The LLM is merely a tool. And you can use it for domain discovery that enables efficient deep learning at an unprecedented rate or you can develop a cursory understanding of a topic and think yourself an expert.

It’s true of most things. Running, dieting, weightlifting being uncomfortable is a sign of progress.

You can ask the LLM how to do this. Start with a topic you know well to get the mechanism working and trust it well.

Indeed it's often a waste of time to just focus on talking people fully if you want to learn fast, reading and especially deliberate practice are better for that. But if you don't have the time, energy or focus, then listening to interviews in the background can be useful supplementally

I have done this for all my work this week and it works quite well.

For one it lets you actually query the LLM as to why, their plans give a high level not every single change and it allows you to correct it as you go and the plan will change.

What is your process for creating these resources?

(Yes I confirmed it was ok to use AI with the material)