ES version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
56% Positive
Analyzed from 3697 words in the discussion.
Trending Topics
#more#temperature#resume#model#don#deterministic#llms#hiring#llm#different
Discussion (99 Comments)Read Original on HackerNews
> temperature 0.1 — low, supposedly nudging the model toward deterministic outputs
This is not correct (and is briefly touched on later in the piece when he sets temperature to 0), temperature is not some kind of "deterministic" switch, but rather it affects the sampling distribution (which becomes more "spiky"—but is still very much a distribution).
Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample (using the actual formula that works for non-zero values would cause a zero division).
However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs.
It does exist very much, even if you go to pure math. Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. I.e. in a discrete setting (like for LLMs with a finite set of output tokens), probability P becomes one for argmax and 0 for everything else. Only in coding practice it is easer to implement T=0 as a simple if check that directly chooses argmax instead of calculating the limit of some function that includes 1/T quotients. But setting T to zero is in both, theory and practice, turning the usual probability function into greedy sampling.
But theoretically, the output of every LLM is seed-driven (or could be if you wrote the software to isolate it) just like any computer software. It's just none of the software written (even llama.cpp AFAIK) chooses to support stable-seeding due to the changes in stuff like CPU/Vulkan/CUDA/Metal differences making it difficult to make consistent.
They could though! Hopefully one day someone implements it into the mainstream LLM-engine software and it gets exposed in the APIs serving the models. It'd do a lot to show folks the "internals" of these models.
It is a common misconception, but it is not true even in principle. If I have 2 or more logits which are equal to the maximum of my logits, I will sample uniformly random from them with any temperature, even zero. Sampling from softmax([1, 0, 1]) is still stochastic at temperature 0, because the limit is to sample uniformly from the first or the last element.
Anyway: "GPUs don't do deterministic matrix multiplications" is the biggest source of randomness in LLMs. GPUs put the associativity of the sums in matrix multiplications in arbitrary order, and this has a huge impact on the logits coming out of the neural network.
I don't know for sure, but I would be surprised if it was illegal in my particular US state. You might be able to argue the AI has inherent biases that introduce illegal discrimination in the hiring process, but my understanding is winning I case like that would be very difficult, especially since most employers are very cagey about their hiring process and why they mades a decision.
Show someone a list of resumes with an "applicant score*" and they'll naturally ignore the ones with a low ranking
*scores are generated with AI, mistakes may be made, use only as a guide and verify results
I've been studying AI for 20 years. What really needs to be added to this statement is:
"An alarming number of people don't understand that LLMs work via purely stochastic processes - and so does human thinking. People do NOT arrive at the same conclusion if merely the weather's different. Worse: with human thinking not only do most people not think this is real, a subset of people will actively fight the idea. Of course, depending on the weather"
If you train two different LLMs and replace what data they "see" in batch n, that doesn't affect the data they see in batch n+1, or any further batches. In LLMs, you can introduce "noise" into the training process, but that noise doesn't really compound.
Humans learn from experience, not from data, and their experiences at age n shape what experiences they seek (and hence train on) at age n+1. A small amount of "noise" injected into their "training", let's say hearing a group of friends discuss a movie while their identical tween goes to the bathroom, can compound into them watching that movie, which can compound into them forming an identity around that genre, and so on, until they're two completely different people, trained on completely different "data mixtures".
That’s a tiny model. No LLM is going to be a perfect and repeatable judge, but a tiny 4B model is like plugging an RNG into this system.
This whole exercise feels like someone vibe coded an ATS and got it to the point where the tests were passing because they decided they should have an open source ATS project.
> *SCORES MUST NEVER DEPEND ON THE FOLLOWING FACTORS:*
> - College, university, or educational institution name
> - CGPA, GPA, or academic grades
I don't understand why they would omit these factors from the evaluation.
Just kidding, my resumes are sent to /dev/null like everybody else’s.
As someone who’s run hiring pipelines for technical roles in the past few years, that’s actually a fantastic number. I objectively hate saying that, but it’s true.
35% chance of elevating a technical individual to the next stage with no effort? I’ve seen as many as 100+ applicants an hour even when including a domain specific screener question. That’s 35 “screened” applicants in an hour. Were valid candidates screened out? Yes. Does you still have a candidate pool 35x larger than you need? Unfortunately, also yes.
The volume of applicants is SO HIGH such that your chances of getting moved to the next stage are actually markedly worse if AI isn’t involved. If you didn’t apply immediately (using an AI bot) there’s 50+ people ahead of you, and an exhausted technical leader if they ever make it to your resume.
Referral bonuses exist for a reason.
*According to our proprietary, undisclosed, non-deterministic metric, which may or may not be Math.random
https://stackoverflow.com/questions/16833100/why-does-the-mo...
Gates that reduce resume flow-through are only useful if their reduction is correlated with quality. Otherwise they're just dragging out your hiring process or unnecessarily causing you to ultimately lower your hiring bars.
The volume is infeasible to review everyone for quality, even at an hour scale. The conclusion and solution is inevitable, though I wish it were different. 35% is actually really good if you’re not coming in through a referral.
The current reality is <1% and the person reviewing you is exhausted.
Corpo bullshittery at its finest.
If the first 50 people who apply are all bots, why are you reading resumes in order of submission?
1. Give them some easy leetcode questions. Nothing that a competent programmer would have any problem with.
2. If they pass, ask for a deposit of like $20. Shouldn't be an issue for people who are actually serious.
3. Do more simple leetcode questions but this time on zoom so you can tell if they are using AI. If they pass that they get the deposit back.
(Yeah I know there are real-time interview cheat AI programs but based on what I've seen on demos of them it's super obvious when they're being used.)
Probably not practical but just a thought!
I don’t think the point of a lot of this is to optimize your resume. It’s to show how arbitrary these systems are.
You read my mind. If the answer is “no”, then we can ignore this.
> LLM is called six times to extract structured information
Followed by
> The default model is gemma3:4b, running at temperature 0.1 — low, supposedly nudging the model toward deterministic outputs.
This is exactly why hiring is even more broken: Because the people looking for candidates are also just as unqualified if not, more.
Using an LLMs to replace the person in charge of the final judgement call is the wrong technology. In fact, it is the wrong solution as this is a plain old social problem.
Even if you wanted to use LLMs for this case, the default configuration, model choice is laughably flawed. The LLM doesn’t even know what it is reading.
The correct solution is a far stronger LLM, with an experienced person making the final judgement call in case the LLM hallucinates or misses a critical detail.
Worse model may not "know" enough to distinguish between a 70 and a 100 candidate, so it's expected that it's output has high variance. But a better model might "know" enough, so it can be more confident and thus more consistent.
Well done you! It is difficult to avoid architectural complexity, but imho well worth it.
Typically, retrieval should be tied to evaluation metrics, evidence should be linked to scores, and you also need to account for parsing errors.
But personally, I'm weak to these kinds of ATS systems (ugly appearance, non-native English speaker, didn't go to a good university), so if this kind of filtering existed, I probably would have never had a job in my entire life. Come to think of it, even now I don't have a proper job—I just bid on projects at the lowest price and implement them. So maybe it doesn't really matter whether such a system exists or not
The only drawback I see is that you should compare every pair of CVs for best results, and that grows quadraticly with number of CVs. Of course you can settle for fewer comparisons and not perfect results. But then I'm not sure if you can hit a good ratio of quality and token spend.
However given the time constraints reviewers have, yes, the former (making a resume easy to consume quickly) is a huge help.
https://en.wikipedia.org/wiki/Secretary_problem
> 35 points for open source contributions
> 30 for personal projects
I don't contribute to open source or have personal projects because I don't spend my free time doing what I do 40 hours a week to make a living. My 15 years of work experience is worth a maximum of 25%, so any company using this idiotic system would pass on me immediately. Open source and personal projects are fine, but in no sane world are they worth 65% of a resume's score.
Free software work doesn't imply we work for free. We work on our projects, the stuff that we actually enjoy working on. Nobody is going to work on corporate products without adequate compensation.
I guess there sadly are many nobodies who do this to hope to become somebody.
Now all my "non-work" time is spent on startup work. And none of that is visible via GitHub.
Nice cherrypicking the number. If the cutoff is at 75, you fail almost never.
It's always fun to see people who've never done any hiring to express their uninformed opinions.
Here is my informed opinion after being heavily involved in hiring for the last 7 years: Hiring is not so much about finding good candidates, it's more about filtering out all the bad ones. And believe me, you can't imagine how many bad ones, and how bad, until you are involved in hiring.
While resume's are being filtered left and right, they just make TikTok's on company's dime [1]. What a sad state of affairs.
[1] https://www.youtube.com/shorts/wSug80Vg5JU
Also if HR was really useless (or actively hurting the company) they wouldn't still have a job (or they'll lose it eventually). No one likes burning money for no reason. So obviously they are doing something useful.
Speculative thought only, of course.
https://github.com/interviewstreet/hiring-agent/blob/main/pr...