Small AI Models Gain Traction In places with unreliable networks

ssscaryterry about 6 hours ago 19 commentsRead Article on spectrum.ieee.org

RU version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

100% Positive

Analyzed from 726 words in the discussion.

Discussion (19 Comments)Read Original on HackerNews

N_Lens•37 minutes ago

I strongly believe this premise in the article is correct - we will see a lot of tiny, hyper specialized models for individual tasks, and perhaps that will converge with an orchestration layer for a generalized intelligence that controls these specialized tiny models, that will be quite capable.

I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet).

It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. The overall coordination between these sub-brains creates emergent skills/abilities.

tim-fan•about 4 hours ago

Is anyone making LLM-in-a-box for emergency supply kits yet?

I feel that would be handy in all sorts of situations when networks are down.

Terr_•about 3 hours ago

> LLM-in-a-box for emergency

For most actual emergency scenarios, a device that focuses on storage of large amounts of prepared normal reference material [0] will be wayyyyy cheaper, more durable, portable, and able to run on batteries or being constantly plugged into a somehow-still-normal electrical grid. (Think an e-ink tablet that can run off a 5V battery pack buffering a literal handcrank.)

In contrast, imagine spending the money to build a beefy LLM-running computer with good GPU/RAM, and somehow mothballing it (to depreciate, unused) in a "safe" location for the big earthquake/flood/etc... Then when the disaster strikes and you dig it out, how will you power it when you need it, and for long enough to do anything useful?

Even if wall-current civilization is 20 miles away on the other side of the mountain, are you going to carry it on your back, or are you going to carry food and water to live? If you do drag it there, are they going to let you run it when it cuts into light for surgery or heat to sterilize drinking water?

skybrian•about 3 hours ago

You will probably want a search engine though. Perhaps a small LLM would work well as a component for that?

rtpg•33 minutes ago

grep works well!

weikju•about 2 hours ago

> If you do carry it to an enclave of civilization that has the right power, are they going to let you run it when it cuts into light for surgeons or heat to sterilize water?

Knowing humans? They'd probably take it by force and run it for themselves instead of providing light and heat to surgeons and water sterilizers...

/daily dose of cynism

zmgsabst•about 1 hour ago

You can run a small model off a home generator — so in an emergency, you’d turn on both the generator and information service, eg, a mesh for “quick” responses querying that huge collection of information.

That way your machine that, eg, normally plays video games or does AI work can support relief efforts by supporting emergency response IT. You don’t need to mothball the machine, just have an “emergency” boot USB than can run the services from your home generator.

You don’t even need to bring it with you: turn it on and leave it “best effort” at home, while you continue to use it via WAN.

SwellJoe•about 3 hours ago

This is couched in prepper nonsense, but it's got LLM, WikiPedia, maps, etc. A bunch of genuinely useful stuff to keep on a USB stick or whatever: https://www.projectnomad.us/

But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart for its size, excellent vision capabilities, good tool user, surprisingly good reasoning.

vessenes•about 3 hours ago

I've been mulling over a good use of a large philanthropy spend in the next decade, and I would love to build a bunch of hardware "oracles" that include an LLM. Ideally solid state, visual/audio, solar + usb-c, so, good in a lot of doomsday scenarios as well as just out hiking. It's a fun thought experiment. I imagine making like 1 million of them, they could be sold and genuinely useful, but also given away; once owned, you could use them, or store and put in an emergency box, bury next to the 10k year clock.. a lot of possibilities.

adrianN•about 1 hour ago

I feel like you could get a lot more quality of life improvement for more people with the money if you spent it on low tech solutions, eg more efficient cooking stoves for people still cooking with biomass, or solar microgrids for areas without electricity.

burgerone•33 minutes ago

This is HN, not Reddit.

rasz•40 minutes ago

Smallest local model able to work with offline wikipedia dump would be one step above just having an offline wikipedia dump.

cdnsteve•about 4 hours ago

Can you expand what you mean?

wahnfrieden•about 4 hours ago

They want to ask the iOS Foundation model (frontier on device intelligence for something small) for instance about emergency procedures and life-saving info. I wouldn’t trust that model with much at all though. More likely to find what you need from miniature survival guides.

bix6•about 2 hours ago

Has anyone used the Rx Scanner mentioned in the opening?

https://rxall.net/rxscanner/

jdonaldson•about 3 hours ago

I think neuro-symbolic AI has a lot of potential here, since small models can handle a lot of conversational inputs, while relying on wired-in solvers for more complex symbolic math/computation needs. https://jjd.io/posts/swollm-bbh-leaderboard.html

mountainriver•about 1 hour ago

I looked into this a bit but unfortunately because of starlink most of this won’t be needed

bombcar•about 5 hours ago

99% of the model "work" (meaning the connection to your computer) is just spinning a spinner - something that makes me want to wrap it with a mosh shell so I can just keep moving from network to network.

enoint•about 4 hours ago

Fascinating to wonder whether the bigger model finds fewer or more counterfeits than the on-device one.