We're at an inflection point in the last few months that means Apple might finally be able to do something good with Siri.
Fundamentally, Siri's great promise (well in the last few years) has been that it could hook into all of the scripting facility made available by iOS apps. There are tons and tons of AppleScript hooks on many, many apps -- if only Siri could, you know, access them.
This is traditionally not Apple's strong suit - API access oriented thinking - but I will also say that it turns out to have been a pretty hard problem to actually parse out what 'hooks' to access.
We now refer to this as "tool calling", and know that it's a specific thing an LLM (or even MLM/SLM in many cases) can be trained to do a pretty good job at.
If Apple wants to do all this on device, I think we're not far off in terms of hardware and architectural knowhow to get this done now. I would bet that the Siri team spent a number of years on alternates, essentially relearning the bitter lesson.
I'd guess the best way to make this work on device would be to train and finetune a small model on a massive number of apple app store apps so that it gets good at figuring out which apps it will want to query for tooling functions calls - that's too much to inject in context directly especially on device - then it would call out for a list of available calls -> that's just code -> then it could use a local LLM to choose the proper call and call it -> then if needed it could offload for more heavyweight thinking. To start you could offload the tool call decisions as well, or really the whole thing, except for the local function call.
This would be vastly vastly better than what we have now, and I think does not require huge reintegration efforts from app developers.
"Siri what month is it?"
[... calendar_getcurrentdate requested ]
-> [ Mar 21 2025 ]
Fundamentally, Siri's great promise (well in the last few years) has been that it could hook into all of the scripting facility made available by iOS apps. There are tons and tons of AppleScript hooks on many, many apps -- if only Siri could, you know, access them.
This is traditionally not Apple's strong suit - API access oriented thinking - but I will also say that it turns out to have been a pretty hard problem to actually parse out what 'hooks' to access.
We now refer to this as "tool calling", and know that it's a specific thing an LLM (or even MLM/SLM in many cases) can be trained to do a pretty good job at.
If Apple wants to do all this on device, I think we're not far off in terms of hardware and architectural knowhow to get this done now. I would bet that the Siri team spent a number of years on alternates, essentially relearning the bitter lesson.
I'd guess the best way to make this work on device would be to train and finetune a small model on a massive number of apple app store apps so that it gets good at figuring out which apps it will want to query for tooling functions calls - that's too much to inject in context directly especially on device - then it would call out for a list of available calls -> that's just code -> then it could use a local LLM to choose the proper call and call it -> then if needed it could offload for more heavyweight thinking. To start you could offload the tool call decisions as well, or really the whole thing, except for the local function call.
This would be vastly vastly better than what we have now, and I think does not require huge reintegration efforts from app developers.
"Siri what month is it?"
[... calendar_getcurrentdate requested ] -> [ Mar 21 2025 ]
"It's March".