bullshitai.dev

RESEARCH Hermes · skill and tool routing

I tried wiring Jev into Hermes.

I don't know how to use it man

Parody cut-out portrait of Steve Carell as Michael Scott from The Office, grimacing, in a dark jacket.

The prototypes work and Jev answers in a third of a second. After three rounds of experiments I still can't show a useful, reliable Hermes speedup that would justify switching it on.

Jev is fast. In our one-request selector test it returned a full set of skill judgments with a median HTTP time of 0.31 seconds. That number is real, and it’s why I kept trying. The question was never “can Jev answer quickly”. It was whether a quick Jev answer can take some real work off Hermes, so that the agent reaches the right skills and tools sooner without losing the ones it needs.

So far it hasn’t. This isn’t a story about Jev being useless or about broken code. The plugins run, our selector runs, the replays are reproducible. What’s missing is a demonstrated case where routing through Jev makes Hermes better or faster at the job it already does. Below is who tried what, how each piece works, and where the evidence stops.

Skills, tools, the main model, and Jev

Some vocabulary first, because the results don’t make sense without it.

Hermes is the agent I use day to day. The main model is the large model that reads the conversation and writes the answer. In the replay described later, that was gpt-6.1-sol at high reasoning effort, through my existing subscription.

Tools are actions the main model can take, such as running a shell command, opening a browser or searching. Each one is described to the model by a schema, and every schema sent with a request takes up prompt space whether or not the model uses it. Hermes can also keep some schemas out of a request and let the model discover them later.

Skills are written instructions: how to do a particular job. The main model always sees a catalog of skill names with short descriptions. When it decides one applies, it loads the full text with a tool call (skill_view). So the catalog is always in the prompt, and the full instructions only arrive when asked for.

Jev is TypeSafe’s small decision model. You don’t chat with it. You send it some shared context and a list of typed questions, such as yes/no or pick-one, and it answers each with a probability. TypeSafe calls the yes/no type Noul. Jev is quick and cheap: a documented $0.042 per million input tokens, and output is free.

Luna is another decision model I already had, on an existing subscription. It isn’t free, and it doesn’t answer in the same format as Jev, so swapping one for the other also changes how you ask.

The idea was simple: let a fast model decide which skills or tools matter for this turn, so the main model has less to sort through and gets to useful work sooner. Saving tokens would be a nice side effect. The goal was faster selection that’s still useful.

What the ready-made plugins actually do

Two community plugins already existed: hermes-jev by DoGMaTiiC and Tool Slimmer by aliasocracy. I tried each one as published, without changing its code, at the version linked here. Later we built a third option of our own. They answer different questions, so it helps to see them side by side before the numbers.

Three separate add-ons, three different answers

Skill written instructions for a job, loaded only when needed

Tool an action the model can take, like running a command or searching

  • Community plugin

    hermes-jev

    Chooses from Skills

    Hands back one skill, at most

    Not loaded. The main model decides.

  • Community plugin

    Tool Slimmer

    Chooses from Tools

    Hands back a shorter tool list, plus the tools it always keeps

    The main model works with that list.

  • Our prototype

    Shortlist selector

    Chooses from Skills

    Hands back a skill shortlist, possibly empty

    Only a suggestion, off unless switched on. The main model still chooses.

Figure 1. These are alternatives, tried one at a time, not steps in one pipeline. The shapes show what kind of answer each one gives, not amounts. Only the third is ours: a prototype that is off unless switched on and only suggests.

hermes-jev suggests a skill. It first asks Jev to score every skill it knows about, then asks again to reorder the best few, and passes on at most one suggestion. It doesn’t replace the catalog or load the skill for you; the main model still decides. It finds skills by scanning a skills folder, and in my setup that missed some skills I had enabled from elsewhere. Eleven of the 24 test requests needed more than one skill, which a one-suggestion design can’t cover by construction.

Tool Slimmer trims the list of tools, not skills. It picks candidate tools by keyword matching (BM25, a standard search ranking), asks Jev one batch of yes/no questions about them, and always keeps a fixed set of tools no matter what. It also has a cache and a way to recover a tool it dropped. Its keyword matcher only recognises Latin letters and digits, so a request written only in Ukrainian can produce no candidates at all, and Jev never sees it. That’s a limit of the keyword matcher. It says nothing about whether Jev understands Ukrainian.

I tested both on 20 held-out requests from my real sessions, picked on purpose rather than at random. The results:

Holdout, 20 requestsRequired items foundComplete nonempty required sets
Tool Slimmer, keyword only16/31 tools6/18
Tool Slimmer, with Jev17/31 tools5/18
hermes-jev skill suggestion6/28 skills2/17

Jev found one more required tool than keywords alone and completed one fewer full set. It did cut the selections outside the annotated sets from 147 to 86. hermes-jev’s 6/28 is mostly a design limit: one suggestion at most, from a skill list that was missing some enabled skills. Twelve of its 16 holdout suggestions still landed on skills labeled required or optional, so the suggestions weren’t nonsense. Skills and tools have different catalogs and jobs, so I don’t rank the rows against each other.

These results describe the versions I tried; both plugins may have changed since. Hermes also documents a separate Jev TypeSafe plugin, where you ask Jev to check, route, score or evaluate something by hand. hermes-jev itself also includes a risk judge that can run watch-only, or enforce its decisions. Those are different, narrower decisions. I haven’t benchmarked them, so I have nothing to report on them either way.

Our own selector

Next we built a selector for the problem I actually have. It’s a separate plugin that only makes suggestions, and it stays off unless you switch it on. It doesn’t replace how Hermes normally picks skills, and it doesn’t load any skill text ahead of time.

It sends Jev a limited recent part of the conversation, plus a list of the skills that are already loaded or about to be, with their descriptions. Those skills are left out of the questions, so Jev isn’t asked about them again. “Nothing new is needed” is a valid answer.

One request is enough to pick several skills: it carries the shared context and one yes/no question for each remaining skill. What we didn’t test was asking Jev to send back a single list of skill IDs instead. That’s a different answer format, and it isn’t needed to stay at one request.

The practical limit was size. We kept the shared context plus any single question under a 32k standard limit, and the context plus all the questions under 64k. We measured those sizes on our side, approximately, so they aren’t exact token counts.

Score each option, or ask for the shortlist

Illustrative example, not a measured result

Request: Find sources and write a summary

Skills to choose from: Research Writing 3D printing

Score each option Jev Boolean batch

One request: the task, plus a yes/no question for each skill

Jev answers with a score for each

  • Researchabove the line
  • Writingabove the line
  • 3D printingbelow the line

Our code keeps scores at 0.5 or above

Research Writing 3D printing

Ask for the shortlist Luna selected_ids

One request: the task, plus the list of skills

Luna answers with the chosen skills

Research Writing

Our code uses the list as is

Research Writing 3D printing

Figure 2. Two ways to get several skills from one request, shown on a made-up example. Both sides end with the same pair here only to make the comparison easy to follow; it doesn’t mean the two methods agreed in the test. The figure counts requests we send, not how much work the provider does inside.

On the left, Jev doesn’t pick anything itself. It returns a score for every skill, and our code keeps those at 0.5 or above. On the right, Luna returns the chosen list in one answer, and we use it unchanged.

Same 30 scenarios, three practical arms

Then we compared three ways of making that single selection request, on the same 30 scenarios: 24 requests from my history and 6 synthetic ones. “Jev Boolean batch” is the selector above. The two Luna versions ask Luna instead: “scalar vector” gets a score for every skill, and we keep those at 0.5 or above; and “selected_ids” gets back the list of chosen skills directly. The labels come from a single annotator and are partial. After leaving out what was already loaded, they name 25 required items, and 19 scenarios need at least one.

Required items recovered and complete nonempty sets for three one-request arms

Required items recovered scale 0–25

Jev Boolean batch16/25
Luna scalar vector17/25
Luna selected_ids16/25

Complete nonempty required sets scale 0–19

Jev Boolean batch11/19
Luna scalar vector11/19
Luna selected_ids12/19
Figure 3. Delivered coverage on all 30 scenarios. Each Luna configuration’s single ~115-second timeout stays in these counts. Labels are nonexhaustive, so “outside annotations” doesn’t mean wrong.
Arm, same 30 scenariosRequired foundComplete nonempty setsOutside annotations
Jev Boolean batch16/2511/19174
Luna scalar vector17/2511/1972
Luna selected_ids16/2512/1933

The three arms are within one item of each other on required coverage. They differ much more in how much they select beyond the labels: Jev’s batch picked 174 things outside the known required and optional sets, Luna’s ID array 33. Some of those are probably reasonable picks the labels missed. I can’t call them false positives. But a selector that hands the main model a big pile of maybes isn’t doing much narrowing.

How fast the selection call was

For timing, we used the 26 scenarios where all five configurations succeeded. That drops the two Luna timeouts, and one case where reading the answer was interrupted, from every arm alike.

Median selection HTTP time, seconds, on 26 common successful scenarios

Jev Boolean batch0.313 s · n=26
Luna scalar vector6.139 s · n=26
Luna selected_ids2.383 s · n=26
Figure 4. Median HTTP time for the selection request only, linear scale from zero. This is not Hermes task time: it leaves out the main model, adapter overhead and everything after selection.
Arm, 26 common scenariosMedian selection HTTP timeMean input tokens
Jev Boolean batch0.312507 s10,733.50
Luna scalar vector6.139126 s11,167.96
Luna selected_ids2.383038 s3,975.27

Jev wins this part clearly, and the gap is large. The input token means use different protocols, encodings and accounting, so they don’t show that one format compresses better. Price matters less. Using the documented Jev tariff and the reported input usage, the Boolean batch extrapolates to about $0.45 per 1,000 comparable selections. That’s arithmetic on reported usage; I haven’t seen a bill for it. Luna’s per-request dollar cost on the subscription is unknown. It isn’t zero.

The detour that wasn't my design

Two more arms are in the data: Jev Choice and Luna enum, each asked repeatedly to pick one more skill until it says none. The agent running the experiment introduced them. I had said I didn’t want cycles, and this is a cycle. They’re reported because they were measured: on all 30 scenarios they recovered 12/25 and 11/25 required items, and on the common 26 their summed median times were 0.710420 s and 7.623803 s. They used more requests and found fewer required items. I’m not pursuing them.

Replaying real history

The test that settled it for now: does a Jev shortlist help the main model at a real decision point?

We took 12 moments from 10 of my past sessions where the main model was about to pick a skill, and rebuilt the conversation up to that point from stored history. They’re reconstructions, not the exact requests sent at the time. Then we made 24 fresh attempts with gpt-6.1-sol at high reasoning effort. Arm A got the full current catalog of 151 skills. Arm B got exactly the same, except the skill catalog was replaced by the shortlist Jev produced. Ten episodes got a real Jev shortlist. The other two had long conversations with skills already loaded, were too large to send to Jev without cutting, and fell back to the full catalog.

Historical replay stages: shortlist contents, full-catalog selection and shortlist selection

12 historical episodes scale 0–12

Real Jev shortlist10 episodes
Full-catalog fallback (long loaded state)2 episodes

Annotated required skills in the 10 shortlist episodes scale 0–17

Stage 1 · Jev shortlist contained14/17
Stage 2 · A: main model, full catalog, selected12/17
Stage 2 · B: same main model, shortlist only, selected11/17
Figure 5. Shortlist coverage and two alternative main-model branches, not three competing models or a serial pipeline. A receives the full catalog independently; B receives Jev’s shortlist. Counts are over 17 annotated required skills in the 10 episodes with a real shortlist. The two fallback episodes are outside these coverage bars.
Stage, 10 effective episodesAnnotated required skills covered
Jev shortlist contained14/17
Main model with full catalog selected (A)12/17
Same main model with the shortlist selected (B)11/17

The shortlist kept 14 of 17 required skills. With the full catalog, the main model picked 12 at the first selection boundary; with the shortlist, the same model picked 11. In four episodes B also selected at least one skill that wasn’t on its shortlist, so the shortlist narrowed what B was shown rather than what it could load. A and B chose identical sets in five of the ten episodes.

Jev wasn’t playing on equal terms. The main model in both arms also saw my saved instructions outside the catalog and the list of available tools; Jev’s selector didn’t. So I can’t pin the misses on Jev alone. And because both loaded-state cases fell back, the shortlist’s quality with long, loaded state is untested.

Timing is one observation per heterogeneous episode. Across the ten shortlist episodes, median A selection time was 3.718 s, against 4.061 s for router plus B. That’s the selection phase only, without local packing or skill-read overhead and without repetitions. I can’t call it a speed win, or a stable slowdown either.

Tokens tell a similar story. Main-model input was 341,689 for A and 310,481 for B, but Jev read another 125,023 input tokens to build the shortlists. A also got 67,584 cached input tokens and B got none. A smaller main prompt isn’t a system-wide saving. Final-answer quality and combined skill-plus-tool routing were never measured.

Why it feels underwhelming

Every piece does what it says. hermes-jev suggests a skill, Tool Slimmer trims the tools, our selector returns a set in one request, and Jev answers much faster than the Luna configurations. But the only part where Jev clearly wins is the selection call itself, and that call is extra work in front of a main model that was already making this decision on its own.

To beat the full catalog, a shortlist has to keep almost everything the task needs while making the downstream turn noticeably cheaper or faster. In the replay it kept 14 of 17 required skills, the main model then dropped one more than it did with the full list, and in these single observations the main call didn’t get faster by enough to pay for the router call. The one-request arms, meanwhile, either overselect or need seconds of Luna time. None of this is damning. It just doesn’t add up to anything I’d switch on.

What would make me switch it on

I’d want four answers before activating anything:

  1. What existing work disappears? Name the specific call, prompt section or decision that no longer happens, and measure it, rather than adding a step in front of an unchanged pipeline.
  2. Does coverage hold? The selection has to keep the skills and tools the task actually needs, against labels that are better than one person’s partial set.
  3. What happens in hard state? Uncertain requests, long conversations and already-loaded skills need defined behavior and their own test, not a silent fallback.
  4. Does the same task get better end to end? Same task, same main model, repeated runs, final answers compared, not just the first selection boundary.

One plausible candidate exists in principle: a narrow decision that’s currently expensive, where a fast yes/no answer could replace a large-model call outright. I haven’t found or proven that case. Until something passes this test, keeping the default full catalog is a reasonable outcome. Nothing has been activated as a production router.

Methods and limits

  • The plugin test. Both community plugins ran as published, at the versions linked in the sources, with one fixed Jev version. I used 24 sanitized requests from my history: 4 to calibrate and 20 held out. They were picked on purpose, not at random, and requests from the same session are related. The labels were frozen before the first call. A later correction to the tool list, a snapshot of 100 tool schemas, was also frozen before the extra responses were collected. A single annotator labeled everything.
  • The one-request comparison. Five configurations on the same 30 scenarios (24 historical, 6 synthetic). Jev’s cost uses TypeSafe’s documented price for the Jev version we called. Configurations that return scores count a skill as picked at 0.5 or above. Timeouts count against coverage; the timing and token averages use only the 26 scenarios that succeeded in every configuration. These numbers were recomputed offline from saved responses, not from a new run.
  • The replay. 12 episodes from 10 sessions, 24 main-model attempts with gpt-6.1-sol at high effort, and 10 Jev selector requests. Both arms had the same 54 tool schemas, the same instructions and the same read-only way to look things up. The labels were frozen before the A/B runs and are my own partial judgments. No tool from the old sessions was actually run again.
  • What was never measured: final answer quality, complete Hermes task latency, combined skill-plus-tool selection, repeated timing runs, and actual billed dollars.
  • Scope of this article: it reuses the existing reports. No new inference was run for it.

Notes

The banner is a parody built on a small, low-resolution reference still of Steve Carell as Michael Scott from The Office, cut out and placed on a dark backdrop. It isn’t a licensed photograph. There’s no affiliation with or endorsement from the actor or the show.

Sources: hermes-jev at the reviewed revision, Tool Slimmer at the reviewed revision, TypeSafe models and pricing, TypeSafe primitives, Hermes Jev TypeSafe plugin docs.