Valentius Kryptix

Most people meet language models as chat windows: you type a question and get an answer. An agent is a different kind of program. It is handed a goal and a toolbox, and then it decides for itself what to do next: which tool to call, what to pass into it, whether the result is enough, and when to stop. During my internship at Valentius Kryptix, my milestone, “Agentic AI Workflow”, was to build one of these and to prove that it works instead of just claiming it does.

The milestone had three parts. The agent had to complete a task using at least two tools. It needed one customisation of its behaviour. And it had to be measured against at least ten test cases, with its intermediate steps logged so that any failure could be explained.

Picking a problem worth an agent

A good first agent task has two properties: it needs several steps, and its answers can be checked. Expense claims fit both. A company policy is a set of rules with numbers in it, an employee’s claim is messy English, and the correct reimbursement can be worked out by hand. If the agent says $648, I can verify it in ten seconds.

I invented a small policy for a fictional company. Meals are capped per day, hotel nights per night, taxi rides per trip and home internet per month. Mileage is paid per mile. There are also rules about alcohol, receipts, business-class flights and a 30-day submission deadline, and domestic and international limits differ for meals and hotels. Making the policy fictional kept the project reproducible, because anyone can clone the repository and get the same answers without paid data or live exchange rates.

What’s under the hood

ExpenseBot is written in Python, with LangGraph providing the loop in which the model reasons, calls a tool, reads the result and continues. Three tools do the work.

The first searches the policy file. Given a short query such as “lodging international”, it scores each rule by word overlap and returns the best matches, or the phrase “NO MATCHING POLICY” if nothing fits. The second converts euros, pounds, rupees or yen into dollars at fixed rates. Fixed rates are a deliberate choice: with live rates, a test that passes today might fail tomorrow for reasons unrelated to the agent.

The third is a calculator, and it taught me something about trust. The obvious way to evaluate a math string in Python is eval, which will happily run any code the model writes. Instead, I parse the expression into a syntax tree and allow only numbers and basic operators:

_OPS = {ast.Add: operator.add, ast.Sub: operator.sub,
        ast.Mult: operator.mul, ast.Div: operator.truediv}

def _eval(node):
    if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
        return node.value
    if isinstance(node, ast.BinOp) and type(node.op) in _OPS:
        return _OPS[type(node.op)](_eval(node.left), _eval(node.right))
    raise ValueError("unsupported expression")

Anything else, such as a function call or an import, raises an error that the model sees and can recover from. When a model’s output feeds directly into code, it should be treated as untrusted input.

Giving the agent a job description

The customisation requirement pushed me to write a system prompt that reads like a job description for an auditor. It sets a persona (precise, neutral, concise) and a numbered procedure: identify each expense item, look up the policy for each one, convert any foreign amount before comparing it with a dollar limit, do all arithmetic through the calculator, and take the smaller of the claimed amount and the allowed amount. The answer must come in a fixed shape: a decision, a reimbursable amount, and a one-line reason.

The most important instruction is a negative one. If the policy search finds nothing, the agent must say so and send the user to Finance, never improvising a rule. One of my test cases asks about stock option reimbursements, which the policy doesn’t mention. A helpful-sounding but fabricated answer would be the worst possible behaviour for a system that approves spending.

One claim, step by step

Here is how the hotel claim plays out: three nights in an international hotel at 200 euros a night.

1. The agent searches “lodging international” and gets back the lodging rule: $250 per night for international stays.
2. It calls the converter with 200 EUR and receives 216.00 USD.
3. It sees that 216 is under the 250 cap, so the full amount is allowed.
4. It sends “3 * 216” to the calculator and gets 648.
5. It writes: Approved, $648, with the cap cited as the reason.

If the hotel had cost 300 euros a night, which is $324, step 3 would flip. The cap would apply, the agent would calculate 3 times 250, and the decision would be “partially approved” at $750. The design means every number in the answer comes from a tool, so it can be traced back to a line in the log instead of to the model’s mental arithmetic.

Designing the test set

I wrote twelve test claims before running anything, each with an expected result. They cover five kinds of behaviour: cap enforcement (hotel, meals, taxi, internet), straightforward arithmetic (mileage at $0.67 a mile), currency handling (euros, rupees, pounds, yen), pure rule lookups (business class needs manager approval on flights over six hours, receipts are needed above $25, claims expire after 30 days), and two boundary cases, an alcohol rejection and a topic the policy doesn’t cover.

A test passes only if two things hold: the final answer contains the expected value, and the agent actually used the tools that value depends on. That second condition is stricter than most examples I had seen, and it turned out to matter, as the third failure below shows. The harness prints a pass or fail for each case with the reason, and saves the details to files.

Three failures with three different causes

My first complete run on the larger open model scored 10 out of 12. I treated each failure as a detective case, and none had the cause I expected.

The taxi that didn’t exist. A claim for a 5,000 rupee taxi ride got the reply that no taxi policy was found. My first thought was to rewrite the prompt. The log said otherwise: the agent had searched “taxi domestic”, and my scoring gave the word “domestic” the same weight as “taxi”. The meals and lodging rules both contain “domestic”, so three rules tied, and I was returning only the top two. The taxi rule lost the tie. The model had behaved exactly as instructed with the bad evidence it was given. I fixed the search by ignoring generic words, adding synonyms like “cab” and “ride”, and returning three results. The prompt didn’t change at all.

The empty answer. On a mileage claim, the agent found the right rule and then returned a blank message, twice in a row. Nothing in my tools was wrong. The model was occasionally ending its turn without finishing. I added a retry: if the final message is empty, the agent is told to continue, up to two times, and the retry count is saved in the log. It’s a patch rather than a cure, and I documented it that way.

The right number, the wrong way. Running the same agent on a smaller open model, two tests failed even though the answers were numerically correct: $270 for three days of meals and $648 for the hotel. The model had done the multiplication itself without calling the calculator. I kept those as failures. A system that sometimes guesses correctly is not the same as a system that computes, and the difference shows up later, on a claim with awkward numbers, when nobody is checking. This is also where I found a flaw in my own test: in one answer the explanation described the claim as $200 a night when it was 200 euros. The total was right, but a check on the final text could never catch the sloppy reasoning.

The unglamorous parts

Two non-AI problems took real time. My free API plan has a daily token allowance, and a few evaluation runs died halfway with rate-limit errors. My script originally recorded those as test failures, which would have written misleading results over good ones. I rewrote it to stop cleanly on a rate limit, save everything finished so far, and pick up later with a resume option. The write-up says plainly which cases ran in which session.

The second problem was a rejected push. GitHub scans for secrets, and it found a real key sitting in the template file I had meant to fill with a placeholder. Annoying at the time, correct in hindsight. I replaced the key, rewrote the single commit before it went public, and revoked the old key. The habit I took from it: real credentials live only in an ignored file, and anything that has been exposed gets rotated, not just hidden.

What the numbers do and don’t say

Across runs on the larger model, the score went 10, then 11, then 11 out of 12, as I fixed retrieval along the way. The smaller model scored 10. Those numbers shift even with temperature set to zero, so I read them as a rough range, not a precise measurement. Twelve hand-written cases are enough to find bugs; they are not enough to claim the agent is reliable in general. I would rather say that clearly than round up.

Limits, and what I’d build next

ExpenseBot is a prototype. Its search is keyword-based with a hand-built synonym list, so an unusual phrasing can still miss a rule. The policy and exchange rates are invented. It judges one claim at a time with no memory, so it can’t enforce a monthly limit across several claims. And its tests inspect the final number and the tools used, not the quality of the explanation.

The next steps follow directly from those gaps: embedding-based search instead of word overlap, claims with several expense types at once, a checker that also reads the reasoning, and repeating each evaluation several times to separate real improvement from noise.

Takeaways

If you are building your first agent, three habits will save you time. Read the log of tool calls before you touch the prompt, because the tools often hold the real problem. Test the process as well as the output, because a correct answer reached by guessing is a warning. And report your results the way you’d want to read them: with the failures, the causes and the limits.

Thank you to Valentius Kryptix for the milestone. The code, test cases and results are in the repository: https://github.com/Yogendra404/expense-agent