Skip to Content

AI projects and traditional software: a concrete comparison

April 22, 2026 — QTNOVA

When we start a sales conversation with a client who wants "an artificial intelligence project," that client rarely knows what it involves or what to expect. "At the end of the day, it's a software project like any other, right?"

The honest answer is: yes and no. In the abstract, the life cycle is the same — you start from a business problem, define the requirements, write code, test, release and maintain. But what happens inside that cycle, and above all how you measure whether the system works, changes substantially when there's a language model at the center.

In this article we walk through two real projects we took into production in 2025-2026, with very different levels of complexity, and try to show concretely where the difference lies.

To protect our clients' confidentiality, we'll use the labels Client A (a manufacturing company in Lombardy) and Client B (an Italian manufacturer of elevator components). All the technical references are genuine.

Introduction: deterministic vs. probabilistic software

Traditional software development rests on one fundamental property: it is deterministic. Given an input and the state of the system, the output is predictable. A function that adds two numbers will always return the same result for the same arguments. This property is what lets us write a specification, encode tests, and state with reasonable confidence that "the system works" when all the tests pass.

This is the foundation of a practice such as Test-Driven Development, formalized by Kent Beck: first write a failing test (red), then the minimum code to make it pass (green), then clean up the code (refactor). The cycle works because every assertion is unambiguously true or false.

A word of caution, though: even in classic software, there are insurmountable theoretical limits to what a test can guarantee. Two historic results remind us of this:

  • The halting problem (Turing, 1936) shows that there can be no general algorithm capable of deciding, for an arbitrary program, whether that program will halt or not;
  • The theorem proved by Rice (1951) generalizes the result: any non-trivial semantic property of a program is undecidable.

In practice: there is not, and cannot be, a "super-test" that automatically verifies whether a program does what we expect. Tests cover specific statements — precise assertions about precise behaviors — and our confidence depends on the density and quality of those assertions. But they remain an approximation. TDD works because it accepts this limit and iteratively builds a mesh of assertions dense enough to make regressions immediately visible.

Now let's turn to LLM-based systems. Here it's a different story, for at least three reasons:

1. The output is not deterministic, even for the same input. Even with parameters such as the model's "creativity" (in the jargon, its "temperature") set to zero and with fixed seeds, the same question (prompt) can produce different answers from one call to the next. The reason runs deeper than you might think: it has to do with how GPU kernels handle batching during inference, and it can't be solved with a couple of flags. If you want to dig deeper, this article by Thinking Machines Lab goes into the technical details, while a recent post by Sara Zan explains it in accessible terms.

2. The output is often natural language. Not a boolean, not an integer, not a JSON object: a sentence, a paragraph, an explanation. How do you tell whether an answer in Italian to a technical question is "correct"? There are infinitely many equivalent ways to phrase it, and string-to-string comparison makes no sense at all.

3. The system's behavior depends on variables that don't exist in classic software. The model chosen, the model version, the system prompt, the temperature, the connected tools, the context retrieved from a vector store, the order in which information is presented. Every choice affects quality, and every change can improve some answers while making others worse.

That's why the industry has stopped talking about testing and started talking about evaluation. As OpenAI's documentation puts it: traditional testing methods are insufficient for AI systems, and you need evals. Anthropic has published an entire paper on the challenges of evaluating AI systems that shows how even small changes to a prompt's formatting can shift a model's metrics by several percentage points. Anyone who has worked on LLM systems in production — see the collaborative write-up Applied LLMs by Yan, Bischof, Husain and others — agrees on one thing: the quality of an AI system in production rests on the quality of its eval system, not on the quality of the model.

There's even a name for this approach: Evaluation-Driven Development, by analogy with TDD. You'll find it formalized on the MLflow blog, the tool we used for the second project.

With this context in mind, let's move on to the two concrete cases.

OK, but in practice?

Today, April 22, 2026, we want to tell you about two artificial intelligence projects with two very different levels of difficulty.

The first is currently in production at Client A, a manufacturing company in Lombardy, and turns order emails and quote requests — which arrive in mixed, unstructured formats — into clean, structured records in the company's ERP.

The second is in pre-production at Client B, an Italian manufacturer of elevator components, and is a virtual diagnostic assistant that helps field maintenance technicians interpret wiring diagrams and technical manuals.

Why did we choose these two projects in particular, if on the surface they both sit within the same "AI life cycle"? Because they illustrate two very different levels of difficulty perfectly.

Client A's project has a relatively modest technical core from an AI standpoint: it's about extracting information from an unstructured (or semi-structured) format into a structured one. The expected output is a JSON with well-defined fields — a product code, a quantity, a price — that feeds a traditional software flow. Evaluation is binary: either the extracted value matches the expected one, or it doesn't.

Client B's project, on the other hand, is far more complex. There's a RAG component (Retrieval-Augmented Generation, introduced in the paper by Lewis et al. in 2020), with embeddings and a vector store for the technical manuals. There's a context optimization component, because the document corpus is large and the model must be shown only what is actually relevant to the specific question. There's a prompt engineering component, with domain-specific rules for interpreting wiring diagrams. There's an image preprocessing component based on computer vision. There's a tool calling component that lets the model use deterministic tools (such as code execution) to improve the quality of its output. And above all, there's an unstructured output component: the answer is a natural-language explanation for a technician in the field, and it has to be evaluated as such.

We'll see how in the first case evaluation is almost "old school" — you measure a precise accuracy rate on a dataset and iterate — while in the second case evaluation becomes an engineering activity in its own right, with multiple scorers, systematic comparisons between configurations and, above all, quantitative data showing the impact of every single change to the prompt, preprocessing, model or tools.

Client A — Structured extraction from the chaos of email

The problem

Client A is an industrial group that manages its master data, catalog and orders on a legacy ERP built on SQL Server. The ERP works, it's deeply integrated with inventory, shipping and invoicing, and replacing it in the short term is not an option. The sales front end, however, is inefficient: customer orders arrive by email, with PDF or Excel attachments, or directly in the body of the message as free text. Every customer uses its own template. Some list the products in a table, others in a bulleted list, others still in a running sentence.

Client A's operators spent a significant part of their day reading these emails and manually transcribing the orders into the ERP. Slow, repetitive, error-prone work — and above all, hard to scale during peak periods.

The solution

We designed a hybrid system with two components:

  1. A modern Odoo-based front end, where customer emails land. An AI assistant automatically reads the content of the email and its attachments and extracts a structured JSON with a header (customer, delivery address, date, order reference) and lines (item code, quantity, unit price, any food-contact certifications, delivery date). An operator, whom we call the Requests Operator, reviews the ambiguous cases — unidentified customer, SKU with multiple matches, address not on file — and confirms the order.
  2. A Python sync container that reads incrementally from the legacy ERP at regular intervals and updates Odoo: items, partners, addresses, payment terms, order history. When the operator confirms the order in Odoo, the system reports back to the ERP through a REST call that creates the quote request in the original system. The legacy system remains the master for fulfillment and invoicing; Odoo becomes the master for order intake.

The measurable value is clear-cut: shorter processing time per order, far fewer transcription errors, and real-time visibility for sales and operations without forcing them to work directly in the legacy system (plus all the benefits that come with software like Odoo: change history, notes on quotes and orders, and the ability for users to chat with each other inside the software).

The process

How does a project like this unfold? Let's go through it phase by phase, highlighting where the AI component brings in new considerations.

Phase 1 — Collecting the sample. Before making any technical choice, we worked with the client to collect a diverse sample of real documents: PDFs, Excel files, plain-text emails, orders with a tabular layout, orders with lines written "as prose," cases with food-contact certifications, cases with multiple delivery addresses. Without a representative sample, any later benchmark would have been skewed.

Phase 2 — First natural-language extraction tests. At this stage we don't care about the shape of the output yet: we want to find out whether the chosen model — in our case GPT-5.2, through the ecosystem of adapters we developed in-house — is able to understand the documents. We give it a PDF and ask, in plain language: "what is this customer ordering?" If the answer shows "understanding," we're good; if the model makes up products or ignores attachments, we know the problem is at the root and no downstream optimization will fix it.

This step is also when we confirm the project's commercial feasibility. If the AI can't even read the client's documents, there's no point going any further.

Phase 3 — Defining the structured output. Once past the feasibility stage, we move on to making the AI format its output to a strict schema: a header with customerVatCode, orderReference, deliveryAddress, deliveryDate, and lines with skuReference, description, quantity, unitPrice, discount, revision, plus a set of booleans for food-contact certifications (Food Grade, MOCA, Material Detectable, FDA). This schema is the contract between the probabilistic part of the system (the AI) and the deterministic part (our software), which takes the output, cross-references the data with the catalog and order history, applies specific business flows — recognizing the customer from the email domain, matching the customer's code to the product variants — and finally writes to the ERP.

Phase 4 — Evaluation and iteration. This is where the interesting part comes in — though it's also, relatively speaking, the simplest part of this project. Because for Client A, the output cannot be ambiguous. Whether the original document writes the price as 15.34, 15,340 or 15,34 € (note the different use of periods and commas), we expect the model to always extract 15.34. The same goes for quantities, codes and dates: there is an objectively correct answer, and the model has to produce it.

So we build a ground-truth dataset, labeled by hand, from the documents collected in Phase 1, and iterate on the prompt and the model parameters, measuring the exact match rate field by field. A couple of prompt iterations got us to 100% accuracy on our dataset.

A concrete example of an iteration: some specific customers send orders in which the same product appears with duplicate or variant codes (the customer's internal code plus a standard code). On the first pass, the model picked one at random. Only someone who knows the domain and has experience with that customer knows that, in that case, one specific code of the two is the relevant one. A rule in the prompt — "for customer X, when you find both codes, pick the second one" — solved the problem.

Phase 5 — Staging and go-live. Once acceptable accuracy thresholds are reached, we bring the software and the model together in a staging environment identical to production, run UAT (User Acceptance Testing) with the real operators, and release. From then on, the system is monitored in production, and when new cases come up that trip the model up, they are added to the evaluation dataset and we iterate again.

As you can see, for a project of this kind the paradigm stays very close to the classic one: there's a specification (the JSON schema), a test dataset (the labeled sample), and a binary metric (match / no match). The AI is "just" the component that produces the output; everything else is deterministic software that we can test as we always have.

Client B's project forces us to step outside this pattern.

Client B — The diagnostic assistant for maintenance technicians

The problem

Client B makes components and electrical control panels for elevators. Every installed system is delivered with:

  • a wiring diagram (a multi-page PDF) specific to that model;
  • a set of technical manuals for the installed components (inverters, relays, sensors, control boards) from different brands — Mitsubishi, FRENIC, EURO and many others.

When an elevator breaks down, a field maintenance technician steps in — an employee of the installation company, which is itself a customer of Client B. This technician has to:

  1. Figure out which component is faulty;
  2. Read the wiring diagram to trace the connections;
  3. Consult the technical manuals for the components involved;
  4. Diagnose the cause and carry out the repair.

Every step takes time, experience and access to documentation that is often physically far away or out of date. In complex cases, the technician calls a senior engineer at Client B, who is available during business hours. Every job produces know-how — the diagnosis, the fix, the notes — but that know-how is not captured: it stays in the individual engineer's head or, at best, in an archived email. And given the number of installations in the field, the time the engineers spend is a significant cost for Client B.

The solution

We built a platform, again based on Odoo, organized into three layers.

Layer 1 — AI-powered knowledge base. Client B has historically kept the components' technical manuals on an internal file server, fed day after day by its technical staff. At QTNOVA, we believe a good AI project should optimize existing workflows, not upend them: so we made sure the system draws directly from that same data source, without asking the client to change its working habits. The manuals go through a pipeline that extracts the text (falling back to OCR when needed), asks the model to extract structured metadata (component name, manufacturer, aliases), creates the component's master record and indexes the document in a vector store provided by Google.

The wiring diagrams, on the other hand, go through a preprocessing pipeline based on computer vision (OpenCV). There's a non-obvious detail here: the original diagrams are black and white, as is typical of industrial technical documentation. To help the model interpret them correctly, our preprocessor applies algorithmic coloring to the wires and junctions, so that every connection is annotated in a visually and textually unambiguous way. It's a piece we invested heavily in: without it, the model constantly gets confused when interpreting the diagrams.

Layer 2 — Diagnostic chatbot. The technician accesses a dedicated interface through live chat. The chatbot walks them through a few steps: accepting the disclaimer, identifying the installation (by PF number, the identifier Client B assigns to every control panel it delivers, or by diagram number), and asking the question. At that point the Gemini 3.1 Pro assistant comes into play — when the project started, it was the model that performed best on this specific task in our preliminary tests, particularly in the multimodal understanding of complex technical documents such as wiring diagrams. The model receives the wiring diagram of the specific installation in its context, has tool access to the manuals for the relevant components (filtered by component, thanks to the diagram → components mapping we build at ingestion time), and answers the technician.

Layer 3 — Escalation to a human, via ticket. When the AI isn't enough — either because the technician explicitly asks for it, or because the system detects that the conversation isn't converging — the chatbot automatically opens a helpdesk ticket, prefilled with the chat transcript, the PF number, the diagram, the warranty status (calculated as the date of the original DDT delivery note + 12 months) and the correct support project. A Client B engineer works the ticket, fills in the resolution and logs the timesheet. When the ticket is closed, the final resolution is indexed in the vector store: future conversations can also retrieve relevant past solutions. The know-how gains value over time.

On top of this come warranty tracking; management of support projects split between "Support" and "Warranty" and between the current year and past years; automatic expiry of AI access for the technicians of customers who stop ordering; and locking of proprietary diagrams for unauthorized users. A complete platform, in short — not just a chat.

The process

This is where the story gets interesting. Because if for Client A the AI was "a feature," for Client B the AI is the heart of the product, and that radically changes how you develop and evaluate it.

"Collateral" deterministic complexity. Before we could even address the quality of the answers, we had to deal with a whole series of classic problems that an LLM-based conversational application brings with it. Keeping a conversation consistent if the technician loses the connection while the model is still generating its answer. Handling demand spikes on the provider side, when new models are released and start rejecting requests due to saturation: we implemented exponential backoff with retries and fallbacks. Capping the number of AI answers per conversation to prevent drift and runaway costs. Rate limiting on the endpoints. All "deterministic" problems that are tackled with classic testing, but that didn't exist in Client A's project.

The real challenge: how do we evaluate the answers? The heart of the project is a natural-language conversation between a technician and a model. There's no unitPrice field to compare. The model's answer can be more or less correct, more or less complete, more or less grounded in the right sources. A typical question a technician might ask is: "what is terminal X on diagram Y connected to?" A seemingly simple question that carries considerable ambiguity:

  • Should the answer include only the direct connections to terminal X, or the indirect ones too (that is, paths that run through jumpers, junctions and other terminals)?
  • Should it cite the PDF pages it took the information from? In what format?
  • Should it use the nomenclature found on the diagram, or the generic terms used in the manuals?
  • If the model has both the diagram (in the conversation context) and the component manuals (in the vector store), which of the two sources should it prioritize for this specific question?

This kind of ambiguity isn't resolved with a unit test. It's resolved by defining explicit, measurable evaluation criteria, building a dataset of questions and expected answers validated together with the domain experts (Client B's engineers), and systematically measuring the impact of every change to the system against those criteria.

The tool: MLflow and the evaluation matrix. For this project we built an evaluation framework based on MLflow that, for each system configuration, runs the full set of test conversations and computes a battery of automatic metrics. The configurations vary along several axes:

  • Model (GPT-5.2, GPT-5.4, Gemini 3.1 Pro — with OpenAI and Google as independent providers);
  • System prompt (several variants with different levels of detail on the rules for interpreting the diagrams);
  • Code execution on / off;
  • Diagram preprocessing (none, wire detection, junction detection);
  • External tools on or off;
  • Model temperature.

Every turn of every conversation is assessed by several scorers:

  • Correctness (LLM judge, based on LLM-as-a-Judge): is the answer factually correct compared with the expected answer? Returns Pass/Fail.
  • Fact recall (custom LLM judge): what fraction of the "expected facts" — a list of atomic facts that a correct answer should contain — actually appears in the answer? Returns a value between 0 and 1 (number of facts found / total number of facts).
  • Completeness: does the answer list all the relevant connections, include page references, and distinguish direct from indirect connections?
  • Plain text format: deterministic; checks that the answer contains no Markdown (an explicit requirement from Client B, for compatibility with the technician's chat).
  • Response length and turn duration: operational metrics.

The dataset: 14 conversations chosen precisely because they're hard. One fundamental point to stress, to put the numbers we're about to show in context: the evaluation dataset is not representative of the average questions a technician might ask. It's a dataset built together with Client B's engineers by selecting cases where early tests had shown that the baseline model, unaided, got it wrong almost systematically. These are 14 multi-turn conversations — more than 50 individual turns — which we chose precisely because we wanted to measure improvement on what was hard, not on what already worked.

This is an important principle: to build a solid AI system, you have to evaluate it where it's fragile, not where it's trivial. And you have to do it with a dataset labeled by domain experts — work that takes days of joint sessions with the client's technicians.

The iterations and the numbers. We ran several iterations on the system, each one holding the other variables constant and changing only the piece we wanted to measure. Here are the four most significant.

Iteration v1 — Baseline with GPT-5.2. This is the starting configuration: no diagram preprocessing, a minimal prompt, no code execution. Its job is to mark the bottom of the scale.

  • Correctness: 14% (Pass/Fail)
  • Average fact recall: 0.26

On the "hard" test dataset we had selected, the unaided model passes one conversation in seven and recovers about a quarter of the expected facts. It's a very low starting point, and an expected one, which gives us an honest baseline for comparison.

Iteration v2 — Switching the model to Gemini 3.1 Pro. We keep everything else identical and change only the underlying model. Rationale: Gemini 3.1 Pro has a deep thinking mode and native handling of multimodal files (diagram PDFs passed as attachments), which we expected could improve the visual understanding of the wiring diagrams.

  • Correctness: 36% (+22 points)
  • Average fact recall: 0.38 (+0.12)

Switching the model alone produces a substantial improvement. Interestingly, fact recall grows less than correctness: the model more often gives answers that are "correct overall" but not necessarily more complete in the details.

Iteration v3 — Diagram preprocessing with wire detection. This is where the custom computer vision component we built comes in. Before the PDF diagram is passed to the model, a preprocessor detects the colored wires and junctions and produces an annotated version of the diagram, in which every connection is made explicit in a visually and textually unambiguous way. We also refine the system prompt with specific rules on treating "Sheet 001" (the legend of Client B's diagrams) as the source of truth.

  • Correctness: 43% (+7 points)
  • Average fact recall: 0.63 (+0.25)

The truly significant figure here is the jump in fact recall: +0.25 in a single iteration. Preprocessing gives the model a much more "readable" input to start from, and that translates into answers far denser in correct facts. Correctness grows more modestly because some borderline cases are still classified as Fail even with more correct facts in them.

Iteration v4 — Final refinement. At this point we further refined the prompt, the per-component filters on the vector store and the tool selection rules. Some conversations in the original dataset were revised because the question itself turned out, in hindsight, to be ambiguous even for the client's engineers — a sign that evaluation isn't only for assessing the model, but also for uncovering ambiguities in the domain that would otherwise have stayed hidden.

  • Correctness: 58% (+15 points)
  • Average fact recall: 0.81 (+0.18)

Compared with the v1 baseline, on a dataset deliberately selected to be tough, we went from 14% to 58% correctness and from 0.26 to 0.81 fact recall. These metrics aren't the finish line — the project is in pre-production and the work continues — but they prove that every architectural change to the system produces measurable effects: effects we can isolate, effects we can defend in front of a client.

Why all this matters. Let's go back to where we started. In Client A's project we had a binary metric (match / no match), and a few iterations were enough to take it to 100%. In Client B's project the metric itself is composite (correctness, fact recall, completeness, groundedness), 100% isn't a realistic target on a hard dataset, and every architectural change can improve one aspect while worsening another.

Without an eval framework like MLflow, deciding whether a change of model or prompt "is an improvement" would come down to gut feeling. With the framework, it becomes a matter of numerical comparison: measurable, traceable and — this is the crucial point for an IT manager — auditable. If the system in production starts performing worse tomorrow, we have a historical record of what changed and how each change affected the metrics. It's the equivalent of regression testing, but for a probabilistic system.

Conclusion

The two projects show two different levels of AI complexity, but they share a common message: in the world of LLM-based systems, the difference between a prototype that works in demos and a system that works in production lies in the quality of the eval system.

Traditional software has taught us that you can't prove a program correct in an absolute sense — as Rice's theorem reminds us — but you can build a web of tests dense enough to catch regressions and guide development. TDD turned this principle into an operational practice.

With AI systems the challenge is similar, but on a different level: the output is no longer true or false, but more or less good according to multiple criteria. So you need new tools — labeled datasets, LLM-as-a-judge scorers, frameworks like MLflow — and above all a change of mindset on the team's part. You don't "test" an AI system in the traditional sense; you evaluate, compare and iterate.

For an IT manager weighing an investment in an AI project, the right questions to ask the vendor are not "which model do you use?" or "what accuracy do you achieve?" They are:

  • How do you measure quality metrics, and how do you track them over time?
  • If we change the underlying model six months from now — and that will happen — do you have a way to show whether the system is doing better or worse?
  • Do you have a plan for monitoring in production, not just for development?

A soundly set-up AI project shows in the answers to these questions, not in the initial demo. And that, all the underlying technical complexity aside, is perhaps the most important difference to grasp compared with classic software development.

If the topic interests you: at QTNOVA we are building systematic experience in putting LLM-based systems into production. For a no-commitment conversation about your specific case, write to us.


Want to know more? Leave us your email below and we'll get back to you as soon as possible.

Send 

Thank you for your message

Our team will get back to you as soon as possible.
In the meantime, we invite you to visit our website.

Odoo in a nutshell - How to manage sales agent commissions