Customize Amazon Nova models to improve tool usage

Fashionable huge language fashions (LLMs) excel in language processing however are restricted via their static practicing information. Alternatively, as industries require extra adaptive, decision-making AI, integrating gear and exterior APIs has grow to be very important. This has resulted in the evolution and fast upward push of agentic workflows, the place AI methods autonomously plan, execute, and refine duties. Correct device use is foundational for boosting the decision-making and operational potency of those self sustaining brokers and construction a hit and sophisticated agentic workflows.

On this publish, we dissect the technical mechanisms of device calling the usage of Amazon Nova fashions via Amazon Bedrock, along strategies for type customization to refine device calling precision.

Increasing LLM functions with device use

LLMs excel at herbal language duties however grow to be considerably extra tough with device integration, equivalent to APIs and computational frameworks. Gear allow LLMs to get entry to real-time information, carry out domain-specific computations, and retrieve actual data, bettering their reliability and flexibility. For instance, integrating a climate API permits for correct, real-time forecasts, or a Wikipedia API supplies up-to-date data for advanced queries. In clinical contexts, gear like calculators or symbolic engines cope with numerical inaccuracies in LLMs. Those integrations become LLMs into powerful, domain-aware methods able to dealing with dynamic, specialised duties with real-world application.

Amazon Nova fashions and Amazon Bedrock

Amazon Nova models, unveiled at AWS re:Invent in December 2024, are optimized to ship remarkable price-performance price, providing state of the art functionality on key text-understanding benchmarks at low price. The sequence accommodates 3 variants: Micro (text-only, ultra-efficient for edge use), Lite (multimodal, balanced for versatility), and Professional (multimodal, high-performance for advanced duties).

Amazon Nova fashions can be utilized for number of duties, from technology to creating agentic workflows. As such, those fashions have the potential to interface with exterior gear or services and products and use them via tool calling. This can also be completed in the course of the Amazon Bedrock console (see Getting started with Amazon Nova in the Amazon Bedrock console) and APIs equivalent to Converse and Invoke.

Along with the usage of the pre-trained fashions, builders give you the option to fine-tune those fashions with multimodal information (Professional and Lite) or textual content information (Professional, Lite, and Micro), offering the versatility to reach desired accuracy, latency, and value. Builders too can run self-service customized fine-tuning and distillation of bigger fashions to smaller ones the usage of the Amazon Bedrock console and APIs.

Answer review

The next diagram illustrates the answer structure.

For this publish, we first ready a customized dataset for device utilization. We used the take a look at set to judge Amazon Nova fashions via Amazon Bedrock the usage of the Communicate and Invoke APIs. We then fine-tuned Amazon Nova Micro and Amazon Nova Lite fashions via Amazon Bedrock with our fine-tuning dataset. After the fine-tuning procedure used to be entire, we evaluated those custom designed fashions via provisioned throughput. Within the following sections, we undergo those steps in additional element.

Gear

Instrument utilization in LLMs comes to two essential operations: device variety and argument extraction or technology. For example, believe a device designed to retrieve climate data for a particular location. When offered with a question equivalent to “What’s the elements in Alexandria, VA?”, the LLM evaluates its repertoire of gear to decide whether or not a suitable device is to be had. Upon figuring out an appropriate device, the type selects it and extracts the specified arguments—right here, “Alexandria” and “VA” as structured information varieties (for instance, strings)—to build the device name.

Every device is carefully outlined with a proper specification that outlines its meant capability, the required or not obligatory arguments, and the related information varieties. Such actual definitions, referred to as device config, ensure that device calls are achieved as it should be and that argument parsing aligns with the device’s operational necessities. Following this requirement, the dataset used for this case defines 8 gear with their arguments and configures them in a structured JSON layout. We outline the next 8 gear (we use seven of them for fine-tuning and dangle out the weather_api_call device throughout checking out with the intention to assessment the accuracy on unseen device use):

• weather_api_call – Customized device for purchasing climate data
• stat_pull – Customized device for figuring out stats
• text_to_sql – Customized text-to-SQL device
• terminal – Instrument for executing scripts in a terminal
• wikipidea – Wikipedia API device to look via Wikipedia pages
• duckduckgo_results_json – Web seek device that executes a DuckDuckGo seek
• youtube_search – YouTube API seek device that searches video listings
• pubmed_search – PubMed seek device that searches PubMed abstracts

The next code is an instance of what a tool configuration for terminal may seem like:

{'toolSpec': {'title': 'terminal',
'description': 'Run shell instructions in this MacOS device ',
'inputSchema': {'json': {'kind': 'object',
'homes': {'instructions': {'kind': 'string',
'description': 'Record of shell instructions to run. Deserialized the usage of json.rather a lot'}},
'required': ['commands']}}}},

Dataset

The dataset is a man-made device calling dataset created with the help of a basis type (FM) from Amazon Bedrock and manually validated and altered. This dataset used to be created for our set of 8 gear as mentioned within the earlier phase, with the objective of constructing a various set of questions and power invocations that let every other type to be informed from those examples and generalize to unseen device invocations.

Every access within the dataset is structured as a JSON object with key-value pairs that outline the query (a herbal language person question for the type), the bottom fact device required to respond to the person question, its arguments (dictionary containing the parameters required to execute the device), and extra constraints like order_matters: boolean, indicating if argument order is important, and arg_pattern: not obligatory, a typical expression (regex) for argument validation or formatting. Later on this publish, we use those flooring fact labels to oversee the educational of pre-trained Amazon Nova fashions, adapting them for device use. This procedure, referred to as supervised fine-tuning, might be explored intimately within the following sections.

The scale of the educational set is 560 questions and the take a look at set is 120 questions. The take a look at set is composed of 15 questions according to device class, totaling 120 questions. The next are some examples from the dataset:

{
"query": "Give an explanation for the method of photosynthesis",
"resolution": "wikipedia",
"args": {'question': 'strategy of photosynthesis'
},
"order_matters":False,
"arg_pattern":None
}
{
"query": "Show machine date and time",
"resolution": "terminal",
"args": {'instructions': ['date'
]
},
"order_matters":True,
"arg_pattern":None
}
{
"query": "Improve the requests library the usage of pip",
"resolution": "terminal",
"args": {'instructions': ['pip install --upgrade requests'
]
},
"order_matters":True,
"arg_pattern": [r'pip(3?) install --upgrade requests'
]
}

Get ready the dataset for Amazon Nova

To make use of this dataset with Amazon Nova fashions, we want to moreover format the data in keeping with a selected chat template. Local device calling has a translation layer that codecs the inputs to the suitable layout prior to passing the type. Right here, we make use of a DIY device use means with a customized urged template. In particular, we want to upload the machine urged, the person message embedded with the device config, and the bottom fact labels because the assistant message. The next is a coaching instance formatted for Amazon Nova. Because of house constraints, we solely display the toolspec for one device.

{"machine": [{"text": "You are a bot that can handle different requests
with tools."}],
"messages": [{"role": "user",
"content": [{"text": "Given the following functions within ,
please respond with a JSON for a function call with its proper arguments
that best answers the given prompt.

Respond in the format
{"name": function name,"parameters": dictionary of argument name and
its value}.
Do not use variables. Donot give any explanations.

ONLY output the resulting
JSON structure and nothing else.

Donot use the word 'json' anywhere in the
result.

    {"tools": [{"toolSpec":{"name":"youtube_search",
    "description": " search for youtube videos associated with a person.
    the input to this tool should be a comma separated list, the first part
    contains a person name and the second a number that is the maximum number
    of video results to return aka num_results. the second part is optional", 
    "inputSchema":
    {"json":{"type":"object","properties": {"query":
    {"type": "string",
     "description": "youtube search query to look up"}},
    "required": ["query"]}}}},]}

Generate resolution for the next query.

Record any merchandise that experience won persistently damaging evaluations
"}]},
{"position": "assistant", "content material": [{"text": "{'name':text_to_sql,'parameters':
{'table': 'product_reviews','condition':
'GROUP BY product_id HAVING AVG(rating) < 2'}}"}]}],
"schemaVersion": "tooluse-dataset-2024"}

Add dataset to Amazon S3

This step is wanted later for the fine-tuning for Amazon Bedrock to get entry to the educational information. You’ll be able to add your dataset both in the course of the Amazon Simple Storage Service (Amazon S3) console or via code.

Instrument calling with base fashions in the course of the Amazon Bedrock API

Now that we have got created the device use dataset and formatted it as required, let’s use it to check out the Amazon Nova fashions. As discussed up to now, we will use each the Communicate and Invoke APIs for device use in Amazon Bedrock. The Communicate API allows dynamic, context-aware conversations, permitting fashions to interact in multi-turn dialogues, and the Invoke API permits the person to name and engage with the underlying fashions inside Amazon Bedrock.

To make use of the Communicate API, you merely ship the messages, machine urged (if any), and the device config immediately within the Communicate API. See the next instance code:

reaction = bedrock_runtime.speak(
modelId=model_id,
messages=messages,
machine=system_prompt,
toolConfig=tool_config,
)

To parse the device and arguments from the LLM reaction, you’ll use the next instance code:

for content_block in reaction['output']['message']["content"]:

if "toolUse" in content_block:
out_tool_name=content_block['toolUse']['name']
out_tool_inputs_dict=content_block['toolUse']['input']
print(out_tool_name,out_tool_inputs_dict.keys())

For the query: “Hiya, what is the temperature in Paris presently?”, you get the next output:

weather_api_call dict_keys(['country', 'city'])

To execute device use in the course of the Invoke API, first you want to organize the request frame with the person query in addition to the device config that used to be ready prior to. The next code snippet presentations how one can convert the device config JSON to thread layout, which can be utilized within the message frame:

# Convert gear configuration to JSON string
formatted_tool_config = json.dumps(tool_config, indent=2)
urged = prompt_template.change("{query}", query)
urged = urged.change("{tool_config}", formatted_tool_config)
# message template
messages = [{"role": "user", "content": [{"text": prompt}]}]
# Get ready request frame
model_kwargs = {"machine":system_prompt, "messages": messages, "inferenceConfig": inferenceConfig,} frame = json.dumps(model_kwargs)
reaction = bedrock_runtime.invoke_model(
frame=frame,
modelId=model_id,
settle for=settle for,
contentType=contentType
)

The usage of both of the 2 APIs, you’ll take a look at and benchmark the bottom Amazon Nova fashions with the device use dataset. Within the subsequent sections, we display how you’ll customise those base fashions particularly for the device use area.

Supervised fine-tuning the usage of the Amazon Bedrock console

Amazon Bedrock provides 3 other customization tactics: supervised fine-tuning, model distillation, and continued pre-training. On the time of writing, the primary two strategies are to be had for customizing Amazon Nova fashions. Supervised fine-tuning is a well-liked way in switch finding out, the place a pre-trained type is customized to a particular assignment or area via practicing it additional on a smaller, task-specific dataset. The method makes use of the representations realized throughout pre-training on huge datasets to give a boost to functionality within the new area. All over fine-tuning, the type’s parameters (both all or decided on layers) are up to date the usage of backpropagation to reduce the loss.

On this publish, we use the categorized datasets that we created and formatted up to now to run supervised fine-tuning to evolve Amazon Nova fashions for the device use area.

Create a fine-tuning activity

Whole the next steps to create a fine-tuning activity:

1. Open the Amazon Bedrock console.
2. Make a selection us-east-1 because the AWS Area.
3. Underneath Basis fashions within the navigation pane, make a choice Customized fashions.
4. Make a selection Create Superb-tuning activity underneath Customization strategies. 

On the time of writing, Amazon Nova type fine-tuning is solely to be had within the us-east-1 Area.

5. Make a selection Make a choice type and make a choice Amazon because the type supplier.
6. Make a selection your type (for this publish, Amazon Nova Micro) and make a choice Observe.

7. For Superb-tuned type title, input a novel title.
8. For Task title¸ input a reputation for the fine-tuning activity.
9. Within the Enter information phase, input following main points:
   • For S3 location, input the supply S3 bucket containing the educational information.
   • For Validation dataset location, optionally input the S3 bucket containing a validation dataset.

10. Within the Hyperparameters phase, you’ll customise the next hyperparameters:
    • For Epochs¸ input a worth between 1–5.
    • For Batch measurement, the worth is fastened at 1.
    • For Studying price multiplier, input a worth between 0.000001–0.0001
    • For Studying price warmup steps, input a worth between 0–100.

We advise beginning with the default parameter values after which converting the settings iteratively. It’s a just right follow to switch just one or a few parameters at a time, with the intention to isolate the parameter results. Take note, hyperparameter tuning is type and use case particular.

11. Within the Output information phase, input the objective S3 bucket for type outputs and coaching metrics.
12. Make a selection Create fine-tuning activity.

Run the fine-tuning activity

After you get started the fine-tuning activity, it is possible for you to to look your activity underneath Jobs and the standing as Coaching. When it finishes, the standing adjustments to Whole.

You’ll be able to now cross to the educational activity and optionally get entry to the training-related artifacts which can be stored within the output folder.

You’ll be able to in finding each practicing and validation (we extremely suggest the usage of a validation set) artifacts right here.

You’ll be able to use the educational and validation artifacts to evaluate your fine-tuning activity via loss curves (as proven within the following determine), which observe practicing loss (orange) and validation loss (blue) through the years. A gentle decline in each signifies efficient finding out and just right generalization. A small hole between them suggests minimum overfitting, while a emerging validation loss with lowering practicing loss alerts overfitting. If each losses stay excessive, it signifies underfitting. Tracking those curves is helping you briefly diagnose type functionality and regulate practicing methods for optimum effects.

Host the fine-tuned type and run inference

Now that you’ve finished the fine-tuning, you’ll host the type and use it for inference. Apply those steps:

1. At the Amazon Bedrock console, underneath Basis fashions within the navigation pane, make a choice Customized fashions
2. At the Fashions tab, make a choice the type you fine-tuned.

3. Make a selection Acquire provisioned throughput.

4. Specify a dedication time period (no dedication, 1 month, 6 months) and assessment the related price for internet hosting the fine-tuned fashions.

After the custom designed type is hosted via provisioned throughput, a type ID might be assigned, which might be used for inference. For inference with fashions hosted with provisioned throughput, we need to use the Invoke API in the similar means we described up to now on this publish—merely change the type ID with the custom designed type ID.

The aforementioned fine-tuning and inference steps will also be finished programmatically. Consult with the next GitHub repo for extra element.

Analysis framework

Comparing fine-tuned device calling LLMs calls for a complete strategy to assess their functionality throughout quite a lot of dimensions. The main metric to judge device calling is accuracy, together with each device variety and argument technology accuracy. This measures how successfully the type selects the right kind device and generates legitimate arguments. Latency and token utilization (enter and output tokens) are two different vital metrics.

Instrument name accuracy evaluates if the device predicted via the LLM fits the bottom fact device for each and every query; a rating of one is given in the event that they fit and nil once they don’t. After processing the questions, we will use the next equation: Instrument Name Accuracy=∑(Right kind Instrument Calls)/(General collection of take a look at questions).

Argument name accuracy assesses whether or not the arguments supplied to the gear are proper, in keeping with both actual fits or regex trend matching. For each and every device name, the type’s predicted arguments are extracted. It makes use of the next argument matching strategies:

• Regex matching – If the bottom fact comprises regex patterns, the expected arguments are matched in opposition to those patterns. A a hit fit will increase the rating.
• Inclusive string matching – If no regex trend is equipped, the expected argument is in comparison to the bottom fact argument. Credit score is given if the expected argument accommodates the bottom fact argument. That is to permit for arguments, like seek phrases, not to be penalized for including further specificity.

The rating for each and every argument is normalized in keeping with the collection of arguments, permitting partial credit score when a couple of arguments are required. The cumulative proper argument rankings are averaged throughout all questions: Argument Name Accuracy = ∑Right kind Arguments/(General Selection of Questions).

Under we display some instance questions and accuracy rankings:

Instance 1:

Consumer query: Execute this run.py script with an argparse arg including two gpus
GT device: terminal   LLM output device: terminal
Pred args:  ['python run.py —gpus 2']
Floor fact trend: python(3?) run.py —gpus 2
Arg matching way: regex fit
Arg matching rating: 1.0

Instance 2:

Consumer query: Who had probably the most dashing touchdowns for the bengals in 2017 season?
GT device: stat_pull   LLM output device: stat_pull
Pred args:  ['NFL']
directly fit
arg rating 0.3333333333333333
Pred args:  ['2017']
Immediately fit
Arg rating 0.6666666666666666
Pred args:  ['Cincinnati Bengals']
Immediately fit
Arg rating 1.0

Effects

We are actually able to visualise the effects and evaluate the functionality of base Amazon Nova fashions to their fine-tuned opposite numbers.

Base fashions

The next figures illustrate the functionality comparability of the bottom Amazon Nova fashions.

The comparability unearths a transparent trade-off between accuracy and latency, formed via type measurement. Amazon Nova Professional, the most important type, delivers the best possible accuracy in each device name and argument name duties, reflecting its complex computational functions. Alternatively, this comes with larger latency.

Against this, Amazon Nova Micro, the smallest type, achieves the bottom latency, which excellent for quick, resource-constrained environments, even though it sacrifices some accuracy in comparison to its higher opposite numbers.

Superb-tuned fashions vs. base fashions

The next determine visualizes accuracy growth after fine-tuning.

The comparative research of the Amazon Nova type variants unearths really extensive functionality enhancements via fine-tuning, with probably the most vital positive aspects noticed within the smaller Amazon Nova Micro type. The fine-tuned Amazon Nova type confirmed exceptional expansion in device name accuracy, expanding from 75.8% to 95%, which is a 25.38% growth. In a similar way, its argument name accuracy rose from 77.8% to 87.7%, reflecting a 12.74% build up.

Against this, the fine-tuned Amazon Nova Lite type exhibited extra modest positive aspects, with device name accuracy bettering from 90.8% to 96.66%—a 6.46% build up—and argument name accuracy emerging from 85% to 89.9%, marking a 5.76% growth. Each fine-tuned fashions surpassed the accuracy completed via the Amazon Nova Professional base type.

Those effects spotlight that fine-tuning can considerably toughen the functionality of light-weight fashions, making them sturdy contenders for packages the place each accuracy and latency are essential.

Conclusion

On this publish, we demonstrated type customization (fine-tuning) for device use with Amazon Nova. We first offered a device utilization use case, and gave information about the dataset. We walked via the main points of Amazon Nova particular information formatting and confirmed how one can do device calling in the course of the Communicate and Invoke APIs in Amazon Bedrock. After you have the baseline effects from Amazon Nova fashions, we defined intimately the fine-tuning procedure, internet hosting fine-tuned fashions with provisioned throughput, and the usage of the fine-tuned Amazon Nova fashions for inference. As well as, we touched after getting insights from practicing and validation artifacts from a fine-tuning activity in Amazon Bedrock.

Take a look at the detailed notebook for device utilization to be informed extra. For more info on Amazon Bedrock and the newest Amazon Nova fashions, consult with the Amazon Bedrock User Guide and Amazon Nova User Guide. The Generative AI Innovation Middle has a bunch of AWS science and technique mavens with complete experience spanning the generative AI adventure, serving to consumers prioritize use instances, construct roadmaps, and transfer answers into manufacturing. See Generative AI Innovation Center for our newest paintings and buyer good fortune tales.

In regards to the Authors

Baishali Chaudhury is an Implemented Scientist on the Generative AI Innovation Middle at AWS, the place she specializes in advancing Generative AI answers for real-world packages. She has a powerful background in laptop imaginative and prescient, device finding out, and AI for healthcare. Baishali holds a PhD in Pc Science from College of South Florida and PostDoc from Moffitt Most cancers Centre.

Isaac Privitera is a Important Information Scientist with the AWS Generative AI Innovation Middle, the place he develops bespoke generative AI-based answers to deal with consumers’ trade issues. His number one focal point lies in construction accountable AI methods, the usage of tactics equivalent to RAG, multi-agent methods, and type fine-tuning. When no longer immersed on the planet of AI, Isaac can also be discovered at the golfing direction, playing a soccer recreation, or climbing trails along with his dependable dog spouse, Barry.

Mengdie (Plant life) Wang is a Information Scientist at AWS Generative AI Innovation Middle, the place she works with consumers to architect and put into effect scalableGenerative AI answers that cope with their distinctive trade demanding situations. She focuses on type customization tactics and agent-based AI methods, serving to organizations harness the entire doable of generative AI era. Previous to AWS, Plant life earned her Grasp’s stage in Pc Science from the College of Minnesota, the place she advanced her experience in device finding out and synthetic intelligence.