Using Amazon Bedrock, Anthropic Claude for Langchain Tools and Agents

Amazon Bedrock

Overview

This article is about how we can use LLMs to create an application that can interact with users to fetch proprietary data via APIs and let the LLM consume or pass along that data to users based on user's question. We will use Amazon Bedrock, Anthropic Claude, Langchain and Monex Insight Market Data APIs to create a simple proof of concept on how that can be achieved.

Amazon Bedrock

Amazon Bedrock is a fully managed service that offers choice of foundation models (FMs) from leading AI companies like Ai21 Labs, Anthropic, Cohere, Meta, etc and also Amazon via a single API. More information about Amazon Bedrock can be found in References.

Anthropic Claude

For this blog, we will focus on Anthopic's large language model (LLM), Claude. As of writing this article, Anthropic released Claude v3 Sonnet and Haiku, which are, according to some statistics comparable or even better than Chat GPT v4. For this article, we will use Claude v2.1 and will not focus on which LLM is better in this article and just focus on Claude and how we can use it.

LangChain

LangChain is a framework for developing applications powered by language models. It has libraries written in both Python and Javascript. For this article, we will use LangChain's Javascript library.

Agents

Agents are LangChain components that uses LLMs as a reasoning engine to determine which actions to take and in which order.

Tools

Tools are LangChain components and are functions that an Agent can invoke. Tools can be customized to fetch data from DB, call an API (which this article is about), and other custom logics to help provide a relevant response to questions and queries from User.

Monex Insight Market Data API

Insight is one of internal teams in Monex Securities that provides market data directly to users through it's front end systems, and to other Monex Teams and 3rd parties through Market Data APIs. For this article, we will use Scans API that returns symbols based on their Average True Ratio (ATR) and Security Type (ETF, REIT, etc).

Flow Diagram

Below is a high level diagram on the flow and interaction of the components used by this sample program.

User question ---(1)--> App/Langchain ---(2)--> LLM
                        App/Langchain <--(3)--- LLM
                        App/Langchain ---(4)--> Tool/Agent ---(5)--> API
     response <--(8)--- App/Langchain <--(7)--- Tool/Agent <--(6)--- API

Description of each step

  1. User submits a question
  2. App through LangChain sends the prompt that includes user's question to LLM
  3. LLM responds with instructions to LangChain on what will be the next action to take (e.g. call Tools/Agents) based on user question
  4. LangChain calls Tools/Agent
  5. Tool/Agent calls API, passing necessary parameters to API
  6. API responds with data
  7. Tool/Agent pass the API response data back to LangChain
  8. App returns the data from LangChain to User

Implementation

Langchain's Amazon Bedrock module

To use Amazon Bedrock in LangChain, import the Bedrock package included in LangChain, and create an instance of Bedrock that uses Anthropic Claude model.

import { Bedrock } from 'langchain/llms/bedrock';

const model = new Bedrock({
  model: 'anthropic.claude-v2:1',
  region: 'ap-northeast-1',
  modelKwargs: { max_tokens_to_sample: 4096 },
  maxTokens: 40000 // set max token to some reasonably high value where we can pass the below prompt (plus chain of thoughts) to the LLM
});

Note: modelKwargs: { max_tokens_to_sample: 4096 }, is added to avoid getting Too many requests error from Bedrock.

Prompt

Create a prompt that provides detailed instruction to LLM to define its behavior on how it should respond based on use case or service we want to provide.

const prefix = `You are a Stock Market financial assistant, who can help answer stock market related question. 

Please make sure you complete the above objective with the following rules:
1/ You should use the provided tools to get information about the question.
2/ Answer ONLY based on the response from these tools. DO NOT make up answers.

When your task is complete, set 'Final Answer:' field with the formatted response, nothing else. DO NOT truncate response.`;
const suffix = `Begin!

Human: {input}

{agent_scratchpad}

Assistant:`;

const createPromptArgs = {
  suffix,
  prefix,
  inputVariables: ['input', 'agent_scratchpad'],
};

const prompt = ZeroShotAgent.createPrompt(tools, createPromptArgs);

AWS and Anthropic provides guidelines on how to do prompt engineering (see References) but from this sample prompt: - Claude prompt expects Human and Assistance to be included in the prompt. If those are missing, Clause will return bad request response.

Human: {input}
Assistance:
  • Instruct the LLM to use only the tools provided
  • Instruct the LLM to base its responses to queries only based on the data from the Agent/Tools and not make up answers. This is to avoid any hallucinations.

Chains

Chains help make a sequence of calls to LLM or our custom tools. To initialize, we pass the LLM model to use (which is Bedrock), and the prompt.

const llmChain = new LLMChain({ llm: model, prompt });

Agents and Tools

Next, define the agent and our custom tools which the Chain can call if needed.

Sample Tool

Below is a sample code on how to define custom tool that calls an API. More about this can be found in References.

const securityType = new DynamicTool({
  name: 'security-type',
  returnDirect: true, // set to true so that llm chain will stop once it receives response from this tool.
  // If this is set to false, the llm chain will continue to run as it cannot determinate some terminal state.
  description:
    "Useful when getting the list of symbols with their Security Type such as 'ETF', 'REIT', or 'Normal' stock. Inputs should be a JSON formatted string that has optional 'securityTypes' array property, which when provided will return symbols that matches Security Types specified. Output, in JSON format, is an array of symbols and their Security Type.",
  func: async (inputAsJsonString) => {
    console.log('SECURITY TYPE TOOL input', inputAsJsonString);

    console.log(`Fetching symbols with their security type...`);

    let result = await fetch();
    const input = JSON.parse(inputAsJsonString);
    const allowedTypes = input.securityTypes ?? [];

    // transform security type codes into human readable
    result = transform(result);

    if (allowedTypes.length > 0) {
      // filter out symbols that does not match allowed security types
      result = result.filter((item) => allowedTypes.includes(item.value));
    }

    const markdown = formatJsonToMarkdown(result, ['symbol', 'value']);
    return markdown;
  }
});

Agent

import securityType from './scans/securityType.js';

const tools = [securityType];
...

const agent = new ZeroShotAgent({
  llmChain,
  allowedTools: ['security-type'],
  verbose: VERBOSE
});
const agentExecutor = AgentExecutor.fromAgentAndTools({
  agent,
  tools,
  verbose: VERBOSE,
  maxIterations: 3,
  returnIntermediateSteps: VERBOSE,
  handleParsingErrors: true
});

Pass user input to LangChain

The basic application ready to accept user input. Pass the user input to LangChain via the agent executor that was created earlier. Based on user input, and through LLM, LangChain will get instructions on what will be the next action to take.

const result = await agentExecutor.invoke({ input: line });

Sample input:

QUESTION> get all symbols with Security Types ETF and REIT

Below is a log output that shows the interaction between LangChain and LLM based based on user's input:

[chain/start] [1:chain:AgentExecutor] Entering Chain run with input: {
  "input": "get all symbols with Security Types ETF and REIT"
}
[chain/start] [1:chain:AgentExecutor > 2:chain:LLMChain] Entering Chain run with input: {
  "input": "get all symbols with Security Types ETF and REIT",
  "agent_scratchpad": "",
  "stop": [
    "\nObservation: "
  ]
}
[llm/start] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:Bedrock] Entering LLM run with input: {
  "prompts": [
    "You are a Stock Market financial assistant, who can help answer stock market related question. \n\nPlease make sure you complete the above objective with the following rules:\n1/ You should use the provided tools to get information about the question.\n2/ Answer ONLY based on the response from these tools. DO NOT make up answers.\n\nWhen your task is complete, set 'Final Answer:' field with the formatted response, nothing else. DO NOT truncate response.\n\natr: Useful when getting the ATR or Average True Range values of symbols. Inputs should be a JSON formatted string that has optional 'minimum' and 'maximum' properties, which when provided will return symbols with ATR values between those minimum and maximum values. Output, in JSON format, is an array of symbols and their ATR values in percent unit.\nsecurity-type: Useful when getting the list of symbols with their Security Type such as 'ETF', 'REIT', or 'Normal' stock. Inputs should be a JSON formatted string that has optional 'securityTypes' array property, which when provided will return symbols that matches Security Types specified. Output, in JSON format, is an array of symbols and their Security Type.\n\nUse the following format in your response:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [atr,security-type]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nHuman: get all symbols with Security Types ETF and REIT\n\n\n\nAssistant:"
  ]
}
[llm/end] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:Bedrock] [3.71s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": " Question: get all symbols with Security Types ETF and REIT\n\nThought: To get a list of symbols filtered by security type, I should use the security-type tool. I need to specify ETF and REIT as the securityTypes filter.\n\nAction: security-type\nAction Input: {\"securityTypes\":[\"ETF\",\"REIT\"]}\n"
      }
    ]
  ]
}
[chain/end] [1:chain:AgentExecutor > 2:chain:LLMChain] [3.71s] Exiting Chain run with output: {
  "text": " Question: get all symbols with Security Types ETF and REIT\n\nThought: To get a list of symbols filtered by security type, I should use the security-type tool. I need to specify ETF and REIT as the securityTypes filter.\n\nAction: security-type\nAction Input: {\"securityTypes\":[\"ETF\",\"REIT\"]}\n"
}

As shown in the above LLM's output, LLM is telling LangChain that the next action is to execute tool called security-type and the input to the tool is {"securityTypes":["ETF","REIT"]}

[agent/action] [1:chain:AgentExecutor] Agent selected action: {
  "tool": "security-type",
  "toolInput": "{\"securityTypes\":[\"ETF\",\"REIT\"]}",
  "log": " Question: get all symbols with Security Types ETF and REIT\n\nThought: To get a list of symbols filtered by security type, I should use the security-type tool. I need to specify ETF and REIT as the securityTypes filter.\n\nAction: security-type\nAction Input: {\"securityTypes\":[\"ETF\",\"REIT\"]}\n"
}
[tool/start] [1:chain:AgentExecutor > 4:tool:security-type] Entering Tool run with input: "{"securityTypes":["ETF","REIT"]}"

The above log shows that Agent runs the tool security-type passing the arguments as provided by the LLM.

SECURITY TYPE TOOL input {"securityTypes":["ETF","REIT"]}
Fetching symbols with their security type...
>>INSIGHT: Fetching Security Types from https://api.monexinsight.net/market/scans/SecurityType
[tool/end] [1:chain:AgentExecutor > 4:tool:security-type] [348ms] Exiting Tool run with output: "| symbol | value |
| --- | --- |
| JP:1305-TS | ETF |
| JP:1306-TS | ETF |
| JP:1308-TS | ETF |
| JP:1309-TS | ETF |
| JP:1311-TS | ETF |
| JP:1319-TS | ETF |
| JP:1320-TS | ETF |
| JP:1321-TS | ETF |
...
[chain/end] [1:chain:AgentExecutor] [4.06s] Exiting Chain run with output: {
  "output": "| symbol | value |\n| --- | --- |\n| JP:1305-TS | ETF |\n| JP:1306-TS | ETF |\n| JP:1308-TS | ETF |\n| JP:1309-TS | ETF |\n| JP:1311-TS | ETF |\n| JP:1319-TS | ETF |\n| JP:1320-TS | ETF |\n| JP:1321-TS | ETF |\n| JP:1322-TS | ETF |\n| JP:1325-TS | ETF |\n| JP:1328-TS | ETF |\n| JP:1329-TS | ETF |\n| JP:1330-TS | ETF |\n| JP:133A-TS | ETF |\n| JP:1343-TS | ETF |\n| JP:1345-TS | ETF |\n| JP:1346-TS | ETF |\n| JP:1348-TS | ETF |\n| JP:1356-TS | ETF |\n| JP:1357-TS | ETF |\n| JP:1358-TS | ETF 
  ...
  "intermediateSteps": [
    {
      "action": {
        "tool": "security-type",
        "toolInput": "{\"securityTypes\":[\"ETF\",\"REIT\"]}",
        "log": " Question: get all symbols with Security Types ETF and REIT\n\nThought: To get a list of symbols filtered by security type, I should use the security-type tool. I need to specify ETF and REIT as the securityTypes filter.\n\nAction: security-type\nAction Input: {\"securityTypes\":[\"ETF\",\"REIT\"]}\n"
      },
      "observation": "| symbol | value |\n| --- | --- |\n| JP:1305-TS | ETF |\n| JP:1306-TS | ETF |\n| JP:1308-TS | ETF |\n| JP:1309-TS | ETF |\n| JP:1311-TS | ETF |\n| JP:1319-TS | ETF |\n| JP:1320-TS | ETF |\n| JP:1321-TS | ETF |\n| JP:1322-TS | ETF |\n| JP:1325-TS | ETF |\n| JP:1328-TS | ETF |\n| JP:1329-TS | ETF |\n| JP:1330-TS | ETF |\n| JP:133A-TS |
  ...

The above log shows the actual tool has been invoked and API returned some data. The program then shows the API response back to user, which is something similar as shown below.

ANSWER>> | symbol | value |
| --- | --- |
| JP:1305-TS | ETF |
| JP:1306-TS | ETF |
| JP:1308-TS | ETF |
| JP:1309-TS | ETF |
| JP:1311-TS | ETF |
| JP:1319-TS | ETF |
...
| JP:2971-TS | REIT |
| JP:2972-TS | REIT |
| JP:2979-TS | REIT |
| JP:2989-TS | REIT |
| JP:3226-TS | REIT |
| JP:3234-TS | REIT |

Summary

The above sample program shows that through frameworks such as LangChain, and LLMs such as Amazon Bedrock and Anthropic Claude, we can create tools that can provide data by calling APIs, Data Sources or execute proprietary logic/algorithm and return data to users based on user input.

References


Monex Insight Senior Engineer
Jay Landicho

人材募集