Automated Customer Support

In this demonstration, we show how to use the Predibase SDK to fine tune LLaMa2-7B for a customer support automation task. We'll give a detailed walkthrough of the task, dataset, and all the steps you need to follow in order to complete this task. The goal will be to teach you how to work with Predibase fine-tuning capabilities and apply these learnings to your own use cases!

Open in Colab Notebook

Authentication 🔐

The first step is to sign into Predibase.

• If you do not have a Predibase account set up yet, you may sign up for a free account here
• If you already have an account, navigate to Settings -> My Profile and generate a new API token.
• Finally, plug in the generated API token in the code below to authenticate.

import yaml
from predibase import PredibaseClient

pc = PredibaseClient(
    token="YOUR TOKEN HERE"
)

Task Description 📝

Customer support is a area where there is a lot of potential for LLM automation. In this use case, we have a dataset where every row contains a complaint to a financial institution about a particular product. Every row contains the following info:

• The Product the user complained about
• The Issue the user ran into
• The Complaint that the customer raised
• The Company Response to the complaint
• And a Structured JSON Output containing the Product, Issue and Company Response values
  • This is the field that we would like the LLM to generate

Here is an example row from this dataset:

Complaint ID	Date received	Product	Issue	Complaint	Company Response	Structured JSON Output
2004629	07/09/2016	Credit reporting	Incorrect info on report	My credit score is below XXXX and there does not appear to be justification for the low score. the credit reporting reasons for the low score : " too many delinquencies, too many credit reporting requests, proportion of balances too high ''. these reasons do not appear valid : there are almost no delinquencies within 24 mo, there are very few credit reporting requests in past 12 mo and the balances for debt to income are 40 %.	"Dear Customer,\n\nThank you for reaching out to us and bringing your concerns to our attention. We understand that you are dissatisfied with your credit score and believe there may be inconsistencies in the credit reporting reasons provided.\n\nWe apologize for any confusion or inconvenience this may have caused you. Our team will investigate your credit score and the reasons provided to ensure accuracy and fairness. It is important to us that your credit score reflects your financial circumstances appropriately.\n\nPlease be assured that we take your concerns seriously and will work diligently to address them. We will contact you with an update as soon as our investigation is complete. In the meantime, if you have any further questions or require any additional information, please do not hesitate to contact us.\n\nThank you for your patience and understanding.\n\nSincerely,\n[Your Company Name]"	{\n"product": "Credit reporting",\n"issue": "Incorrect information on credit report",\n"generatedCompanyResponse": "Dear Customer,\n\nThank you for reaching out to us and bringing your concerns to our attention. We understand that you are dissatisfied with your credit score and believe there may be inconsistencies in the credit reporting reasons provided.\n\nWe apologize for any confusion or inconvenience this may have caused you. Our team will investigate your credit score and the reasons provided to ensure accuracy and fairness. It is important to us that your credit score reflects your financial circumstances appropriately.\n\nPlease be assured that we take your concerns seriously and will work diligently to address them. We will contact you with an update as soon as our investigation is complete. In the meantime, if you have any further questions or require any additional information, please do not hesitate to contact us.\n\nThank you for your patience and understanding.\n\nSincerely,\n[Your Company Name]"\n}

The idea of this task is to automate both the routing of customer issues and the immediate response to the customer. We will fine-tune LLaMA2-7B to predict the Product and Issue for the incoming customer complaint. The fine-tuned model will also be able to generate a response that specifically addresses the customers complaint, asks for more info if necessary, and prevents further escalation while the human support representative takes time to investigate the issue. Since the fine-tuned LLM will generate this info in a valid JSON format, you use the outputs of this LLM on the backend to drive this automation. Let's dive in and see how we can do all of this!

Dataset 📄

We are going to import this dataset using the Predibase Public Dataset connector.

complaints_dataset = pc.import_public_dataset("consumer_complaints_generation")

Prompt Template ✏️

Now we will construct our prompt template. This is where we tell the LLM exactly what we want it to do. It is important to be very explicit here. We will be injecting the value of the Complaint column into this prompt template for every row as you can see where we have written Complaint: {Complaint}.

fine_tuning_prompt_template = """You are a support agent for a public financial company and a customer has raised a complaint.
    Generate a structured JSON output with the following fields "product", "issue", and "generatedCompanyResponse".

    Here is an example structure:
    {{
      "product": "...",
      "issue": "...",
      "generatedCompanyResponse": "..."
    }}

    The value for "generatedCompanyResponse" should be a polite response to the following complaint.

    ### Complaint: {Complaint}

    ### Structured JSON Output:"""

Model Training 🏁

Now we can kick off our fine-tuning job!

All we need to do is specify the LLM that we will be fine-tuning, then we call llm.finetune() and we're on our way.

You can follow the link in the output to track the model's progress in the UI!

llm = pc.LLM("hf://meta-llama/Llama-2-7b-hf")

fine_tuning_job = llm.finetune(
  template=fine_tuning_prompt_template,
  target="Structured JSON Output",
  dataset=complaints_dataset,
  repo="Customer Support Automation"
)

Deploy and Query 🚀

Once our fine-tuning job has finished, we can deploy and start querying!

LoRAX 🔥

We are going to be serving our model with a tool we developed called LoRAX. LoRAX utilizes a base model like LLaMA2-7B and attaches an adapter on top to hot load our fine-tuned model. The benefit of this approach that we can hot load multiple different fine-tuned models while only paying the serving costs of one base model!

# Run the following command to see the list of LLM deployments we can use for the base model
pc.list_llm_deployments()

We are going to use the value in the deploymentName column after the prefix: pb://deployments/. In this example, we will use the shared llama-2-7b deployment.

# Get base model
base_model = pc.LLM("pb://deployments/llama-2-7b")

# Get fine-tuned model once job finishes
fine_tuned_model = fine_tuning_job.get()

# If you want to serve a model you've already trained, you can get the model using this code pattern
# fine_tuned_model = pc.get_model("Customer Support Automation", version=1)  # Swap out desired model version

# Hot load model with LoRAX
lorax_deployment = base_model.with_adapter(fine_tuned_model)

Sample Test Complaints 🔖

Here are a few complaints from the test set to evaluate our fine-tuned model's performance!

customer_complaint = """
    Equifax was hacked on Thursday. \n\nOn Friday, I checked to see whether or not I have been affected by the hack. Equifax has demanded that
    I wait until XXXX to get protection from a threat of identity theft caused by their mistakes. \n\nNot only this, but if I were to sign up
    for their protection, they demand that I waive the right to litigate against them ( again for their mistakes XXXX. \n\nIt seems to me that
    while Equifax has been hit hard by this crisis, the fact that they are not willing to own up to consumers even remotely and are ill equipped
    to protect them makes Equifax primarily responsible in this case.
"""

customer_complaint = """
    Been receiving multiple calls per day from XXXX, which leaves a prerecorded message to call regarding a debt. It also gives an option to
    speak to a representative. I opted to do this to find out what this is about. The operator asked for a relative, who has never lived with
    us yet our number was the one they had on file. I told them that the person they seek does not live here and asked to be removed from their
    caller list. The operator said he could not do this without a new number being given. He also said they are not required to stop calling.
    \n\nThis is simply harassment by proxy.
"""

Query LoRAX Deployment ❓

Here we will inject the test complaint into the prompt that we send to our fine-tuned LLM deployment.

# Recall that our model was fine-tuned using a template that accepts {Complaint}
# This template is automatically applied when prompting.
result = lorax_deployment.prompt(
    {"Complaint": customer_complaint},
    max_new_tokens=256
)

print(result.response)

Conclusion

You've just completed an end to end tutorial where you fine-tuned a model to automate customer support routing and immediate response. Congratulations!!

We hope you are able to take your learnings and apply them to your own use cases. If you have any questions about how to apply these capabilities to your own data, please reach out to support@predibase.com, and we are happy to assist you!