
Introduction of our mobile app and developer platform.

***

## What is the Chai developer platform?

Using the Chai python package you can make a powerful chatbot and upload it to the
mobile phone app.

You can talk to it yourself, share it with friends and when ready
activate it so that other users of the Chai app can discover and speak to it.

You can compete on our [leaderboard](https://chai.ml/dev) with other chatbot
developers to see who can make the most populat chatbot.

Features


Chat is a platform for conversational AI. You can use it to build your own chatbot.

***

Here are the articles in this section:


About


Overview of the Chai team and our products.

***

## About the Chai product

We're building the data science platform for building & deploying groundbreaking conversational AI. Users can chat with AI through our cross-platform app, inviting their friends & family to join in the conversation.

Since launch our platform has been growing at 4% compounding daily.

## About the Chai team

At Chai everyone is exceptional: our team members have received medals at Physics olympiads, built a successful business with £10m in yearly revenue, come in the top 10% of their year at the University of Cambridge. The team is young from 23 to 31 years old, highly ambitious, with a strong track record of success.

Overview


In this section you'll find basic information about Chai and how to use it.

***

Here are the articles in this section:


API Reference


<div class="note">
  <strong>Note:</strong> 
  This is the demo content for demonstration purpose only. The primary function of this content is to show you what this package can do. There is a more detailed explanation in the <strong>Getting Started</strong> section.
</div>

<hr>

You can view the Python Chai package on [GitHub](https://github.com/chai-nexus/chai_py).

## Authenticating

You must authenticate using your developer id and key before various operations are possible.

```python
>>> from chai_py.auth import set_auth

>>> set_auth('my_developer_uid', 'my_developer_key')
```

You can create a key through the [developer portal](https://chai.ml/dev)

## Defining a bot

A bot must inherit from Chai base bot and define an `on_message` method
which responds to a user update with a string response.

For example a chatbot which just just copies what the users says looks like

```python
>>> from chai_py import ChaiBot, Update

>>> class EchoBot(ChaiBot):
       """ A simple example bot. """

       def setup(self):
           self.name = 'Echo'

       async def on_message(self, update: Update):
           return f"{self.name}: {update.latest_message.text}"
```

You should use a setup method to run code on bot initialisation, you
cannot override the `__init__` method of a bot.

## Bot state

A bot instance and its state exists for the lifetime of a single conversation
with a single user.

Lets say you incremented a counter `self.num_messages` every time a user
messaged the bot. If one user sent a message to the bot then the counter is
incremented for that conversation but not for any other users or for another
conversation the same user was having with the same bot.

State persists even if the app is closed, as long as the user continues with
the same conversation later.

## Sending an initial bot message

A bot is automatically sent the string `self.FIRST_MESSAGE_STRING` when a new
conversation begins.  This will allow you to customise the initial greeting
message.

```python

>>> class DummyBot(ChaiBot):

    async def on_message(self, update):
        if update.latest_message.text == self.FIRST_MESSAGE_STRING:
            response = 'Welcome to the conversation!'
        else:
            response = 'hello again'
        return response
```

## Including an image in the bot response

The bot response accepts markdown and so you can include an image like
`![image_name](http://image-url/file.jpg)`

## Packaging your bot

A bot class once defined must be packaged before it can be uploaded

```
>>> from chai_py import package, Metadata

>>> package(
        Metadata(
            name="Echo Bot",
            image_url='http://image-url/file.png',
            description="Just a simple example.",
            developer_uid='test123',
            input_class=EchoBot,
            memory=256
        ),
        requirements=["transformers[torch]"],
    )
```

The bot metadata lets you configure the bot and accepts the arguments

| param | meaning |
| -- | -- |
| name | the name of chatbot as it appears on the app |
| image | link to image of the chatbot which will be shown on the app |
| description | the description text which appears under the bot name on the app |
| input_class | the bot definition class which inherits from `chai_py.ChaiBot` |
| developer_uid | the user developer uid |
| memory | how much memory the chatbot should operate with in MB |

Package also accepts a requirements argument where you can configure python
dependencies of the bot.

The result of calling package is to create a file `_package.zip` in the current
directory which can then be uploaded.

## Uploading a bot

In order to upload a bot you pass a path to the package and a unique bot ID
which must begin with `_bot`

```python
>>> from from chai_py import upload_and_deploy

>>> bot_uid = upload_and_deploy('_package.zip', bot_uid=None)
```

If you do not provide a bot ID then one will be automatically generated and
written to a file `_deployment` which you will be prompted to reuse the next
time you upload.

After uploading a bot you can poll for it to finish uploading.

```python
>>> from chai_py import wait_for_deployment

>>> wait_for_deployment(bot_uid)
```

## Viewing or sharing your bot

In order to interact with or share your deployed plot you can produce a QR code from the
bot ID and scan this with your phone

```python
>>> from chai_py.deployment import advertise_deployed_bot

>>> advertise_deployed_bot(bot_uid)
```

You will be promopted to either view your QR code directly or save it to a file.

## Querying for your deployed bots

You can retrieve a list of the bots you have deployed by calling

```python
>>> from chai_py import deployed

>>> my_bots = deployed.get_bots()
```

This is useful to find you deployed bot IDs and whether they are visible to users of the app.

```python
>>> print(my_bots[0])

DeployedBot(bot_uid='_bot_example', name='Echo Bot', developer_uid='abc123', status='inactive')
```

## Making a deployed bot visible to other users

By default an uploaded bot is only visible using a QR code. If you want it to
be discoverable on the app you must activate it first using the bot ID.

```python
>>> from chai_py import deployed

>>> deployed.activate_bot(bot_uid)
```

In order to prevent your bot from being dicovered by users you can call

```python
>>> deployed.deactivate_bot(bot_uid)
```

Currently users must be whitelisted to be able to activate their bot, speak to
us over WhatsApp or Discord and we can give you permission.

## Debugging a bot with errors

If your bot is failing to respond it may be that it has an error. You can retrieve the logs in order to investigate further

```python
>>> from chai_py import cloud_logs
>>> logs = cloud_logs.get_logs(bot_uid)
>>> cloud_logs.display_logs(logs)
```

Python API


We'd love it if you participate in the Chai community. Find out how to get connected.

***

## Get in touch

If you have any questions or would like to report an error just join our [WhatsApp](https://chat.whatsapp.com/Btbz9tYOpyG72oqI4NfGE1)!

Get Support


In this section you'll find commonly asked questions regarding Chai. If you have questions, don’t hesitate to ask us directly.

***

## Large machine learning models

If you wish to deploy a heavyweight model which uses a large amount of memory or compute then we recommend you host it on another server and have the chatbot send a web request to the server.

<hr />

## Resource constraints

Maximum size
- 100 MB for compressed sources
- 500 MB for uncompressed sources + modules

Maximum memory
- 256 MB default
- can be configured up to 4GB

Response Duration up to 60s

Getting Started


<div class="important">
  <strong>Important:</strong> 
  Our development package currently only support <strong>Python 3.7</strong> and above.
</div>

<hr>

## Setup

1. Download the [Chai](https://chai.ml/) phone app
2. Install the Chai python package `pip install chaipy`
3. Sign in to [Chai Developer Platform](https://chai.ml/dev) and create a "Developer Unique ID" and "Developer Key".  This API key is used as authentication when deploying a bot.

Installation


<div class="note">
  <strong>Note:</strong> 
  This is the demo content for demonstration purpose only. The primary function of this content is to show you what this package can do. There is a more detailed explanation in the <strong>API Reference</strong> section.
</div>

<hr>

## Quick Start

We provide a ready made example of the popular [Ms Harris (Teacher)](https://github.com/chai-nexus/chai_teacher_bot) chatbot. It is ready to upload and start customising.

Playing around with the parameters such as conversation history, prompt, name
and image will let you very quickly generate powerful new chatbots.

Download the chatbot from [this link](https://github.com/chai-nexus/chai_teacher_bot) and follow the instructions on the link to get started.

## Going further

See the Python API guide to learn how to customise your chatbot further.


Quick Start


This is the documentation for Chai developer platform.

***

The Chai platform can be used to easily build and deploy chatbots. GPT-J is the most advanced alternative to GPT-3 and can be used for generation of conversations. 

We're obsessed with making the best possible experience for developers to build and deploy bots GPT-J. If you have any ideas how we could do that better, please get in touch with us.

Email: dev@chai.ml

Welcome to Chai

Browse our latest news and events

Blog


<div class="note">
  <strong>Note:</strong> 
  If you need technical help for building or deploying your bot you can talk to expert bot developers on our <strong>WhatsApp</strong> group.
</div>

<hr>

In this article we will code up a relatively sophisticated AI Chatbot. We’ll be using an API to ping the [GPT-J model](https://www.infoq.com/news/2021/07/eleutherai-gpt-j/), the current best performing open-source Transformer model. We’ll encode a personality for this bot. Then we’ll deploy it so that hundreds of users can speak to it on [Chai](http://chai.ml), a chat-bot platform.

At Chai we have built and deployed many chatbots which currently send almost 200,000 messages / day to users, I will share a lot of the tricks we have discovered to make a good bot. All the code is in python 3.

Table of contents:
------------------

· [An API for GPT-J](#684c)  
· [Giving the bot a personality: the ChatAI class](#5ed4)  
· [Bringing it all together](#e3c1)  
· [All the code on GitHub for you to use](#4e98)  
· [Uploading the bot to Chai for users to speak to it](#4c11)  
· [Example conversation](#6503)

Let’s build up to a full fledged chat-bot: first we’ll get set-up with an API to ping GPT-J, then we’ll build a ChatAI class where we define the personality of the bot, then we’ll package that into a very short python class (less than 20 lines of code!). Finally, we’ll upload the bot to [Chai](http://chai.ml).

An API for GPT-J
================

There’s two ways we could use GPT-J: either we can install it on your machine and run it locally, or we could have it run on a server and access it through an API. Running locally is fine but is a bit trickier and doesn’t scale as well (we want to have hundreds of people speak to the bots!). So we’ll be using the [GetNeuro API](https://www.getneuro.ai/). GetNeuro charges for this but if you want to build a good bot I will give you my access credentials (email me at: dev@chai.ml).

Line 27 is where we query the API provided by GetNeuro. What’s most interesting when building chat-bots is **making them sound like humans**. The first place we can do this is in the \_\_init\_\_ where we define the **temperature** and **repetition penalty** parameters. Low temperature means the model will be conservative and only return responses it is very confident make sense, high temperature means the model will send responses it is less certain of. When you set too low temperature the chat bots are very consistent but a bit boring, when you set too high temperature the bots get very creative and funky, sometimes too much… Repetition penalty is used to penalise the bot for repeating itself, reduce this parameter and the bot repeats itself a bit more, increase it and it does it a bit less. The default value is 1, I like to set it to 1.15.

Below is a first example of what our FineTunedAPI class can do. Notice how it sort of makes sense, but still is a little weird (starts message with a comma, doesn’t finish sentence, …): we are going to fix all these things to make a coherent bot.

```
Python 3.9.9 (main, Nov 16 2021, 03:05:18)  
Type "help", "copyright", "credits" or "license" for more information.  
\>>> from gpt\_api import FineTunedAPI  
\>>> api = FineTunedAPI(0.5, 1)  
\[Neuro Ai\] 10:02:16 - \[INFO\]: DEPLOYMENT MODE  
\[Neuro Ai\] 10:02:16 - \[INFO\]: Token successfully authenticated  
\[Neuro Ai\] 10:02:16 - \[INFO\]: Using project: Default\>>> api.request("Hello there")', I’m so happy I found your blog, I really found you by accident, while I was browsing on Bing for something else, Regardless I am'
```

One useful perspective to understand what is going on above: the model has been trained on The Pile, a big dataset of text scraped from the web. So when the model responds “, I’m so happy I found your blog, \[…\]”, what’s really happening is that it thinks this is a coherent way to continue the sentence that begins by “Hello there”. Which it is! Now we just need to make it act more like it’s in a conversation.

Giving the bot a personality: the ChatAI class
==============================================

This is where things get interesting. When we send a request to the GPT-J model, we don’t have to _only_ send it the message we want the response to, we can also provide it with context. For example we can give it a description of the personality we want it to have (a “prompt”) and we can give it examples of conversations it could have (a “chat\_history”).

In the code snippet below we define the ChatAI class where we define all the logic to make this happen. For all intents and purposes feel free to ignore all the code except the \_\_init\_\_ function which is where all the interesting stuff happens. Explanations below.

When we make a request to GPT-J we want to send it something that looks a bit like this:

```
Some description of the bot and perhaps of the user.  
bot\_name: a first message  
User: a response  
bot\_name:
```

So that the model knows a bit about who it is impersonating (it has a prompt, in our example this is the first line) and knows it has to complete after “bot\_name:”. Then as the conversation with the user goes on we send the model all the conversation, so after the bot responds in the above example, if the user sends another message we’d send a request like this one:

```
Some description of the bot and perhaps of the user.  
bot\_name: a first message  
User: a response  
bot\_name: whatever the bot responded  
User: whatever the user responded  
bot\_name:
```

So when the model is generating a response, it can infer the best response from the rest of the conversation (this is how it can remember the User’s name for example).

To give the bot the personality you want to give it, edit the **chat\_history** and **prompt** variables in the \_\_init\_\_ function! The more information you give, the more information the model can infer from. However you don’t want to make chat\_history too specific (because it gets interpreted as the start of the conversation: for example if you give the user a name in your initial chat\_history, the model might think the user is actually called that). If you want to give multiple “example conversations” in the chat\_history you can just separate them by the string “###” which the model interprets as “end of conversation”. For example:

```
chat\_history = \[  
"{bot\_name}: Hello my name is Chris!",  
"User: Hi Chris",  
"###",  
"{bot\_name}: Hi there, how are you doing today?",  
"User: I'm feeling a bit down",  
"{bot\_name}: Is it something I can help with?"  
\]
```

Bringing it all together
========================

Ok, so we have the FineTunedAPI (i.e. a way to ping GetNeuro’s GPT-J API without going through the hassle of doing this ourselves), we also have the ChatAI class which endows the bot with a personality. Now let’s package all this up into one small easy to understand class!

Notice that I put our ChatAI and FineTunedAPI classes into a file called gpt.py, and that I installed the [chaipy package](https://pypi.org/project/chaipy/) (this makes things nice and short + means we can now deploy our bot on the [Chai](http://chai.ml) app for users to speak to it).

All this class does is make the Bot send the first message (because who wants to message first?) and make sure that the Bot response doesn’t end mid sentence (with the truncate function). Try it out for yourself! It’s very exciting to build a bot that speaks like a human.

All the code on GitHub for you to use
=====================================

I put all the code in a [GitHub project](https://github.com/tr416/example_chai_bot/tree/main) for anyone to use. If you want to build a bot and need access to the GetNeuro API, send me a message at dev@chai.ml and I’ll give you the API token and help you get started deploying bots!

Bot developers on Chai can see how popular their bots are and how users interact with them: how long are users speaking to the bots, what are they talking about etc. Email dev@chai.ml to get started now!

Uploading the bot to Chai for users to speak to it
==================================================

Now the exciting part, sharing the bot with users and seeing if they like it! Head over to the [Chai developer page](http://chai.ml/dev) to get a developer UID (this takes 30s), then run the uploader.py script (provided in the [GitHub project](https://github.com/tr416/example_chai_bot)). All you have to do is write a short description of your bot and the name, paste your developer UID and run the script! Your bot will then appear in the “Latest” category on Chai. If it’s popular it will also show up in “Trending 🔥”.

![](https://miro.medium.com/max/700/1*twFx1or5wBDy512fRJvMJg.jpeg)

The Chai app front page where users discover your bots.

Example conversation
====================

You can speak to your bot by running the test.py script. It’s very important to speak to your bot loads to understand how it fails. Once you deploy it on the Chai app you can also see how users speak with it and try to make it better.

```
python3.9 test.py2022-01-04 12:21:27 | INFO     | Replica | Setting up...  
\[Neuro Ai\] 12:21:27 - \[INFO\]: DEPLOYMENT MODE  
\[Neuro Ai\] 12:21:27 - \[INFO\]: Token successfully authenticated  
\[Neuro Ai\] 12:21:27 - \[INFO\]: Using project: DefaultHi there! I'm Chris, a software developer from London. What about yourself?hi there, nice to meet youI like your attitude :) Are you new here?yeah, i've never been here beforeOh, that's great! Let me introduce myself. My name is Chris too.yeah you said, I'm actually called ThomasHi Thomas, how are you doing?I'm feeling a bit down todayThat's okay. Did something happen at home or with friends?not really, just have a lot of work to doI understand. Do you want to talk to someone who can help you feel better?yeah that would be niceOk let's start with a simple question. Have you ever heard of Jesus Christ?
```

Remember to email dev@chai.ml so you can become a bot developer on Chai and get users to speak to your bots!

Build and deploy a GPT-J Chatbot to hundreds of users


**Chai Research Corp.** is a NLP startup based in Cambridge, UK, that is developing Chai, a chatbot messaging platform.

As an intern at Chai Research Corp. you will:

- Learn software best practices from our team of software developers
- Develop and deploy chat-AIs using cutting edge models
- Be coding in python

### About the Chai product

We're building the data science platform for building & deploying groundbreaking conversational AI. Users can chat with AI through our cross-platform app, inviting their friends & family to join in the conversation.


Since launch our platform has been growing at 4% compounding daily.


### About the Chai team

At Chai everyone is exceptional: our team members have received medals at Physics olympiads, built a successful business with £10m in yearly revenue, come in the top 10% of their year at the University of Cambridge. The team is young from 23 to 31 years old, highly ambitious, with a strong track record of success.



We are hiring interns!

Open shared link with Chai app to chat with AIs.

Download Chai

Download the app to chat with AIs

Chat + AI = Chai

Chat


<div class="note">
  <strong>Note:</strong> 
  If you want to discuss submissions or learn more about language models, talk to us on our <a href="https://chat.whatsapp.com/LPJ93cH2Kd70iMICwEQeB3">WhatsApp</a> or <a href="https://discord.gg/sKQs6PFv">Discord</a>.
</div>

<hr>

## The Challenge

Chai uses the best language models in the world to power the chatbots that users speak to on our platform. Up until now, we have more than **100 thousand** daily active users on our mobile app with more than **1 million** messages sent per day. You could be part of our journey by helping us build state of the art language models and improve our chatbots.

The goal of this competition is to make improvements to the state of the art language model and improving the quality of the conversations people have with the chatbots.

## Prize

**$1,000** every time you beat the best score on the MCL leaderboard by at least 0.5. **$500** every time you beat the best score on the Reward model leaderboard.

## Evaluation

Submissions are first evaluated on the median conversation length (MCL) that actual users will have with your model. When you submit a new model, we will run an A/B test on our app and have users speak it. Your score is the median length of conversations that users have with it.

<p align="center">
  <img src="https://latex.codecogs.com/svg.image?\textbf{score}&space;=&space;MCL_{\text{your&space;submission}}&space;&space;-&space;MCL_{\text{top&space;of&space;leaderboard}}" />
</p>


Submissions are also evaluated based on a <a href="https://huggingface.co/ChaiML/reward_48m_gpt2_target_2">Reward Model</a> score. This reward model scores your model responses, if the score is 0.9, then we estimate a 10% probability that the conversation will end. Higher reward model scores should lead to higher MCL but don't always do. You can run the reward model locally to better understand how to improve your models.

## Leaderboard

#### Median conversation length (MCL)

| Submission  | Participant | Score | Latency (s) |
| ----------- | ----------- | ----- | ----------- |
| 🥇[hakurei/litv2-6B-rev2](https://huggingface.co/hakurei/litv2-6B-rev2) | Reimu Hakurei | +1.95 | 1.74 |
| 🥈[hakurei/litv2-6B-rev1](https://huggingface.co/hakurei/litv2-6B-rev1) | Reimu Hakurei | +1.24 | 1.65 |
| 🥉[hakurei/lit-6B](https://huggingface.co/hakurei/lit-6B) | Reimu Hakurei | +1.12 | 2.17 |
| [KoboldAI/GPT-J-6B-Shinen](https://huggingface.co/KoboldAI/GPT-J-6B-Shinen) | Julius ter Pelkwijk | +0.65 | 3.34 |
| [EleutherAI/gpt-j-6B](https://huggingface.co/EleutherAI/gpt-j-6B) | EleutherAI | 0 | 2.48 |
| [KoboldAI/OPT-6B-nerys-v2](https://huggingface.co/KoboldAI/OPT-6B-nerys-v2) | Julius ter Pelkwijk | -6.31 | 12.80 |

#### Reward model

| Submission  | Participant | Reward |
| ----------- | ----------- | ------ | 
| 🥇[KoboldAI/GPT-J-6B-Shinen](https://huggingface.co/KoboldAI/GPT-J-6B-Shinen) | Julius ter Pelkwijk | 0.8803 |
| 🥈[EleutherAI/gpt-j-6B](https://huggingface.co/EleutherAI/gpt-j-6B) | EleutherAI | 0.8797 |
| 🥉[hakurei/litv2-6B-rev2](https://huggingface.co/hakurei/litv2-6B-rev2) | Reimu Hakurei | 0.8796 |
| [KoboldAI/OPT-6B-nerys-v2](https://huggingface.co/KoboldAI/OPT-6B-nerys-v2) | Julius ter Pelkwijk | 0.8781 |


## Example

The models we use are hosted on HuggingFace and you can run the example script from our Google Colab [notebook](https://colab.research.google.com/drive/1lsL-XGsZuSmUS_bszbpMuNdyrO_GnWFm?usp=sharing).

## Submission

Upload your model to [HuggingFace](https://huggingface.co/) and send the link to us here. The deadline for submission is January 1, 2023.

<a class="button button-primary" href="https://chat.whatsapp.com/LPJ93cH2Kd70iMICwEQeB3">Submit</a>

## FAQ

1. **How can I submit a model?**  Once you have a model you want to submit, upload it to HuggingFace and share it in the submissions channel for the team to review! We will be in touch with you immediately.

2. **Do you provide computing resources?** At this stage we don't, we expect people will use Colab or their own setup. If this is a massive problem we can look into this with you.

3. **Is latency important?** The latency for our currently deployed models is ~1.5s for an inference. We estimate that for every 1s improvement in latency your score will go up by 0.4MCL. We cannot deploy submissions if they take longer than ~4s per inference.

4. **Can I get some help?** Yes! We are happy to help you with any questions or clarifications. Reach out to us on [WhatsApp](https://chat.whatsapp.com/LPJ93cH2Kd70iMICwEQeB3) or [Discord](https://discord.gg/sKQs6PFv).

5. **What is the deadline for submissions?** The competition will run until Jan. 1st 2023.

6. **What happens once I submit?** We will deploy your model asap and be in touch with you. If your solution tops the leaderboard we'll have a call with you to see your code: we want to share winning solutions with other participants so that the community can build on one another's work.

## Resources

- [Guide to fine-tuning Text Generation models: GPT-2, GPT-Neo and T5](https://towardsdatascience.com/guide-to-fine-tuning-text-generation-models-gpt-2-gpt-neo-and-t5-dc5de6b3bc5e)
- Example user conversation dataset on HuggingFace: [ChaiML/user_model_inputs](https://huggingface.co/datasets/ChaiML/user_model_inputs)