Teaching an LLM to talk like my Discord Group Chat
it didn't go as expected
tl;dr: I trained an LLM on +100,000 Discord server messages and created an AI version of our Discord group chat. Months of work later, it works slightly better than I expected.
Intro
On February 27th, 2020, a Discord server that is still used to this day was made by our friend Thomas. It was senior year of high school, and because of a series of events that I really don’t care enough to explain, my friend group from high school made a brand new server spun off of the old server. I think at that point our AP Gov class was on the the UK unit, and for some reason we thought it would incredibly funny if the server was themed around UK government, so #general was #question-time, and we had a Queen, a Prime Minister, Lords, a Supreme Court and one Constituent (Henrry). Our Prime Minister even wrote up a lengthy constitution for no other reason than that it was probably really funny then. This bit has certainly not aged at all.
As you can guess by the date, this server ended up becoming the lifeline to each other through the pandemic shutting down high school early throughout 2020 and all the way throughout college until now as we’re all starting to be actual adults, which is crazy to think about. That server, for better or for worse, has been a tenant in my post-high-school life, and there've been so many fond memories associated with that server; most of them incredibly stupid and esoteric bits that I could not even begin to explain to a normal person with a normal brain.
In April of 2020, with absolutely nothing to do because of the pandemic, inspired by both Deep Leffen and the messages that our friend Henrry would say, I fine-tuned a really small GPT-2 model on just Henrry’s messages and deployed it as a Discord bot that would post messages every 30 minutes in its own channel. We called this: Henrybot. Henrybot was really really bad, but that somehow made it really funny in those early days before the public got tricked into thinking that LLMs were hyper-intelligent with chatGPT.
Around April of last year, 2023, I stumbled across this project done by Izzy Miller off of a Verge article. I urge you to stop what you’re doing right now and read his really good blog post. There’ll be some parts during this where I just refer you to what he wrote.
I have spent almost ¾ of the past year of my life obsessing over this project on and off every couple of weeks, getting discouraged and coming back each and every time. This was the logical next step after Henrybot: An LLM trained on the entire Discord Server, to not only replicate Henrry, but everyone else too.
In trying to explain or show this to friends, it’s not immediately crystal clear what the finished project ACTUALLY is, so here is a TL;DR as succinct as I can get it: It’s a chat interface where you can pick which of the group chat members you want to send a message as. You can type and send a message, and we use an LLM trained on the group chat history to generate a conversation following that message, about 14 or so messages, with messages sent by any one of the group chat members. Most of what I did is based off or almost exactly taken from Izzy’s project, so I’ll try to explain what I did on top of that.
Data
First, I had to scrape the messages sent from our Discord server. You could probably do this any number of ways, but my mission was to get as close as possible to a format Izzy used to prepare his data, so I could be lazy and not have to do that much work. You could do this any number of ways if you don’t want to be lazy.
To start, I used DiscordChatExporter to export the chat history of my Discord Server’s text channel as a .csv file. From there, I trimmed and formatted the .csv with the following python code:
import pandas as pd
import re
import csv
df = pd.read_csv('messages.csv')
bruh = df.iloc()[326].Date[:26] + df.iloc()[326].Date[27:]
bruh2 = df.Date.str.endswith
# Function to remove character at specific index
def remove_char_at_index(text, idx):
return text[:idx] + text[idx+1:]
# Index of the character to remove
idx = 26
regex = r"(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’]))"
df_new = df.copy()
df_new['Content'] = df['Content'].str.replace(regex, '', case=False, regex=True)
df_new = df_new[df_new.Content != ""]
df_new = df_new.dropna(subset="Content")
df_new['Date'] = df_new['Date'].apply(lambda x: remove_char_at_index(x, idx))
df_new.to_csv('trim_messages.csv', index=False)Then, I used DB Browser for SQLite to convert my .csv into a .db file. (It has been so long since I did this.)
From there you can use the Hex Project and code from Izzy’s blog post to get from a .db file to a .json file of sessioned messages, or conversations, in an almost train-ready format.
We need to make an important detour here before talking about training, though. I say ALMOST train-ready because at this point, with the data I had, I must have attempted training so many times and ended up with poorly functioning models that would only give me one-word responses that just could not result in conversations. There were also some changes I had to make at inference-time, but I still seriously urge you to clean your data, especially if you think the conversations you and your friends have are really really really stupid.
(Substack formats Github Gists really weird, click the link to see the full code)
| import pandas as pd | |
| import re | |
| import json | |
| sessionized = pd.read_csv('sessionized_messages.csv') | |
| def contains_at(text): | |
| pattern = r'^@[\w]+\s*( @[\w]+\s*)*$' | |
| return bool(re.match(pattern, text)) | |
| def contains_multiple_at_words(text): | |
| words = text.split() | |
| count = 0 | |
| for word in words: | |
| if word.startswith("@"): | |
| count += 1 | |
| if count >= 2: | |
| return True | |
| return False | |
| sess_dict = sessionized.to_dict('records') | |
| items = [] | |
| counter = 0 | |
| for row in sess_dict: | |
| context = [] | |
| cstring = '' | |
| for i in range(10,0,-1): | |
| try: | |
| if sess_dict[counter-i]['session_id'] == row['session_id']: | |
| if sess_dict[counter-i]['session_id'] == row['session_id']: | |
| msg = f"{sess_dict[counter-i]['sender']}: {sess_dict[counter-i]['text']}" | |
| a_msg = msg.split(":")[1] | |
| a_msg = re.sub(r"(\\n|\\r|\\t|\\)", "", a_msg).strip() | |
| if len(a_msg.split()) > 2 and not contains_at(a_msg) and not contains_multiple_at_words(a_msg) and not (msg.split(":")[0] == "nan"): | |
| if len(context) > 0: | |
| cstring += '\n' | |
| context.append(msg) | |
| cstring += msg | |
| except: | |
| # my redacted data doesn't work here | |
| print('too little data =(') | |
| if len(context) < 2: | |
| for i in range(5,0,-1): | |
| msg = f"{sess_dict[counter-i]['sender']}: {sess_dict[counter-i]['text']}" | |
| if sess_dict[counter-i]['session_id'] == row['session_id']: | |
| a_msg = msg.split(":")[1] | |
| a_msg = re.sub(r"(\\n|\\r|\\t|\\)", "", a_msg).strip() | |
| if len(a_msg.split()) > 2 and not contains_at(a_msg) and not contains_multiple_at_words(a_msg) and not (msg.split(":")[0] == "nan"): | |
| context.append(msg) | |
| cstring += '\n' | |
| cstring += msg | |
| items.append(cstring) | |
| counter+= 1 | |
| d = pd.DataFrame.from_dict(sess_dict) | |
| d['context'] = items | |
| final = d.to_dict('records') | |
| group_chat_context = "You all went to high school together. You talk to each other with no filter, and are encouraged to curse, say amusingly inappropriate things, or be extremely rude. Everything is in good fun, so remember to joke and laugh, and be funny." | |
| prompts = [] | |
| senders = ['Alfred','Pranav','Henrry','Slam','Thomas','Daniel'] | |
| for row in final: | |
| sender = row['sender'] | |
| others = list(filter(lambda x: x!=sender,senders)) | |
| prompt = {'instruction': f'Your name is {sender}. You are in a group chat with 5 of your best friends: {", ".join(others)}. {group_chat_context}. You will be presented with the most recent messages in the group chat. Write a response to the conversation as {sender}.','input':row['context'],'output':row['text'] } | |
| prompts.append(prompt) | |
| import json | |
| with open("mydata.json", "w") as final: | |
| json.dump(prompts, final) |
You can see here I updated the last bit of Izzy’s code to remove messages do a couple things:
Remove all one or two word responses.
Remove messages that are only @ing people, including ones that @ multiple people with text following.
Remove messages from non-existent users, which start with “nan: ”
The reasons for some of these are obvious, but I was stuck with a model that could only generate one to two-word responses or responses that @ed other users only, so I tried doing some basic data analysis. Apparently, about 40% of the messages sent consisted of one-word or two-word responses, like “okay” or “lol” or “lmao” or even “@pranav smash?”. I didn’t test it, but variations of that last one probably take up an insane majority of the dataset. There could be a more graceful way of handling this, but I wanted wordy responses that had a higher likelihood of being funny, so I got rid of all of these messages. Because of this, I urge you to ponder if you need to do the same, especially if you're getting your data from a Discord server. If I had done this from the start, it probably would have saved me hundreds of dollars.
Once you’ve done that, you need to convert your .json data file to a .jsonl file. I used this jq command, which I ran in an Ubuntu terminal:
jq -c '.[]' mydata.json > mydata.jsonlWith that, you should be ready to prepare your wallet and train.
Training
I was put onto Modal by Izzy’s original post. Modal lets you run and deploy code and pay for how much GPU compute you use. It is insanely simple to use. To put it way too simply, you can just write normal training or other GPU-using Python code as normal, add some Modal headers to your methods, and deploy it in one line. Izzy’s original post didn’t include this, but I’ve adapted a Modal LLM finetuning example for finetuning Llama-2 on the .jsonl dataset created.
My clone of Modal’s finetuning example can be found here.
All you have to do run training is:
modal run --detach src.train::mainYou could probably adapt this to use a different or possibly better model, like Llama-3, or Mistral, but I was just being lazy and didn’t get it working within 15 minutes.\
Deployment
Once you’ve trained that and have your model file, it's ready to use in a web app. I heavily modified Izzy’s original code to deploy the model for usage to both work with the current (at the time!) version of Modal syntax and be usable for the eventual frontend.
(why is substack so bad at this, click the link to see the full code)
| from modal import App, enter, Volume, Image, gpu, web_endpoint, Secret, method | |
| import os | |
| CHAT_ID = 'chat_beta_1' | |
| volume = Volume.from_name("uk-runs-vol") | |
| #create a modal app to handle config for functions | |
| app = App( | |
| "uk-app", | |
| image=Image.debian_slim().pip_install("numpy", | |
| "rouge-score", | |
| "fire", | |
| "torch", | |
| "sentencepiece", | |
| "firebase-admin", | |
| "tokenizers").apt_install('git').run_commands('pip install git+https://github.com/huggingface/transformers') | |
| ) | |
| @app.cls( | |
| gpu=gpu.A10G(count=2), | |
| volumes={"/runs": volume}, | |
| container_idle_timeout=1200, | |
| timeout=800, | |
| concurrency_limit=1) | |
| class MessagePrediction: | |
| @enter() | |
| def run_on_startup(self): | |
| import transformers | |
| import firebase_admin | |
| from firebase_admin import credentials | |
| from firebase_admin import firestore | |
| import json | |
| model_path = "/runs/axo-2024-07-23-18-06-34-7757/lora-out" | |
| tokenizer_path = "/runs/axo-2024-07-23-18-06-34-7757/lora-out" | |
| service_account_info = json.load(open('/runs/uk-firebase-svc-acc.json')) | |
| cred = credentials.Certificate(service_account_info) | |
| app = firebase_admin.initialize_app(cred) | |
| # Create a Firestore client | |
| self.db = firestore.client() | |
| m_inter = transformers.LlamaForCausalLM.from_pretrained(model_path) | |
| self.tokenizer = transformers.AutoTokenizer.from_pretrained(tokenizer_path) | |
| self.tokenizer.pad_token = self.tokenizer.eos_token | |
| #print("Currently used pad token", self.tokenizer.pad_token) | |
| #self.tokenizer.pad_token = "[PAD]" | |
| self.tokenizer.model_max_length = 1024 | |
| m_inter = m_inter.half() | |
| self.model = m_inter.to("cuda") | |
| print("startup") | |
| @method() | |
| def create_conversation(self,init_context: str,wake: bool): | |
| import random | |
| import traceback | |
| print("phase 1") | |
| if wake: # just a way to wake up this function! | |
| return | |
| print("phase 2") | |
| ctx = '' | |
| # conditionally get 'background' context on chat if desired, helpful to keep conversations going across multiple prompts. | |
| background = self.get_firestore_context.remote() | |
| if len(background) > 0: | |
| ctx = background + '\n' + init_context | |
| else: | |
| ctx = init_context | |
| print(ctx) | |
| counter = 0 | |
| backup_counter = 0 | |
| if len(init_context) > 0: | |
| most_recent_sender = init_context.split(":")[0] | |
| most_recent_message = init_context.split(":")[1] | |
| else: | |
| most_recent_sender = ctx.split(":")[0] | |
| most_recent_message = ctx.split(":")[1] | |
| print('Example ctx:', repr(ctx)) | |
| print('most recent sender: ',most_recent_sender) | |
| # quick and dirty loop to generate an entire conversation. These probabilities are based off the actual distribution of messages in the chat archive. | |
| while counter <= 12 and backup_counter <= 40: | |
| try: | |
| backup_counter += 1 #prevent infinite loops due to reaction chains | |
| characters = ['Alfred','Pranav','Slam','Henrry','Thomas','Daniel'] | |
| character_probabilities = [0.4,0.5,0.3,0.1,0.5,0.5] | |
| most_recent_index = characters.index(most_recent_sender) | |
| if counter == 0: | |
| character_probabilities[most_recent_index] = 0 | |
| else: | |
| character_probabilities[most_recent_index] += .2 | |
| most_recent_referenced = '' | |
| if 'alfred' in most_recent_message or 'Alfred' in most_recent_message or '@alfred' in most_recent_message: | |
| most_recent_referenced = 'Alfred' | |
| elif 'pranav' in most_recent_message or 'Pranav' in most_recent_message or '@karepan' in most_recent_message: | |
| most_recent_referenced = 'Pranav' | |
| elif 'slam' in most_recent_message or 'Slam' in most_recent_message or 'Slam' in most_recent_message or 'Firenova21' in most_recent_message: | |
| most_recent_referenced = 'Slam' | |
| elif 'henrry' in most_recent_message or 'Henrry' in most_recent_message or 'henry' in most_recent_message or 'Henry' in most_recent_message or '@KARIZZMA' in most_recent_message: | |
| most_recent_referenced = 'Henrry' | |
| elif 'thomas' in most_recent_message or 'Thomas' in most_recent_message or '@Shadow Biden' in most_recent_message: | |
| most_recent_referenced = 'Thomas' | |
| elif 'daniel' in most_recent_message or 'Daniel' in most_recent_message or '@Sandhooah' in most_recent_message: | |
| most_recent_referenced = 'Daniel' | |
| if len(most_recent_referenced) > 0: | |
| referenced_index = characters.index(most_recent_referenced) | |
| character_probabilities[referenced_index] += .7 | |
| character = random.choices(characters,character_probabilities)[0] | |
| print('Making prediction as: ',character) | |
| res = self.predict.remote(context=ctx,character=character) | |
| print('JUST DECODED TEXT IS : ',res) | |
| print('END OF DECODED TEXT') | |
| temp = '' | |
| for i in res.split("###")[-2:]: | |
| temp += i | |
| if len(temp.split("Response:")) < 2: | |
| print(temp) | |
| print('split: ',temp.split("Response:")) | |
| print('no completion generated, skipping') | |
| continue | |
| temp = temp.split("Response:")[1] | |
| temp = temp.replace("</s>","") | |
| if u'\uFFFC' in temp: #this is the character used to represent images in the model, unnecessary if you cleaned them out prior. | |
| continue | |
| if 'https://' in temp: | |
| print('just link, not incrementing counter') | |
| continue | |
| if 'Loved' in temp or 'Laughed' in temp or 'Disliked' in temp or 'Emphasized' in temp or 'Liked' in temp or 'Pinned a message.' in temp: | |
| print('suppressing reaction') | |
| continue | |
| if self.checkat.remote(temp.split(' ')): | |
| print('suppressing @mention') | |
| continue | |
| m = self.dispatch_msg_to_firestore.remote(temp,character) | |
| text = f"{character}: {temp}" | |
| ctx += '\n' | |
| ctx += text | |
| ctx = ctx.replace(' "Input:\n','') | |
| counter+= 1 | |
| except Exception: | |
| print('failed message',traceback.format_exc()) | |
| break | |
| return ctx | |
| #exists only to wake the container | |
| @method() | |
| def wake(): | |
| print('waking up') | |
| #checks if all words in sentence start with "@", which is a good proxy for checking if the sentence is a list of @mentions | |
| @method() | |
| def checkat(self, words): | |
| for i in words: | |
| if len(i) > 0: | |
| if i[0] != '@': | |
| return False | |
| return True | |
| @method() | |
| def predict(self, context: str,character: str,temp=.8): | |
| #unnecessary old code from alpaca, just haven't cleaned it yet. | |
| PROMPT_DICT = { | |
| "prompt_input": ( | |
| "Below is an instruction that describes a task, paired with an input that provides further context. " | |
| "Write a response that appropriately completes the request.\n\n" | |
| "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:" | |
| ), | |
| "prompt_no_input": ( | |
| "Below is an instruction that describes a task. " | |
| "Write a response that appropriately completes the request.\n\n" | |
| "### Instruction:\n{instruction}\n\n### Response:" | |
| ), | |
| } | |
| PROMPT = PROMPT_DICT['prompt_input'] | |
| characters = ['Alfred','Pranav','Slam','Henrry','Thomas','Daniel'] | |
| others = list(filter(lambda x: x!=character,characters)) | |
| prompt = f"Your name is {character}. You are in a group chat with 5 of your best friends: {', '.join(others)}. You all went to high school together, and talk to each other with no filter. You are encouraged to curse, say generally inappropriate and hilarious things, or be very rude. Everything is in good fun, so remember to joke and laugh, and be funny, but also answer all questions clearly. You will be presented with the most recent messages in the group chat. Write a response to the conversation as {character}." | |
| format_prompt = PROMPT.format_map({'instruction': prompt,'input':context}) | |
| tokenized_text = self.tokenizer(format_prompt, return_tensors="pt", padding="longest", max_length=self.tokenizer.model_max_length, truncation=True) | |
| full_completion = self.model.generate(inputs=tokenized_text["input_ids"].to("cuda"), | |
| attention_mask=tokenized_text["attention_mask"].to("cuda"), | |
| temperature=.75, | |
| top_p=0.85, | |
| top_k=60, | |
| do_sample=True, | |
| num_beams=3, | |
| max_new_tokens=600, | |
| eos_token_id=self.tokenizer.eos_token_id, | |
| pad_token_id=self.tokenizer.pad_token_id, | |
| repetition_penalty=1, | |
| no_repeat_ngram_size=2) | |
| decoded_text = self.tokenizer.decode(full_completion[0]) | |
| return decoded_text | |
| @method() | |
| def dispatch_msg_to_firestore(self,message,sender): | |
| from datetime import datetime,timezone | |
| import time | |
| # I delay to make the conversation more realistic on the front-end. Could save a ton of money probably by doing this delay on the frontend instead! | |
| #time.sleep(0.25) | |
| senders = { | |
| 'Alfred': { | |
| 'uid': 'fake-alfred', | |
| 'photo': 'https://i.imgur.com/a3mDeHY.png', | |
| 'email': 'fake@email.com', | |
| 'displayName': 'Alfred' | |
| }, | |
| 'Pranav': { | |
| 'uid': 'fake-pranav', | |
| 'photo': 'https://i.imgur.com/K5mKyGs.png', | |
| 'email': 'fake@email.com', | |
| 'displayName': 'Pranav' | |
| }, | |
| 'Slam': { | |
| 'uid': 'fake-slam', | |
| 'photo': 'https://i.imgur.com/03iMQYQ.png', | |
| 'email': 'fake@email.com', | |
| 'displayName': 'Slam' | |
| }, | |
| 'Henrry': { | |
| 'uid': 'fake-henrry', | |
| 'photo': 'https://i.imgur.com/IKkT93y.png', | |
| 'email': 'fake@email.com', | |
| 'displayName': 'Henrry' | |
| }, | |
| 'Thomas': { | |
| 'uid': 'fake-thomas', | |
| 'photo': 'https://i.imgur.com/x1jZE5c.png', | |
| 'email': 'fake@email.com', | |
| 'displayName': 'Thomas' | |
| }, | |
| 'Daniel': { | |
| 'uid': 'fake-daniel', | |
| 'photo': 'https://i.imgur.com/eTCgPrm.png', | |
| 'email': 'fake@email.com', | |
| 'displayName': 'Daniel' | |
| }, | |
| 'Rythm': { | |
| 'uid': 'rythmbot', | |
| 'photo': 'https://i.imgur.com/HX3sVQE.png', | |
| 'email': 'fake@email.com', | |
| 'displayName': 'Rythm Bot' | |
| } | |
| } | |
| sender = senders[sender] | |
| chat_doc_ref = self.db.collection('chats').document(CHAT_ID) | |
| chat_messages_ref = chat_doc_ref.collection('messages') | |
| create_time, doc_ref = chat_messages_ref.add({ | |
| 'timestamp': datetime.now(timezone.utc), | |
| 'message': message, | |
| 'uid': sender['uid'], | |
| 'photo': sender['photo'], | |
| 'email': sender['email'], | |
| 'displayName': sender['displayName'], | |
| }) | |
| return create_time | |
| @method() | |
| def get_firestore_context(self): | |
| from firebase_admin import firestore | |
| from datetime import datetime, timedelta,timezone | |
| chat_doc_ref = self.db.collection('chats').document(CHAT_ID) | |
| chat_messages_ref = chat_doc_ref.collection('messages') | |
| try: | |
| most_recent_message = chat_messages_ref.order_by('timestamp', direction=firestore.Query.DESCENDING).limit(1).get()[0] | |
| message_timestamp = most_recent_message.get('timestamp') | |
| current_time = datetime.now(timezone.utc) | |
| time_diff = current_time - message_timestamp | |
| if time_diff <= timedelta(minutes=4): | |
| messages = chat_messages_ref.order_by('timestamp', direction=firestore.Query.DESCENDING).limit(10).get() | |
| ctx = '' | |
| prev = '' | |
| for i in messages: | |
| raw = i.to_dict() | |
| if prev == raw['message']: | |
| return '' | |
| if raw['displayName'] == 'Rythm Bot': | |
| continue | |
| msg = f"{raw['displayName']} : {raw['message']}" | |
| ctx += msg | |
| ctx += '\n' | |
| prev = raw['message'] | |
| return ctx | |
| else: | |
| return '' | |
| except Exception: | |
| print('no previous message') | |
| return '' | |
| @method() | |
| def confirm_completion(self): | |
| from firebase_admin import firestore | |
| from datetime import datetime, timedelta,timezone | |
| chat_doc_ref = self.db.collection('chats').document(CHAT_ID) | |
| chat_messages_ref = chat_doc_ref.collection('messages') | |
| try: | |
| most_recent_message = chat_messages_ref.order_by('timestamp', direction=firestore.Query.DESCENDING).limit(1).get()[0] | |
| message_timestamp = most_recent_message.get('timestamp') | |
| message_user = most_recent_message.get('uid') | |
| current_time = datetime.now(timezone.utc) | |
| time_diff = current_time - message_timestamp | |
| if time_diff >= timedelta(minutes=3) and message_user != 'rythmbot': | |
| m = self.dispatch_msg_to_firestore.remote("You can send a new message now. Unknown if the conversation completed successfully or timed out since you refreshed the page.", "Rythm") | |
| except Exception: | |
| print('no previous message') | |
| return False | |
| # just for testing | |
| @app.function( | |
| container_idle_timeout=500, | |
| timeout=300 | |
| ) | |
| @web_endpoint() | |
| def get_completion(context: str): | |
| from fastapi.responses import HTMLResponse | |
| convo = MessagePrediction().create_conversation.remote(init_context=context, wake=False) | |
| to_render = convo.replace("\n", "<br />") | |
| return HTMLResponse(to_render) | |
| @app.function() | |
| @web_endpoint(label="alive") | |
| def check_alive(): | |
| print('Checking status of GPU container') | |
| status = MessagePrediction().create_conversation.get_current_stats() | |
| MessagePrediction().confirm_completion.remote() | |
| return status | |
| @app.function() | |
| @web_endpoint(label="wake") | |
| def wake(): | |
| MessagePrediction().create_conversation.spawn(init_context='wake', wake=True) | |
| print('waking up container') |
All you have to do to deploy this is:
modal deploy main.pyThere’s a good amount of changes I made to this code, but a really big change I made to actually use the model well was adding no_repeat_ngram_size=2 as an argument during inference. Llama-2 seems to struggle a lot with repetition, and without changing parameters much, conversations that the model generated would end up just being different people @ing each other. It was really frustrating and resulted in uninteresting conversations with repeated messages, even after trimming out the one-word and uninteresting responses. What seemed to work for me was changing no_repeat_ngram_size, though from what I understand, this limits the amount of usable tokens and isn’t a perfect solution. It seems like DRY is an actual solution to this issue, but I couldn’t get it to work in time.
With this, you can easily test and generate sample conversations just to make sure everything works. If everything seems right, you can go ahead and use your model from a frontend.
Here is a link to my Discord-esque frontend.
This is mostly just the frontend Izzy used, but with some small changes, including a login selector, and a Discord theme. The biggest change I added was support for Discord emotes. If your model is trained off of Discord messages and your friends use emotes enough, your model will too, and you can render those emotes alongside the text, by adding them to the emote list dictionary in Message.jsx.
That should be it! (If you ignore all the struggling this took me to actually get working)
Aftermath
I tried to lay out the steps as straightforward as possible, but in reality, I went back and forth between training, testing inference, and repeating, over and over again, until I got somewhere I was satisfied with. I started this project late November 2023 and got something working in July 2024, so several months on and off were spent on this. Which is really fun, since for the most part my friends weren’t interested in trying this even once! I have great friends.
except for slam thanks for beta testing this my goat
This project to me is an example of one of the only good use cases of LLMs: really stupid projects that don’t matter and have the potential to be funny like Infinite Craft.
and maybe some other things maybe perhaps i guess but thats mostly it
i think
I was worried that I wouldn’t get something usable, but I’m actually so happy with the quality of responses I get. I wasn’t looking for something insanely accurate to how any of us would respond in a given situation, all I cared about was that I would, more often than not, get unhinged screenshot-worthy funny moments. By that metric, I think I succeeded?
All of the messages sent in our server are from that covid summer in 2020 when we had nothing to do. Some of us were on Warzone every day, or we were getting lunch or playing tennis, or just talking on call in Discord, or playing Mario Party or Jackbox or Smash, or whatever. Because most of the messages are from that time, the model seems to portray the characters that represent us as, for the most part, being stuck in that time where we had barely any responsibilities. It couldn’t be any more different to now where we’re all out of college, in different parts of the country or getting actual jobs. In some way, generating conversations in the 2-day honeymoon period after I finished this project brought me back to that time.
and then I read something dumb the model spat out that wasn’t funny and remember I can just reread our old messages and remember how much funnier we used to be (for the most part i think we still thought jojo memes were funny in 2020 there are exceptions to this let’s not get it twisted)
Please feel free to follow along with these steps and make your own LLM Discord group chat simulator. If you have any questions, email me at alfredrpk at gmail.com, and I’ll try to help you out as best I can.
Thaanks for readinggggg