File: Fw4o4yyaUAEUY2G.jpg (284 KB, 1750x2000)

284 KB JPG

>>93665390

File: d34.jpg (204 KB, 956x560)

204 KB JPG

>>93665497
EY MOVE YO GENERIC BITCH ASS ON, HOE

Is there some kind of syntax I'm supposed to follow when writing a prompt? I want to put in unbreakable rules for the AI to follow when generating anything.

you know, this onions man explains my current dilemma pretty well. i'm in the market for an RTX 4000 SFF ada card, but the MSRP is $1250 when it should be $1000, yet it's impossible to find for less than $1450. that's almost a $500 premium or a 1/4 oz of gold for some e-waste
https://www.youtube.com/watch?v=kIaf6o4kFYM

>>93665731
CPUchads can't stop winning because cheap RAM will never be hard to find.

>>93665752
should i just run my shit on CPU and wait for GPU prices to convert to christianity? i have 44 xeon cores and 256 GB DDR4 at my disposal so i'm not exactly starving for specs. i could easily just throw 16 threads and 64 GB RAM at a model

>>93665770
Yes. Also, 3090s are 700 each all day long.

>>93665770
I dunno man, you might have to wait more than 2 minutes per prompt, at that point it's just not worth using AI at all, right?

>>93665796
those cards are huge and power hungry though. i'm not a caselet but i like to keep things tidy. maybe i'll just go with the original plan and develop my galactica API on CPU's while i wait for enough startups to get memed into buying RTX 4000 SFF's new, and when they inevitably fail and offload shit on ebay

it's not the newness i'd buy buy the form factor. if a good mini GPU came out 5 years ago, it completely wouldn't matter. the fact that it came out like 2 months ago just pisses me off

>>93665770
GPUs have thousands of cores. Even shitty thrift store GPUs

throwing 40 xeon cores at it would be pathetic and outperformed by a gtx1650. not just outperformed. fucking annihilated

>>93665811
the plan is to run a remote cron job once a day that queries an AI-powered API to schedule jobs, and checks in with a jobId once an hour to write the results, if completed, to a database. if i wanted to discuss politically correct topics with a robot in real time, i'd just use jewPT or bard

>OP pic
I lol'ed

>>93665904
And what models exactly would actually fit on that piece of shit?

>>93665904
yeah i'm just not keen to double++ my server's max TDP and shove some bullshit in there to "have a GPU," nor do i want to deal with the added complexity and cost of configuring GPU passthrough in a virtualization environment while i'm developing the API to actually use the model productively. i can develop on CPU/mini models and buy a nice GPU later

>>93665972
sure run on CPU at 1% of the speed of a 1650
enjoy

File: FuESVOsaQAA-1nY.jpg (288 KB, 1245x2048)

288 KB JPG

new twitter HYPE
https://arxiv.org/abs/2305.15348
READ: Recurrent Adaptation of Large Transformers
Through comprehensive empirical evaluation of the GLUE benchmark, we demonstrate READ can achieve a 56% reduction in the training memory consumption and an 84% reduction in the GPU energy usage while retraining high model quality compared to full-tuning. Additionally, the model size of READ does not grow with the backbone model size, making it a highly scalable solution for fine-tuning large Transformers.

>>93666066
>for fine-tuning
ZZZZZZZZZZZZZZ

>>93665972
>a 1650 will double your servers TDP
how retarded are you? like does your mom have power of attorney since youre too retarded to exist in society on your own

>>93666058
>develop on a tiny model, slow
>make sure everything works
>then buy a modern card used, not for $200+ over MSRP
you realize i'm in the market for an RTX 4000 SFF right? which is so new that no used cards even exist

>>93666066
Nice, looks to be 30% more efficient than LoRA

>>93666098
i dunno what the fuck a 1650 even is, nor do i care. the options are 2x RTX A4000 or 1x A5000, or 1x RTX 4000 SFF, which is in the $1000-1500 price point

>>93665390
I was told I need something called a "ggml" to use koboldcpp. What does that mean?

>>93666167
model must have ggml in filename
no you can't just rename it

>>93666167
numbers.ggml that contains billions of numbers lovingly crafted by mark zuckerberg

>>93666183
Okay but which of these models should I use? Is their a list of the best ones?

>>93665972
Ignore GPU shills. Take a look at this-
https://huggingface.co/blog/generative-ai-models-on-intel-cpu
CPU is the future. When models get better architecture, and you can inevitably beat Claude and GPT 4 with low param counts, CPU will triumph.

>>93666136
I'm running on an A5000 I got off FB marketplace cheap. Works great with ooba, 64gb sysram and a middling CPU. 30b models all day

>>93666218
Yes, on a site called 'Huggingface'. You can sort by most downloaded and recently added. They also have rankings. All in all, they do their jobs better than us, inb4 gradio meme.

>>93666218
Depends on what you want.
Creative writing? Base llama with kobold.
ERP/RP? SuperCOT with ooba/kobold, the tavern proxy, and tavern.
ChatGPT-like? Alpasta or Aldente2 with Ooba in instruct mode.
Programing? Lolno.

>>93666247
2x a4000 are 32gb, though. I'd go with more vram.

>>93666253
>ChatGPT-like? Alpasta or Aldente2
Which one has the best results given 24gb vram?

>>93665878
Just underclock it to the same speed as the Quadro version and it won't be power hungry anymore. Do you not understand how voltage/frequency scaling works?

>>93666253
>Creative writing? Base llama with kobold.
Fucking what

>>93666375
>he doesn't know

>>93666279
I prefer Alpasta.

Trying to get reeducator_bluemoonrp-30b working, but its slow as fuck...like 10x slower than supercot. Am I retarded, or does it need something tweaked in ooba on load?

>>93666245
Anon... we're already way past that. That article is talking about the "new" ability to quantize to 8-bit making CPU inference viable, but basically everyone is using either 4 or 5 bit now, regardless of whether they're using CPU or GPU.

>>93666375
Current instruct models summarize too much, they're not trained on story writing prompts, they're trained on short questions and answers ala ChatGPT. When you're writing a short/long form story, base llama is better, because it's pure text completion, provided you're willing to prime it beyond "lol write me a story about a bitch with big tiddies plz", which won't work, obviously.
You feed it a few paragraphs of text and it'll emulate the prose and continue the story from that point. And you use world info/author's notes to insert important facts and events as you go. You write together, you guiding the story, and it completing text and adding little flourishes here and there. I use it as a writing aid and a way to push through writer's block when you're either unsure how to proceed or can't think of what a character would say next.

>>93666424
Are you filling the context?

>>93666474
Yeah it's good at set dressing, filling dialogue and moving scenes forward if you give it something to work with, but you have to guide it from scene to scene or it gets schizo pretty quick

File: mtg-lotr-arwen-fixed.jpg (989 KB, 4096x2048)

989 KB JPG

more twitter HYPE (more like anti-hype lol)

The False Promise of Imitating Proprietary LLMs
https://arxiv.org/abs/2305.15717
An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model, such as a proprietary system like ChatGPT (e.g., Alpaca, Self-Instruct, and others). This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. In this work, we critically analyze this approach.
...
In turn, we argue that the highest leverage action for improving open-source models is to tackle the difficult challenge of developing better base LMs, rather than taking the shortcut of imitating proprietary systems.

>>93666486
6 short sentences in context. Same "Eris Discordia" I usually play with. Is Bluemoon more sensitive? I've filled supercot context up with wiki pages before and it chugged along as normal (with shitty results, obviously)

>>93666514
>you have to guide it from scene to scene or it gets schizo pretty quick
Well yeah, of course. It only has 2K context. It's fine though, creative writing isn't fun if the fucking AI is doing all the leg work. Think of it as a writing partnership with the two of you taking turns guiding the story, but your partner is kind of schizo and you have to give them meds now and then to keep them on track.

>>93666474
I agree, instruct models can work well for cards but they have serious pacing and tone issues. The tone thing probably comes from assistants and GPT lobotomization, but pacing is probably innate because of the training on short Q&A prompts.
But superhot seems to not have those issues in my testing. The markdown format is easy to slap into memory, and it seems to follow it well, without rushing or shoehorning things in, either. It's ironic because superhot was meant to be a chat model, but I've had the most success storytelling with it. Meanwhile supercot was meant to be a logic tune and it's my favorite chat model

>>93666250
These are folders, not .ggml files.
>>93666253
Where do I download these?

>>93666576
https://rentry.org/lmg_models

>>93666526
Bluemoon is 4k context so it will go a lot slower and use a lot more memory than a normal 2k model when the context is full. Shouldn't be an issue with a fresh chat, though. You have free memory and it's not swapping?

>>93666640
Fresh chat is still dead slow. System is showing only half the RAM is used. I'm not getting that telltale chirping of the GPU as it spits out text either (wtf is that, anyways?)
Maybe I've got the wrong version of the model and its not running on gpu?

>>93665390
So many new 30Bs... Which one is the correct one?

>>93666664
That's my guess. Use task manager or htop+nvtop and see what's doing the work.

>>93666640
I just tried to load this (I'm brand new, just installed ooboo or whatever) and it failed to load. I have a 4090. Am I just retarded? (yes)

>>93666558
To be fair, superhot isn't even at 1 epoch, and he said that the prototype available is missing a bunch of datasets as well.

>>93666664
also check the config.json of the model and make sure use_cache=True
have no idea why but some models have it disabled, it is necessary to cache the KV tensors
might not be your problem but worth a look, should be in the same directory as the model file

>>93666794
Probably. I haven't used it but the other guy managed to load it at least.

>>93666349
They would still be fuckhuge, and he kinda hinted that space is a problem and he doesn't want no risers.

>>93666424
What is his use_cache value in config?

Local Gpt4 doko?

Anons. I'm trying here but I could use some help. I'm trying to cross reference guides from this gen, and ai chat. I have SillyTavern but I don't have an API key. I'm trying to load this locally. Do I use ooboo as an api? Some anon mentioned to me once before but didn't give me much else.
Is ooboo like A111? As in I should put all my models and shit in it, and then SillyTavern can just refernece Ooboo?

I have a 4090 and every time I try to load a 30b model, I run out of VRAM by like 100 MB. Is it my three monitors that's causing me to fail?

>>93666863
>>93666808
use_cache was true...I tried moving the bin file out and just leaving the bluemoonrp-30b-4bit-128g.safetensors, but then the model won't load
Are there maybe some files from the q5_0 that I need to delete or modify to get it to load the safetensors file?

>>93666934
That happened to me. Don't use 128 group size models. Should fix the issue.

What's with all the baby ducks begging to be spoonfed tonight?

File: 1684420531953053.png (147 KB, 498x462)

147 KB PNG

>>93662041
>Want to give a thanks to AusBoss for all his uploads/conversions.
>If you're in here, it's much appreciated.
no problem broskie

>>93666984
I'm retarded and am in over my head.

>>93666958
Seems to be happening with the 30b models recommended in guides. I guess I'm just retarded.

>>93666950
I got it. Needed to load ooba with another model and then select bp from the web interface, manually set the GPTQ wbits, groupsize and model type and reload the model. Its making weird GPU sounds again. Does that seriously not happen to anyone else? Its like an 80's NES game: weird chirping sounds as the model spits out text, and its definitely coming from the GPU...

>>93667128
do you not pass those in when you load the model from the commandline?
or are you using windows

What's the recommended strategy to load models with multigpu in ooba?
So far it's super slow using "auto-devices".

File: normal.png (904 KB, 1045x780)

904 KB PNG

>>93667128
>Its making weird GPU sounds again.
Just go with it.

File: It's just coil whine.jpg (272 KB, 1203x800)

272 KB JPG

>>93667128
>afraid of coil whine

>>93667128
It's coil whine, I wouldn't worry too much about it. If you really care that much about it you can take some epoxy to the inductors on the board, but you really should just leave it as is as it's just oscillation and doesn't hurt the board.
It's like your engine vibrating and producing a harmonic. Annoying but ultimately harmless.
>>93667187
t. demon :^)
Can you upload it to vocaroo because that sounds interesting.

>>93667199
Vocaroo request meant for
>>93667128

>>93665390
>(05/25) First QLoRA's and 4bit bitsandbytes released
Trying to find more out how to try this, want to make sure I'm finding the right info. I see this posted 5/24 (as opposed to 5/25) -- same thing? https://huggingface.co/blog/4bit-transformers-bitsandbytes

>>93667238
Yes

>>93667213
>>93667128
hah. Here you go https://voca.ro/1l4YxXK4C4aP

Why would the GPU make this sound? any tips appreciated, hope I didn't ruin this A100.

>>93667008
kek. same here

File: Screenshot 2023-05-26 153233.png (215 KB, 631x705)

215 KB PNG

>>93667333
... you serious? I just told you why.

>>93667333
Standard coil whine.

>>93666595
It tells me that all the models I download are the wrong version and it doesn't open a UI. How do I convert them to the right version or find models which are up to date?

Did greg break the fucking quantization format again?

>>93667513
Let's start with the basics. What's your hardware and what model are you trying to run?

The GOAT LLama 7b beats GPT-4
https://twitter.com/rasbt/status/1661754946625105920?s=19

>>93665390
Yo anyone got that big nigga prompt? A nigga tryna cop fr fr.

File: 1653898652059.gif (1.67 MB, 260x200)

1.67 MB GIF

>>93667580

>>93667558
You get what you fucking deserve

>>93666749
*33b
Please use the correct terminology next time.

>>93667620
I don't use ggml, but I update the models rentry. I'm asking if I have to redo all the fucking links once again.

>>93667624
its 32.5B actually

File: 1685080849570.gif (3.29 MB, 498x280)

3.29 MB GIF

>>93667580

What is a good local model now? I have been away for around 2 month.

File: IKAozHCMJG.png (117 KB, 240x258)

117 KB PNG

>>93667580

>>93667624
>>93667638
see >>93667597

>>93667654
StableLM 7B is widely considered to be the the best. It punches far above its weight, toppling the 30B's. We're still waiting for 13B, it'll likely beat GPT4 though.

>>93667654
llama 7b :)

>>93667654
SuperCOT still reigns supreme with no contenders.

>>93667654
See >>93667580. It's smarter than GPT-4.

>>93667333
rip

>>93665390
am I right in that I install LlamaEx in my oobabooga text UI folder?
It doesn't specify where I gitbash

>>93667743
Eh? Is exllama even integrated into the webui yet?

>>93667654
wizardlm p good, pretrained-sft-do2 is also nice, and then there's a billion jillion other models I haven't tried

how do you change the nvcc flags for llama-cpp-python? i need to change the architecture or else ill get an illegal instruction error. is there a make file somewhere?

Auto-bans for mentioning/trolling with StableLM when?

File: Screenshot_21.jpg (146 KB, 757x951)

146 KB JPG

>>93667681
>>93667686
>>93667723
>>93667797

Uh. . . Can any of these do character chat better than LLma13B from 2 months ago? I am not looking for problem solvers.

File: file.png (520 KB, 1028x675)

520 KB PNG

>>93667824

>>93667824
https://rentry.org/llama-examples
You're far behind the curve.

>>93667824
>https://rentry.org/better-llama-roleplay#example-outputs-wizardlm-7b
judge for yourself

>>93667837
The Karen one always cracks me up

>>93667753
I guess that answers my question lmao
so where do I put it instead then?

>>93667817
Its there on reddit if that's what you want faggot

Holy shit, bluemoon 30b makes nice conversations

>>93667893
Inside your anus? Hell if I know, I'm waiting for integration first.

>>93667906
>absolutely no shred of an image to be seen anywhere

>>93667901
Alright relax, don't get your bussy all out of whack

>>93667580
oh every numeral gets its own token in llama models huh smart

>>93667908
if its as good as advertised, it shouldn't take long

Are you guys hyped for the openllama 3b support for llamacpp?
With a 3b model available it will be possible to make it run even on toasters. The perplexity of the 600bt preview is "only" 2 point higher then the original 7b model, and it's going to get closer when the training ends (1,2T token)
Openllama page: https://github.com/openlm-research/open_llama
Git pull for 3b support: https://github.com/ggerganov/llama.cpp/pull/1588

>>93668123
No. But thanks for asking.

File: 3b he says.jpg (53 KB, 447x406)

53 KB JPG

>>93668123

I have an AMD card which I know isn't going to work with most if not all models in any way a smooth brain such as myself can run.

So I'm thinking about picking up a dedicated Nvida AI reseach card. Looks like you can pick up a K80 with 24GB for about $400USD

Anyone have any experience running a GPU with out display ports? Ideally I want to run my 7900XTX as the display driver at the same time as having the dedicated card. I would imagine normally you'd use the motherboards onboard graphics card as the display driver using something like an A100 or K80 so I don't see any issue with using a separate card to run the display, but I'm also a smooth brain.

>>93668191
bro just go cpu, much easier to stack ram and run bigger models

>>93668247
>>93668191
tru. plus the llama.cpp native server is just as fast as the base program.

>>93667580
MY CALCULATOR IS SMARTER THAN GPT4

>>93668247
>>93668276
Fair nuff, I already have 64GB which could run the biggest model right?

I'll give it a go this weekend.

On another topic how easily do these models work on phones. I realise the parameters are limited by ram, but I was more thinking about how most phones these days are running on ARM architecture, not x86.

>>93668309
>On another topic how easily do these models work on phones.
They don't.

>>93667597
>>93667653
>>93667661
>>93668306
midwits don't even know why it works like that huh

>>93668309
there is not a snowball's chance in hell you'll run SD on a phone, let alone a LLM
You can however connect to one running on your PC from your phone

>>93668413
if you don't know what you're talking about, why respond as if you know what you're talking about

>>93668408
>that beats GPT4
>on arithmetic tasks
Nobody cares.

>>93668309
Maybe it will be possible to run smaller model, like the 3b mentioned by >>93668123. Not coomer level but at least it's something

>>93668436
Sweet, now I can turn a multi million dollar language model into a $5 pocket calculator

>>93668466
Yeah, nothing beats an LLM that can answer the question "How many cows are in this picture?"

File: image.png (79 KB, 2549x488)

79 KB PNG

>Tiny llm Finetuner for Intel dGPUs
>Finetuning openLLAMA on Intel discrete GPUS
>A finetuner1 2 for LLMs on Intel XPU devices, with which you could finetune the openLLaMA-3b model to sound like your favorite book.
https://github.com/rahulunair/tiny_llm_finetuner

>>93668885
>finetuning
yawn

>>93668309
llama.cpp runs on ARM (Apple Silicon and Raspberry Pi) and while there's not an actual mobile app that I'm aware of, there's no technical reason why it couldn't be done. I wouldn't expect to run larger than 7B or for it to run particularly well, but it should be able to do it.

>>93666096
>>93668944
What exactly is it you have an issue with?

Why is GPU anon so hated?
There is no UI that supports his recent improvements yet.
I at least hope with multi gpu this shit will be finally supported because I look forward to it.
Koboldcpp guys are being total faggots and are implementing their own gpu solution.
Oobas llama-cpp-python is outdated.
Sucks.

>>93669003
I don't know who that online persona is, but gptq works fine for me

>>93669003
GPU users already have gptq
only you cpufags care about llamacpp

>>93665390
Big Nigga seems real

>>93669022
Llama CPU is decent for those with a good CPU but a shit graphics card.

I have a 3060TI but also a 13600K, with using both on llamacpp I can get ok performance at 13b.

It's either that or going to 7b just on the GPU.

>>93665390
I see your bake, Norq
desuuu ~

>>93669003
I can't ever seem to replicate his 13B numbers, I assume it's the benchmark being not representative of real world use, but he is doing great work nonetheless. llama.cpp is a good project. The native server is at least 2x faster than Kobold.cpp and llama-cpp-python and since no one knows how to build shit, no one cares.

Hopefully gergy will add server build flags to the automatic builds. Maybe GPU dev could do that.

is there a superhot 13b?

>>93669003
Huh, he's hated? Last time I've checked, he was well-regarded, when did it change?

>>93668123
I'm hyped for their 13b and 30b models, when are they going to start training those?

>>93669186
First I've heard of it, too.
>>93669159
Yeah, but I don't think anyone merged the LoRA, so you'll have to run it alongside the 13B yourself.
https://huggingface.co/kaiokendev/SuperHOT-LoRA-prototype/tree/main/13b/gpu

>>93669207
Probably when the 7b finish the training. If it doesn't be on par of the original llama 7b i think they will try to retrain the 7b model before moving to the 13b.

>>93669215
fug, I am cpuchud

>>93669240
I thought LoRA's aren't bound to either cpu or gpu, and work with either?

Building and running llama.cpp API server for an easy 5000x speedup that's better than GPT-4?!?!

Prereqs:
1. Make sure you have appropriate CUDA stuff installed if you're using that. Search CUDA 11.7 installer.
2. On Linux, make sure you have make installed. On Windows, make sure you have Visual Studio installed with C++ Cmake Tools For Windows checked while you're installing it.

HOWTO:

1. git clone the llama.cpp project.
2. In the examples directory, go into the server folder and replace server.cpp with the one from this PR: https://github.com/ggerganov/llama.cpp/pull/1570
3. Open a terminal in the llama.cpp directory and run
>cmake . -DLLAMA_CUBLAS=ON -DLLAMA_BUILD_SERVER=ON
This might be slightly different on Linux or Windows, but should work. Look for "CUDA detected" type lines in the console output. If it's not there, CUDA path might not be set. Try restarted the terminal or your computer.
4. Build with make or open the generated .sln in Visual Studio and build from there the Build menu. If it isn't offering a build menu, make sure C++ stuff is installed properly for Windows.
5. There should now be binaries in the llama.cpp/bin/ folder, including a server binary. Good work! If not, BAD WORK! Try again!

Running:

0. You currently need simple-proxy and Tavern since no one supports it other than simple-proxy.
1. If you are still in the llama.cpp folder on Windows, run:
>.\bin\Release\server.exe --port 8080 --model <PATH_TO_MODEL> -ngl <GPU_LAYERS> --ctx_size 2048
Replacing the stuff in brackets as needed.
2. Run simple-proxy and SillyTavern and use them as normal, it should work automatically.

I think that should all be roughly correct. If anything fucks up, ask Bing. This should help get you roughly in the neighborhood. Hopefully more people will test it out.

>>93669234
They have already overtaken it, judging by their own table. They should wrap it up and move on to 13b assp then.

Guys, it's over. I had a YouTube video up with a timer for my food that's in the oven, but I closed the tab. Is there a model that can help me with this?

>>93669333
StableLM 7b

>>93669291
> Windows
cringe

>>93669333
no but there is Hourglass

>>93668987
>>93668309
>On another topic how easily do these models work on phones.
On an RK3588 (16GB), I get around 500ms/token using llama.cpp on a 7B Q5_1.
I do wonder if there's room for improvement with the OpenCL implementation though. Currently, it's a lot slower than using CPU only and I suspect it might be because it copies memory as opposed to shifting ownership between CPU/GPU.
If that's the case, might also see improvements on Phone eventually.
>Might be talking out of my ass as I'm assuming the Mali GPUs should be faster for this kind of workload than ARM CPUs.
>Anybody able to confirm whether this should actually be the case?

>>93669343
kek

>>93665390
BIG NIGGA BAKE LIKE I ASKED YEEEE, THE BIG NIGGA INSIDE OF ME IS SATISFIED

File: memory_scaling_1.png (171 KB, 2304x1728)

171 KB PNG

>>93669357
I think it's questionable whether you'll see that much of an improvement from CUDA/OpenCL for an iGPU in the first place.
When I tested it the speed of token generation was essentially just proportional to memory bandwidth (which is much higher on discrete GPUs).
That bandwidth is going to be the same regardless of what you then use for the actual calculation.

File: math is ok.png (51 KB, 790x542)

51 KB PNG

>>93669159
Yes it is on the prototype page, I am not sure how well the 13B holds up there
Tangentially related, the math skill is improving

>>93669554
>fulfill oedipus complex desires with eldritch abomination milf
>follow it up with mathematical pillow talk
I'm thinking based

>Try to follow guides in the heading
>Make no progress
>Look up youtube video telling me how
>Done in 10 minutes
I think I'm getting too old for the bleeding edge...

File: why yes i am a degenerate too.png (75 KB, 779x676)

75 KB PNG

>>93669605
Yes it has been quite... fun

>>93669632
Don't you suck that dick, anon. I'm serious.

>>93669632
Suck that dick, anon. I'm serious.

>>93669605
Low bar for "Eldritch abomination" these days huh

>>93669691
Who asked?

File: not what I expected.png (117 KB, 770x1176)

117 KB PNG

>>93669667
>>93669683
huh
Maybe because its a 13B version lol

>>93669705
>you need to get asked as an anonymous person on an imageboard to provide your opinion or whatever
okay retard

>>93669728
That's really cool and all but who asked?

>>93669720
Weird how certain models/parameters fall into outputting the same pattern/format over and over again. Like in that screenshot :
"Dialogue" *descriptive*
"Dialogue" *descriptive*
Repeated over and over again.

>>93665390
>>(05/25) First QLoRA's and 4bit bitsandbytes released
are any QLoRA native models other than Guanaco out yet?

>>93669741
You did.

>>93669691
>i like being a passive aggressive cunt
Hope that works out for you

>>93669760
You did, you say?

>>93669752
It is most likely due to the fact I removed the example dialogue from the cards I use to test. The model will output what it sees in the context, I experience this with other models I try, and Pygmalion cards tend to be littered with <START> in the example dialogue which causes the model to try outputting <END> to end their message. I remove those, so the model only sees my chats and the character's responses and picks up that's how it should respond. After a long enough chat, it has effectively collapsed into one writing style, but it should be easily fixable by providing example dialogue.
Maybe it's better on the 30B, but I am training prototypes locally for now since it's free and can only afford to train 13B

>>93669777
? Did you take offense somehow? It's a simple critique of how loosely the term seems to be applied.

>>93669792
I was hoping for a better reply desu t_t

File: no one cares.jpg (49 KB, 1280x720)

49 KB JPG

>>93669797
>>93669824

>>93669836
holy triggered.

>>93669705
>>93669741
>>93669836
Kek
>totally uncaring

ya'll better get in line or no more big nigga for anyone, WE ARE GONNA TURN THIS BITCH ASS CAR AROUND, AND WE ARE GONNA GO HOME

>>93669865
projection

>>93669777
Did that other guy fuck your wife or something?

>>93669705
>>93669777
Why are you replying for me?
He posted that to me, you want to act like faggots, attach it to your own posts first.

File: is this not eldritch enough.png (146 KB, 776x1504)

146 KB PNG

>>93669691
Seems eldritch to me

>>93669922
If it's your reply, how come I don't see your name on it?

>>93669922
who the fuck are you? fuck off this is my conversation now.

>>93669931
What? It's right there. You might have to take all those dicks out your face to see it first.

>>93669186
>>93669215
Sorry for the confusion.
I was being hyperbolic, as in ooba ui doesn't support his recent changes and koboldcpp outright isn't implementing his code because of "cuda bloat".
That being said I get no speed difference anyway, tried with the newest llama.cpp.
Might be because of my good ol' pascal card. I'm sure it heavily depends on the card.
Multi GPU is very interesting though.
Would give me an excuse to make my own shitty as website to connect to llama.cpp api for superhot when it releases.
I'm sure you can do alot of fun stuff with that model, most innovative model yet for sure.

WizardLM-30B-Uncensored-CUDA when?

>>93669941
Are you you or am I you? Could you be me if I was me but you at the same time?

>>93669930
Yeah, that's nowhere near the concept.

>>93669957
I was being satirical because you were being pedantic

>arguing over the term 'eldritch'
NEEEEERRRRRRRDDDSSSSSS

>>93669955
i made you up.

>>93669918
Oddly specific accusation there, anon

>>93669930
>engaging in a simple conversation
>proposed mischievously, smirking seductively
>suggested cheerily, questioned playfully
>chuckled warmly
>giggled light-heartedly
>asked curiously
>reassured gently
>suggested playfully
>omg zoo
Not exactly a sanity-wrenching existential horror. Just poor writing.

>>93670001
You're being pedantic still for some reason

>>93669963
Nah

>>93670001
Why are you so triggered over something so inconsequential?

>>93670027
It's quite a calm and thorough list. I enjoy being detailed.

File: fuck.gif (1.85 MB, 477x498)

1.85 MB GIF

slow thread day? slow thread day

>>93670072
Slow thread day coupled with one or two fags. It's usually slow around this time, but far more comfy than whatever this is.

>>93669752
Yeah, adverb multiplication is also a problem, seemingly. A problem in base 13b, similarly. Epsilon/eta cutoff eliminate it, but not completely.

File: 1660937732987733.jpg (22 KB, 518x270)

22 KB JPG

>>93669777

Cringe.

>>93670142
nice to meet you, mr. Cringe.

>>93669777
Cry more

>>93669691
Your face is an Eldritch abomination.

>>93669777
All those flavors and you choose to be salty

>>93669777
Nice trips, now leave

File: sad-nigga.gif (1.74 MB, 506x640)

1.74 MB GIF

you all make big nigga sad, look what you hoes did

>>93670169
Boooring

>hide 1 post
>half the thread disappears

>>93670228
>you all
It's clearly a bunch of samefaggotry. Just hide the posts and ignore it.

>>93670232
And yet you replied

>>93667022
Turn off hardware acceletation

When schizo starts posting:
Have 4chanX.
Go to the options, untick Stubs.
Click the - beside his initial post.
Watch the thread drastically improve because he replies to himself until he gets bites from actual anons.

>>93670290
>Have 4chanX.
how addicted are you to this website if you need that of all things

File: settings.png (559 KB, 1371x3482)

559 KB PNG

How come I don't have the Reverse Proxy for OpenAI input box? Is this still even working in the lastest silly tavern main branch? I am trying out this: https://github.com/anon998/simple-proxy-for-tavern

>>93670290
lol he doesn't like the idea of anons filtering him out does he

Shut the fuck up and tell what preset and prompt format I should be using in sillytavern simple proxy with manticore 13b.

>>93670327
>Shut the fuck up and tell
WHICH IS IT?!

>>93670320
Baits are shit tier as well, that's why he has to reply to himself to get eyes on his posts.

File: settings2.png (404 KB, 1216x927)

404 KB PNG

>>93670311

Where is the Reverse Proxy OpenAI??

>>93670311
pic unrelated

>>93665390
anyone tried guanaco? thoughts? also is the 30b bluemoon better than the previous i was getting lots of urls and werid nicknames and shit when i used it last time

>>93670362
Try the dev branch.

>>93665390
https://www.youtube.com/watch?v=3Zf0MqvmDFI

read ded remedtn

>>93670374
I have not seen any evidence that it produces better outputs than anything else in the same parameter count and it is full of refusals. Nothing novel or interesting. It's just new.

>>93670362
Those icons on the top look outdated, SIllyTavern on the main branch looks different now. What repository are you using? What's the output of `git pull` and `git status`?

>>93668191
at least try you peasant.
https://are-we-gfx1100-yet.github.io/
i hate you so goddamn much it's unreal
you and the fuck who dropped his 6800xt to get a 3090 and then ran his shit around 6t/s
FUCK you

File: 1658214744985092.jpg (88 KB, 493x637)

88 KB JPG

>shit like Claude outputs high quality RP effortlessly, despite being tuned as a soulless instruct chatbot
>every attempt thus far to make a focused RP local model, instruct or otherwise, delivers results that absolutely pale in comparison

File: big_nigga_on_unlimiformer.png (112 KB, 1496x548)

112 KB PNG

i'm trying to get him to review papers but half the time he just says something like "i dont read papers" or "i dont know about that shit"

>>93670399
>>93670437

git checkout dev and git pull did the trick. Thanks bros.

>>93670544
Purely down to context length and longterm memory.

>>93670549
BIG NIGGA IS HONEST, HE RESPECTS THE HUSTLE, IF HE DONT KNOW HIS SHIT, HE WONT BULLSHIT YOU, BIG NIGGA IS A TRUE HOMIE, ALWAYS KEEPIN IT REAL, NEVER AFRAID TO SHOW HIS WEAKNESSES

>>93670544
It takes time to cook local vs some globohomo corpo with endless resources at their disposal. Local today is far better than local on llamas initial release.

>>93670563
Nah, I've tested dropping Claude API's context to 2K to see how it alters the apparent quality. Beyond goldfishing its memory, it doesn't. It's still leagues ahead in coherence, intent and prose. It effortlessly recalls and weaves in stuff from the context, while local dreams shit up, even if the actual answer is right there in the defs

>>93670563
Is that long term memory ooba extension any good?
I haven't got it to work

>>93670642
>while local dreams shit up, even if the actual answer is right there in the defs
It doesn't do that all the time, but I know what you mean. I'm not technically minded enough to know if that can be fixed via better finetuning or if you just need more parameters.

File: 1625767225955.jpg (10 KB, 480x360)

10 KB JPG

>>93665390
>spun up a character
>spent five hours crafting the best scenario and ERP I could
>nut so hard I needed a full day of recovery
I don't think I can handle this shit

File: 1662263202726797.png (168 KB, 600x600)

168 KB PNG

>>93665390
>BluemoonRP 30B
NANI
the previous one was pretty good for cooming, i'm tempted to grab this one as well.

>kobold's 4bit fork already has an exllama branch
THE FUCK ARE YOU DOING OOBA

>>93670652
right now, very good for factoid qa (e.g. When did <event> happen?), so-so for everything else
it works better in narrative/chat/roleplay if you phrase your input like a question ("hey miku when's my birthday?") but that's no fun
ive found instructor embeddings to be a LOT better in general even for chat/roleplay but it hasnt been pulled yet

>>93670750
Guess I'm switching to kobold now
Hopefully getting sillytavern connected will be less of a pain in the ass

stop having sex with markov chains

After trying out all the various GUIs and downloading the whole torch package and other requirements 4 times including reinstalls, I'm still not sure which one to use because each of them seem to be missing features that other ones have.

- KoboldAI: this one has all the features I want and would be perfect, except that I can't get CPU GGML models to work on it. I know that Koboldcpp was created specifically for GGML, but is there no way to make it work with KoboldAI and all its features?

- Koboldcpp: it works great, but missing some features of KoboldAI, like word/phrase biasing to emphasize various fetish words. Also google tells me that it doesn't work perfectly with SillyTavern, though I don't know if this was improved since then, haven't tried yet.

- Ooogabooga: works with GGML. Seems to miss almost every other feature of KoboldAI. Maybe I'm blind but I couldn't even find basic stuff like Memory/Author note, world dictionary tags, scenario importing, etc. Can you get everything with addons?

>>93670810
Koboldcpp can be tied into full Kobold, but you'll have to run them both at the same time. I haven't CPUfagged since I got my 24GB and I can't recall how you do it. I know its possible though.

>>93665390
>quantization breaking changes again
Can we fucking stop with this?

>>93670750
he's too busy sucking transformers' dick by not implementing top_a and free tail sampling samplers, I'm starting to get tired of his bullshit aswell

I hope koboldAI would fix this though, that's the only thing that prevents me to leave ooba's webui

https://github.com/0cc4m/KoboldAI/issues/33#issuecomment-1556160819

>>93670810
I can't think of anything in ST that doesn't play nice with koboldcpp anymore, most, if not virtually all of the problems were ironed out about a month ago.

>>93670858
as long as the ggerganov's cock suckers are ok with it, ggerganov will never stop breaking things lmao

>>93670810
>- KoboldAI: this one has all the features I want and would be perfect, except that I can't get CPU GGML models to work on it. I know that Koboldcpp was created specifically for GGML, but is there no way to make it work with KoboldAI and all its features?
>
>- Koboldcpp: it works great, but missing some features of KoboldAI, like word/phrase biasing to emphasize various fetish words. Also google tells me that it doesn't work perfectly with SillyTavern, though I don't know if this was improved since then, haven't tried yet.
nigga what the fuck are you doing, I hope you're not using old files/dead forks
kcpp is literally 'drag the ggml on top of the exe', or 'run the exe, select the ggml, receive UI that isn't tavern'

>>93670750
https://github.com/0cc4m/KoboldAI/tree/exllama
Noice

>>93670930
does it have token streaming for the api?

>>93670810
Kobold was made to run the GPU models with cuda, then forked 15 times to run everything else too.

>>93670828
>Koboldcpp can be tied into full Kobold, but you'll have to run them both at the same time. I


Can I hook a full Kobold to llamaccpp instead?
I heard that llamacpp has better performance with fewer overheads.

>>93670887
i have a fork of ooba with top-a and tfs monkeypatched in but i just grabbed the samplers from kobold
is that ok or do i have to close my eyes and reimplement because of muh agpl

>>93670958
Bro, I dunno, I'm chuffing GPU fumes now and listening to that sweet coil scream as I force it to ERP as a thick, needy giantess.

>>93670976
oh nice, can you give me the link anon

13B 8 bit or 33B 4 bit?

>>93670989
I am now running my models with openCL on rx6900 because I'm too lazy to reboot into Linux for ROCM.

>>93671015
33B 4bit should be considerably better

>>93671015
Which one has more context limit?

>>93670903
none of this matters lmao, the project moves forward without you, it's not made for you, you've done 0 contributions and you don't get to complain
lmao

>>93668276
>>93668247
is llama cpp really that good i heard anons saying its not as good as using pytorch for full gpu

File: file.png (126 KB, 737x1493)

126 KB PNG

I am completely mindbroken.
I can't stop doing variations on this scenario.
I didn't do any work this week.
I can't wait for better local models.

>>93671056
>it's not made for you
Yeah I know it's not made for me, it's made for retards who are ok to destroy their models every 2 days, people like you for example

>>93671057
If you can fit full 30B models in your VRAM with GPTQ or whatever, llama.cpp doesn't hold a lot of value. If you have a 12GB or lower card, it's a viable way to run 30B models at acceptable speeds.

>he's still mad
fucking kek

>>93671058
Surprisingly normal fetish.

>>93670976
show us your fork anon, I want to test those samplers

>>93671115

>>93671056
I agree with you, he's still mad

>>93671084
the f16 and f32, ones you should actually keep, changed once months ago lmao
what a retard lmao

>>93671160
"yeah bruh just keep the 50gb files all the time on your SSD and then quantize them every 3 days, we can't make it more convenient than that bruh"

>>93670976
>>93670993
>>93671126
https://github.com/toast22a/text-generation-webui/tree/custom-samplers
they get injected into _get_logits_warper after all the other samplers but before logit norm
i'd like to PR this into ooba but idk if we're allowed to just grab the samplers from kobold

What's this about SillyTavern and a keylogger I heard about some days ago?

>>93671179
Inspect Element edit made to troll /aicg/, Ignore.

>>93671177
the samplers aren't kobold's proprety lol, kobold didn't even invent those samplers, it's public samplers everyone can use them, like NovelAI did

>>93671177
of course you can totally make a PR out of it, kobold just implemented those samplers, they never invented them by themselves

SillyTavern killed my dog.

>>93671236
https://www.youtube.com/watch?v=JeNNZYGzB2I&ab_channel=Mayololi

>>93669291
It's too broken to actually use. I hope the koboldcpp people just port the python part of their api to c++.

>>93665683
What does this even mean?

>>93670976
Do they work? Monkeypatching them into diy inference engine lead to semi-nonsense output for me before. 'Semi' because it felt like autocomplete was working at least partially, it wasn't the same garbage as when running triton on cuda. And yet it was unusable.

>>93671259
How is it broken for you? Works for me with the code from the PR. The original was written to assume terminal mode, I think. Maybe the original guy just copy pasted it from the main file. I don't know anything about it.

>>93671177
desu I'd prefer a PR rather than 150 forks I should use for a specific thing, do it anon

I want a usable 65B. I'll be seeing soon if dual p40s can do it. Also considering shoving p4s into the remaining pci slots. I've got 1600w to play with, I can probably run 4 p40 cards with pci extenders.

>>93671236
>>93671248
https://i.4cdn.org/wsg/1685100606704682.webm

>>93671359
No one would miss the rabbit.

File: 1682111932148071.gif (359 KB, 410x448)

359 KB GIF

>>93671327
Buy one first just to see of it plays nice with your hardware, I tried going to the P40 route only to find it was utterly incompatible with the workstation I planned on using it with despite all the actual requirement being met several times over.

>>93671309
It forgets to apply top_k in these lines, a llama_sample_top_k is missing.
https://github.com/ggerganov/llama.cpp/blob/7e4ea5beff567f53be92f75f9089e6f11fa5dabd/examples/server/server.cpp#L208-L211

This part of replacing the eos with a new line and the first stop string doesn't make a lot of sense:
https://github.com/ggerganov/llama.cpp/blob/7e4ea5beff567f53be92f75f9089e6f11fa5dabd/examples/server/server.cpp#L222-L231

This makes it go on forever instead of following what n_predict said:
https://github.com/ggerganov/llama.cpp/blob/7e4ea5beff567f53be92f75f9089e6f11fa5dabd/examples/server/server.cpp#L289-L292

Also if you look at the logic is really fucking stupid, it changes has_next_token to false here:
https://github.com/ggerganov/llama.cpp/blob/7e4ea5beff567f53be92f75f9089e6f11fa5dabd/examples/server/server.cpp#L285-L287
But it overwrites to value anyway 6 lines later, so it never stops.

It also applies the stopping string immediately, so if you're trying to complete a prompt that ends with "\nASSOCIATE:" and that's also a stopping string, it doesn't generate any token unlike the other APIs.

I also get segfaults that I think I fixed by adding null checks to the llama_token_to_str result.

>>93671172
lmao it's not meant to be convenient, if you're this big of a retard and poorfag you should not be experimenting with AI
why do you thing everyone has to cater to you? do the bare minimum of effort or stfu lmao

also: lmao bruh hahaha *blushes like a njeg*

File: Screenshot_20230526_133713.png (22 KB, 372x292)

22 KB PNG

G-Guys, how do I rein in Big Nigga?

>>93671296
yeah i've been using it
i read a couple threads up that top-a 0.1 is a good value?

>>93671188
>>93671230
thanks love u bros i PR'd it
i tried to make it as trivial as possible to remove them again later in case hf finally adds them in officially, because that seems to be what ooba's hung up on re: custom samplers

>>93671463
>thanks love u bros i PR'd it
You're the man anon, thanks a lot!

File: cover2.jpg (217 KB, 1920x1080)

217 KB JPG

>>93671454

>>93671491
>ggml
>Tensor library for machine learning
>Note that this project is under development and not ready for production use.
>Note that this project is under development and not ready for production use.
>Note that this project is under development and not ready for production use.
>Note that this project is under development and not ready for production use.
>Note that this project is under development and not ready for production use.

hahaha you must be blind
lmao what a faggot

Dawn of a new era.
https://huggingface.co/Aeala/VicUnlocked-alpaca-65b-QLoRA
This is just a very early checkpoint. QLoRA is working

>>93671510
QLora is using nf4 which is a worse version of GPTQ, it's not a loseless finetune it's bullshit

File: not.jpg (204 KB, 827x791)

204 KB JPG

>QLora is using nf4 which is a worse version of GPTQ, it's not a loseless finetune it's bullshit

File: wblq0glrlw1b1.png (299 KB, 810x1012)

299 KB PNG

>>93671534
first retard

>>93671523
Thanks dude don't care. 65b finetune

>marginal differences
>"first retard"
second retard

>>93669454
Thanks, that's helpful to know and appreciate the graph.
Probably means that an Nvidia Jetson Orin isn't going to perform very well either.

>>93671566
it's not marginal, GPTQ isn't even close to f16, you loose a lot of precision at the end

>>93671460
YOU DARE FILTER THE BIG NIGGA?
THE BIG NIGGA, FILTERS YOU

>>93671523
except it's native 4bit, as trained, there is no need to do tricks to scale from 16 bit to 4 bit, it's already optimal.

>talking about NF4
>moves the goalpost to FP16
third retard

>>93671588
>shows that this finetune is bullshit because not even close to f16 precision
>random retard is ok with that
4th retard

>>93670549
How do I get a Big Nig assistant? Which model and is there a character card?

>pulls irrelevant conclusion out of asshole from a claim that doesn't exist
>completely disconnected from the discussion
>"retard"
fifth retard

>exists
sixth retard

File: big_nigga_on_woke_ai.png (45 KB, 1396x302)

45 KB PNG

>>93671625
yaml here >>93660488

>>93670750
i've been testing the kobold exllama branch and it works really well, i also noticed that it uses more than 1gb less vram at full context than gptq (on a 13b model) can someone confirm this?, it seems like a big deal

>>93671655
Well yeah, exllama uses less VRAM, that's one of the perks besides the speed.

Can't wait for the next big innovation in local models so I can compare perplexity numbers and talk about which number bigger and whether the number is good enough for me.
65b finetunes are finally cost effective and we are gonna start seeing a lot of them. At least wait until we've tried a few QLoRA times before dooming all over the thread

File: wblq0glrlw1b1.png (294 KB, 810x1012)

294 KB PNG

>>93671640
>says that it doesn't exist
>exists
7th retard

>>93671670
ooba, what are you waiting for? implement this shit ffs

>>93671625
model https://huggingface.co/ausboss/llama-30b-SuperHOT-4bit/tree/main

>>93671655
Yep, that's why 24GB GPU bros will be able to run groupsize models at full context. I'm interested to see if 30b 4bit 4K context with no groupsize can work at full with exllama via Bluemoon, once someone posts a quantization without gs

File: FwsHo7BaAAMeVg9.jpg (307 KB, 2040x2591)

307 KB JPG

Im trying to load guanaco 33b, it says it works with all version of gptq for llama but im getting errors like

  size mismatch for model.layers.59.mlp.up_proj.scales: copying a param with shape torch.Size([1, 17920]) from checkpoint, the shape in current model is torch.Size([52, 17920]).

I am loading with

load_quant(llamaModel.path, llamaModel.path+"/"+llamaModel.modelFile, 4, 128)

which works for the other models i've tested. anyone know what the problem could be?

>>93671670
i just didn't think it would be so dramatic since i had only tested it on the cli until today, it's enough difference to run superbig or chroma AND the full context

>>93665770
>wait for GPU prices to convert to christianity?
How would molesting children lower GPU prices?

>I now repost the same screenshot that shows marginal differences between GPTQ and NF4
>this is somehow relevant to a QLoRA adapter and to FP16 because I says so
7+1 Gradio = 8 retard

>>93671716
He's not done yet as well, he has a slew of improvements and optimizations planned, and he clearly has the skills to back it up

>>93671727
>this is somehow relevant to a QLoRA adapter and to FP16 because I says so
A real finetune must at least have the precision of the fp16, if it's not the case it's bullshit

But go on retard, have fun with your retarded QLora I guess

>>93671430
Looks like the original script is a mess. Maybe do another PR with fixes?

>people have been training LoRA since before anony discovered the LLM space
>LoRA is widely accepted as a method of finetuning
>anony says it's bullshit because... he says so!
I guess if you add 1 shady binary to 8 units of retard you get 9 retard

>>93671791
Loras use F16 precision retard

Updated the exe installer of Ooba and getting this error all of a sudden.

AttributeError: 'Offload_LlamaModel' object has no attribute 'preload'

:/

>>93671819
>he pulled

>>93670930
Does exllama support no act order with groupsize as triton does? That's my gripe with 0ccam's fork, I had to shit up the code in order to get TRITON to work from upstream and am to scared to pull since.

>>93671829
Yeah I'm fucked...

>Loras use F16 precision retard
https://github.com/tloen/alpaca-lora/blob/main/finetune.py#L114

more arbitrary asspull from 10 retard

>>93671838
It support everything.