Offline Models for Apple hardware(CoreML + MLX) Local Inference

Diadon Acs has been working on something similar but not sure if it local or cloud based. Worth talking to him.

I have played around with these locally but not for LENR research:

- PrivateGPT 2.0

- AutoGPT

- LM Studio

AutoGPT was a bit flakey (6 months ago) but the others were alright.

They are a there are a lot more out there. For my day job I use Deepset HayStack framework and this allows for local or cloud but due to nature of the bots we use cloud based models only.

Quote from mkemp

I am interested in exploring models that can be inferred locally and trained on websites and documents that can aid research in the LENR field and answer newer members questions.

Professor David Nagel (Washington USA) is running a fairly big project on similar lines - he is working with a bunch of students training an AI on everything relevant. . There is also this article (by oone of the forum admin team...

Quote from mkemp

I am interested in exploring models that can be inferred locally and trained on websites and documents that can aid research in the LENR field and answer newer members questions.

By this do you mean develop our own "local" AI here for forum purposes? As Alan mentioned, there are others already doing that for the field. The forum though has a wealth of info, but it is basically lost because the "Search" function sucks. A scaled down AI would be nice...if available.

Quote from mkemp

I am interested in exploring models that can be inferred locally and trained on websites and documents that can aid research in the LENR field and answer newer members questions.

I have not fully developed my skillset in this field but have run huggingface models on Nvidia hardware thus far.

I am of the philosophy that you should own your AI.
AI in the cloud is not aligned with you!
it’s aligned with the company that owns it!!

Who's interested in exploring this avenue?

Display More

Hello,

I have a general paper on LENR and Machine learning with a particular focus on LLMs and Multi-Agent Frameworks which you can find here
"Utilizing Machine Learning Techniques for In-Depth Investigation of Low Energy Nuclear Reactions (LENR and Lattice-Assisted Nuclear Reactions(LANR)*Pre-Print*https://conscious.energy/2023/08/29/umlenr/

if your interested. "

I have run many local models and have found some very promising results. ChatGPT4 is still the best GPT/LLM I have tested so far though and most of the LLM benchmarks suggest it is the most advanced, but the cost to use there API calls gets ridiculous. The trade off is training your own models with a lot of compute power that is challenging for most people at home though platforms like Hugging Face, Google Collab, RunPod, relvanceai, ect. do a very good job allowing people to run compute heavy models.

Also, running a windows machine with WSL/Ubuntu has created some issues and my current machine doesn't have AVX2 which limits me a little atm. Though I did just upgrade my GPU and added some SSDs to carry out more experiments.

As it stands now from my point of view, using Mixtral and/or a mix of other lightweight 7B models like Mistral/Guanaco/Falcon/ect. has been very promising. I would say by many metrics they perform as good or better than ChatGPT3.5 or Llama2.
I have been working on making my own mix of fine tuned agent experts that can work together to solve problems and complete tasks. By leveraging open-source GPTs with fine-tuning in a particular expertise, having long-short term vector memory, local memory for embedded knowledge bases, and having the ability to run various functions or executables.

There is a lot of various approaches in this field and it is quite exciting to be able to create your own team of agents to perform digital tasks.
I am assuming you have some knowledge of python, and if so, I highly suggest learning more about Autogen, Langchain, and how to use various cloud services to train models.

You can also use Ollama to run local models that is very easy to use. Though, you have to run WSL/Ubuntu if you are using a Windows OS, as they haven't released Windows compatibility yet.
Though the model training does come at a cost depending on how much data you wish to train and how well the models take to the fine-tuned training datasets one uses.
I.E. How well organized and optimized the datasets are greatly determines the quality of the fine-tuning from my little experience on doing it.
The other option is to buy the GPU hardware needed to train the models, which also comes at a cost as I am sure you are aware of.

So far I have only tried training Gunocao-7b at a cost of about $30 bucks and it didn't turn out very well with the datasets I have on huggingface. It had too many hallucinations to be usable, but I think my next attempt will go much better since I have been going through the datasets again cleaning them further. I have had more success with knowledge embedding and retrieval functions for the LLM to be able to use than attempting to fine to a LLM.

Mathematic trainings and human feedback on the training is likely going to be challenging as there may be both national security concerns and proprietary self interest hurtles.
Which may or may not have been what the OpenAI controversy was about according to one person I talked to who works in the industry. He was of course an Anon on a Twitter space, so take that with a grain of salt. 😅
So having access to advanced mathematical reasoning is something regarded as a contentious topic in the Machine Learning communities.

GPT4 with a WolframAlpha Tool did very well though in many of my tests, but my mathematics is not at a very high level to be able to verify accuracies of complex non-linear and multi-order differential equations. So again, the work I have done is very preliminary.

Your philosophy is shared by many and is often called "the right to compute". Some say it should be amended in our social contracts and is a part of natural law on like the philosophies of personal liberty and the freedom of information.

It's great to see someone else interested and active in this type of work.
I am very much open for collaboration and all my work is open source. Hopefully you can find some value in it with your own explorations.

I have a philosophy that there is an energy freedom paradox which has emerged by our collective psychologies and imaginations of energy scarcity in the universe.
These conditioned psychological models probably evolved to help keep us alive during times of crisis, but we should recognize they exist and that they are not absolute, but collectively self-organized by our socioeconomic design.
You now, that ego thing, only on a large collective organism scale... 😁
That is an entirely separate conversation though, I digress.


Hit me up any time if you want to chat further and you are interested in working on an open-source AI project for LENR.

Cheers,

Diadon

https://jan.ai looks interesting but not tried it yet.

You might find this paper about datasets interesting. AI and LENR - The Bots Get Busy.

Quote from mkemp

Been a while but finally got back round to this, i've found Ollama to be a good project for running models locally, and you can use the continue plugin in vscode for co-pilot like experience. My Macbook Pro M2 16GB is probably too constrained to go above 7B models.
I have a pair of Nvidia Tesla P40's I can use for transfer learning but unsure of a good dataset to use or which tools to go with.

I was hoping I could add the corpus to a tool like PrivateGPT or AnythingLLM and avoid getting lost in langchain and all the sub tools and terminology since I'm not a data scientist just an engineer (Dev/SRE)
I would like to figure out an architecture and process to collate the corpus, produce the new model and push to huggingface for consumers as opensource.

Yes, going to need above 16GB+ GPU to run the larger models.
I'm excited by some of the new hardware coming online from Nvidia and in particular Groq's chip architecture.

One solution to your problem is to spin up rented hardware through one of the various "cloud" providers to fine-tune a pre-existing model.

Either way you slice it there is going to be a price tag on the compute. I would to prefer to do it locally myself, but that is a challenge in training any model, fine tuning or otherwise.
There is a lot of power in embeddings and retrieval processes as well.

You can pick a model and train it from Hugging face under the trained section.

There is even a "no-code" option called auto-train.

Just make sure you have well structed data sets for the particular model you are planning on using usually found in the data card.

Anasse Bari, David Nagel, et al. have structured data which may be useful.

I will reach out and see if there datasets are open-source or if they will allow open access to there SOLR server to make request queries to there server.
This is my first time learning about SOLR and Tika, so I'm not sure about the backend integration necessary yet.
If I understand correctly, I believe they are working on making a chat bot that uses RAG and perhaps is finetuned on the data already?
Not entirely sure tbh.

Most of it is collected via Jed's Online Library using beutifulsoup, Serpapi, and arxiv api using pandas to put the json data into tables exactly as Dave Nagel's team did.
I also integrated it with Langchain using python which did make the job a lot easier because I could use Autogpt to carry out the tasks of collecting and reviewing data in parallel and I would review the output for RLHF to ensure the data was free of error.
This did end up costing me about $100 in total on all the api calls and countless hours. 😅
It was a lot of fun though and I would have liked to use some Finetuned models before ICCF25, but as I said above, it gets expensive to train and deploy them.

If you are trying to avoid as much code as possible and having access to a local LLM, you could look at using LMstudios and the Auto-train method from HuggingFace.
Hopefully that as helpful to you and I have some datasets I have played around with both structured and un-structured on this HuggingFace if your interested.

🍻

Llama3 is now available in both 8B and 70B Parameter models.

It is Open-Source and can be ran locally and beats Mistral and Mixtral on all benchmarks.