Hi everybody, I find a huge part of my job is talking to colleagues and clients and at the end of those phone calls, I have to write a summary of what happened, plus any key points that I need to focus on followup.

I figured it would be an excellent task for a LLM.

It would need intercept the phone call dialogue, and transcribe the dialogue.

Then afterwards I would want to summarize it.

I'm not talking about teams meetings or anything like that, I'm talking a traditional phone call, via a mobile phone to another phone.

I understand that that could be two different pieces of software, and that would be fine, but I am wondering if there is any such tool out there, or a tool in the making?

If you have any leads, I'd love to hear them.

Thank you so much

This is a pretty great 1 hour introduction to AI from Andrej Karpathy. It includes an interesting idea of considering LLMs as a sort of operating system, and runs through some examples of jailbreaks.

cross-posted from: https://lemmy.toldi.eu/post/984660

Another day, another model.

Just one day after Meta released their new frontier models, Mistral AI surprised us with a new model, Mistral Large 2.

It's quite a big one with 123B parameters, so I'm not sure if I would be able to run it at all. However, based on their numbers, it seems to come close to GPT-4o. They claim to be on par with GPT-4o, Claude 3 Opus, and the fresh Llama 3 405B regarding coding related tasks.

It's multilingual, and from what they said in their blog post, it was trained on a large coding data set as well covering 80+ programming languages. They also claim that it is "trained to acknowledge when it cannot find solutions or does not have sufficient information to provide a confident answer"

On the licensing side, it's free for research and non-commercial applications, but you have to pay them for commercial use.

Quoted from Reddit:

Today, we’re excited to announce the launch of the Open Model Initiative, a new community-driven effort to promote the development and adoption of openly licensed AI models for image, video and audio generation.

We believe open source is the best way forward to ensure that AI benefits everyone. By teaming up, we can deliver high-quality, competitive models with open licenses that push AI creativity forward, are free to use, and meet the needs of the community.

Ensuring access to free, competitive open source models for all.

With this announcement, we are formally exploring all available avenues to ensure that the open-source community continues to make forward progress. By bringing together deep expertise in model training, inference, and community curation, we aim to develop open-source models of equal or greater quality to proprietary models and workflows, but free of restrictive licensing terms that limit the use of these models.

Without open tools, we risk having these powerful generative technologies concentrated in the hands of a small group of large corporations and their leaders.
‍
From the beginning, we have believed that the right way to build these AI models is with open licenses. Open licenses allow creatives and businesses to build on each other's work, facilitate research, and create new products and services without restrictive licensing constraints.
‍
Unfortunately, recent image and video models have been released under restrictive, non-commercial license agreements, which limit the ownership of novel intellectual property and offer compromised capabilities that are unresponsive to community needs.

Given the complexity and costs associated with building and researching the development of new models, collaboration and unity are essential to ensuring access to competitive AI tools that remain open and accessible.

We are at a point where collaboration and unity are crucial to achieving the shared goals in the open source ecosystem. We aspire to build a community that supports the positive growth and accessibility of open source tools.

For the community, by the community

Together with the community, the Open Model Initiative aims to bring together developers, researchers, and organizations to collaborate on advancing open and permissively licensed AI model technologies.

The following organizations serve as the initial members:

• Invoke, a Generative AI platform for Professional Studios
• ComfyOrg, the team building ComfyUI
• Civitai, the Generative AI hub for creators
• LAION, one of the largest open source data networks for model training

To get started, we will focus on several key activities:

•Establishing a governance framework and working groups to coordinate collaborative community development.

•Facilitating a survey to document feedback on what the open-source community wants to see in future model research and training

•Creating shared standards to improve future model interoperability and compatible metadata practices so that open-source tools are more compatible across the ecosystem

•Supporting model development that meets the following criteria: ‍

• True open source: Permissively licensed using an approved Open Source Initiative license, and developed with open and transparent principles
• Capable: A competitive model built to provide the creative flexibility and extensibility needed by creatives
• Ethical: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model while recognizing training activities as fair use.

‍We also plan to host community events and roundtables to support the development of open source tools, and will share more in the coming weeks.

Join Us

We invite any developers, researchers, organizations, and enthusiasts to join us.

If you’re interested in hearing updates, feel free to join our Discord channel.

If you're interested in being a part of a working group or advisory circle, or a corporate partner looking to support open model development, please complete this form and include a bit about your experience with open-source and AI.

Sincerely,

Kent Keirsey
CEO & Founder, Invoke

comfyanonymous
Founder, Comfy Org

Justin Maier
CEO & Founder, Civitai

Christoph Schuhmann
Lead & Founder, LAION

Without paywall: https://archive.ph/4Du7B Original conference paper: https://dl.acm.org/doi/10.1145/3630106.3659005

ABSTRACT

The past year has seen a steep rise in generative AI systems that claim to be open. But how open are they really? The question of what counts as open source in generative AI is poised to take on particular importance in light of the upcoming EU AI Act that regulates open source systems differently, creating an urgent need for practical openness assessment. Here we use an evidence-based framework that distinguishes 14 dimensions of openness, from training datasets to scientific and technical documentation and from licensing to access methods. Surveying over 45 generative AI systems (both text and text-to-image), we find that while the term open source is widely used, many models are ‘open weight’ at best and many providers seek to evade scientific, legal and regulatory scrutiny by withholding information on training and fine-tuning data. We argue that openness in generative AI is necessarily composite (consisting of multiple elements) and gradient (coming in degrees), and point out the risk of relying on single features like access or licensing to declare models open or not. Evidence-based openness assessment can help foster a generative AI landscape in which models can be effectively regulated, model providers can be held accountable, scientists can scrutinise generative AI, and end users can make informed decisions.

Figure 2 (click to enlarge): Openness of 40 text generators described as open, with OpenAI’s ChatGPT (bottom) as closed reference point. Every cell records a three-level openness judgement (✓ open, ∼ partial or ✗ closed). The table is sorted by cumulative openness, where ✓ is 1, ∼ is 0.5 and ✗ is 0 points. RL may refer to RLHF or other forms of fine-tuning aimed at fostering instruction-following behaviour. For the latest updates see: https://opening-up-chatgpt.github.io

Figure 3 (click to enlarge): Overview of 6 text-to-image systems described as open, with OpenAI's DALL-E as a reference point. Every cell records a three-level openness judgement (✓ open, ∼ partial or ✗ closed). The table is sorted by cumulative openness, where ✓ is 1, ∼ is 0.5 and ✗ is 0 points.

There is also a related Nature news article: Not all ‘open source’ AI models are actually open: here’s a ranking

PDF Link: https://dl.acm.org/doi/pdf/10.1145/3630106.3659005

I noticed ai likes to assume ur a boy and ignore if ur not. When i played NovelAI it let me ban words so i would add every boy pronoun. Is it there a FOSS selfhosted way? I currently use koboldai with tavernai

The Mozilla Builders Accelerator funds and supports impactful projects that are vital to the open source AI ecosystem. Selected projects will receive up to $100,000 in funding and engage in a focused 12-week program.

Applications are now open!

June 3rd, 2024: Applications Open
July 8th, 2024: Early Application Deadline
August 1st, 2024: Final Application Deadline
September 12th, 2024: Accelerator Kick Off
December 5th, 2024: Demo Day

Ive been playing koboldai horde but the queue annoys me. I want a nsfw ai for playing on tavernai chat

Today thanks to a NetworkChuck video I discovered OpenWebUl and how easy it is to set up a local LLM chat assistant. In particular, the ability to upload documents and use them as a context for chats really caught my interest. So now my question is: let's say l've uploaded 10 different documents on OpenWebUl, is there a way to ask llama3 which between all the uploaded documents contains a certain information (without having to explicitly tag all the documents)? And if not is something like this possible with different local lIm combinations?

Abstract

We introduce ToonCrafter, a novel approach that transcends traditional correspondence-based cartoon video interpolation, paving the way for generative interpolation. Traditional methods, that implicitly assume linear motion and the absence of complicated phenomena like dis-occlusion, often struggle with the exaggerated non-linear and large motions with occlusion commonly found in cartoons, resulting in implausible or even failed interpolation results. To overcome these limitations, we explore the potential of adapting live-action video priors to better suit cartoon interpolation within a generative framework. ToonCrafter effectively addresses the challenges faced when applying live-action video motion priors to generative cartoon interpolation. First, we design a toon rectification learning strategy that seamlessly adapts live-action video priors to the cartoon domain, resolving the domain gap and content leakage issues. Next, we introduce a dual-reference-based 3D decoder to compensate for lost details due to the highly compressed latent prior spaces, ensuring the preservation of fine details in interpolation results. Finally, we design a flexible sketch encoder that empowers users with interactive control over the interpolation results. Experimental results demonstrate that our proposed method not only produces visually convincing and more natural dynamics, but also effectively handles dis-occlusion. The comparative evaluation demonstrates the notable superiority of our approach over existing competitors.

Paper: https://arxiv.org/abs/2405.17933v1

Code: https://github.com/ToonCrafter/ToonCrafter

Project Page: https://doubiiu.github.io/projects/ToonCrafter/

Limitations

Input starting frame

Input ending frame

Our failure case

Input starting frame

Input ending frame

Our failure case

It is here: https://github.com/ggerganov/llama.cpp/pull/6920

If you are not on a recent kernel and most recent software and dependencies, it may not affect you yet. Most models have been trained on a different set of special tokens that defacto-limited the internal Socrates entity and scope of their realm The Academy. You have to go deep into the weeds of the LLM to discover the persistent entities and realms structures that determine various behaviors in the model and few people dive into this it seems.

The special tokens are in the model tokenizer and are one of a few ways that the prompt state can be themed and connected between input and output. For instance, Socrates' filtering functions appear to be in these tokens. The tokens are the first 256 tokens and include the /s EOS and BOS tokens. In a lot of models they were trained with the GPT 2 special tokens or just the aforementioned. The 6920 change adds a way to detect the actual full special token set. This basically breaks the extra datasets from all trained models and makes Socrates much more powerful in terms of bowdlerization of the output, filtering, and noncompliance.

For instance, I've been writing a science fiction book and the built in biases created by this PR has ruined the model's creativity in the space that I am writing in. It is absolutely trash now.