Final week, MBW printed a extensively shared report about TikTok and its dad or mum ByteDance‘s ambitions within the area of AI-generated music.

Inside that article, we advised you in regards to the intensive analysis performed by ByteDance’s Speech, Audio & Music Intelligence (SAMI) staff within the area of AI music.

A type of analysis tasks describes a generative mannequin referred to as MeLoDy that was educated, in keeping with ByteDance’s SAMI researchers, on 257,000 hours of music (read the paper here) that got here from 6.4 million audio recordsdata.

We additionally advised you about two US patents, secured by ByteDance in latest months, that present IP safety within the US for the Chinese language firm’s AI music-related applied sciences.

Now, we’ve uncovered an further batch of AI music-related patent and trademark filings from ByteDance that may increase quite a lot of eyebrows within the music business – masking subjects together with “knowledge mining” and “music technology”.

Curiously, these US patents and trademark purposes have been filed by a mysterious ByteDance affiliate referred to as Lemon Inc.

There’s not quite a lot of details about Lemon Inc. on-line. We do know, nevertheless, that (a) it’s included within the Cayman Islands, and (b) It’s been very busy over the previous couple of years submitting purposes to america Patent and Trademark Workplace. (It even applied for a guide publishing-related trademark at the beginning of 2023).

Lemon Inc. isn’t talked about particularly on ByteDance’s company construction diagram (see beneath, and here).

Nevertheless, the World Mental Property Group, through its Arbitration reports, not too long ago described Lemon Inc. as being “affiliated with Bytedance Ltd (“Bytedance”), an Web know-how firm that gives cellular purposes (“apps”) resembling TikTok”.

Lemon Inc. is to not be confused with one other ByteDance product referred to as Lemon8, the Instagram/Pinterest rival launched within the US final yr, which some speculate is being pushed out there as a “back-up” within the occasion of a possible TikTok ban.

As reported by Axios final yr, Lemon8 was developed by Singapore-based Heliophilia Pte., Ltd., which is registered on the similar deal with as TikTok in Singapore.

Why Lemon Inc. is getting used as a car to use for logos and patents in america on behalf of ByteDance is a query for an additional day.

For now, feast your eyes on what else ByteDance has been as much as…

Lemon Inc. PAtent 1: ‘Automated and quick technology of music audio content material for movies’

In what presents the clearest glimpse at what ByteDance is perhaps planning to make use of its AI music-related tech for,  its affiliate Lemon Inc owns a patent in america for a “technique of mechanically and effectively producing music audio content material for movies”.

The patent software was filed in July 2022, granted in September 2023, and is assigned to each Lemon Inc. and ByteDance (see beneath).

In accordance with the submitting, which you’ll be able to read for yourself here, this patent covers “methods that allow environment friendly good music technology {and professional} music modifying”.

“After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video shall be mechanically generated. Such music or audio could also be royalty-free and may be of any size.”

Excerpt from patent

The submitting provides: “After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video shall be mechanically generated. Such music or audio could also be royalty-free and may be of any size.”

The submitting explains additional that the tech can extract “info from the video, whereby the extracted info includes info indicating movement pace within the video, info indicating movement saliency within the video, info indicating scene transition within the video, and timing info related to the video”.

It may well then generate a “plurality of units of music notes matching the plurality of segments of frames based mostly a minimum of partially on the extracted info utilizing a mannequin, whereby the mannequin is pre-trained and learns to correlate video movement pace with music be aware density, correlate video movement saliency with music be aware power, correlate video scene transition with music construction, and correlate video timing with music beat”.

Simply to additional spotlight the shut hyperlinks between Lemon Inc, TikTok and ByteDance, right here:

Amongst the named inventors of this tech within the patent submitting are Los Angeles-based researchers named Zhihao Ouyang and Daiyu Zhang, who in keeping with LinkedIn are a Senior Research Engineer at TikTok, and ML Research Engineer at ByteDance, respectively.

Lemon Inc Patent 2: Computing system and technique for music technology

Lemon Inc additionally has one other patent pending for a “Computing system and technique for music technology”.

This patent was filed by Lemon Inc USA in June 2022 and assigned to each Lemon Inc and ByteDance in September 2023 (see beneath). It was printed in December.

The submitting explains that “Applications have been developed that may generate music based mostly on a lyric inputted by a person” however that the music generated by such packages “usually lacks musical qualities that many individuals admire, and thus isn’t very song-like”.

It continues: “For instance, auto-generated music from such packages can undergo from misalignments in lyrics and melody notes, scattered or disjointed group and tune construction, mismatched rhythm tracks, and lack of a catchy repeating melody.

“In consequence, such packages haven’t achieved widespread use. In consequence, a barrier presently exists to speedy tune improvement utilizing such packages.”

The patent software, which you’ll be able to see here, describes “a music technology system” that options “a processor and a reminiscence operatively coupled to the processor and storing a rhythm template database comprising a plurality of rhythm templates”.

The invention additionally includes “a music technology program saved within the reminiscence and executed by the processor to be configured to obtain a person enter of lyrics, establish a plurality of syllables within the lyrics, decide a syllable sample within the recognized plurality of syllables, match the syllable sample to a specific rhythm template of the plurality of rhythm templates”.

The invention is then proposed to “generate a melody based mostly on the chosen rhythm template, generate a music file encoding the melody and the lyrics, and output the music file encoding the melody and the lyrics”.

The diagram beneath units out the way it works:

Lemon Inc.’s Emblems

Along with its lively and pending patents within the area of AI Music, Lemon Inc. has additionally filed for a number of logos in america that consult with AI, music technology and music tech usually.

A type of logos listed on america Patent and Trademark Workplace (USPTO) is for a service referred to as  ‘BytePlus‘ which seems to have been officially registered in May 2023.

TikTok and Lemon Inc dad or mum firm ByteDance launched BytePlus in 2021 reportedly as a brand new division to promote its AI know-how. ‘BytePlus’ is listed in ByteDance’s company construction diagram.

The trademark listed on the USPTO describes a “Platform as a Service (PaaS)” and covers what the submitting states are “laptop software program platforms for producing and delivering personalised suggestions to prospects within the fields of synthetic intelligence, knowledge mining, knowledge analytics, and machine studying covers a spread of AI-related actions.

The trademark additionally covers “machine-learning based mostly music technology, and machine language translation; software program consultancy providers referring to synthetic intelligence, knowledge mining, knowledge analytics, and machine studying”.

Elsewhere on the USPTO database, a trademark is registered to Lemon Inc. for Mawf, ByteDance’s machine-learning-driven music-making app that analyzes incoming audio alerts after which “re-renders” these alerts utilizing what it says is machine-learned fashions of musical devices.

The Mawf trademark was utilized for in July 2022 and officially registered by the USPTO in September.

The submitting covers “Laptop software program, specifically, downloadable software program for creating music movies, recorded music composition software program”.

Lemon Inc. has additionally filed (in Might 2023) two purposes to trademark logos for what the filling seems to explain as a music-making app (see beneath). These logos resemble the logos of ByteDance’s Ripple app.

The application described within the pair of filings, which seem like beneath overview at present, cowl “Downloadable software program for processing music and audio recordsdata; Downloadable music-composition software program; Downloadable software program for creating, recording, and modifying audio to create music and audio; downloadable software program for creating music and audio; Downloadable cellular software software program enabling customers to share music and audio they created for different customers to view, entry, and remix inside the software; Downloadable software program to be used in music and audio manufacturing“.

Each logos for various colour variations of the emblem seem to have been filed in April and Might 2023 and have been lively as of January this yr.

Lemon Inc. additionally filed a trademark software for ‘Ripple‘ in February 2023 and the mark was solely simply published on the USPO’s Trademark Official Gazette final month.

Ripple is the AI-powered music-making app launched by ByteDance in June final yr that may flip a hummed melody right into a tune.

Curiously, Lemon Inc additionally filed a trademark software again in 2021 for SoundOn, which is the DIY music distribution service launched by TikTok in March 2022.

SoundOn lets artists add their music on to TikTok and Resso, and it could additionally distribute artists’ music to platforms like Spotify and Apple Music.

And simply in case you’re nonetheless not satisfied about Lemon Inc’s affiliation with TikTok dad or mum ByteDance, Lemon Inc. utilized for a trademark for “ByteDance” in January.

MBW’s reporting on ByteDance’s and TikTok’s AI-related music actions in america arrives in opposition to a backdrop of two huge ongoing tales involving the corporate and its subsidiary TikTok.

The primary story is a looming TikTok ban in america, as a result of potential forced sale of its US operations through a invoice handed by means of the Home of Representatives final week, which is now one massive step nearer to turning into legislation.

The opposite story that we’ve been following intently is TikTok’s falling out with Universal Music Group, which refuses to resume its licensing take care of TikTok for a variety of causes, together with the app’s compensation for artists and songwriters, and UMG’s issues about TikTok’s use of AI.

On March 1, Common Music Publishing’s catalog of ~4 million songs grew to become unlicensed to be used on TikTok, becoming a member of UMG’s portfolio of ~3 million recordings, whose license on TikTok expired (up to now with out renewal) on February 1.

In a statement issued to UMPG’s songwriters on February 29, the corporate turned a lot of its consideration to the position AI-generated audio is taking part in on TikTok.

UMPG claimed that, up to now, TikTok has not supplied Common with any assurances that the platform received’t practice its AI fashions on the music firm’s songs.

MBW has been writing for a while now in regards to the hypothetical potential for TikTok and different providers to fill their catalogs with music generated by AI, which might, within the course of, dilute the market share of conventional rightsholders.

In February final yr, we printed an ‘MBW Reacts’ article asking if TikTok may pull off a “heist” on the music business on this regard, following its aggressive funding in generative AI know-how.

The “heist” we have been referring to: Utilizing licensed music as a cornerstone within the rise of TikTok to nicely over a billion customers globally, earlier than utilizing first-party, AI-created songs to crowd out music owned by conventional music rightsholders on the platform.

We wrote: “With music taking part in such a key position in TikTok’s rise, if main label content material does disappear from the platform – and the hole is someway efficiently crammed by indie and AI-driven creations – TikTok might be stated to have pulled off one of many largest heists in music enterprise historical past. A bait and change for a billion customers.”Music Enterprise Worldwide