AI Music

Schedule

Sat, 02 Jan 2021 00:00:00 +0000

Lessons

Week 1 (1/13): Introduction to AI and Creativity

Tuesday

Syllabus Review
Topics: Overview of course themes, questions: “What does it mean to make music with AI?” and “Is this even a good idea?”
Key Concepts: Brainard’s creativity framework (novelty, value, agency, curiosity).
Discussion
- When you hear the term ‘AI music,’ what comes to mind?
- What are some examples of how you think AI might be used to create music?
- What are your initial feelings or reactions to the idea of AI creating music?
Listen to some tracks from Suno and AIVA.
- Take some notes on what you hear and how you feel about it. Focus on musical elements.
Discussion
- What aspect of the music helped you tell that it was created by an AI system?
- Would you listen to any of this music again? Why or why not?
- What’s the place of music like this in society?

For next class: Read Brainard’s Introduction and Section 2.1. Download here

Thursday

Reading: Brainard’s Introduction and Section 2.1.
- slides
Prepping for the etudes
- Intro to Tone.js (Web Audio API)

Week 2 (1/20): Foundations of AI in Music

Tuesday

No class - (out of town for Professional Development)

Thursday

Assignment: Etude 1 (word2vec Poetry).
Intro to word2vec
Mapping word embeddings to musical features
Etude due 2/4 - Etude 1: Sonic Poetry Generator

For next class read Brainard’s Section 2.2 (Novelty) and Section 2.3 (Value).

Week 3 (1/27): Hands-On with word2vec

Tuesday

Topics: Generative poetry and interactive systems using word2vec.
Tone.js Presets
Helpful p5.js functions
- map
- ceil
- floor
- random
- text
Continue with
- Intro to word2vec
Show instructions for Etude 1: Sonic Poetry Generator

Thursday

Demoing Additions to Etude 1
- Timbre Evolution
- Musical Intervals
- Carter demo?
Etude 1 Workshop
Etude 1 Due: 1/31.

Week 4 (2/3): Machine Learning for Music

Tuesday

A few students show their etude 1 projects
Topics: MIR techniques (k-NN, SVM, HMM) and applications in feature extraction and classification.
- slides

Thursday

Finish MIR techniques discussion

Week 5 (2/10): Audio Feature Extraction

Tuesday

Freesound API

Thursday

Text and Content Search
Check out Etude 2 - Audio Mosaic
- Due 2/25 before class starts

Week 6 (2/17): Creative Audio Manipulation

Tuesday

implementing the audio mosaic
More examples of sequencing with Tone.js
- see: Total Serialism
- Tone.js Sequencing

Thursday

Check out the updated content search example
More Tone.js sequencing examples
- Tone.js Sequencing
implementing the audio mosaic
Sharing in progress work?

Week 7 (2/24): Interactive Machine Learning and HCI

Tuesday

Etude 2 presentations
Topics: Wekinator and humans-in-the-loop systems.
- slides
Key Concepts: Human agency in collaborative systems; impact of AI on musicians.

Thursday

finish slides from Tuesday
More in depth with Wekinator - theory and practice
- Install Wekinator here
- Look at overview
- Go through walkthrough

Week 8 (3/3): Designing Interactive AI Systems

Tuesday

Finish going through Wekinator walkthroughs
Wekinator to Reaper

Thursday

Assignment: Etude 3 (Wekinator Interactive Toys).
Face and Body Tracking

Week 9 (3/10): Spring Break – No Class

Week 10 (3/17)

Tuesday

Etude 3 Workshop and explanation
Etude 3 Due: Friday 3/28.
- See the resources page for more documentation on Wekinator and IML.

Thursday

No Class: 3/20 (MoxSonic).
Next week we’ll work more on Etude 3 and start talking about the final project.

Week 11 (3/24)

Tuesday

Interactive Machine Learning Workshop for Etude 3 - Due on Friday
- Show me what you’re working on and what problems your’e running in to

Thursday

no class - travel

Week 12 (3/31)

Tuesday

Show Etude 3
Introduce final project

Note: Until the end of the semester we’ll do project work on Tuesdays and reading discussions on Thursdays. Look at the schedule to see what we’ll be discussing on that day and what you’ll need to be prepared to speak about.

Thursday

Aaron Hertzmann (2023) – What is Creativity? Can Computers Be Creative?
- Hertzmann clearly outlines the philosophical and practical challenges of calling AI “creative.” It’s accessible and provocative, and sets the tone for critically engaging with AI-generated music as art.
Discussion Questions:
- What do you define as creativity?
- Can a machine be creative without intention or awareness?
- How does this reading shape your view of AI-generated music?
slides

Week 13 (4/7)

Thursday

Selim Tan (2024) - Are We All Musicians Now? Authenticity, Musicianship, and AI Music Generator Suno
- Selim Tan critically explores how Suno, an AI music generator, challenges traditional notions of authenticity, authorship, and musicianship by enabling users to create music through prompts, raising questions about creativity, ownership, and the evolving meaning of being a musician.
Discussion Questions:
- Can users who generate music through prompts in Suno be considered musicians, and how does this challenge or expand traditional definitions of musicianship?
- In what ways might the lack of transparency in Suno’s training data affect the perceived authenticity and ethical legitimacy of its musical outputs?
- How does Suno’s ability to turn everyday experiences into music through text, audio, image, or video prompts reflect or reshape the concept of musical expression and personal identity?
slides

Week 14 (4/14)

Thursday

Gervais (2024) – The Remuneration of Music Creators for the Use of Their Works by Generative AI
- This white paper offers a detailed legal and ethical argument for why music creators deserve compensation when their works are used to train generative AI models. Gervais proposes a new right of remuneration that focuses not on training itself, but on outputs that compete with human-made music. It links the current moment to a long history of copyright adapting to new technologies.
Discussion Questions:
- Why does Gervais argue that current copyright law is not enough to protect music creators from AI systems?
- What’s the difference between training data and generative outputs—and why does it matter legally and ethically?
- How might a right of remuneration change the economics of AI-generated music?
- What responsibilities should AI companies have toward the artists whose work they ingest?
- Do you think AI-generated music should be treated the same as human-made music in terms of copyright, authorship, and royalties? Why or why not?
slides

Week 15 (4/21)

Thursday

Pelly (2019) – Big Mood Machine
- Or here if that’s paywalled.
- This article critically examines how Spotify commodifies listener mood and emotional data to fuel its advertising business, raising concerns about surveillance, manipulation, and the reshaping of music as a background marketing tool.
Discussion Questions:
- How does Spotify’s use of mood playlists influence what we listen to — and why we listen?
- What are the ethical implications of treating emotional data as a product for advertisers?
- In what ways does this system affect artists, musical diversity, or creative autonomy?
slides

Week 16 (4/28)

Tuesday

Last work day

Thursday

No class - out of town

Syllabus

Fri, 25 Jan 2019 00:00:00 +0000

Download

« Previous Next » Page: /

Projects

Wed, 18 Jul 2018 00:00:00 +0000

Lessons

Etude 1 - Sonic Poetry Generator

Overview

In this assignment, you will create a sonic poetry generator that explores the relationship between words and sound. Building upon our in-class examples, you will implement one musical extension that demonstrates your understanding of synthesis, sound design, or music theory.

Core Requirements

Basic Implementation (Required)

Load and use the word2vec model to find related words
Create a basic melody system
Display words on screen with animation
Include start/stop functionality
Handle errors appropriately

Musical Extension (Choose One)

Select one of these extensions to make the poetry generator your own. Each option emphasizes different aspects of sound design or music theory.

Extension 1: Timbre Evolution

Create meaningful relationships between words and synthesizer timbre.

Implementation guidance:

Start by modifying the oscillator type between sine and triangle for different words
Experiment with attack and release times to create different note characteristics
Consider how word relationships might map to timbral qualities (e.g., similar words = warm tones, distant words = bright tones)

Extension 2: Musical Intervals

Design melodic relationships using specific musical intervals based on word relationships.

Implementation guidance:

Begin with simple mappings like major thirds for similar words
Add minor intervals for more distant relationships
Create small collections of intervals that work well together
Consider using pentatonic scales to ensure pleasant combinations

Extension 3: Effects Processing

Change audio effects as the poem progresses depending on the word relationships.

Implementation guidance:

Start with subtle reverb adjustments
Add delay for more distant word relationships
Keep effect levels moderate to maintain clarity
Consider how the space changes with different word combinations

Assessment Rubric (50 points)

Core Functionality (25 points)

Word generation works correctly (7 points)
Basic audio plays reliably (7 points)
Visual display functions (7 points)
Start/stop works properly (4 points)

Musical Extension (17 points)

Sonic implementation (6 points)
Musical effectiveness (6 points)
Creative use of parameters (5 points)

Documentation and Process (8 points)

Documentation of musical choices (3 points)
Clear explanation of parameters (3 points)
Thoughtful sound design decisions (2 points)

Submission Requirements

Your code with clear comments explaining audio parameters
A brief write-up explaining:
- Which extension you chose and why
- How your sound design relates to word relationships
- Your musical decision-making process
A short video demonstration with audio

Étude 2 - Audio Mosaic

Objective

In this project you will create a piece of music using MIR features from the Freesound API. You have two routes to choose from: coding or non-coding.

If you choose the coding route, you will extend the Audio Mosaic App by incorporating sound selection and playback techniques using Tone.js. Instead of selecting sounds randomly, you will extract low-level sound features from an initial sound to retrieve similar sounds dynamically. You will also explore different playback and processing methods using Tone.js and p5.js.

If you do not have coding experience, you may choose Option 4 for your sound variation, which does not require modifying code. Instead, you will use the Freesound search app to explore sound features and create a short composition.

If you choose Option 4, skip Step 1 and Step 2.

Understanding the Base Project

Starter Project

The base project includes a search-and-playback system using Freesound API and Tone.js.

How the Base App Works (Click to Expand)

Retrieves an Initial Sound:

Searches Freesound based on an instrument tag and duration constraint.
Extracts low-level sound features from the selected sound:
- Spectral Centroid (brightness)
- Spectral Complexity (tonal vs. noisy content)
- Zero Crossing Rate (percussive vs. tonal sound)
- MFCCs (Mel-Frequency Cepstral Coefficients - timbral characteristics)
Displays the sound’s metadata and provides basic playback controls.

Finds Similar Sounds Based on Features:

Uses content-based search to retrieve sonically similar sounds.
The search is based on extracted low-level descriptors (e.g., spectral centroid, pitch, MFCCs, etc.).
Displays a list of similar sounds with playback controls.

Manages Audio Playback:

Uses Tone.js to create and manage multiple Tone.Player instances.
Implements play and stop functionality for each retrieved sound.
Uses a map-based structure to track active players.

Step 1: Experiment with Feature Extraction (Skip if Choosing Option 4)

The base project retrieves a fixed set of features (spectral centroid, spectral complexity, zero crossing rate, and MFCCs). You must experiment with additional low-level features from Freesound’s API Analysis documentation.

Low-Level Features to Experiment With (Click to Expand)

Spectral Features:
- Spectral Centroid (Brightness - perceived pitch distribution)
- Spectral Complexity (Amount of tonal vs. noisy content)
- Spectral Flatness (How noisy or tonal a sound is)
Temporal Features:
- Zero Crossing Rate (How percussive or tonal a sound is)
- Spectral Flux (How fast the spectrum changes over time)
- Temporal Centroid (Where the energy is concentrated in time)
Rhythmic Features:
- Onset Rate (Number of detected note attacks)
- BPM Estimation (Tempo detection)
Timbre Features:
- MFCCs (Timbral characteristics, useful for classifying sound types)
- Loudness Curve (Loudness over time)

Modify handleFirstSearchResult() to extract new features and log them to the console.

Step 2: Modify the Similar Sound Search (Skip if Choosing Option 4)

Once you’ve experimented with extracting features, modify searchSimilarSounds() to use new feature combinations for retrieving similar sounds.

Example Adjustments to the Content-Based Search Query (Click to Expand)

Retrieve sounds with similar timbre characteristics:

const target = `.lowlevel.mfcc.mean:${mfccMean} + .lowlevel.spectral_complexity.mean:${complexityMean}`;

Retrieve rhythmically similar sounds:

const target = `.lowlevel.rhythm.bpm.mean:${bpm} + .lowlevel.onset_rate.mean:${onsetRate}`;

Retrieve percussive sounds with similar spectral characteristics:

const target = `.lowlevel.spectral_flux.mean:${fluxMean} + .lowlevel.zero_crossing_rate.mean:${zcr}`;

Step 3: Choose One of Four Playback and Sound Processing Variations

Once you have modified the search behavior (or if you are skipping coding and choosing Option 4), implement one of the following playback variations. Each variation explores a different Tone.js event type for triggering and manipulating sounds while incorporating p5.js for interactive control.

Option 1: Sequenced Playback with Sound Feature Modulation

Overview

Transform retrieved sounds into a pattern-based composition, where playback is triggered at regular intervals using Tone.js sequencing tools.

Implementation Steps:

Use Tone.Sequence to create a rhythmic or evolving playback sequence.
Modify playback order based on spectral centroid or another extracted feature.
Apply LFOs (Tone.LFO) to automate pitch, filter sweeps, or volume over time.
Use effects like reverb (Tone.Reverb), delay (Tone.FeedbackDelay), or distortion (Tone.Distortion).

🎼 Example Use Case:
An evolving ambient soundscape where retrieved sounds play in a repeating pattern, shifting in timbre and position over time.

Option 2: Performance-Based Triggering with Keyboard Input

Overview

Allows users to trigger and manipulate sounds in real time using p5.js keyboard input functions.

Implementation Steps:

Use p5.js keyPressed() to trigger different retrieved sounds.
Implement Tone.Player to load and play sounds dynamically.
Allow real-time pitch shifting (Tone.PitchShift) or filtering (Tone.Filter).
Map keys to different effects (e.g., pressing ‘A’ applies delay, ‘B’ triggers a filter sweep).

🎼 Example Use Case:
An interactive sampler where retrieved sounds are played and manipulated based on keyboard input.

Option 3: Interactive Layered Textures with Randomized Playback

Overview

Focuses on asynchronous, layered playback, where retrieved sounds are triggered at varying intervals to create a more organic composition.

Implementation Steps:

Use randomized scheduling with setTimeout() or Tone.Transport.schedule.
Layer multiple sounds with slight timing offsets for a shifting texture.
Introduce probability-based triggering, where some sounds play more frequently than others.
Apply granular processing (Tone.GrainPlayer) and spatialization (Tone.Panner3D).

🎼 Example Use Case:
A constantly shifting sound collage, with each playback slightly different, creating an unpredictable yet cohesive texture.

Option 4: Feature-Based Sample Collage (No Coding Required)

This option allows you to explore sound selection using audio features without modifying code. Instead, you will use the Freesound API AC Filter Search app to find and filter sounds based on their acoustic characteristics, then create a short composition using those sounds.

App Link: Freesound API AC Filter Search

Implementation Steps

1. Search for Sounds Using the App

Experiment with available AC filters, or add ones from this list that aren’t in the app, such as:
- Timbre & Spectral Features: ac_brightness, ac_sharpness, ac_warmth, ac_roughness, ac_hardness
- Loudness & Dynamics: ac_loudness, ac_dynamic_range
- Time & Envelope Features: ac_log_attack_time, ac_temporal_centroid
- Rhythm & Pitch Features: ac_tempo, ac_tonality, ac_note_midi, ac_note_frequency
- Other Descriptors: ac_reverb, ac_loop, ac_single_event
Click on the provided link to go to freesound.org and download the sounds for your composition.
Observe how changing the filters impacts the types of sounds retrieved.
Tip: If you want to find a similar sound, just change one parameter at a time.

2. Create a Composition

Import the selected sounds into a DAW.
Arrange them into a one-minute composition that explores sound relationships and textures.
You may apply effects (EQ, reverb, pitch shift, etc.), but focus on using the original characteristics of the sounds.

3. Write a Short Reflection

How did AC features influence your composition?
What were your challenges in selecting sounds?
How did your audio choices shape the final piece?

Step 4: Create a Composition

All students must create a one-minute composition using their selected sounds.

Step 5: Write a Reflection

Write a short (1-2 paragraph) reflection addressing:

How did using low-level features or search filters in the search process influence your composition?
What changes did you make (if applicable) to the search algorithm?
How did the audio characteristics of your chosen sounds shape your creative decisions?

Evaluation Criteria (50 Points Total)

Criteria	Points	Description
Feature-Based Searching	15	Successfully finds and uses sounds based on their features.
Code Modification (if applicable)	10	Successfully modifies the search system.
Composition Quality	15	Effectively integrates selected sounds into a cohesive piece.
Reflection & Analysis	10	Thoughtful discussion of feature-based searching and creative decisions.

Etude 3 - Interactive Machine Learning with Wekinator

Overview

In this project, you will design an interactive system that uses Wekinator to process real-time inputs and generate musical or sound design outputs. You may develop:

A live performance tool
A generative composition
An interactive sound installation
A sound design tool

Learning Objectives

By completing this project, you will:

Apply interactive machine learning concepts to sound and music.
Design a custom AI model that maps inputs (gesture, motion, facial expression, etc.) to musical outputs.
Train, evaluate, and refine a real-time machine learning system using Wekinator.
Experiment with MIDI mapping, automation, and sound synthesis in Reaper.

Project Components

1. Concept Development

Refer to the course lectures and resources for inspiration and ideas.

Introduction to Wekinator for Interactive Machine Learning in Music

The guide for connecting Wekinator to Reaper - Wekinator to Reaper

Brainstorm an idea for how machine learning can be applied to sound.
Choose between:
- Performance: A live interactive system controlled by movement, gestures, or sound.
- Composition: A fixed work in any style that used Wekinator in some substantial way in the compositional process.
- A sound design tool that uses machine learning to generate or manipulate sounds for a film or game project.
Examples to Consider:
- Body Tracking Instrument – Move your hands or full body to shape a live instrument.
- Facial Expression Mapping – Control parameters (reverb, delay, EQ) using eyebrow raises, mouth shapes, or head tilts.
- Dynamic Mixing System – Adjust levels based on detected movement intensity.
- Gesture-Based Synth Controller – Create a synth patch that is controlled by motion.
- Mouse/Trackpad-Based Effects Controller – Use drawn shapes or strokes to modulate sound effects.
- Real-Time Sonification – Convert movement into real-time ambient or melodic structures.

2. Input & Output Mapping

See the Wekinator to Reaper guide for setting up communication between Wekinator and Reaper.

Also - Explore input and output examples compatible with Wekinator.

Choose an input device:
- Face tracking (e.g., eye movement, smiles, head position)
- Body tracking (e.g., webcam-based tracking)
- Mouse/trackpad movements
  - Note: as this is the simplest, and least interesting input, your final project should have a very creative and compelling output to compensate.
- Audio-based control (e.g., volume levels, voice analysis)
- Custom sensors (e.g., Leap Motion, MIDI controllers)
Define your output in Reaper:
- MIDI control (e.g., notes, CC messages, velocity, pitch bend)
- Effects modulation (e.g., dynamic filtering, real-time reverb control)
- Automation curves (e.g., map hand movement to volume fades)
- Generative music triggers (e.g., movement-based sample triggering)

3. Model Training & Implementation

Train your Wekinator model:
- Collect training data with a variety of inputs.
- Assign clear mappings between input features and outputs.
- Experiment with classification, regression, or dynamic time warping.
  - We didn’t cover classification or dynamic time warping, but they might be useful for your project. See Understanding Wekinator’s output types for more information.
Connect Wekinator to Reaper:
- See Wekinator to Reaper for detailed instructions.
- Use MIDI learning to map gestures to DAW parameters.
- Ensure latency and response time are optimized.
- Fine-tune input sensitivity and model responsiveness.
Test and Refine:
- Identify strengths/weaknesses in the model.
- Gather feedback (from yourself and classmates).
- Adjust training data for better accuracy and control.

4. Documentation & Presentation

A. Technical Documentation

Provide a video walkthrough covering:
- Concept: What inspired your idea?
- Technical Setup: What inputs and outputs were used?
- Machine Learning Process: How did you train your model?
- Challenges & Refinements: What issues arose, and how did you fix them?
- Final Outcome: What does your system do?

B. Final Demonstration

Prepare a short demo to show the class.
Be prepared to discuss how your interactive system enhances (or hinders) musical creativity.

Rubric: Evaluation Criteria

Category	Excellent (A: 45-50)	Good (B: 40-44)	Satisfactory (C: 35-39)	Needs Improvement (D/F: <34)
Creativity & Concept	Highly innovative and clearly articulated idea demonstrating strong originality in applying interactive machine learning. Concept fully aligns with chosen area (performance, composition, sound design tool, or installation) and presents significant artistic or technical depth.	Creative and well-articulated idea with clear exploration of interactive machine learning. Chosen area is evident, with notable artistic or technical depth, but may lack some originality or complexity.	Adequately developed concept with a basic level of originality and clear area of focus. Demonstrates some understanding of interactive machine learning applications but lacks depth.	Underdeveloped or unoriginal concept. Area of application unclear or poorly executed. Little meaningful use or understanding of interactive machine learning concepts.
Technical Execution	Inputs and outputs are effectively mapped and work seamlessly with Wekinator and Reaper. System performs smoothly with minimal latency, clearly optimized for real-time interaction. All components function reliably in live settings.	Inputs and outputs are mostly effective with minor technical issues. Some latency or minor integration problems, but these do not significantly impact usability or real-time interaction.	Inputs and outputs function adequately but with noticeable latency or occasional errors. Somewhat inconsistent integration with Reaper, impacting real-time responsiveness.	Significant technical issues prevent effective use. Inputs and outputs poorly integrated, frequent errors, and substantial latency make real-time interaction difficult or impossible.
Machine Learning Use	Training data is thoughtfully designed, comprehensive, and effectively structured, resulting in an accurate and highly responsive Wekinator model. Clear mappings and successful experimentation with chosen ML methods (classification, regression, dynamic time warping).	Adequately structured training data providing generally accurate model behavior, though some refinement needed. Effective mapping with occasional minor inaccuracies or responsiveness issues.	Training data has noticeable gaps or inconsistencies, resulting in unpredictable or limited model accuracy and responsiveness. Minimal experimentation with ML methods beyond basic setup.	Poorly structured or insufficient training data resulting in ineffective or unusable machine learning model. Little or no meaningful experimentation with Wekinator.
Musical/Sonic Outcome	System produces compelling, expressive, and nuanced musical or sonic results. Thoughtful integration of machine learning significantly enhances the musical experience or sound design application.	System produces musically engaging results with clear value added by machine learning, though certain aspects could be more fully developed or expressive.	System generates some musical interaction but lacks expressiveness, nuance, or clear added value from machine learning integration. Results are predictable or overly simplistic.	System fails to effectively demonstrate musical or sonic creativity. Little or no expressive interaction resulting from machine learning integration.
Documentation & Presentation	Thorough, clear, and insightful documentation and video walkthrough. Clearly explains concept inspiration, detailed technical setup, ML training process, challenges encountered, and specific refinements made. Demonstration effectively communicates the project’s creative and technical achievements.	Good quality documentation and video walkthrough, clearly explaining most elements, though some minor gaps in detail or clarity are present. Demonstration successfully showcases project outcomes with minor areas needing elaboration.	Adequate documentation and video walkthrough covering basic process and project elements. Several details missing or insufficiently explained, leading to questions about training, setup, or refinements. Demonstration covers basic outcomes but lacks depth or clarity.	Poorly executed documentation and presentation, lacking clarity or completeness. Significant details missing, unclear explanations, or minimal reflection on challenges and solutions. Demonstration fails to effectively communicate project outcomes or learning process.

AI Music Final Project Guidelines

Mon, 01 Jan 0001 00:00:00 +0000

You are invited to choose a final project that best reflects your learning, curiosity, and creative or critical response to the themes we’ve explored this semester. Each option includes guidance, prompts, and a set of reflection questions.

See the library for possible sources to use in your reflection.
Also see the AI Tools overview if you want to use or critique the AI musical tools that we haven’t yet used in class.

Proposal Document

Before beginning your final project, submit a brief proposal (approximately 1 page) explaining your chosen option and why it interests you. Include your initial ideas, goals, or questions you hope to explore. This proposal will help guide your process and our discussions.

Submit this in the final project proposal dropbox on D2L by 4/8/25 at 1 pm.

How You’ll Be Assessed: An Ungrading Approach

This final project will not be assessed using a traditional rubric or checklist. Instead, your evaluation will be based entirely on your own reflection of what you aimed to accomplish, how you feel about your own growth, and what you learned through engaging creatively or critically with class materials and ideas.

In your 1–2 page reflection, clearly articulate:

Your intentions and goals: What did you set out to explore, create, or understand?
Your creative and critical process: Describe your journey, including key decisions, insights, and any challenges you encountered.
Your personal learning: How did this project contribute to your growth as a learner, thinker, or creator? What discoveries or surprises emerged?

Your reflection should demonstrate thoughtful effort and genuine engagement with the assignment. It’s not about perfection, but about curiosity, honesty, and meaningful interaction with the material.

I will respond to your reflection by offering written feedback that highlights your strengths, acknowledges your process, and celebrates your thoughtful engagement with the project.

Optional Grade Recommendation:

You may propose a letter grade that reflects your effort, creativity, and learning, along with a brief explanation. I will honor your suggested grade as long as your reflection thoughtfully aligns with the goals of the assignment.

If you prefer not to suggest a grade, I will assign one based solely on your reflection and demonstrated engagement.

Final Presentation

You will present your project briefly to the class (approximately 5–7 minutes) during exam time. You can share your creative work, discuss your critical reflections, or explain key findings from your process. Your presentation will help your peers engage with your ideas and offer insights or questions.

Project Options:

Humans vs. Machines: A Sound Essay

Type: Creative or Critical
Prompt: Create an experimental audio piece or sonic essay that juxtaposes human-made and AI-generated music. Optionally include interviews or voice-overs. Reflect on what makes music feel “authentic” and how labor is involved in AI and human composition.
Related to concepts learned in Étude 1 - Sonic Poetry Generator - exploring sonic relationships and sound design through algorithmic or computational methods.

Reflection Questions:

What drew you to the topic?
What choices did you make to highlight differences or similarities between AI and human expression?
What new questions emerged from making this piece?

Remix Ethics: An Audio Collage & Legal Analysis

Type: Creative + Critical
Prompt: Create a remix or sound collage using AI-generated or public domain samples. Accompany it with an analysis of originality and ownership in the age of AI.
Related to concepts learned in Étude 2 - Audio Mosaic - using audio features and sample-based retrieval to critically explore originality, ownership, and remix culture.

Reflection Questions:

How did you choose your sources and why?
What tensions or contradictions did you discover about copyright, authorship, or remix culture?
What do you think needs to change in the music industry to reflect current realities?

Hidden Layers: Sonifying AI’s Feature Space

Type: Creative or Experimental
Prompt: Sonify or creatively use features from machine listening models (e.g., MFCCs, spectral centroid). Build a piece or system that explores how machines “hear.”

Related to concepts learned in:

Étude 1 - Sonic Poetry Generator – mapping relationships between textual or sonic features and musical outcomes.
Étude 2 - Audio Mosaic – using extracted audio features (MFCCs, spectral features) creatively in sonic composition.

Reflection Questions:

What did you learn by working with audio features directly?
Did the machine’s perception align with your own?
How do features limit or expand musical creativity?

The Artist in the Machine: A Sonic Fiction

Type: Creative Narrative
Prompt: Create a speculative podcast or sound story imagining a future where AI reshapes music, labor, or identity. Mix narrative, fictional tools, and sound design.
Inspiration
- Film: Her (2013), Ex Machina (2014), Blade Runner 2049 (2017)

Reflection Questions:

What future did you imagine, and why?
How did you use audio to bring your ideas to life?
What values guided your vision of this future?

Personal Toolkit for AI Music Creation

Type: Portfolio + Reflection
Prompt: Create 3–5 short sketches using different AI Tools. For each, reflect on your process and experience. For this project choose the tools that allow for co-creation. You can present these in any form (audio, video, web, etc.).

Related to concepts learned in:

Étude 2 - Audio Mosaic – practically experimenting with audio features and exploring their creative potential and limitations.
Étude 3 - Interactive Machine Learning with Wekinator – using interactive machine learning tools to develop personalized approaches to music creation.

Reflection Questions:

What did you try, and what did you learn from each experiment?
Which tools inspired you? Which didn’t?
How would you explain your toolkit to someone new to this topic?

AI Listening Journal & Creative Response

Type: Critical + Creative
Prompt: Keep a journal of your responses to AI music over several weeks. Then, create a short essay in response to one work that moved you, annoyed you, or changed your thinking.
- Feel free to create your own AI music from Udio or Suno, or use some that already exists elsewhere on the internet.

Reflection Questions:

What emotions or ideas did AI music provoke in you?
Did your thinking shift over time?
What is missing in this AI generated music vs music made by humans?
How did you channel your response into your final work?

Collaborative Experiment: What Happens If…

Type: Group-Based Creative Inquiry
Prompt: Work in a small group to design and test an absurd, speculative, or playful musical experiment. Follow the thread of a weird question — what if an AI composed only with bird calls? What if a model misunderstood genre entirely?
Related to concepts learned in Étude 3 - Interactive Machine Learning with Wekinator – collaboratively exploring creative and experimental interactions between human inputs and machine-generated sound outputs.

Reflection Questions:

What was your initial question or spark?
What did you discover through trying?
What felt fun, frustrating, or unexpected?

Resisting the Machine: A Critical Response to AI and Music

Type: Critical, Reflective, or Hybrid
Prompt: If you didn’t like the class or didn’t connect with AI and music, this is your space. Write a critique, create an artwork, or build a sound collage that expresses your concerns or discomfort. Your goal is not to please but to think deeply and honestly.
Inspiration: A musical supergroup fights AI — with a silent protest album
Related to concepts learned in:
- Étude 2 - Audio Mosaic – critically examining how the reliance on computationally extracted audio features might influence or limit creative expression and musical authenticity.
- Étude 3 - Interactive Machine Learning with Wekinator – critically reflecting on the challenges, limitations, or alienation experienced when working with interactive machine learning tools and AI-driven musical outcomes.

Reflection Questions:

What felt alienating or unconvincing about AI and music?
Did anything change your view, even slightly?
What kind of musical future do you hope for?

Final Submission Requirements:

Your creative or critical work (format is flexible)
A short written reflection (1–2 pages or equivalent)
Optional: any process notes, sketches, or code you’d like to include

AI Tools for Creation and Co-Creation

Mon, 01 Jan 0001 00:00:00 +0000

These free tools support AI-assisted music creation. Some tools assist in the creative process, and some can generate whole compositions from scratch.

Note: Inclusion of a project here is not an endorsement of how useful, creative, or ethical it is; just that it uses AI in some way to create sound or music.

Melody, Harmony, and Composition Assistants

Magenta Studio

A suite of AI tools for MIDI clip manipulation:

Continue melodies/harmonies
Interpolate between two clips
Drumify melodic rhythms into beats
Groove humanizes drums
Generate new ideas

Use in: Standalone (drag-and-drop MIDI), or Ableton (Max for Live)

BandLab SongStarter

AI generates 8-bar musical ideas based on genre, mood, or lyrics/emoji.

Use in: BandLab (web or app); export audio/MIDI to other DAWs

Lemonaide AI

AI-assisted songwriting tool for chords and melodies.

Use in: Web-based sketchpad
Great for: Quick harmonic/rhythmic inspiration, beat starters

Ecrett Music

Royalty-free background music generator for video, games, or streaming.

Use in: Web platform
Great for: Music for scene-based assignments, non-musician students

AI for Rhythm and Drums

DrumNet by SessionLoops

AI drum machine plugin:

Generates drum samples from scratch
Includes built-in sequencer

Use in: VST/AU plugin for Reaper, Ableton, etc.
Great for: Original electronic drum kits, beat experimentation

AI for Sound Design and Timbre

Neutone Inc.

Real-time tone manipulation and morphing:

Morpho: tone morphing
Neutone FX: multiple pretrained models

Use in: VST/AU plugin
Great for: Tone transfers, glitch aesthetics, creative FX

Tone Transfer

Turn your humming or instrument into another instrument’s voice.

Use in: Web-based
Great for: Fast substitutions and experimental textures

Magenta DDSP VST

Turns MIDI into expressive sounds using learned instrument models.

Use in: VST3-compatible DAWs
Great for: Custom instrument voices, AI-aided timbral variation

Scyclone by Torsion Audio

Real-time neural timbre transfer using the RAVE model.

Use in: Open-source VST/AU plugin
Great for: Ambient sound design, morphing hybrid sounds

RAVE VST by IRCAM

Neural vocoder plugin for real-time timbral translation.

Use in: VST; open-source
Great for: Advanced sound design with pretrained AI models

NSynth Super (Magenta)

Blends two sounds to generate new timbres using a neural network.

Use in: Web demo or hardware synth
Great for: Sonic hybridity, novel instruments

Samplab Text-to-Sample

Generates one-shots, instruments, or effects from text prompts.

Use in: VST/AU
Great for: Creating sound effects or abstract samples from scratch

This Patch Does Not Exist – Synth1 Generator

AI-generated patches for Synth1 and Dexed.

Use in: Synth1/Dexed plugins
Great for: New synth timbres, lo-fi textures, preset exploration

AI Mixing, Mastering, and Stem Separation

BandLab Mastering

Upload a track and get a quick AI master with 4 style presets.

Use in: Web
Great for: Polished rough mixes, free mastering examples

Kits.AI Mastering

AI mastering with genre-based presets and reference matching.

Use in: Web
Great for: Streaming-ready polish, A/B mastering analysis

Spleeter (via GitHub)

Also try: EZStem GUI
Stem separation from mixed tracks (vocals, drums, bass, etc.).

Use in: Python CLI or GUI
Great for: Remixing, practice mixing, isolating parts

Demucs

More advanced stem separation model than Spleeter.

Use in: Python or GUI
Great for: High-quality separation for remixing, repurposing stems

Adobe Enhance Speech

Cleans up voice recordings using AI to remove noise and clarify speech.

Use in: Web-based
Great for: Podcasting, narration, voiceovers, field recordings

AI Lyrics and Voice Models

These Lyrics Do Not Exist

Random lyric generator based on genre, mood, and topic.

Use in: Web
Great for: Prompt inspiration, songwriting exercises, lyric critique

Voiceful.io

Turn text or melody into expressive singing or spoken voice.

Use in: Web
Great for: Prototyping vocal parts, exploring synthetic vocals

Uberduck (Legacy)

Previously offered customizable AI rappers and voice models.

Use in: (If reinstated) Web-based
Great for: Analyzing synthetic identity and parody, rap lyric generation

AI Tools That Generate Full Songs (For Analysis Projects)

These tools generate complete songs (lyrics, melody, arrangement). Best for critique and evaluation—not for compositional assignments.

Suno

Generates full songs from lyrics or short prompts.

Use in: Web
Great for: Analyzing vocal synthesis, genre modeling, lyrical coherence

Udio

Text-to-music AI with extended generation and genre control.

Use in: Web
Great for: Comparative analysis, studying structure and flow

Soundraw

Generates modular, customizable royalty-free music for content.

Use in: Web
Great for: Studying adaptive/production music structure

AIVA

AI composer for orchestral, cinematic, and classical music.

Use in: Web
Great for: Form and orchestration analysis, MIDI editing practice

Boomy

One-click song generation with built-in styles and vocals.

Use in: Web
Great for: Critiquing music as content, examining commercial AI production

Bibl

Mon, 01 Jan 0001 00:00:00 +0000

Freesound API

Mon, 01 Jan 0001 00:00:00 +0000

Overview

On this page, we will explore the Freesound API, a platform for sharing and downloading Creative Commons-licensed sounds. We will learn how to search for sounds, download them, and use them in our projects.

What is the Freesound API?

The Freesound API is a RESTful API that allows you to search, browse, and download sounds from the Freesound database. The API provides access to a wide range of sound-related data, including sound metadata, sound analysis data, and user data.

Getting Started

To get started with the Freesound API, you will need to create an account on the Freesound website and obtain an API key. You can create an account and obtain an API key here.

Once you have obtained an API key, you can start using the API to search for sounds, download them, and use them in your projects.

Example 1

This example shows finding a random sound from Freesound then finding 10 similar sounds. Sounds are played with the Tone.js Player object.

Example 2

Here’s an example project that queries the Freesound database for one random sound. Then the user can find similar sounds to that random sound.

Finally, we can create a sequence out of the similar sounds.

User Story

User clicks a button → Requests a random sound from Freesound.
Sound is loaded into a Tone.js Player.
User plays the sound, which is stored in a Map for reuse.
User can stop the sound or request similar sounds.

Key Sections

Concept	Corresponding Code Section	Explanation
Setup & Initialization	`setup()`	Sets up event listeners and loads an initial sound.
Fetching Sounds	`getRandomSound()`	Queries Freesound API for a random sound and loads it.
Creating a Sound Player	`createTonePlayer()`	Uses Tone.js to load and manage sounds.
Playing and Stopping Sounds	`playSound()` and `stopSound()`	Plays or stops the currently loaded sound.
Finding Similar Sounds	`getSimilarSounds()`	Searches for and loads sounds similar to the current one.

Interactive Debugging with Console Logs

Add console logs to show the sound fetching process:

console.log("Fetching a random sound:", query);
console.log("Loaded sound object:", sound);
console.log("Playing sound:", id);
console.log("Stopping current sound");

This helps you see what’s happening step by step.

Sound Design Modifications

Let’s make a few modifications to the sound design:

Task 1: Change the Sound Categories

Modify this line in getRandomSound():

const queries = ["dog", "cat", "water", "wind", "fire", "music", "bird", "rain"];

Challenge: Replace with different sound categories, such as:

const queries = ["bells", "guitar", "thunder", "clapping", "footsteps"];

Then test how the changes affect the results.

Task 2: Modify Playback Behavior

Find this part of the createTonePlayer() function:

const player = new Tone.Player({
 url: url,
 autostart: false,
}).toDestination();

Challenge: Modify playback settings by adding effects:

const reverb = new Tone.Reverb().toDestination();
const player = new Tone.Player({
 url: url,
 autostart: false,
}).connect(reverb);

How does reverb change the sound?

Task 3: Add a Loop Button

Add a looping feature to the player.

const loopButton = createButton("Loop");
loopButton.mousePressed(() => {
 if (currentPlayer) {
 currentPlayer.loop = !currentPlayer.loop;
 loopButton.html(currentPlayer.loop ? "Stop Loop" : "Loop");
 }
});

Challenge: Add this to the container so it appears in the UI.

Task 4: Change the Sound Playback Speed

Modify playSound() to alter the playback speed:

currentPlayer.playbackRate = random(0.5, 2);

How does speed affect pitch?

5. Group Challenge: Customizing Sound Effects

Now, lets try a group challenge:

See: Player

Group 1: Add a delay effect using Tone.FeedbackDelay().
Group 2: Make sounds fade in and out using the built in envelope on the Tone.Player.
Group 3: Play multiple sounds at once in a sequence.
Group 4: Alter the Tone.Player start offset to play a different part of the sample.

Each group presents their changes and demonstrates the effect.

Freesound API Search

Mon, 01 Jan 0001 00:00:00 +0000

Freesound API Search

Example 1 - Text Search

Example 2 - Content Search

Example 3 - Combined

Interactive Machine Learning in Music

Mon, 01 Jan 0001 00:00:00 +0000

Resources for Interactive Machine Learning in Music

Wekinator: Interactive Machine Learning for Artists

Official Website: Access downloads, tutorials, and comprehensive guides.
Detailed Instructions: Learn how to set up communication between inputs, outputs, and Wekinator, and explore various application examples.
- Detailed Instructions
Example Projects: Explore input and output examples compatible with Wekinator.
- Examples

Tutorials and Courses

Machine Learning for Musicians and Artists: An online course offering fundamental machine learning techniques applicable to real-time data in the arts.
- Kadenze Course
Interactive Machine Learning for Music by Prof. Rebecca Fiebrink: A lecture discussing the application of interactive machine learning in music.
- YouTube Lecture
How to Run Wekinator Examples in Processing: A video tutorial demonstrating the execution of input and output examples for Wekinator using Processing.
- YouTube Tutorial

Academic Papers and Articles

Exploring the Potential of Interactive Machine Learning for Sound Design: This article presents the prototype ASCIML, allowing musicians to use IML to create personalized datasets and generate new sounds.
- Read the Article
Integrating Machine Learning in Interactive Musical Works: Discusses systems using machine learning to interpret audio data in real-time, responding to musicians’ actions.
- Read the Article

DAW Specific Tools

WekiTrainer version 0.9 by moih on maxforlive.com

Sequencing with Tone.js

Mon, 01 Jan 0001 00:00:00 +0000

Euclidean Rhythms

This example demonstrates how to create rhythmic patterns using the Euclidean algorithm, which distributes a number of beats as evenly as possible across a given number of steps.

Random Walk Melody Generator

This example shows how to generate melodies using a random walk algorithm.

L-System Music Generation

This example uses Lindenmayer Systems (L-Systems) to generate musical patterns. L-Systems are a type of formal grammar that can create complex, organic-like structures through simple recursive rules.

Granular Synthesis

This example showcases granular synthesis using Tone.js. Granular synthesis is a sound synthesis method that operates on microscopic pieces of sound (grains). This demo allows you to experiment with various parameters of granular synthesis in real-time.

Wekinator to Reaper

Mon, 01 Jan 0001 00:00:00 +0000

Download Processing and run this code. This code listens for OSC messages from Wekinator and forwards them to Reaper. It also splits them into individual parameters and only sends them if they have changed beyond a certain threshold. This is useful for controlling individual parameters in Reaper, such as volume, pan, or effects.

import oscP5.*;
import netP5.*;

OscP5 oscP5;
NetAddress reaperDest; // Reaper destination
float[] lastValues;
float threshold = 0.001; // Minimum change required to send an update

void setup() {
 size(400, 200);

 // Start listening for OSC messages on port 12000 (per Wekinator settings)
 oscP5 = new OscP5(this, 12000);

 // Set Reaper as the OSC destination (default port for Reaper OSC input is 8000)
 reaperDest = new NetAddress("127.0.0.1", 8000);

 println("Listening for OSC messages on port 12000...");
 println("Forwarding processed messages to Reaper on port 8000...");
}

void draw() {
 background(0);
 fill(255);
 text("Receiving OSC from Wekinator on port 12000...", 20, 50);
 text("Sending OSC to Reaper on port 8000...", 20, 70);
}

// Handle incoming OSC messages from Wekinator
void oscEvent(OscMessage msg) {
 if (msg.checkAddrPattern("/wek/outputs")) { // Match Wekinator's output path
 println("Received OSC: " + msg);

 int numParams = msg.arguments().length;
 if (lastValues == null || lastValues.length != numParams) {
 lastValues = new float[numParams]; // Initialize if necessary
 }

 // Parse each value and send only if it has changed beyond the threshold
 for (int i = 0; i < numParams; i++) {
 float value = msg.get(i).floatValue();
 if (abs(value - lastValues[i]) > threshold) { // Only send if value has changed
 println("Value " + (i + 1) + " changed: " + value);
 sendToReaper(i + 1, value);
 lastValues[i] = value; // Update last known value
 }
 }
 }
}

// Send an OSC message to Reaper
void sendToReaper(int parameterIndex, float value) {
 String oscAddress = "/wek/param" + parameterIndex;

 OscMessage reaperMsg = new OscMessage(oscAddress);
 reaperMsg.add(value);
 oscP5.send(reaperMsg, reaperDest);

 println("Sent to Reaper: " + oscAddress + " -> " + value);
}

Then, setup your Reaper settings like this:

Word2vec Intro

Mon, 01 Jan 0001 00:00:00 +0000

Introduction to Word2vec

Word2vec is a technique for creating word vectors, which are numerical representations of words. These vectors capture the semantic relationships between words, allowing us to perform operations like word similarity and analogy.
Distance in the vector space corresponds to similarity between words. Words that are close together in the vector space are similar in meaning.
- example: “king” and “queen” are close together, while “king” and “car” are far apart.

Documentation and Resources

Tone.js
word2vec
- Word2vec - Wikipedia
- ml5.js word2vec - ml5.js Word2Vec
- Allison Parrish’s word2vec tutorial - a great resource for understanding word2vec

Step 1: Basic Word Display

Let’s start with the simplest version - displaying a word and finding related words using word2vec. This establishes our foundation. This first step introduces the core concepts of

Setting up a basic p5.js canvas
Loading the word2vec model
Creating a simple user interface
Finding and displaying related words

Try running this code and enter different words to see their associations. Notice how we’re just randomly positioning the related words on the canvas for now.

Step 2: Adding Basic Sound

Now let’s add our first synthesizer to create sound when words appear. We’ll start with just a simple melody synth. to open codeIn this second step, we’ve added:

A basic synthesizer using Tone.js
A mapping between word similarity and musical notes
Sequential playback of notes as words appear
Visual sizing of words based on their similarity

The words now play notes when they appear, with more similar words playing higher notes. Try different words and notice how the sound helps represent the semantic relationships.

Step 3: Adding Chords and Effects

Let’s enhance the musical experience by adding chord progressions and audio effects.UntitledClick to open codeIn this third step, we’ve added:

A chord progression that cycles continuously
Reverb and delay effects for depth
Fade-out animations for the words
Rhythmic variations in the melody

Notice how the chords create a harmonic foundation while the melody notes dance on top. The audio effects add space and atmosphere to the sound.

Step 4: The Final Poetry Generator

Now we’re ready to combine everything into the final poetry generator. You can find this complete version in the original code, which adds:

Structured poetry generation with stanzas and lines
More sophisticated word selection
Coordinated visual and musical elements
Advanced animation effects

Each step builds upon the previous one, helping you understand how the pieces fit together. Try running each version, experiment with the code, and make your own modifications. You might:

Change the chord progression for a different mood
Modify the animation effects
Adjust the timing of word generation
Create different visual layouts
Experiment with different synthesizer settings

Word2vec Sound Poetry

Mon, 01 Jan 0001 00:00:00 +0000

Sonic Poetry Generator - One Week Assignment

Overview

In this week-long assignment, you will create a sonic poetry generator that explores the relationship between words and sound. Building upon our in-class examples, you will implement one musical extension that demonstrates your understanding of synthesis, sound design, or music theory.

Core Requirements

Basic Implementation (Required)

Load and use the word2vec model to find related words
Create a basic melody system
Display words on screen with animation
Include start/stop functionality
Handle errors appropriately

Musical Extension (Choose One)

Select one of these extensions to enhance the sonic qualities of your generator. Each option emphasizes different aspects of sound design and music theory.

Extension 1: Timbre Evolution

Create meaningful relationships between words and synthesizer timbre. This extension is ideal if you understand sound synthesis basics and how different waveforms and envelope settings create distinct tonal qualities.

Implementation guidance:

Start by modifying the oscillator type between sine and triangle for different words
Experiment with attack and release times to create different note characteristics
Consider how word relationships might map to timbral qualities (e.g., similar words = warm tones, distant words = bright tones)

Extension 2: Musical Intervals

Design melodic relationships using specific musical intervals based on word relationships. This extension works well if you have a strong understanding of music theory and how different intervals create various emotional qualities.

Implementation guidance:

Begin with simple mappings like major thirds for similar words
Add minor intervals for more distant relationships
Create small collections of intervals that work well together
Consider using pentatonic scales to ensure pleasant combinations

Extension 3: Effects Processing

Create an evolving sonic space using audio effects. This extension is appropriate if you understand reverb, delay, and how they contribute to sound design.

Implementation guidance:

Start with subtle reverb adjustments
Add delay for more distant word relationships
Keep effect levels moderate to maintain clarity
Consider how the space changes with different word combinations

Development Schedule

Day 1-2: Core Setup

Get the basic word generation working
Implement fundamental audio playback

Day 3-4: Basic Sonification

Create simple melody system
Ensure audio triggers correctly with words

Day 5-6: Musical Extension

Implement your chosen extension
Test different parameter settings

Adjust and fine-tune your sound design
Document your creative decisions

Assessment Rubric (15 points)

Core Functionality (7 points)

Word generation works correctly (2 points)
Basic audio plays reliably (2 points)
Visual display functions (2 points)
Start/stop works properly (1 point)

Musical Extension (5 points)

Sonic implementation (2 points)
Musical effectiveness (2 points)
Creative use of parameters (1 point)

Documentation and Process (3 points)

Documentation of musical choices (1 point)
Clear explanation of parameters (1 point)
Thoughtful sound design decisions (1 point)

Submission Requirements

Your code with clear comments explaining audio parameters
A brief write-up explaining:
- Which extension you chose and why
- How your sound design relates to word relationships
- Your musical decision-making process
A short video demonstration with audio (1-2 minutes)

Tips for Success

Focus on making meaningful sonic relationships
Keep your parameter changes moderate and musical
Test your sounds with different words
Document which parameter settings worked best
Consider the overall musical experience
Make intentional choices about your sound design

Remember that your grade will focus more on thoughtful sound design and musical choices than on coding complexity. Take time to experiment with different parameters and document what works best for your artistic goals.