Getting ACE-Step to make actual music on a home RTX 3090

I almost wrote ACE-Step off.

The first renders were strange in a very specific way. They were not noise. They were not silence. They had tones, little instrument fragments, the ghost of a beat here and there. But they were not music. Nothing settled. The groove would appear for a second and then fall apart. It sounded like a model trying to remember a song while someone kept changing the room around it.

That kind of failure is annoying because it is close enough to make you doubt the model, and broken enough to make you doubt yourself. In this case the model was not the problem. I was driving it wrong.

What finally worked

The setup that finally crossed the line for me was quite specific:

• acestep-v15-xl-turbo for the music model
• acestep-5Hz-lm-4B for the planner
• an RTX 3090 with 24 GB of VRAM
• the active checkpoints on the Linux filesystem, not a slow mounted drive
• thinking=true
• 8 inference steps for XL Turbo
• BPM, key, time signature and lyrics metadata, even for instrumental clips
• CPU offload and tiled VAE decode enabled

The important part is that those settings belong together. XL Turbo is built for the short 8-step path. The base/SFT-style checkpoints are not. If you run the base model like Turbo, the output can sound expensive and broken at the same time. For base/SFT I would start more in the 32 to 64 step range.

This is the shape I now use as a sane starting point for XL Turbo:

{
  "model": "acestep-v15-xl-turbo",
  "lm_model_path": "acestep-5Hz-lm-4B",
  "prompt": "A detailed musical caption...",
  "lyrics": "[Instrumental]",
  "bpm": 88,
  "key_scale": "D minor",
  "time_signature": "4",
  "audio_duration": 24,
  "audio_format": "mp3",
  "thinking": true,
  "inference_steps": 8,
  "guidance_scale": 7.0,
  "shift": 3.0,
  "batch_size": 1
}

The field that cost me the most time was thinking. Not think. Not a cute checkbox in my own UI that never reaches the backend. The API parameter has to be thinking=true, because that turns on the 5 Hz language-model planning path.

I had the wrong name wired through at one point. One missing suffix. That was enough to keep the planner off and make the model sound like it was throwing musical scraps onto the floor.

Why the first attempts were garbled

There was no single magic fix. Three boring things were wrong at the same time.

First, I mixed up checkpoint behavior and step count. Turbo wants 8 steps. Base does not. Eight steps on the wrong checkpoint gave me fragments.

Second, the planner was not actually enabled. I thought it was. The UI said it was. The backend disagreed, because computers are petty and exact.

Third, the prompt was too thin. I was sending little vibe prompts and expecting the model to invent the rest. The examples from the project are not that sparse. They include a proper musical caption, structured lyrics, BPM, key and time signature. Even if the lyrics are just [Instrumental], the extra structure matters.

So I changed the dashboard to stop pretending this is a one-line toy prompt problem. In Auto mode, it first asks the planner to write the caption, lyrics and metadata. Then it submits a normal generation job with that plan. Manual mode exposes the same fields directly. It feels less magical, but much more like telling a band what to play.

What it sounds like now

These are raw 24-second generations from the working setup. No mastering, no DAW arrangement afterwards, no polishing pass. Just model outputs that finally sound like music instead of almost-music.

I am not pretending these are finished tracks. They are not. But they have a beat I can follow, a tonal center, a shape, and some sense of arrangement. That was the line I wanted to cross.

Hardware reality

I would not describe this as "runs on any GPU" software. For XL Turbo, I would treat 20 GB of VRAM as the real floor. My 3090 has 24 GB, and it works, but I still cap CUDA memory and let ACE-Step offload aggressively to the CPU. That is what keeps it reliable instead of trying to keep every piece resident on the GPU.

The disk side is easy to underestimate. The XL Turbo checkpoint is roughly 19 GB. The 5 Hz planner is roughly 7.9 GB. The smaller Turbo checkpoint is around 4.5 GB. In practice I would keep at least 40 to 60 GB of fast local disk free if you want to experiment without constantly cleaning up.

Also: put the active weights on the Linux filesystem. Loading a large checkpoint through a slow mounted host drive looked like a hang and wasted more of my afternoon than I want to admit.

System RAM matters too because the CPU offload is real work, not a checkbox for decoration. I would not build this with less than 32 GB. 64 GB is more comfortable if the machine is doing other AI work at the same time.

The small dashboard

I put a tiny web UI in front of the API because curl is a miserable way to listen to music.

The dashboard has two modes. Auto mode takes a one-line vibe, lets the 5 Hz LM write the musical plan, then generates from that. Manual mode lets me fill in the caption, lyrics, BPM, key and time signature myself.

For XL Turbo it defaults to thinking=true and 8 steps. If I switch to a base or SFT checkpoint, I raise the step count instead of pretending the Turbo defaults transfer cleanly.

The raw ACE-Step API is also worth treating honestly: it is an async job API. Submit, poll, download. It is not a chat-completions endpoint, and trying to force it into one is just asking for timeouts and weird UX.

For remote use I put the dashboard behind TLS and Basic auth. It is still a tool, not a product surface. It talks to a model server in my own lab, on hardware I own. That is part of why I like it. It is not cloud magic. It is a machine doing real work, with the rough edges still visible.

What I learned

Bad generative audio does not always sound like a crash. Sometimes it sounds like a band falling down the stairs in tune. That is worse, because you start blaming the model before you check the wiring.

In this case the fix was not mystical. Use the right checkpoint. Use the right step count. Actually enable the planner. Give the model enough musical structure to work with.

The artwork at the top was generated locally with SANA. The clips were generated locally with ACE-Step. That is the kind of AI setup I keep coming back to: useful, local, a little rough, and close enough to the metal that I can fix it when it lies to me.