Skip to content

Commit bcb3a21

Browse files
agicquelamzmtdowling
authored andcommitted
tecs.audio: load a clip from the bytes of an encoded file in memory
Audio:loadBytes(name, bytes, options?) decodes a clip from a whole encoded file a game already holds, container and all, through the same symphonia probe a file goes through, so every format load reads from disk reads from memory too. A game holding raw samples writes the container itself; for PCM that is a 44-byte WAV header. The name is the clip's identity on the same terms as a path is for load, in the same namespace: loading a name again returns the clip already held without reading the new bytes, a failure is cached the same way, and a Sound row and a snapshot carry the name. A memory clip is always resident and keeps no copy of the bytes, so options.stream = true raises and reload reports false with a reason. Clip.memory tells the two kinds apart in clips() and clip(id). The backend contract gains an optional loadClipBytes, which the silent backend and the native binding implement; a backend without it makes loadBytes return nil and a reason. tecs-audio adds decode::decode_bytes, Engine::load_clip_bytes and tecs_audio_load_clip_bytes, which takes a pointer and a length because an encoded file is full of zeroes.
1 parent f0ea6fa commit bcb3a21

9 files changed

Lines changed: 829 additions & 33 deletions

File tree

‎README.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,8 @@ test suite reach the same world without starting a graphics stack.
5757
layers, object factories, runtime tile edits and authored collision outlines.
5858
- [Retained UI backed by Taffy](docs/ui/index.md), intrinsic text and images,
5959
scrolling, clipping, pointer capture, keyboard focus and controller activation.
60+
- Audio clips decoded from a file or from the bytes of a whole encoded file held
61+
in memory, for a game that unpacks its own archives.
6062

6163
The 2D shape materials include rectangles, circles, rings, rounded rectangles,
6264
triangles and stars. 3D rendering, custom post-processing and multi-camera
@@ -465,6 +467,22 @@ decode graph, so adapting either meant writing custom sources anyway and taking
465467
the dependency. What remained was roughly 450 lines: resampling, a gain
466468
envelope, a speaker pair, and a loop wrap.
467469

470+
A clip from memory is the bytes of a whole encoded file, container and all,
471+
rather than raw samples. `loadBytes` hands them to the same `symphonia` probe a
472+
file goes through, so every format `load` reads from disk reads from memory too,
473+
and there is one decode path rather than two. The alternative was a
474+
`loadSamples` taking floats with a rate and a channel count, and it lost on
475+
three counts: it needs a sample-format vocabulary (width, signedness, endianness,
476+
interleaving) that a WAV header already is, it copies a Nupp table across the
477+
seam one float at a time where a string crosses as one pointer and a length,
478+
and a game format that is not raw PCM would still need the decoders anyway. A
479+
game holding raw samples writes the 44-byte header itself, which is the cost
480+
this pushes onto it. The clip is always resident, because streaming would mean
481+
Rust keeping the bytes alive for every voice; the bytes are copied once for the
482+
decode and dropped. Its name shares the clip namespace with paths, because a
483+
`Sound` row and a snapshot carry a clip as a string, and so `reload` reports
484+
false for it rather than having a file to re-read.
485+
468486
The capture buffer is bounded rather than growing without limit as the SDL
469487
stream did. It holds one second by default, drops the oldest, and reports the
470488
loss through `Microphone.overruns`. An unbounded allocation driven by a

‎native/rust/tecs-audio/src/decode.rs‎

Lines changed: 123 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
//! Symphonia-backed audio decoding for the Tecs audio runtime.
22
//!
3-
//! This module turns a file on disk into interleaved f32 samples at the source
4-
//! layout. It never downmixes and never resamples: the mixer above owns both.
3+
//! This module turns an encoded file, on disk or held in memory, into
4+
//! interleaved f32 samples at the source layout. It never downmixes and never
5+
//! resamples: the mixer above owns both.
56
67
use std::fs::File;
8+
use std::io::Cursor;
79
use std::path::Path;
810

911
use symphonia::core::audio::GenericAudioBufferRef;
@@ -22,7 +24,7 @@ use symphonia::core::codecs::CodecParameters;
2224
use symphonia::core::errors::Error as SymphoniaError;
2325
use symphonia::core::formats::probe::Hint;
2426
use symphonia::core::formats::{FormatOptions, FormatReader, SeekMode, SeekTo, TrackType};
25-
use symphonia::core::io::{MediaSourceStream, MediaSourceStreamOptions};
27+
use symphonia::core::io::{MediaSource, MediaSourceStream, MediaSourceStreamOptions};
2628
use symphonia::core::meta::MetadataOptions;
2729
use symphonia::core::units::{Duration, Time, TimeBase};
2830

@@ -77,8 +79,32 @@ pub struct Decoded {
7779
/// cannot be identified, it carries no audio track, or no linked decoder
7880
/// handles its codec.
7981
pub fn decode_all(path: &str) -> Result<Decoded, String> {
80-
let mut streamer = Streamer::open(path)?;
82+
read_to_end(Streamer::open(path)?)
83+
}
8184

85+
/// Fully decodes the bytes of a whole encoded file held in memory.
86+
///
87+
/// The bytes are what a file on disk would hold, container and all, in any
88+
/// format a linked decoder reads, so a caller that has raw samples builds the
89+
/// container itself. `name` stands where a path would: every error names it,
90+
/// and its extension, when it carries one, hints the probe the way a path's
91+
/// does. The probe reads the content either way.
92+
///
93+
/// The bytes are copied once for the length of the decode, because the reader
94+
/// takes an owned source, and nothing of them is kept afterwards. Returns an
95+
/// error naming `name` when the bytes are empty, their container cannot be
96+
/// identified, they carry no audio track, or no linked decoder handles their
97+
/// codec.
98+
pub fn decode_bytes(name: &str, bytes: &[u8]) -> Result<Decoded, String> {
99+
if bytes.is_empty() {
100+
return Err(format!("tecs-audio: \"{name}\" holds no bytes to decode"));
101+
}
102+
let reader = probe(name, Box::new(Cursor::new(bytes.to_vec())))?;
103+
read_to_end(Streamer::from_reader(name, reader)?)
104+
}
105+
106+
/// Reads a streamer to its end and returns everything it decoded.
107+
fn read_to_end(mut streamer: Streamer) -> Result<Decoded, String> {
82108
let mut samples = Vec::new();
83109
while streamer.read(&mut samples, usize::MAX)? {}
84110

@@ -151,8 +177,14 @@ impl Streamer {
151177
/// cannot be opened, its container cannot be identified, it carries no audio
152178
/// track, or no linked decoder handles its codec.
153179
pub fn open(path: &str) -> Result<Streamer, String> {
154-
let reader = open_reader(path)?;
180+
Streamer::from_reader(path, open_reader(path)?)
181+
}
155182

183+
/// Builds a streamer over a reader whose container is already identified.
184+
///
185+
/// `path` labels every error the streamer reports, and is the name a
186+
/// memory input was given when there is no path.
187+
fn from_reader(path: &str, reader: Box<dyn FormatReader>) -> Result<Streamer, String> {
156188
let track = reader
157189
.default_track(TrackType::Audio)
158190
.ok_or_else(|| no_audio_track(path))?;
@@ -390,10 +422,19 @@ fn open_reader(path: &str) -> Result<Box<dyn FormatReader>, String> {
390422
let file =
391423
File::open(path).map_err(|err| format!("tecs-audio: failed to open \"{path}\": {err}"))?;
392424

393-
let source = MediaSourceStream::new(Box::new(file), MediaSourceStreamOptions::default());
425+
probe(path, Box::new(file))
426+
}
427+
428+
/// Identifies the container format of an opened input.
429+
///
430+
/// Passes the extension of `label`, a path or a memory input's name, to the
431+
/// probe as a hint when it carries a usable one, and probes on content alone
432+
/// otherwise.
433+
fn probe(label: &str, input: Box<dyn MediaSource>) -> Result<Box<dyn FormatReader>, String> {
434+
let source = MediaSourceStream::new(input, MediaSourceStreamOptions::default());
394435

395436
let mut hint = Hint::new();
396-
if let Some(extension) = Path::new(path).extension().and_then(|ext| ext.to_str()) {
437+
if let Some(extension) = Path::new(label).extension().and_then(|ext| ext.to_str()) {
397438
if !extension.is_empty() {
398439
hint.with_extension(extension);
399440
}
@@ -406,7 +447,7 @@ fn open_reader(path: &str) -> Result<Box<dyn FormatReader>, String> {
406447
FormatOptions::default(),
407448
MetadataOptions::default(),
408449
)
409-
.map_err(|err| format!("tecs-audio: failed to identify the format of \"{path}\": {err}"))
450+
.map_err(|err| format!("tecs-audio: failed to identify the format of \"{label}\": {err}"))
410451
}
411452

412453
/// Converts a container-stated duration into seconds.
@@ -776,6 +817,80 @@ mod tests {
776817
assert!(!error.is_empty());
777818
}
778819

820+
#[test]
821+
fn decode_bytes_matches_decoding_the_same_file_sample_for_sample() {
822+
let expected = stereo_tone(4800);
823+
let wav = TempWav::write(&expected, 44_100, 2, false);
824+
let bytes = fs::read(&wav.path).expect("the generated WAV reads back");
825+
826+
let from_file = decode_all(wav.as_str()).expect("the generated WAV decodes");
827+
let from_memory =
828+
decode_bytes("music/tone.wav", &bytes).expect("the same bytes decode from memory");
829+
830+
assert_eq!(from_memory.sample_rate, from_file.sample_rate);
831+
assert_eq!(from_memory.channels, from_file.channels);
832+
assert_eq!(from_memory.samples, from_file.samples);
833+
}
834+
835+
#[test]
836+
fn decode_bytes_reads_a_name_with_no_extension_by_content() {
837+
let expected = stereo_tone(480);
838+
let bytes = encode_wav(&expected, 48_000, 2, true);
839+
840+
let decoded = decode_bytes("generated tone", &bytes).expect("the probe reads the content");
841+
842+
assert_eq!(decoded.channels, 2);
843+
assert_eq!(decoded.samples, expected);
844+
}
845+
846+
#[test]
847+
fn decode_bytes_reads_a_mono_8_bit_unsigned_wav_built_by_hand() {
848+
// RIFF/WAVE, format tag 1 (PCM), one channel, 11025 Hz, 8 bits, five
849+
// frames. Unsigned 8-bit PCM centres on 128, so these five bytes are
850+
// the bottom of the range, halfway down, silence, halfway up and the
851+
// top of the range.
852+
let data: [u8; 5] = [0, 64, 128, 192, 255];
853+
let mut bytes = Vec::new();
854+
bytes.extend_from_slice(b"RIFF");
855+
bytes.extend_from_slice(&(36u32 + data.len() as u32).to_le_bytes());
856+
bytes.extend_from_slice(b"WAVE");
857+
bytes.extend_from_slice(b"fmt ");
858+
bytes.extend_from_slice(&16u32.to_le_bytes());
859+
bytes.extend_from_slice(&1u16.to_le_bytes());
860+
bytes.extend_from_slice(&1u16.to_le_bytes());
861+
bytes.extend_from_slice(&11_025u32.to_le_bytes());
862+
bytes.extend_from_slice(&11_025u32.to_le_bytes());
863+
bytes.extend_from_slice(&1u16.to_le_bytes());
864+
bytes.extend_from_slice(&8u16.to_le_bytes());
865+
bytes.extend_from_slice(b"data");
866+
bytes.extend_from_slice(&(data.len() as u32).to_le_bytes());
867+
bytes.extend_from_slice(&data);
868+
869+
let decoded = decode_bytes("SOUND.BIN", &bytes).expect("the hand-built WAV decodes");
870+
871+
assert_eq!(decoded.sample_rate, 11_025);
872+
assert_eq!(decoded.channels, 1);
873+
assert_eq!(decoded.samples, vec![-1.0, -0.5, 0.0, 0.5, 127.0 / 128.0]);
874+
}
875+
876+
#[test]
877+
fn decode_bytes_reports_bytes_that_are_not_audio_by_name() {
878+
let error = expect_error(
879+
decode_bytes("junk.wav", b"these bytes are not audio at all"),
880+
"text decoded as audio",
881+
);
882+
assert!(
883+
error.contains("junk.wav"),
884+
"the error does not name the input: {error}"
885+
);
886+
887+
let error = expect_error(decode_bytes("empty.wav", &[]), "no bytes decoded");
888+
assert!(
889+
error.contains("empty.wav") && error.contains("no bytes"),
890+
"the error does not say the input was empty: {error}"
891+
);
892+
}
893+
779894
#[test]
780895
fn decoder_names_reports_the_linked_decoders() {
781896
let names = decoder_names();

0 commit comments

Comments
 (0)