Large Array Memory Problem (FIXED)

[trogluddite]

I've just posted a bug report to Malc for one of my big schematics that has a problem...

EDIT: Problem fixed - but see HERE about being careful with your variable names!!

I was trying to clear out the schematic, to change old modules for newer, better versions, when I found that deleting certain modules made the schematic crash every single time that the audio was started.

After a lot of head-scratching, I found the source of the problem...

The schematic won't let me use LESS memory (more is no problem!).
Many of the modules have assembly and code with big arrays (a looping plugin, so lots of them and very BIG). If I delete any of the modules that have arrays in them, SM will crash as soon as the schematic is re-compiled.

I can even delete all of the code/assembly content, leaving nothing but the array declaration - and all is fine. But delete an array, or even knock a few kB off the size, and it crashes every time. Removing the links from the module also causes a crash, as disconnected modules don't run, causing the array to no longer be declared.
It just refuses to allow that memory to be freed up, no matter how I try to do it.

I went looking for a 'last good' save in my archive - but the problem even affects versions from over a year ago.
Many modules have had hundreds of edits in that time - including restructuring a lot of the arrays. However, I realised that whatever structure the arrays had, the total amount of memory required was exactly the same - or got bigger when I added features. Until now, I never decreased the memory footprint, so no way to see the bug lying in wait for me!

Apart from this one problem, the schematic is utterly stable in .osm, exe, and VST format - I can run it for hours without a single glitch. As long as my edits only keep adding stuff, it just fine - I just can't delete any out of date rubbish from it at all!

Fingers crossed, Malc will tell me that my file just got corrupted somewhere down the line - and these arrays really are a bit huge (30sec stereo audio * 8 audio loops) and plentiful (64 smaller automation loops, 8 * delays, reverb), so not a 'typical' schematic.

However, SM has had memory management problems before - so I though it best to post a warning.

[infuzion]

Fingers crossed, Malc will tell me that my file just got corrupted somewhere down the line - and these arrays really are a bit huge (30sec stereo audio * 8 audio loops) and plentiful (64 smaller automation loops, 8 * delays, reverb), so not a 'typical' schematic.
However, SM has had memory management problems before - so I though it best to post a warning.

Ah, I'm glad you have cornered one of SM' memory issues!
I don't think those arrays are large compared to most ROMplers these days; 3-4 samples/octave, with 2-6 layers per sample, with 3-4 octaves makes for a large patch. Almost forgot; 24bit also. & I am not exaggerating!
http://www.vir2.com/instruments/electri6ity

But an OSM that "got corrupted somewhere" is not a vote of confidence IMHO.

[rl]

Sigh! I'm with you.
Does the schematic also crash when using a selector that switches off code parts?

I had a stability issue some time ago ending up reconstructing a whole schematic from scratch, copying/pasting only the good parts, etc....

This case could be another good reason for a new feature to export/import schematics to/from a human readable format. So at least we could use a text editor as a last resort. It is not rocket science, e.g. have a look at PD (puredata.org).

[trogluddite]

This case could be another good reason for a new feature to export/import schematics to/from a human readable format. So at least we could use a text editor as a last resort. It is not rocket science, e.g. have a look at PD (puredata.org).

I also suggested to Malc, to add a kind of 'safe mode', where the schematic is fully recompiled and the memory allocations double-checked at load time - maybe that might fix schematics where the 'dynamic allocation' has its sums wrong?

Does the schematic also crash when using a selector that switches off code parts?

I'll have to check this out - one of the 'crashy' parts did used to have a bypass selector that worked OK. Maybe that will at least help me to find a 'last good save' to work from.

Unless Malc can fix the memory management in SM2.0.6....

[MegaHurtz]

Maybe your best bet for now would be to revert versions or wait for the next one?
It is after all a beta testing yob youre doing.

[trogluddite]

I'll wait to see what Malc has to say first.
The problem affects shematic versions that are very old, so a re-build from the basic components would be the only sensible way to re-construct it without the bug. Luckily I have all my assembly source code in Notepad++ format, and most of the green stuff I could transfer via the toolbox.
Previous versions of SM also had similar memory management problems, so at this stage, I'm not convinced that reverting SM will make things any better.

It is after all a beta testing yob youre doing

Well, we all need to face the fact that we are ALL beta testers for EVERY single piece of software we ever use.
The forces of consumerism mean that version (n+1) of any software always gets released too soon (got to hit the market before your competitors get in there!). The endless stream of updates and bugfixes is the pay-back for our gnawing 'need' for ever shinier toys. OutSim are not alone in that respect!
IMHO, that's one of the big advantages of open source - it acknowledges that software development is a two way process between developer and end-users.
I didn't pay too much for my copy of SM (SM1 'personal edition' upgraded for £20 to SM2 'Pro'), so I don't have a problem with spending some time properly getting to know a bug so that the report is more likely to aid a quick fix. I really need this problem sorting out - I could just sit and curse, but dong something to help the dev's just seems a more constructive way of dealing with it.

[MegaHurtz]

Sigh.. But i do respect your focus.

[infuzion]

It is after all a beta testing yob you're doing

Well, we all need to face the fact that we are ALL beta testers for EVERY single piece of software we ever use. The forces of consumerism mean that version (n+1) of any software always gets released too soon (got to hit the market before your competitors get in there!). The endless stream of updates and bugfixes is the pay-back for our gnawing 'need' for ever shinier toys. OutSim are not alone in that respect!

I would have to disagree. If the price for SM was $49USD or perhaps even $99, these are hobbyist prices, so we would expect a part-time programmer who would fix bugs as he can. For $300USD, I expect a professional program that is well tested & consistently fixed by a team of programmers & has staff for the forums/bug submission (eg Ableton Live, Max/MSP).

I'm 90% certain your bug was reported SIX years ago:
viewtopic.php?f=2&t=2389&hilit=page+faults
viewtopic.php?f=2&t=1735&hilit=page+faults

While I'm glad you helped isolate the issue, this proves that we users are beta testers. I do not want to beta test anymore; I want to get stuff done. If I had this many issues with WinXP, I would never got a new Windows computer & just went OSX.
Would be OK if there was more than one programmer, so it can be squashed fast. Granted, SM is much more stable than before, but we should be much more ahead of the bug-free & features game at the price point requested & time in production by now IMHO.

[trogluddite]

I'm 90% certain your bug was reported SIX years ago:

Yes, that was the historical 'memory management' problem I was hinting at before.
The one reported here is almost certainly related, but not quite the same thing (maybe the problem got 'half-fixed'). I know this, because early versions of my schematic had exactly the problem reported in those posts; specifically, constantly increasing memory requirements while running, and random crashes following a certain number of re-compiles (used to crash every dozen or so stream edits).
This time it is subtly different - the schematic has been utterly stable when running and when editing - I've had 12 hour sessions without the sightest hiccup (apart from the typos in my assembly optimisations!). It is only the array re-sizing that causes the problem this time (well, in this specific case anyway!)

I do not want to beta test anymore;

Oh yes, I can quite sympathise with your points - my perpective as a keen amateur who gets a buzz out of 'sleuthing' will always have a different slant that that of a commercial dev.
Not sure the comparison with WinXP is very fair though - 1000's of programmers writing software that is guaranteed a market of hundreds of millions vs. one guy making software that will only sell to geeks with a particularly rare DSP fetish!

[infuzion]

I do not want to beta test anymore;

Not sure the comparison with WinXP is very fair though - 1000's of programmers writing software that is guaranteed a market of hundreds of millions vs. one guy making software that will only sell to geeks with a particularly rare DSP fetish!

Yes, perhaps WinXP comparison is poor. Max/MSP comparison is spot-on though; given the price points & capabilities.
I do have some fetishes; not having my VSTs crash on stage in front of 1000s of people, & being able to use the same VSTs when I upgrade to Win7_64.

[trogluddite]

I do have some fetishes; not having my VSTs crash on stage in front of 1000s of people,

File along with "unbreakable guitar strings!"

Puts me in mind of a gig I played years ago with one of the old punk bands I was in. I had made myself the mother of all bass stomp pedal boxes out of the guts of an old mixing desk. (and particle board!)
Sadly, it decided to break down half way through a song, and being somewhat 'tired and emotional' from being stuck in the back of a van for the best part of a week, I had a 'prima donna' moment, saw red, and set about violently de-constructing it into very tiny pieces
There were those who said that my 'performance' that night was the highlight of the tour.

Anyhow, yes you make your point well - buy a TV or a car, and if it doesn't work you quite rightly expect to be able to take it back and have your cash refunded. Yet somehow it has been accepted that software (and to some extent even PC hardware) is an exception to this rule. It' seems to have got to the point where only a major 'boycott' by consumers could ever change that, but a whole generation is growing up accepting it as normal.
I'd certainly be happy to pay more for SM updates if there were some kind of guarantee of more development (or even just de-bugging) effort being put in - that model seems to have worked very well for Reaper.

OTOH, I always have this niggly feeling that there's just too much of an expectation of perfection in everything these days - it's odd how so many of the gigs that really stick with me are the ones where something blew up, or a MIDI rig started churning out random 12-tone improvisations (1st gig I ever went to - best song New Order ever played IMHO).

Maybe you have a secret masochistic streak - after all, you are still here in SM land, when you surely have the skills and talent to do the full on C++ thing.

[infuzion]

It' seems to have got to the point where only a major 'boycott' by consumers could ever change that, but a whole generation is growing up accepting it as normal.

Happened to MySpace; they built that crap on top of ColdFusion, which is the worst ever web platform for anything that gets more than 500 hits/day. MySpace is more or less an abondoned island now; everyone is on FaceBook or Twitter or G+. Partly because they simply work.

I'd certainly be happy to pay more for SM updates if there were some kind of guarantee of more development (or even just de-bugging) effort being put in - that model seems to have worked very well for Reaper.

Reaper is $60 or so every 2-3 years; great value IMHO. & You don't even have to pay to use it.

Maybe you have a secret masochistic streak - after all, you are still here in SM land, when you surely have the skills and talent to do the full on C++ thing.

I kind of feel I don't have the time to learn C++, but I still have ideas to make VSTs... but I want to publish to the biggest market possible, & not use abandon-ware to built it. That is why I joined SM the first place 6 years ago; I was using Buzz which was abandon-ware at that time, & saw there was 2-3 devs on SM.

[MegaHurtz]

did you use random names for your arrays or do you have doubles?
The reason I ask this is because i have little trouble changing the sizes, but then again i have only a limited number of arrays, that are about 10 seconds a piece.

[trogluddite]

Hi everyone - panic over, Malc has managed to isolate the problem with my schematic.

The memory management problem was caused elsewhere in the schematic by....
....a duplicated float variable declaration! (i.e. a typo)

Apparently, the assembler was not handling the double declaration at all well, leading to various memory conflicts with other modules in the schematic. I guess because arrays are so big, the odds are much higher that they would clash with the 'false' variable, hence it looked like an array problem

nevertheless, Malc is going to add a routine to the next update that will ensure that this can't happen, as it is such a simple mistake to make when pasting code from an extermal text editor.

I usually reserve a block of syntax definitions in notepad++ for variable names, to make sure my spelling is consistent - hence I though my code was thoroughly checked. But of course,a duplicate is still spelled right, so my check wasn't as thorough as I thought!

Which reminds me, I ought to get back onto Malc - we have requested proper variable name checking for the assembler (like the code does) several times before...

EDIT:
@MHz - this was two regular float variables in the same primitive with the same name - variables/arrays in different primitives are always independent even if they share a name - though I would love to have the ability to declare 'global' arrays, that would be so useful (if used very very carefully!!).
My usual practice is to use CamelCase variable/array names that are descriptive, it make debugging so much easier - e.g. LeftBuffer[DelayCount], AllDoneFlag, etc...

[MegaHurtz]

Sounds like something im capable of doing.