Broken methods

[Kirtai]

I was looking through the Smalltalk-78 image on the Zoo and noticed that some of the methods could not be found when selected in the browser. I also noticed a lot of uncategorised methods and missing Class comments.

Would you be willing to accept fixes from the ST-76 sources extracted from the xmsmall.dsk disk image? Example is attached.

It's Point with the broken #fromPress:value:, #hideData: and #pressCode replaced and recategorized, a class comment added, and #copy recategorized.

This was with a fully updated base image. I checked a fully updated full image but it seemed to be missing even more in the way of temps etc.
Is there a preferred image to work on?

Point.zip

[codefrau]

The original image (or rather, the two files found on the disk pack, object table and object space) had no categories or method sources at all. To keep the system small (to make it fit into the Notetaker's memory) everything was decompiled from bytecode. It is in the Lively version at the Smalltalk Zoo (the code to read it is in vm.js, see class ObjectTableReader).

Come to think of it, we should add those two files to the GitHub repo so we can replicate updating from the original to the current version (see updates/makeBaseImage-bf.st). We switched the update stream URL a few times, but we can map them all to the current URL in the VM primitive). This would give us a reproducible way to build an updated image.

In that original version, Point>>fromPress:value is there (see my screenshot). The sources must have been broken by something in the update stream. Ted imported a lot of sources from St76 (e.g. 5621fdb), my guess is the broken sources come from that. See section 4.3 and Appendix B in our “Reviving Smalltalk-78” paper.

And yes, I'd happily accept fixes. Preferably as updates in the update stream. The way we used to work is by pushing updates to the update stream, and later me committing them to git and pushing to github. I could explain how to do that – or we could come up with a process to commit to git first and then pull that into the update stream? Ideally, the image would load updates from and push to GitHub directly, but I'm not entirely sure how much effort that would be.

[codefrau]

Come to think of it, we should add those two files to the GitHub repo

I just did that, and you can run the original now. After pasting the contents of updates/makeBaseImage-bf.st and executing it, I was able to update all the way from the original to the current version (there's one notifier where you need to proceed manually).

[Kirtai]

In that original version, Point>>fromPress:value is there (see my screenshot). The sources must have been broken by something in the update stream.

I see. I was just looking at an image with the latest updates and I see the original images have the decompiled source so yes, it does seem to have been an update that did it.

For now I'll read up on how to make updates/changesets and either add them here in an issue or as a pull request, whichever you prefer. (I'm afraid of adding directly to the update stream because I'm just a beginner at ST-76/78)

Checking further I also noticed that the following look in different places for the update stream.

At https://smalltalkzoo.thechm.org/HOPL-St78.html all of the following look in https://lively-web.org/users/bert/St78/updates/

• Sim Kit
• ThingLab
• base image

In this repo all of the following look in https://lively-web.org/users/bert/St78/updates/

• updates/0001InitUpdates-bf.st
• updates/makeBaseImage-bf.st
• updates/updateStreamInit-bf.st

The following at https://smalltalkzoo.thechm.org/HOPL-St78.html looks in https://smalltalkzoo.thechm.org/users/codefrau/St78/updates/ instead.

• The default image
• fetch updated st78 from server image (these two might be the same?)

I believe these should all point to codefrau.

[codefrau]

For now I'll read up on how to make updates/changesets and either add them here in an issue or as a pull request, whichever you prefer. (I'm afraid of adding directly to the update stream because I'm just a beginner at ST-76/78)

Great. Let’s start by attaching them to an issue until we figure out a good process.

Checking further I also noticed that the following look in different places for the update stream. https://lively-web.org/users/bert/St78/updates/

<https://lively-web.org/users/bert/St78/updates/>

https://smalltalkzoo.thechm.org/users/codefrau/St78/updates/ instead. I believe these should all point to codefrau.

Agreed – but I think I’d only update the .st sources for now. The images, even if they still point to Lively Web, will automatically switch to the Smalltalk Zoo update stream, because I manually added an additional changeset to the Lively stream that does that. So if someone tried to update those images it should work and they will point to the zoo after.

…

[Kirtai]

With the missing methods it appears that they were deleted by updates from the image but not removed from the ClassOrganization. Would simply removing those selectors from their categories in the ClassOrganization be sufficient?

Also I just noticed that there are two "As yet unclassified" method categories. One starting with uppercase and one with lowercase A.
Some updates use one, some the other.

When making updates, would you prefer that I use the ST76 categories?

[codefrau]

With the missing methods it appears that they were deleted by updates from the image but not removed from the ClassOrganization. Would simply removing those selectors from their categories in the ClassOrganization be sufficient?

Should be sufficient, yes. Instead of doing this manually, we could put an expression into an update that goes through all classes.

Also I just noticed that there are two "As yet unclassified" method categories. One starting with uppercase and one with lowercase A. Some updates use one, some the other.

I believe the capitalized one is the right one.

When making updates, would you prefer that I use the ST76 categories?

Yes, this would make sense.

Thank you!

[Kirtai]

Some interesting breakage here :)

It seems some method sources reference missing classes so the class names were replaced by 'unknown', see Object>>notify:at:in:for: which contains

t5 ← unknown new

where unknown should be SyntaxWindow according to the ST-76 sources.
Other missing ones I've found so far are ParsedObjectReference and ParsedFieldReference in Decompiler>>remoteReference: and loadByte:code:

Attached is a fix containing all three classes and the methods fixed to reference them properly.
(All comes from Smalltalk-76 source)

fix-unknowns-ff.st.zip

Also recategorizations of Object and Parser. (Yes, I did them in an odd order but I was tracking a bug) :)

object-recategorization-ff.zip
parser-recategorization-ff.zip

[codefrau]

It's not so much that they were "replaced" by unknown, but rather that classes like SyntaxWindow never existed in Smalltalk-78. The St78 image was created by exporting only a subset of the St76 classes and methods, and my guess is that those methods were simply overlooked, and should have been removed. Checking the senders of notify:at:in: I'm not sure it's even being used? But it's good to have them back. I'm just not sure how to test them.

I tried your object-recategorization-ff.st but it did not seem to actually change the categories of methods? And parser-recategorization-ff.st has all the method sources even if they are the same it seems? It does not change only the categories.

Nonetheless, this is exciting! Thank you!

[Kirtai]

Hmm, very odd. I must have done something wrong with Parser, it should only have the recategorizations.

I'll have a look at them and see if they need an extra doit to make the changes. (Currently I'm having trouble with the lively file upload not working properly in both chromium and firefox).

Speaking of tests, is there anything like a port of SUnit to ST78? It looks simple enough except for figuring out how exceptions work.

Edit: yes, I messed up Parser somehow. The reorg of object should work, I needed to refresh the pane to see the changes though.

[Kirtai]

Ok, I may have found some weirdness in the updates.

Form>#figure is in the browsers method list but gives a "not found" error in an updated image.
This method was removed in 0332ReshapeForm.st but in 0343Source-ok-07A-tk.st: I find

Form understands: 'figure: figure 
        ["change the figure color (color associated with black) for this Image"]' classified: false

which should add it. But at the top of the update are the words

'Methods with original comments and temp names.
Only those that do not affect actual byteCode will be installed.'.

Could this have left Form>#figure with completely blank source?

I'm seeing similar things with other methods that give this error.
I'm puzzled here :)

[codefrau]

(Currently I'm having trouble with the lively file upload not working properly in both chromium and firefox).

You might like the devserver I just committed (377c5ce) ... it allows you to work with files on your local disk via http:-prefixed file names in the image, just like it works on the lively site.

Speaking of tests, is there anything like a port of SUnit to ST78?

Not that I know of – unit testing was invented 20 years later, and St76/78 were Xerox-internal prototypes, only St80 was released to the public back then.

It looks simple enough except for figuring out how exceptions work.

There isn't really an exception mechanism but it should not be too hard to add. Even the debugger is not working entirely correctly (although Dan got it working to some extent). The reason for that is that the debugger basically comes straight from St76, but St78 uses a linearized stack rather than actual context objects.

Edit: yes, I messed up Parser somehow. The reorg of object should work, I needed to refresh the pane to see the changes though.

Ah I messed up – the file browser menu items were broken when browsing a different directory than the default one (I put your files on my local disk under test/ and then switched a file browser to http:test/). I pushed fc2308a to fix that.

And I pushed your reorg changeset now as 6c0c82a. I should have designated you as the GIT_AUTHOR but I don't think I have your email ... would you want attribution? If you prefer to not reveal your actual email, you could use a private GitHub email.

[codefrau]

Re Form>>figure and Form>>figure::

• 0240Removals-di.st they were added to SymbolTable>>unforgetableMessages (which we should fix now I guess)
• 0294Source-ok-07A-tk.st both got source code
• 0295Source-ok-07B-tk.st a typo was fixed in their comment
• 0332ReshapeForm.st: removed both
• 0344Source-ok-07B-tk.st has source for them but should not have affected the system because MergingSources was true

The idea of MergingSources is to be able to file in ST76 sources to get their comments, without affecting the actual ST78 behavior, in case they were actually changed between 76 and 78. The way that's done is by compiling the St76 source, and only accepting it if the method produced is exactly the same as the existing method, by comparing their decompiled code. This logic was defined in 0283Merge-srcs-fix-tk.st and then employed in Ted's various Source-ok change sets.

In Class>>install:method:literals:code:backpointers: there is a check messagedict has: sel so this would not have brought back these methods (and indeed, we don't have them)

And Class>>derstands: (I love that cheeky name for un-understands) has had 4 versions but each one has the line self organization delete: sel

So it all seems like it should be okay, but it's not, obviously. Somehow both figure and figure: snuck back into Form's organization, or were never removed. We could figure out what happened using user loadUpdatesThrough: N but it seems to be easier to just fix it:

| nm cls sel .
for⦂ nm from: Smalltalk allClassNames do⦂ [
	cls ← Smalltalk ◦ nm.
	for⦂ sel from: cls organization asStream do⦂ [
		(cls canunderstand: sel) ⇒ []
		cls organization delete: sel.
			user show: 'del '; show: nm; space; show: sel; cr ]]

seems to do the trick. What do you think?

[Kirtai]

Dev-server

That's amazing. It took me a bit to figure out how to set the buttons the same as on the Lively site but I got it in the end. Great stuff :)

Unit tests

I read the original paper and the core of SUnit seems very very small. Unfortunately it relies on exceptions to handle failures but correct tests should run.

Exceptions

I imported the ExceptionHandler and related classes to the image but it falls foul of the same problem that hits the debugger. I expect that if the debugger was fixed, this would follow.

Attributions

I got a github email of "3085457+Kirtai@users.noreply.github.com" which would be fine. :)

Missing methods

Yep, your script works :)
It's particularly handy since I've been going through the ST-76 code, ripping out everything but the class names, method names and categories and converting them into simple scripts to reorganize a class. This will help find and remove any selectors in ST76 that aren't in ST78 that try sneaking in :)

(they're along the lines of

Class organization classify: ↪ed: asVector , ↪edit: , ↪edit:para:formerly: , ↪execute: under: 'Editing'.

followed by going in and manually fixing up anything left.)

End

BTW, I was wondering, is restoring removed classes fine? It would be nice if ST78 was ST76-Plus rather than ST76-Minus but I don't know if the system is up to it. Things like the Events, Picture-Editor and Exception-Handling categories.

Also, here's the reorganisation of Class and VariableLengthClass :)

class-recategorization-ff.st.zip
variablelengthclass-recategorization-ff.st.zip

[codefrau]

Exceptions

I imported the ExceptionHandler and related classes to the image but it falls foul of the same problem that hits the debugger. I expect that if the debugger was fixed, this would follow.

I added some details about that in #7.

Missing methods

Yep, your script works :) It's particularly handy since I've been going through the ST-76 code, ripping out everything but the class names, method names and categories and converting them into simple scripts to reorganize a class. This will help find and remove any selectors in ST76 that aren't in ST78 that try sneaking in :)

The question is if I should run this script as an update, or if we want to fix the broken class orgs manually ... I'm tending to just publish it.

End

BTW, I was wondering, is restoring removed classes fine? It would be nice if ST78 was ST76-Plus rather than ST76-Minus but I don't know if the system is up to it.

The system should be able to handle that fine – the only limitation would be total number of objects (30720) and classes (2048), where we currently use about 43% of object oops and 6% of classOops. I'd like to keep the base system 16 bit with enough room to do meaningful work (even though my 32 bit largeOops work lifts that limit).

Things like the Events, Picture-Editor and Exception-Handling categories.

Is Picture-Editor the same as our BitRectEditor? And what are the Events about? I don't have a working St76 here.

Also, here's the reorganisation of Class and VariableLengthClass :)

class-recategorization-ff.st.zip variablelengthclass-recategorization-ff.st.zip

Thank you! Published as cdeb490

[Kirtai]

For access to the source, if you go to this repo the readme contains details on how to extract and convert the source of Smalltalk-76 from the xmsmall.dsk using Contralto.

It also contains pre-extracted source but I did the extraction myself to make sure it was correct and have been using that. Note that while it also has the bootstrap files as text, these files have an error where ↪ has been substituted for ".

While the source is also on a linked website, the web pages uses the wrong kind of single quote so methods with them don't copy/paste cleanly.

It also has the source for quicksilver which seems to be a publishing tool?

The Events category has the classes 'EventQueue', 'MessageTally', 'PriorityInterrupt', 'PriorityScheduler', 'Timer' & 'UserEvent'. So scheduling and system events it seems. These are not in the image.

Yes, Picture-Editor is a category for the BitRectEditor & related tools (along with RadioButtons for some reason). These are in the image. They could be left alone or popped into the original category. (Sorry for not digging deeper in that before mentioning it, I never thought of classes being moved to different categories too)

Would it be okay to change the class categories to match ST76? Most of them are fine but a few seem to be merged. e.g. 'Picture-Editor' was merged into 'Windows'. It would make comparing easier and match the ST76 documentation better.

The system should be able to handle that fine – the only limitation would be total number of objects (30720) and classes (2048), where we currently use about 43% of object oops and 6% of classOops. I'd like to keep the base system 16 bit with enough room to do meaningful work (even though my 32 bit largeOops work lifts that limit).

That shouldn't be a problem. Many of the missing classes relate to hardware access (e.g. ethernet) or no longer extant filesystems (Juniper, Alto, IFS) so won't be needed. Some missing classes that maybe should be added are Path, Image and maybe things like Timer.

Regarding the script, I'll defer to you but the missing methods should be cleaned up with the reorganization I'm doing but publishing it wouldn't hurt afaict.

Edit: I just saw the mention of porting ST76 forward to ST78 on the zoo page.
That's what I kinda have in mind :)