GtForth - Forth on top of the vCPU

Using, learning, programming and modding the Gigatron and anything related.
Forum rules
Be nice. No drama.
Post Reply
hnaves
Posts: 10
Joined: 25 Apr 2021, 00:57

GtForth - Forth on top of the vCPU

Post by hnaves »

Hi everyone,

I would like to share a preliminary implementation of GtForth -- a Forth compiler / interpreter for the Gigatron, running on top of the vCPU.
It is still in very early stage of development and so it still lacks a few features. Here is a preview of GtForth running on the gtemuAT67
screenshot1.png
screenshot1.png (17.68 KiB) Viewed 503 times
Since I do not actually own a Gigatron device myself, I have only tested it on emulators (it should be compatible with all the rom versions from ROMv1 to ROMv5). I am hoping it also works on the actual hardware. In addition, it is not as fast as it would be as if it were running natively on the Gigatron. Ideally, Forth should be ported to run directly from the ROM (as far as I know, quertyface is leading the effort on that front).

The user can check the list of implemented words via the WORDS command, and the free memory is given by FREE:

Code: Select all

WORDS
FREE U. CR
I hope GtForth can be useful to you, and any feedback is welcomed! Also, please let me know if you find any bugs or quirks in the program.

PS: I would like to submit the source code (and my other contributions) to the official gigatron-rom Git repository. What is the right way to proceed? Should I do a pull request?
Attachments
GtForth.gt1
(4.57 KiB) Downloaded 23 times
Last edited by hnaves on 12 May 2021, 21:32, edited 1 time in total.
at67
Posts: 332
Joined: 14 May 2018, 08:29

Re: GtForth - Forth on top of vCPU

Post by at67 »

hnaves wrote: 12 May 2021, 17:38 I hope GtForth can be useful to you, and any feedback is welcomed! Also, please let me know if you find any bugs or quirks in the program.
Works great over here, with a slightly modified print routine you could use the blank scanlines as free RAM as well, (of course if the user manually modified the scanline mode it would produce a pretty pixel mess, but the extra RAM available might be worth the risk).
hnaves wrote: 12 May 2021, 17:38 PS: I would like to submit the source code (and my other contributions) to the official gigatron-rom Git repository. What is the right way to proceed? Should I do a pull request?
You'll need to have git installed and a github account, (you can manage everything from the git command line and/or through github's browser interface).

Clone the entire repo at https://github.com/kervinck/gigatron-rom into a suitable directory.

Code: Select all

git clone https://github.com/kervinck/gigatron-rom
Make a directory under gigatron-rom\Contrib, e.g.

Code: Select all

md/mkdir gigatron-rom\Contrib\hnaves
Copy all your code and resources into this new directory, (use whatever directory structure that makes sense for you and your projects that you like). Don't meddle with anyone else's Contrib areas or the main repo itself, but if you want to experiment with other people's stuff, go ahead and do it on a different local copy of the main repo, (you can make as many clones of the main repo as you like), don't experiment on the repo that you will be making pull requests from.

Check your local repo is not modifying any files outside of your Contrib area:

Code: Select all

git status
Update your local repo with your changes:

Code: Select all

git add .
git commit -a
Do a push of your local repo, (the gigatron-rom you just cloned and your new Contrib\hanves changes), to your remote repo in github.

Code: Select all

git push
Once you are ready to make the pull request, do the following:

Code: Select all

- login to github in your favourite browser.

- go to your gigatron-rom repo and click on the Pull Requests tab across the top.

- click on the green New Pull Request button, (this should give you a list of changes that you have made to the repo).

- make sure you have only made changes to the Contrib\hnaves area.

- add a title/version/description and click Create Pull Request
Your code will be checked for merge conflicts, (which shouldn't exist as you're only making changes to your own area under Contrib), it will then be merged into the main repo, (Pull Requests are checked daily).

In the context of this message:
- main repo means https://github.com/kervinck/gigatron-rom
- remote repo means your github repo
- local repo means your local physical copy of the repo

More information can be found here.
https://www.atlassian.com/git/tutorials ... ll-request
hnaves
Posts: 10
Joined: 25 Apr 2021, 00:57

Re: GtForth - Forth on top of vCPU

Post by hnaves »

Thanks for the quick reply!
at67 wrote: 12 May 2021, 18:50 Works great over here, with a slightly modified print routine you could use the blank scanlines as free RAM as well, (of course if the user manually modified the scanline mode it would produce a pretty pixel mess, but the extra RAM available might be worth the risk).
Is there a way to prevent the user from changing modes? Which mode would you recommend in this case?

Anyway, I submitted a pull request with the source code of GtForth.

Thank you
at67
Posts: 332
Joined: 14 May 2018, 08:29

Re: GtForth - Forth on top of vCPU

Post by at67 »

hnaves wrote: 12 May 2021, 20:12 Thanks for the quick reply!
Is there a way to prevent the user from changing modes? Which mode would you recommend in this case?
You can keep calling the SYS call SYS_SetMode_v2_80 in your main loop, it will be annoying for the user and frowned upon for most Gigatron applications, but in your use case I think it is a perfectly valid thing to do; as long as the user/programmer knows about it before hand.

Who is going to say no to an extra 14Kbytes of free RAM or so? Maybe you could build two versions, one with and one without the forced mode switch and thus cater for everyone?
hnaves wrote: 12 May 2021, 20:12 Anyway, I submitted a pull request with the source code of GtForth.
Cheers.

Opps, I really shouldn't post ideas when I have just awoken, you can still do what I suggested, just not in the way I suggested it. You would need to do it in the same way that Marcel did for his MSBASIC implementation, (which is more complicated than my fail suggestion).
hnaves
Posts: 10
Joined: 25 Apr 2021, 00:57

Re: GtForth - Forth on top of the vCPU

Post by hnaves »

at67 wrote: 12 May 2021, 20:22 Opps, I really shouldn't post ideas when I have just awoken, you can still do what I suggested, just not in the way I suggested it. You would need to do it in the same way that Marcel did for his MSBASIC implementation, (which is more complicated than my fail suggestion).
It seems that Marcel only set the mode at the start of MSBASIC. You can manually change the mode later, and MSBASIC will not revert it back.
at67 wrote: 12 May 2021, 20:22 Who is going to say no to an extra 14Kbytes of free RAM or so? Maybe you could build two versions, one with and one without the forced mode switch and thus cater for everyone?
I really like the idea of having more memory and, on top of that, running GtForth significantly faster.
qwertyface
Posts: 27
Joined: 16 Jul 2019, 09:19

Re: GtForth - Forth on top of the vCPU

Post by qwertyface »

This is really cool, congratulations! It must have been a ton of work. If I may say so, it's lovely neat source as well. I'm sure I have a lot to learn from it, in particular the work on how the Python assembler module packs code into memory. I also like your approach to self-hosting. For obvious reasons that isn't possible for me.

Do you have an idea of what version of the language you're targeting? I've been following the online Forth 2012 standard documentation. It would probably be good if our implementations spoke the same language. Are there particular sources of inspiration you've found useful?

I'm sure I'll have a lot more to say once I've had a chance to play with it.
hnaves
Posts: 10
Joined: 25 Apr 2021, 00:57

Re: GtForth - Forth on top of the vCPU

Post by hnaves »

qwertyface wrote: 13 May 2021, 21:27 This is really cool, congratulations! It must have been a ton of work. If I may say so, it's lovely neat source as well. I'm sure I have a lot to learn from it, in particular the work on how the Python assembler module packs code into memory. I also like your approach to self-hosting. For obvious reasons that isn't possible for me.
Thanks! Writing GtForth took a bit of effort at first, but it became easier once I had a more systematic approach to the code (with some good coding conventions and the automatic placement of functions). Also having the vCPU emulator helped me test the system more quickly.
qwertyface wrote: 13 May 2021, 21:27 Do you have an idea of what version of the language you're targeting? I've been following the online Forth 2012 standard documentation. It would probably be good if our implementations spoke the same language. Are there particular sources of inspiration you've found useful?
I did not base GtForth on any preexisting Forth standard (as the joke says: if you have seen one Forth implementation, you have seen one Forth implementation). I took inspiration mostly from jonesforth and cmForth. Are you planning to implement the entire word set of Forth 2012? That is quite an undertaking.... Anyway, I would be more than happy if our implementations could speak the same language. Did you publish your implementation somewhere?
qwertyface
Posts: 27
Joined: 16 Jul 2019, 09:19

Re: GtForth - Forth on top of the vCPU

Post by qwertyface »

hnaves wrote: 14 May 2021, 00:04 Also having the vCPU emulator helped me test the system more quickly.
I've also benefited from using an emulator for testing. I've got pytest/hypothesis unit tests for all of the words that I've implemented, most of which are written in a fairly high-level way. Perhaps this is something that we could collaborate on? I can imagine a common abstraction that allows the same tests to validate your implementations using the vCPU emulator, and my implementations using gtemu. It would quickly find any observable differences.
hnaves wrote: 14 May 2021, 00:04 I did not base GtForth on any preexisting Forth standard (as the joke says: if you have seen one Forth implementation, you have seen one Forth implementation). I took inspiration mostly from jonesforth and cmForth. Are you planning to implement the entire word set of Forth 2012? That is quite an undertaking.... Anyway, I would be more than happy if our implementations could speak the same language. Did you publish your implementation somewhere?
I love Jonesforth. Reading it for the first time was revelatory!

I was basically planning on implementing all of the core word set from the 2012 standard, space permitting. Most of them seem like things you'd want to have (but I think I've done all of the easy ones now). I was certainly going to try to avoid being incompatible with it. I was also planning on using the test suite to help with bring-up. Perhaps I might try running it against your implementation first, to validate the approach.

My code is available in the master branch of my Github fork but it's far from complete, and quite messy. It might be rather hard to understand. I have a lot of words implemented (some of which are written in Forth), and a framework for cross-compiling more. But I'm lacking anything in the "RAM world", including the dictionary, the outer interpreter, the compiler, variables, or any way to start the system up. It's been stalled for quite a while.
hnaves
Posts: 10
Joined: 25 Apr 2021, 00:57

Re: GtForth - Forth on top of the vCPU

Post by hnaves »

qwertyface wrote: 14 May 2021, 08:33 Perhaps this is something that we could collaborate on? I can imagine a common abstraction that allows the same tests to validate your implementations using the vCPU emulator, and my implementations using gtemu. It would quickly find any observable differences.
Sure, we can collaborate on that. I tried to compile your version of the ROM by running the dev.py script in your Contrib directory, but I got an error:

Code: Select all

[hnaves@blade15 Forth]$ python dev.py                                                                                                                                                                               
runVcpu at $0100 net cycles 282 info ---- novideo                                                                                                                                                                   
runVcpu at $014d net cycles  98 info ---D line 0 no timeout                                                                                                                                                         
runVcpu at $019f net cycles 134 info AB-D line 1-36                                                                                                                                                                 
runVcpu at $01aa net cycles 130 info --C- line 3-39                                                                                                                                                                 
runVcpu at $01e5 net cycles 104 info ---D line 40                                                                                                                                                                   
runVcpu at $02f6 net cycles 148 info ABCD line 40-520                                                                                                                                                               
runVcpu at $1204 net cycles  90 info ---D line 0 timeout but no irq                                                                                                                                                 
runVcpu at $121b net cycles  72 info ---D line 0 timeout with irq                                                                                                                                                   
Warning: redefining forth.restart-or-quit (old 5120 new 5376)                                                                                                                                                       
Warning: redefining .quit (old 5122 new 5378)                                                                                                                                                                       
Warning: redefining forth.restart-or-quit (old 5376 new 5632)                                                                                                                                                       
Warning: redefining .quit (old 5378 new 5636)                                                                                                                                                                       
Warning: redefining forth.restart-or-quit (old 5632 new 5888)                                                                                                                                                       
Warning: redefining .quit (old 5636 new 5890)                                                                                                                                                                       
OK                                                                                                                                                                                                                  
OK     

...


Error: Undefined symbol 'Reset'
1 error(s)
                             
Anyway, it still generated the dev.lst file, which is quite useful.

I started reading your code, and I found the way you defined the bootstrap compiler in forth/bootstrapforth very interesting. I am still far from understanding the big picture though... How do you plan to implement the words? Are you going to implement the full set of words of the Forth standard in native code (ROM)? Or are you going to implement just a small subset of primitive words in ROM, and then build more complex words in RAM using these primitive words? In the dev.lst file, I see labels such as forth.DO-DOCOL-ROM and forth.DO-DOCOL-RAM, which makes me wonder if some of the words have two versions... Additionally, I see labels such as forth.core.RSHIFT, forth.core.ABS, which are non-trivial words, so it seems that you are implementing everything in ROM....

Have you thought about implementing a Forth virtual CPU in native code, something like a J1 clone (such as H2)?
qwertyface
Posts: 27
Joined: 16 Jul 2019, 09:19

Re: GtForth - Forth on top of the vCPU

Post by qwertyface »

hnaves wrote: 14 May 2021, 21:30 I am still far from understanding the big picture though... How do you plan to implement the words? Are you going to implement the full set of words of the Forth standard in native code (ROM)?
It's hard to see the big picture when it's only half drawn. :D In ROM, yes, in native code, no.

To the extent that there is a big picture, it's roughly this: there are more or less two Forths - and only one of them has been written. View this as a proof of concept - that hasn't yet proven the concept.

The idea is that when the user starts the system, there is nothing in memory except the dictionary. It's all available to the user. All of the code lives in, and runs from ROM. From the user's point of view this will be a fairly conventional indirect-threaded code system - when the user types a colon definition, it's compiled as a list of pointers to pointers to code as in Jonesforth (and your system, I believe). The only odd part is that the first pointer is a RAM address (usually a pointer into the dictionary entry), the second a ROM address (because the code is in ROM). This is what I call RAM mode, and only bits of it exist.

The other part is what I call ROM mode. This is compiled Forth code that lives in ROM. This is a direct-threaded code system, and it's the bit that does exist. In this system threads are compiled as pointers to code, or they would be if pointers were a thing in the ROM. I encode them as sequences like this (and yes, I know what you're going to say, this is not a dense encoding; in a lot of cases things could be done a lot better).

Code: Select all

st $low-byte [Y, X++]
jmp Y, <address of a zero-page routine that moves the IP>
st $high-byte [Y, X++]
This is what that bootstrap compiler is doing.
hnaves wrote: 14 May 2021, 21:30 I see labels such as forth.DO-DOCOL-ROM and forth.DO-DOCOL-RAM, which makes me wonder if some of the words have two versions... Additionally, I see labels such as forth.core.RSHIFT, forth.core.ABS, which are non-trivial words, so it seems that you are implementing everything in ROM....
In a DTC system, every thread starts with some machinecode which updates the returnstack, and moves the IP etc. That's all the DO-DOCOL stuff you're seeing in the listing. Because we have to do different stuff when being called from RAM mode we actually have two prefixes - in principle some words might only need one or the other, but for now I think I always put in both.

As you'll surmise there are two implementations of NEXT, one that treats the IP as a RAM pointer, and reads the next address in RAM, and on that executes a "ROM pointer". Which is in use is defined by a zero-page variable.

There will be other things that need to be duplicated, for example, I have a ROM-mode ?BRANCH, but it will need a different implementation for RAM mode. Most of RAM mode can be written in Forth though.

Roughly I'm trying to have a sensible core written in assembly, but use Forth where possible (like ABS, see core.f). Some things (like RSHIFT) are in assembly because it's fun (have a look! I felt pretty proud of it at the time. forth/_shift.py).
hnaves wrote: 14 May 2021, 21:30 I started reading your code, and I found the way you defined the bootstrap compiler in forth/bootstrapforth very interesting.
Interesting is one way of putting it. It seemed like such a simple idea, but the way I had to keep recompiling the same files with different dictionaries to get a version that does the necessary things was... well, confusing to say the least. I feel that there's got to be a better way.
hnaves wrote: 14 May 2021, 21:30 Have you thought about implementing a Forth virtual CPU in native code, something like a J1 clone (such as H2)?
No, but it sounds like a fun idea. It would be interesting to see how well it compares. I'm glad that you've made GtForth, because otherwise I was going to have to complete a vCPU Forth after the ROM forth, just to see if I wasted my time!
Post Reply