partcl-nqp

pmichaud and I and a few others are trying out the latest parrot compiler tools to see if we can create up with a version of partcl that is more maintainable, and will benefit more directly from toolchain improvements that target rakudo (Perl 6 on parrot) and other languages.

The base tool for this is NQP-rx; it's a small compiler for a subset of Perl 6 that targets PIR; It provides a much improved syntax over PIR, even when calling PIR itself - it provides a lot more sugar.

This means that nearly all of the handrolled PIR that was created is no longer needed. There are few cases where it is, and that is pushing development of NPQ-rx so that we can eliminate that PIR and rewrite it NQP.

Currently, there are currently two NQP-rx grammars, one for the language itself, and one for the tcl-specific regular expression language (ARE).

While the ARE is a work in progress (bits are added as needed to pass tests), adding this at all would have been very painful in old partcl (which currently relies on the perl5-like expression engine available via PGE in parrot.) We'll be adding more grammars for globbing, and probably the [format] and [binary] mini-languages as well.

The builtins (that are done) are now all written in NQP - and not having to write them in assembly (even macro'd assembly) makes them much shorter, easier to understand and maintain.

Big Goals:

1. pass all the tcl based tests written for partcl (Already running 12 of the original all-tcl tests)
2. start running the spec tests again.
3. pass almost as many spec tests as partcl...

At which point this new version will become the primary version.

If you're interested in Tcl, parrot, or Perl 6, check us out. There are plenty of small bits of NQP that can be written, and I'd be happy for the help. If you know Tcl but not Perl 6, or Perl 6, but not Tcl, doesn't matter, you can contribute. ^_^

partcl now passing 4,089 spec tests

With the cleanup of several segfaults in parrot and some recent improvements to ::tcl::mathop, partcl is now passing over 4K of the official 8.5.6 test suite!

That leaves expr.test as the last big source of potentially passing tests that aren't being counted; 347 tests in that file pass before an error due to promotion to bigints causes the file to abort.

Special thanks to moritz for providing CPU cycles for testing.

svn2git

I've moved the repository from svn hosted at googlecode to git at github.

Thanks to dukeleto at parrot.org for doing the heavy lifting on the repository migration.

For now, everything else is staying at googlecode, and I have a pointer on the main page to the github repo.

I had been using git-svn for a while to interact with the old svn repo, and while I really loved the git-like features it provided, there were enough annoyances with git-svn that making the full switch makes development easier for me.

Also, it might encourage Matt to commit some fixes. =-)

Now passing 3,363 tcl spec tests

Immediately after parrot 1.5.0 was released, many branches were merged back into trunk, which temporarily disrupted our ability to run the spec test suite.

Thanks to NotFound++ on the parrot team, all of these new failures have been resolved, and we're now able to run the spec test suite again!

Since the speed of partcl has improved a little since we started tracking spec test results, and to give future segfaults more of an opportunity to present themselves, we're now running every file in the test suite that runs to completion (even if they're all skipped).

With various advancements, and a lack of (random) segfaults, we're now running more test files and passing more tests than ever:

Previous High: Files 67 Total 5145 Passed 2972 Skipped 1352 Failed 821 Time: 4374s

Current: Files 93 Total 7397 Passed 3363 Skipped 1989 Failed 2045 Time: 6651s

There are still a few segfaults (but these are repeatable) and out of memory errors remaining, and they're on files with a relatively large set of already passing tests; getting those to run to completion will give us another big bump on the numbers.

Improved interactive mode

In real tcl, you can use interactive mode to run commands like a shell; There's now just enough logic in partcl to do this as well.

Most of the infrastructure for this was already in place; just added a minimal version of

• [file executable] - only checks executable status for other, not group or owner.
• [file split] - isn't windows-friendly, nor does it deal with potentially confusing chars like '~'
• [exec] - ignores all of the shell metachars and just invokes parrot's spawnw opcode as is.

$ ~/bird/bin/parrot tcl.pbc

% ls src

binary.c binary.h binary.o class grammar macros.pir mathops.pir ops pmc returncodes.pasm tclsh.pir

% rm src/binary.o

% ls src

binary.c binary.h class grammar macros.pir mathops.pir ops pmc returncodes.pasm tclsh.pir

Most of the heavy lifting is done by init.tcl from the standard library.

Partcl now passing 2,972 tcl spec tests

Thanks to prompting from kbk on irc (#tcl at freenode), I finally fixed the parsing of string.test.

It had been complaining about the \udead escape in that file; That isn't really a valid codepoint, and while Tcl accepts it as if it were, parrot (which uses ICU internally) throws an exception when trying to create a string that contains it. So, (at kbk's suggestion), I settled for now to convert any codepoints in the \udead range to \ufffd. That allowed the test to parse...

... but it then ran out of memory, as several other tests have been doing since I started running the tests again.

I've been working with chromatic to try to provide test cases to show where the memory leaks have been coming from, but I've only been going after things I could spot with valgrind. Today, someone on irc (#parrot at irc.parrot.org) posted a very small sample program that eventually ran out of memory... because the garbage collector apparently wasn't.

The small PIR example allowed chromatic to identify and fix that issue in parrot, and partcl has now reclaimed several files which had been exhausting a 1/2 gig of memory.

Current results, over 67 test files that run to completion:

Total 5145 Passed 2972 Skipped 1352 Failed 821

That's the highest number of attempted & passing tests since I started keeping track. It's an additional 1181 passing since the 8th, when I started running the spec tests again.

SpecTest updates

It's been about 4 and a half months since the last "official" spec test run -- partcl was offline for most of that time.

I've finally caught back up with parrot development, can build against svn-latest, an installed parrot, have added a few minor features.

We've lost several files (See http://code.google.com/p/partcl/wiki/SpecTestStatus). I'm running the tests on a machine with a 0.5G ram limit - previously these were running on a machine with no virtual memory limit, so they'd be able to complete. I've opened a ticket with the parrot developers for the biggest parrot memory leak I was able to find with a "hello word" in partcl (See https://trac.parrot.org/parrot/ticket/732). Hopefully they can get that and some other leaks closed.

So, since a lot has changed (including the hardware) between those last two runs, there's not much to compare directly. I'll try to get more, regular updates going forward.

Edit: Actual Spectest results